Matt Joiner
b69bb34eaf
Rejig some storage options
2021-01-25 15:54:37 +11:00
Matt Joiner
19f5984aec
Include context.Context and pprof label in withConn
2021-01-25 15:54:37 +11:00
Matt Joiner
2ddd3169ef
Move storage piece benchmarks to storage/test and add a lot more dials
2021-01-25 15:54:37 +11:00
Matt Joiner
cb5f80ec11
Improve configurability and add PutSized to sqlite storage
2021-01-25 15:54:37 +11:00
Matt Joiner
cb1d6bfc8f
Fix boundary conditions trimming sqlite3 storage cache
2021-01-18 14:48:30 +11:00
Matt Joiner
41ef267c1b
sqlite storage: Ensure that chunks are consecutive
2020-11-21 13:44:23 +11:00
Matt Joiner
8ef37cb2e6
sqlite storage: Remove num batched write queries log message
2020-11-16 16:37:11 +11:00
Matt Joiner
d84367c9a6
Update to race fixes in filecache and sqlite
2020-11-16 16:37:11 +11:00
Matt Joiner
258585fd8b
sqlite storage: Buffer write requests
2020-11-16 16:37:11 +11:00
Matt Joiner
b0ebc856d9
sqlite storage: Add some expvars
2020-11-16 16:37:11 +11:00
Matt Joiner
e65fac26ac
sqlite storage: Add error return from withConn
2020-11-16 16:37:11 +11:00
Matt Joiner
7cb74b158f
Replace storage.IncompletePieceToWriter with io.Writer
...
It was incorrect to assume piece hashing only operates on incomplete chunk data. This actually uncovered a bug where duplicate hash checks occurred, and the redundant checks would fail due to not reading the completed data.
2020-11-06 16:23:38 +11:00
Matt Joiner
96b640065a
sqlite storage: Add capacity management
2020-11-03 15:17:12 +11:00
Matt Joiner
8afb05a8a5
sqlite storage: Batch writes synchronously instead of buffering them first
2020-11-03 15:16:57 +11:00
Matt Joiner
784345e9f7
sqlite storage: Init schema in NewPool instead of NewProvider and add an option to disable
2020-11-03 13:11:44 +11:00
Matt Joiner
d04622e4b3
sqlite storage: Track total blob data size manually
...
Works around possible full table scan at startup, and possible lack of caching around cast(data as blob).
2020-11-03 13:10:17 +11:00
Matt Joiner
636b20b860
Add write incomplete/consecutive chunks interfaces
2020-11-02 15:35:07 +11:00
Matt Joiner
17e22516ff
sqlite storage: Add batched writes
2020-10-30 19:46:51 +11:00
Matt Joiner
ba70ad9b41
sqlite storage: Set mmap_size to a very large size
...
Seems to fall back to ~2GiB on my system.
2020-10-30 18:40:47 +11:00
Matt Joiner
b75ebbf9e7
sqlite storage: Provide helpers and reasonable defaults
2020-10-30 12:20:54 +11:00
Matt Joiner
fc039262d9
sqlite storage: Force data to be used as a blob
...
There's a bug in crawshaw.io/sqlite, and some forks where inserting []byte results in a text type instead of blob. To ensure things work correctly, we coerce data to blob wherever we can. See https://github.com/crawshaw/sqlite/issues/94 and the fork that fixes it.
2020-10-30 10:47:50 +11:00
Matt Joiner
8e1a8440bf
Performance fiddling on sqlite storage
2020-10-27 17:07:49 +11:00
Matt Joiner
e30084223d
sqlite storage: Include capacity management
2020-10-27 11:08:37 +11:00
Matt Joiner
55d4bcaf26
sqlite storage: Do ReadAt without using incremental I/O
2020-10-27 11:08:08 +11:00
Matt Joiner
1fc63bdded
sqlite storage: Add NewProviderPool
2020-10-23 09:03:44 +11:00
Matt Joiner
bb2ce15114
sqlite storage: Rename table to blob
...
Avoiding plural table names.
2020-10-23 09:01:15 +11:00
Matt Joiner
3820017a0c
Add last_used to sqlite storage
2020-10-13 09:36:58 +11:00
Matt Joiner
d820f7861c
Add a sqlite data storage implementation
2020-10-11 12:58:27 +11:00
