Commit Graph

45 Commits

Author SHA1 Message Date
Etan Kissling d66a29db7c
avoid `XDeclaredButNotUsed` in `kvstore_sqlite3` with empty tuple (#680)
Suppresses an annoying hint that gets triggered when `params is tuple`
but it's an empty tuple without fields.

```
Hint: 'i' is declared but not used [XDeclaredButNotUsed]
```
2024-02-28 22:03:04 +00:00
Kim De Mey a6942e3a30
Fix missing std/times import for the metrics 0.0.1 case (#656) 2024-01-12 17:35:14 +01:00
Jacek Sieka 0fc5dc03c5
support nim-metrics 0.1.0+ (#651) 2024-01-05 15:17:27 +01:00
Kim De Mey e5c2b1784e
Replace registerCustomScalarFunction with createCustomFunction (#649)
* Replace registerCustomScalarFunction with createCustomFunction

The nice thing about `registerCustomScalarFunction` and
`registerCustomScalarFunction` was that it kept the sqlite calls
of type sqlite3_value_xxx inside kvstore_sqlite.

The big downside however is that the "custom" call is awfully
specific due to the specific function signature it demands.
It is likely that for each different use case, a new version
needs to be added.

In this PR the code gets altered to `createCustomFunction` which
allows to just create a custom function with directly xFunc
signature of sqlite. The downside of this is that you still need
to import sqlite_abi in your local code to have access to each of
the sqlite3_value_xxx and sqlite3_result_xxx calls. These could of
course also be wrapped.

* Fix custom sql func test
2023-11-21 14:39:24 +01:00
andri lim 2ed8e991b5
add mixin env to sqlite3 generic procs (#627) 2023-08-03 22:00:47 +07:00
Jacek Sieka 15a09fab73
sqlite3: better error message (#622)
* sqlite3: better error message

* fix dispose
2023-07-06 06:53:29 +00:00
Kim De Mey d57a1094b1
Remove all Defect raises to avoid unnecessary warnings (#607)
As this module dropped support for Nim versions < 1.6, all Defect
raises can be removed to avoid unnecessary warnings
2023-05-10 15:50:04 +02:00
tersec 7564b21ade
support only Nim 1.6+ (#606) 2023-05-09 20:19:44 +00:00
Jacek Sieka 72c9858927
kvstore: remove unnecessary EH (#592) 2023-02-22 09:03:13 +00:00
Jacek Sieka ca1c205441
fix missing gcsafe (#591) 2023-02-22 08:11:32 +00:00
Jacek Sieka ade6d19691
Defect fixes in kvstore (#589)
* Defect fixes in kvstore

* compile fixes
2023-02-21 18:54:30 +01:00
Jacek Sieka 8f0ae55353
kvstore: return bool from del, add clear (#574)
This allows making decisions based on whether an element was present
during the `del`
2023-01-03 16:38:27 +01:00
Jacek Sieka 8f4ef19fc9
sqlite: support read-only kvstores (#563)
in a read-only database, we cannot create the table but we can still
reason about elements in it - they simply don't exist
2022-11-28 21:15:53 +01:00
Justin Traglia e7d3de6ebf
Fix some typos (#557) 2022-11-16 17:44:00 +01:00
Zahary Karadjov 3a0afa4aa2 Add sqlite3.isInsideTransaction 2022-08-22 18:09:13 +07:00
tersec 6e813353eb
make some safe cstring conversions explicit (#515) 2022-06-28 13:03:02 +02:00
KonradStaniec dacf827a86
Add custom scalar functions to sqlite (#509) 2022-06-02 14:14:15 +02:00
Jacek Sieka 41b8588ade
move rocksdb to nimbus-eth1 (#460)
sqlite ftw!
2022-04-04 13:46:57 +02:00
Jacek Sieka ce4acc168c
kvstore: simplified stmt exec iterator (#467) 2022-01-28 15:23:41 +02:00
Kim De Mey 2c236f6495
Style fixes according to --styleCheck:usages (#452)
Currently only setting `--styleCheck:hint` as there are some
dependency fixes required and the compiler seems to trip over the
findnode MessageKind, findnode Message field and the findNode
proc. Also over protocol.Protocol usage.
2021-12-20 13:14:50 +01:00
Ștefan Talpalaru 2088d7568d
CI: test with multiple Nim version (#429)
* CI: test with multiple Nim version

* clean up the testing tree a little

* replace "unittest" with "unittest2"
2021-12-11 19:12:55 +01:00
Zahary Karadjov fe1bb4c4e7
Allow Sqlite keystores to be used in read-only mode
This is useful for tools such as `ncli_db` that can work with the database
of a running Nimbus instances.
2021-11-16 13:45:46 +02:00
Ștefan Talpalaru 2dfd352fd0 metrics: initialise lock for custom collector 2021-07-06 17:46:23 +03:00
Jacek Sieka 3514ee6484
sqlite: quick exec with result (#361) 2021-05-27 11:31:34 +02:00
Jacek Sieka 68e6aadc29 sqlite3: support option type
.. and any integer when writing but don't support int32 on reads -
internally sqlite will truncate on overflow which isn't nice.
2021-05-26 13:32:31 +03:00
Jacek Sieka 2a292cfb62
sqlite: fix readonly mode (#359)
* sqlite: fix readonly mode

* document kvstore versions
2021-05-25 20:57:28 +02:00
Jacek Sieka 1995afb87e
kvstore fixes (#350)
Storing large blobs in a "WITHOUT ROWID" table turns out to be extremely
slow when the tree must be rebalanced.

* Split out keystore capability into separate interface, making each
keystore a separate instance
* Disable "WITHOUT ROWID" optimization by default
* Implement prefix lookup that allows iterating over all values with a
certain prefix in their key
2021-05-17 15:55:57 +02:00
Jacek Sieka 8890175b6a remove WITHOUT ROWID
This is a minimal performance hotfix for storing large blobs in kvstore
2021-05-14 20:04:14 +03:00
Kim De Mey 762415319c
Add build_dcli target and add it to CI (#344)
* Add build_dcli target and add it to CI

* Fix local imports for dcli

* And use local imports for all other files too

* Use local imports in tests and rlpx protocols
2021-04-06 13:33:24 +02:00
Jacek Sieka 0f48ccecc0 sqlite: manual checkpointing 2020-12-18 19:30:53 +02:00
Mamy Ratsimbazafy 3b15abab10 Address review comment 2020-11-29 11:00:09 +02:00
Mamy André-Ratsimbazafy 482ea988c0 Add support for array[N, byte] in SQLite queries 2020-11-29 11:00:09 +02:00
Zahary Karadjov 49c40c9b95
Restore a commit accidentally omitted from a merged branch 2020-11-27 20:49:08 +02:00
Zahary Karadjov 345dac702b Don't use destructors 2020-10-16 21:21:00 +03:00
Zahary Karadjov cc0d15ccac
Richer Sqlite API
* Adds a strongly typed API for creating custom SQL queries and
  executing them

* Uses destructors to simplify the error handling in the init logic
2020-10-13 21:44:42 +03:00
Zahary Karadjov c103721391
Add support for multiple keyspaces in the Sqlite keystore 2020-10-12 04:02:33 +03:00
Jacek Sieka 3ddb498f2a
close sqlite transactions earlier (#294)
If the database is locked for reading, as it is when step returns ROW,
writes cannot checkpoint the wal leading to ever-increasing wal sizes
and a long delay at shutdown.

By resetting the transaction early, writes become more independent of
reads, memory is released earlier and wal can be checkpointed.
2020-09-21 08:21:47 +02:00
tersec ea0dbb256e
support closing kvstore databases (#293)
* support closing kvstore databases

* clear MemStore databases on close
2020-09-11 13:05:52 +00:00
Dustin Brody 64d880b24d
tweak phrasing to trigger CI to run again 2020-08-24 13:34:45 +02:00
Dustin Brody 9c9b9d5e2e
WAL pragma setting for in-memory databases results in "memory", not "wal" 2020-08-24 13:08:35 +02:00
Dustin Brody e2ccbbb418
use sqlite3 write-ahead logging 2020-08-24 12:37:40 +02:00
Ștefan Talpalaru 8ded65fe0e SQLite3 metrics: cosmetic changes 2020-05-13 11:37:35 +03:00
Ștefan Talpalaru 9da92b5415 SQLite3 mem usage metric 2020-05-13 11:37:35 +03:00
Jacek Sieka f730557baa sqlite: separate contains query
Using `select 1` for contains query tells sqlite not to actually load
the data for simple contains queries
2020-05-08 22:22:55 +03:00
Jacek Sieka 205b57fe71
port kvstore from nim-beacon-chain (#230)
* port kvstore from nim-beacon-chain
* remove old database backends
* use kvstore in trie database
* add sqlite dep
* avoid template param double evaluation
* clean up heterogenous lookup todo
2020-04-27 15:16:11 +02:00