<titledata-rh="true">PostgreSQL | Waku Documentation</title><metadata-rh="true"name="viewport"content="width=device-width,initial-scale=1"><metadata-rh="true"name="twitter:card"content="summary_large_image"><metadata-rh="true"property="og:url"content="https://docs.waku.org/research/benchmarks/postgres-adoption"><metadata-rh="true"property="og:locale"content="en_GB"><metadata-rh="true"name="docusaurus_locale"content="en-GB"><metadata-rh="true"name="docsearch:language"content="en-GB"><metadata-rh="true"name="keywords"content="waku, web3"><metadata-rh="true"name="image"content="https://docs.waku.org/_og/b671a20fd6154a55ddefec1309e744e45b4b6b50.png"><metadata-rh="true"name="docusaurus_version"content="current"><metadata-rh="true"name="docusaurus_tag"content="docs-default-current"><metadata-rh="true"name="docsearch:version"content="current"><metadata-rh="true"name="docsearch:docusaurus_tag"content="docs-default-current"><metadata-rh="true"property="og:title"content="PostgreSQL | Waku Documentation"><metadata-rh="true"name="description"content="Document that describes why Nim-Waku started to use Postgres and shows some benchmark and comparison results."><metadata-rh="true"property="og:description"content="Document that describes why Nim-Waku started to use Postgres and shows some benchmark and comparison results."><linkdata-rh="true"rel="icon"href="/theme/image/favicon.ico"><linkdata-rh="true"rel="canonical"href="https://docs.waku.org/research/benchmarks/postgres-adoption"><linkdata-rh="true"rel="alternate"href="https://docs.waku.org/research/benchmarks/postgres-adoption"hreflang="en-GB"><linkdata-rh="true"rel="alternate"href="https://docs.waku.org/research/benchmarks/postgres-adoption"hreflang="x-default"><linkrel="alternate icon"type="image/png"href="/theme/image/favicon.png">
<p>The <em>Nim Waku Node</em>, <em>nwaku</em>, has the capability of archiving messages until a certain limit (e.g. 30 days) so that other nodes can synchronize their message history throughout the <em>Store</em> protocol.</p>
<p>The <em>nwaku</em> originally used <em>SQLite</em> to archive messages but this has an impact on the node. <em>Nwaku</em> is single-threaded and therefore, any <em>SQLite</em> operation impacts the performance of other protocols, like <em>Relay.</em></p>
<p>Therefore, the <em>Postgres</em> adoption is needed to enhance that.</p>
<h2class="anchor anchorWithHideOnScrollNavbar_WYt5"id="how-to-connect-the-nwaku-to-postgres">How to connect the <em>nwaku</em> to <em>Postgres</em><ahref="#how-to-connect-the-nwaku-to-postgres"class="hash-link"aria-label="Direct link to how-to-connect-the-nwaku-to-postgres"title="Direct link to how-to-connect-the-nwaku-to-postgres"></a></h2>
<p>Simply pass the next parameter to <em>nwaku</em></p>
<p>(start the <em>nwaku</em> node with <code>--help</code> parameter for more <em>Store</em> options)</p>
<h2class="anchor anchorWithHideOnScrollNavbar_WYt5"id="examples-of-nwaku-using-postgres">Examples of <em>nwaku</em> using <em>Postgres</em><ahref="#examples-of-nwaku-using-postgres"class="hash-link"aria-label="Direct link to examples-of-nwaku-using-postgres"title="Direct link to examples-of-nwaku-using-postgres"></a></h2>
<h2class="anchor anchorWithHideOnScrollNavbar_WYt5"id="stress-tests">Stress tests<ahref="#stress-tests"class="hash-link"aria-label="Direct link to Stress tests"title="Direct link to Stress tests"></a></h2>
<p>The following repository was created as a tool to stress and compare performance between <em>nwaku</em>+<em>Postgres</em> and <em>nwaku</em>+<em>SQLite</em>:</p>
<h3class="anchor anchorWithHideOnScrollNavbar_WYt5"id="insert-test-results">Insert test results<ahref="#insert-test-results"class="hash-link"aria-label="Direct link to Insert test results"title="Direct link to Insert test results"></a></h3>
<h4class="anchor anchorWithHideOnScrollNavbar_WYt5"id="maximum-insert-throughput">Maximum insert throughput<ahref="#maximum-insert-throughput"class="hash-link"aria-label="Direct link to Maximum insert throughput"title="Direct link to Maximum insert throughput"></a></h4>
<p><strong>Scenario</strong></p>
<ul>
<li>1 node subscribed to pubsubtopic ‘x’ and the <em>Store</em> protocol mounted.</li>
<li>‘n’ nodes connected to the “store” node, and publishing messages simultaneously to pubsubtopic ‘x’.</li>
<li>All nodes running locally in a <em>Dell Latitude 7640</em>.</li>
<li>Each published message is fixed, 1.4 KB: <ahref="https://github.com/waku-org/test-waku-query/blob/master/sh/publish_one_client.sh"target="_blank"rel="noopener noreferrer">publish_one_client.sh</a></li>
<li>The next script is used to simulate multiple nodes publishing messages: <ahref="https://github.com/waku-org/test-waku-query/blob/fe7061a21eb14395e723402face755c826077aec/sh/publish_multiple_clients.sh"target="_blank"rel="noopener noreferrer">publish_multiple_clients.sh</a></li>
</ul>
<p><strong>Sought goal</strong></p>
<p>Find out the maximum number of concurrent inserts that both <em>SQLite</em> and <em>Postgres</em> could support, and check whether <em>Postgres</em> behaves better than <em>SQLite</em> or not.</p>
<p><strong>Conclusion</strong></p>
<p>Messages are lost after a certain threshold, and this message loss is due to limitations in the <em>Relay</em> protocol (GossipSub - libp2p.)</p>
<p>For example, if we set 30 nodes publishing 300 messages simultaneously, then 8997 rows were stored and not the expected 9000, in both <em>SQLite</em> and <em>Postgres</em> databases.</p>
<p>The reason why few messages were lost is because the message rate was higher than the <em>relay</em> protocol can support, and therefore a few messages were not stored. In this example, the test took 38.8’’, and therefore, the node was receiving 232 msgs/sec, which is much more than the normal rate a node will work with, which is ~10 msgs/sec (rate extracted from Grafana’s stats for the <em>status.prod</em> fleet.)</p>
<p>As a conclusion, the bottleneck is within the <em>Relay</em> protocol itself and not the underlying databases. Or, in other words, both <em>SQLite</em> and <em>Postgres</em> can support the maximum insert rate a Waku node will operate within normal conditions.</p>
<h3class="anchor anchorWithHideOnScrollNavbar_WYt5"id="query-test-results-jmeter">Query test results (jmeter)<ahref="#query-test-results-jmeter"class="hash-link"aria-label="Direct link to Query test results (jmeter)"title="Direct link to Query test results (jmeter)"></a></h3>
<p>In this case, we are comparing <em>Store</em> performance by means of Rest service.</p>
<p><strong>Scenario</strong></p>
<ul>
<li>node_a: one <em>nwaku</em> node with <em>Store</em> and connected to <em>Postgres.</em></li>
<li>node_b: one <em>nwaku</em> node with <em>Store</em> and using <em>SQLite</em>.</li>
<li>Both <em>Postgres</em> and <em>SQLite</em> contain +1 million rows.</li>
<li>node_c: one <em>nwaku</em> node with <em>REST</em> enabled and acting as a <em>Store client</em> for node_a.</li>
<li>node_d: one <em>nwaku</em> node with <em>REST</em> enabled and acting as a <em>Store client</em> for node_b.</li>
<li>With <em>jmeter</em>, 10 users make <em>REST</em><em>Store</em> requests concurrently to each of the “rest” nodes (node_c and node_d.)</li>
<p><ahref="https://github.com/waku-org/test-waku-query/blob/master/docker/jmeter/http_store_requests.jmx"target="_blank"rel="noopener noreferrer">This</a> is the <em>jmeter</em> project used.</p>
<p>With this, the <em>node_b</em> brings a higher throughput than the <em>node_a</em> and that indicates that the node that uses SQLite performs better. The following shows the measures taken by <em>jmeter</em> with regard to the REST requests.</p>
<h3class="anchor anchorWithHideOnScrollNavbar_WYt5"id="query-test-results-only-store-protocol">Query test results (only Store protocol)<ahref="#query-test-results-only-store-protocol"class="hash-link"aria-label="Direct link to Query test results (only Store protocol)"title="Direct link to Query test results (only Store protocol)"></a></h3>
<p>In this test suite, only the Store protocol is being analyzed, i.e. without REST. For that, a go-waku node is used, which acts as <em>Store</em> client. On the other hand, we have another go-waku app that publishes random <em>Relay</em> messages periodically. Therefore, this can be considered a more realistic approach.</p>
<p>The following diagram shows the topology used:</p>
<li><ahref="https://github.com/alrevuelta/waku-publisher/tree/9fb206c14a17dd37d20a9120022e86475ce0503f"target="_blank"rel="noopener noreferrer">Waku-publisher.</a> This app can publish Relay messages with different numbers of clients</li>
<li><ahref="https://github.com/Ivansete-status/waku-store-query-generator/tree/19e6455537b6d44199cf0c8558480af5c6788b0d"target="_blank"rel="noopener noreferrer">Waku-store-query-generator</a>. This app is based on the Waku-publisher but in this case, it can spawn concurrent go-waku Store clients.</li>
</ol>
<p>That topology is defined in <ahref="https://github.com/waku-org/test-waku-query/blob/7090cd125e739306357575730d0e54665c279670/docker/docker-compose-manual-binaries.yml"target="_blank"rel="noopener noreferrer">this</a> docker-compose file.</p>
<p>Notice that the two <code>nwaku</code> nodes run the very same version, which is compiled locally.</p>
<h4class="anchor anchorWithHideOnScrollNavbar_WYt5"id="comparing-archive-sqlite--postgres-performance-in-nwaku-b6dd6899">Comparing archive SQLite & Postgres performance in <ahref="https://github.com/waku-org/nwaku/tree/b6dd6899030ee628813dfd60ad1ad024345e7b41"target="_blank"rel="noopener noreferrer">nwaku-b6dd6899</a><ahref="#comparing-archive-sqlite--postgres-performance-in-nwaku-b6dd6899"class="hash-link"aria-label="Direct link to comparing-archive-sqlite--postgres-performance-in-nwaku-b6dd6899"title="Direct link to comparing-archive-sqlite--postgres-performance-in-nwaku-b6dd6899"></a></h4>
<p>The next results were obtained by running the docker-compose-manual-binaries.yml from <ahref="https://github.com/waku-org/test-waku-query/tree/c07807597faa781ae6c8c32eefdf48ecac03a7ba"target="_blank"rel="noopener noreferrer">test-waku-query-c078075</a> in the sandbox machine (metal-01.he-eu-hel1.misc.wakudev.status.im.)</p>
<p><strong>Scenario 1</strong></p>
<p><strong>Store rate:</strong> 1 user generating 1 store-req/sec.</p>
<p><strong>Relay rate:</strong> 1 user generating 10msg/sec, 10KB each.</p>
<p>In this case, we can see that the SQLite performance is better regarding the store requests.</p>
<p><strong>Relay rate:</strong> 1 user generating 10msg/sec, 10KB each.</p>
<p>In this case, the performance is similar regarding the timings. The store rate is bigger in <em>SQLite</em> and <em>Postgres</em> keeps the same level as in scenario 2.</p>
<h4class="anchor anchorWithHideOnScrollNavbar_WYt5"id="comparing-archive-sqlite--postgres-performance-in-nwaku-b452ed8">Comparing archive SQLite & Postgres performance in <ahref="https://github.com/waku-org/nwaku/tree/b452ed865466a33b7f5b87fa937a8471b28e466e"target="_blank"rel="noopener noreferrer">nwaku-b452ed8</a><ahref="#comparing-archive-sqlite--postgres-performance-in-nwaku-b452ed8"class="hash-link"aria-label="Direct link to comparing-archive-sqlite--postgres-performance-in-nwaku-b452ed8"title="Direct link to comparing-archive-sqlite--postgres-performance-in-nwaku-b452ed8"></a></h4>
<p>This nwaku commit is after a few <strong>Postgres</strong> optimizations were applied.</p>
<p>The next results were obtained by running the docker-compose-manual-binaries.yml from <ahref="https://github.com/waku-org/test-waku-query/tree/c07807597faa781ae6c8c32eefdf48ecac03a7ba"target="_blank"rel="noopener noreferrer">test-waku-query-c078075</a> in the sandbox machine (metal-01.he-eu-hel1.misc.wakudev.status.im.)</p>
<p><strong>Scenario 1</strong></p>
<p><strong>Store rate</strong> 1 user generating 1 store-req/sec. Notice that the current Store query used generates pagination which provokes more subsequent queries than the 1 req/sec that would be expected without pagination.</p>
<p><strong>Relay rate:</strong> 1 user generating 10msg/sec, 10KB each.</p>
<p>It cannot be appreciated but the average <em><strong><strong>Store</strong></strong></em> time was 11ms.</p>
<p><strong>Scenario 2</strong></p>
<p><strong>Store rate:</strong> 10 users generating 1 store-req/sec. Notice that the current Store query used generates pagination which provokes more subsequent queries than the 10 req/sec that would be expected without pagination.</p>
<p><strong>Relay rate:</strong> 1 user generating 10msg/sec, 10KB each.</p>
<p><strong>Store rate:</strong> 25 users generating 1 store-req/sec. Notice that the current Store query used generates pagination which provokes more subsequent queries than the 25 req/sec that would be expected without pagination.</p>
<p><strong>Relay rate:</strong> 1 user generating 10msg/sec, 10KB each.</p>
<h4class="anchor anchorWithHideOnScrollNavbar_WYt5"id="conclusions">Conclusions<ahref="#conclusions"class="hash-link"aria-label="Direct link to Conclusions"title="Direct link to Conclusions"></a></h4>
<p>After comparing both systems, <em>SQLite</em> performs much better than <em>Postgres</em> However, a benefit of using <em>Postgres</em> is that it performs asynchronous operations, and therefore doesn’t consume CPU time that would be better invested in <em>Relay</em> for example.</p>
<p>Remember that <em>nwaku</em> is single-threaded and <em>chronos</em> performs orchestration among a bunch of async tasks, and therefore it is not a good practice to block the whole <em>nwaku</em> process in a query, as happens with <em>SQLite</em></p>
<p>After applying a few <em>Postgres</em> enhancements, it can be noticed that the use of concurrent <em>Store</em> queries doesn’t go below the 250ms barrier. The reason for that is that most of the time is being consumed in <ahref="https://github.com/waku-org/nwaku/blob/6da1aeec5370bb1c116509e770178cca2662b69c/waku/common/databases/db_postgres/dbconn.nim#L124"target="_blank"rel="noopener noreferrer">this point</a>. The <code>libpqisBusy()</code> function indicates that the connection is still busy even the queries finished.</p>
<p>Notice that we usually have a rate below 1100 req/minute in <em>status.prod</em> fleet (checked November 7, 2023.)</p>
<hr>
<h3class="anchor anchorWithHideOnScrollNavbar_WYt5"id="multiple-nodes--one-single-database">Multiple nodes & one single database<ahref="#multiple-nodes--one-single-database"class="hash-link"aria-label="Direct link to Multiple nodes & one single database"title="Direct link to Multiple nodes & one single database"></a></h3>
<p>This study aims to look for possible issues when having only one single database while several Waku nodes insert or retrieve data from it.
The following diagram shows the scenery used for such analysis.</p>
<p>There are three nim-waku nodes that are connected to the same database and all of them are trying to write messages to the same <em>PostgreSQL</em> instance. With that, it is very common to see errors like:</p>
<divclass="language-text codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block"style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><divclass="codeBlockContent_QJqH"><pretabindex="0"class="prism-code language-text codeBlock_bY9V thin-scrollbar"style="color:#F8F8F2;background-color:rgba(var(--lsd-surface-secondary), 0.08)"><codeclass="codeBlockLines_e6Vv"><spanclass="token-line"style="color:#F8F8F2"><spanclass="token plain">ERR 2023-11-27 13:18:07.575+00:00 failed to insert message topics="waku archive" tid=2921 file=archive.nim:111 err="error in runStmt: error in dbConnQueryPrepared calling waitQueryToFinish: error in query: ERROR: duplicate key value violates unique constraint \"messageindex\"\nDETAIL: Key (storedat, id, pubsubtopic)=(1701091087417938405, 479c95bbf74222417abf76c7f9c480a6790e454374dc4f59bbb15ca183ce1abd, /waku/2/default-waku/proto) already exists.\n</span><br></span></code></pre></div></div>
<p>The <code>db-postgres-hammer</code> is aimed to stress the database from the <code>select</code> point of view. It performs <code>N</code> concurrent <code>select</code> queries with a certain rate.</p>
<h4class="anchor anchorWithHideOnScrollNavbar_WYt5"id="results">Results<ahref="#results"class="hash-link"aria-label="Direct link to Results"title="Direct link to Results"></a></h4>
<p>The following results were obtained by using the sandbox machine (metal-01.he-eu-hel1.misc.wakudev) and running nim-waku nodes from <ahref="https://github.com/waku-org/nwaku/tree/b452ed865466a33b7f5b87fa937a8471b28e466e"target="_blank"rel="noopener noreferrer">https://github.com/waku-org/nwaku/tree/b452ed865466a33b7f5b87fa937a8471b28e466e</a> and using the <code>test-waku-query</code> project from <ahref="https://github.com/waku-org/test-waku-query/tree/fef29cea182cc744c7940abc6c96d38a68739356"target="_blank"rel="noopener noreferrer">https://github.com/waku-org/test-waku-query/tree/fef29cea182cc744c7940abc6c96d38a68739356</a></p>
<p>The following shows the results</p>
<ol>
<li>Two <code>nwaku-postgres-additional</code> inserting messages plus 50 <code>db-postgres-hammer</code> making 10 <code>selects</code> per second.</li>
<p>In this case, the insert time gets more spread because the insert operations are shared amongst five more nodes. The <em>Store</em> query time remains the same on average.</p>
<olstart="3">
<li>Five <code>nwaku-postgres-additional</code> inserting messages plus 100 <code>db-postgres-hammer</code> making 10 <code>selects</code> per second.
