mirror of https://github.com/waku-org/nwaku.git
docs: postgres-adoption.md add metadata title, description, and better first-readable-title (#2346)
This commit is contained in:
parent
988580e001
commit
2f8e8bcb52
|
@ -1,4 +1,9 @@
|
|||
# Epic
|
||||
---
|
||||
title: PostgreSQL
|
||||
description: Document that describes why Nim-Waku started to use Postgres and shows some benchmark and comparison results.
|
||||
---
|
||||
|
||||
## Introduction
|
||||
|
||||
The *Nim Waku Node*, *nwaku*, has the capability of archiving messages until a certain limit (e.g. 30 days) so that other nodes can synchronize their message history throughout the *Store* protocol.
|
||||
|
||||
|
@ -8,7 +13,7 @@ Therefore, the *Postgres* adoption is needed to enhance that.
|
|||
|
||||
[https://github.com/waku-org/nwaku/issues/1888](https://github.com/waku-org/nwaku/issues/1888)
|
||||
|
||||
# How to connect the *nwaku* to *Postgres*
|
||||
## How to connect the *nwaku* to *Postgres*
|
||||
|
||||
Simply pass the next parameter to *nwaku*
|
||||
|
||||
|
@ -23,21 +28,21 @@ Notice that this only makes sense if the _nwaku_ has the _Store_ protocol mounte
|
|||
|
||||
(start the _nwaku_ node with `--help` parameter for more _Store_ options)
|
||||
|
||||
# Examples of *nwaku* using *Postgres*
|
||||
## Examples of *nwaku* using *Postgres*
|
||||
|
||||
[https://github.com/waku-org/nwaku-compose](https://github.com/waku-org/nwaku-compose)
|
||||
|
||||
[https://github.com/waku-org/test-waku-query](https://github.com/waku-org/test-waku-query)
|
||||
|
||||
# Stress tests
|
||||
## Stress tests
|
||||
|
||||
The following repository was created as a tool to stress and compare performance between *nwaku*+*Postgres* and *nwaku*+*SQLite*:
|
||||
|
||||
[https://github.com/waku-org/test-waku-query](https://github.com/waku-org/test-waku-query)
|
||||
|
||||
## Insert test results
|
||||
### Insert test results
|
||||
|
||||
### Maximum insert throughput
|
||||
#### Maximum insert throughput
|
||||
|
||||
**Scenario**
|
||||
|
||||
|
@ -61,7 +66,7 @@ The reason why few messages were lost is because the message rate was higher tha
|
|||
|
||||
As a conclusion, the bottleneck is within the *Relay* protocol itself and not the underlying databases. Or, in other words, both *SQLite* and *Postgres* can support the maximum insert rate a Waku node will operate within normal conditions.
|
||||
|
||||
## Query test results (jmeter)
|
||||
### Query test results (jmeter)
|
||||
|
||||
In this case, we are comparing *Store* performance by means of Rest service.
|
||||
|
||||
|
@ -85,7 +90,7 @@ With this, the *node_b* brings a higher throughput than the *node_a* and that in
|
|||
|
||||
![jmeter results](imgs/jmeter-results.png)
|
||||
|
||||
## Query test results (only Store protocol)
|
||||
### Query test results (only Store protocol)
|
||||
|
||||
In this test suite, only the Store protocol is being analyzed, i.e. without REST. For that, a go-waku node is used, which acts as *Store* client. On the other hand, we have another go-waku app that publishes random *Relay* messages periodically. Therefore, this can be considered a more realistic approach.
|
||||
|
||||
|
@ -102,7 +107,7 @@ That topology is defined in [this](https://github.com/waku-org/test-waku-query/b
|
|||
|
||||
Notice that the two `nwaku` nodes run the very same version, which is compiled locally.
|
||||
|
||||
### Comparing archive SQLite & Postgres performance in [nwaku-b6dd6899](https://github.com/waku-org/nwaku/tree/b6dd6899030ee628813dfd60ad1ad024345e7b41)
|
||||
#### Comparing archive SQLite & Postgres performance in [nwaku-b6dd6899](https://github.com/waku-org/nwaku/tree/b6dd6899030ee628813dfd60ad1ad024345e7b41)
|
||||
|
||||
The next results were obtained by running the docker-compose-manual-binaries.yml from [test-waku-queryc078075](https://github.com/waku-org/test-waku-query/tree/c07807597faa781ae6c8c32eefdf48ecac03a7ba) in the sandbox machine (metal-01.he-eu-hel1.wakudev.misc.statusim.net.)
|
||||
|
||||
|
@ -146,7 +151,7 @@ In this case, the performance is similar regarding the timings. The store rate i
|
|||
|
||||
![Query time distribution](imgs/query-time-dist-3.png)
|
||||
|
||||
### Comparing archive SQLite & Postgres performance in [nwaku-b452ed8](https://github.com/waku-org/nwaku/tree/b452ed865466a33b7f5b87fa937a8471b28e466e)
|
||||
#### Comparing archive SQLite & Postgres performance in [nwaku-b452ed8](https://github.com/waku-org/nwaku/tree/b452ed865466a33b7f5b87fa937a8471b28e466e)
|
||||
|
||||
This nwaku commit is after a few **Postgres** optimizations were applied.
|
||||
|
||||
|
@ -184,7 +189,7 @@ It cannot be appreciated but the average *****Store***** time was 11ms.
|
|||
|
||||
![Query time distribution](imgs/query-time-dist-6.png)
|
||||
|
||||
### Conclusions
|
||||
#### Conclusions
|
||||
|
||||
After comparing both systems, *SQLite* performs much better than *Postgres* However, a benefit of using *Postgres* is that it performs asynchronous operations, and therefore doesn’t consume CPU time that would be better invested in *Relay* for example.
|
||||
|
||||
|
@ -196,7 +201,7 @@ Notice that we usually have a rate below 1100 req/minute in _status.prod_ fleet
|
|||
|
||||
-----------------------------
|
||||
|
||||
## Multiple nodes & one single database
|
||||
### Multiple nodes & one single database
|
||||
|
||||
This study aims to look for possible issues when having only one single database while several Waku nodes insert or retrieve data from it.
|
||||
The following diagram shows the scenery used for such analysis.
|
||||
|
@ -210,7 +215,7 @@ ERR 2023-11-27 13:18:07.575+00:00 failed to insert message top
|
|||
|
||||
The `db-postgres-hammer` is aimed to stress the database from the `select` point of view. It performs `N` concurrent `select` queries with a certain rate.
|
||||
|
||||
### Results
|
||||
#### Results
|
||||
|
||||
The following results were obtained by using the sandbox machine (metal-01.he-eu-hel1.wakudev.misc) and running nim-waku nodes from https://github.com/waku-org/nwaku/tree/b452ed865466a33b7f5b87fa937a8471b28e466e and using the `test-waku-query` project from https://github.com/waku-org/test-waku-query/tree/fef29cea182cc744c7940abc6c96d38a68739356
|
||||
|
||||
|
|
Loading…
Reference in New Issue