mirror of
https://github.com/waku-org/nwaku.git
synced 2025-01-14 00:46:44 +00:00
docs: postgres-adoption.md add metadata title, description, and better first-readable-title (#2346)
This commit is contained in:
parent
988580e001
commit
2f8e8bcb52
@ -1,4 +1,9 @@
|
|||||||
# Epic
|
---
|
||||||
|
title: PostgreSQL
|
||||||
|
description: Document that describes why Nim-Waku started to use Postgres and shows some benchmark and comparison results.
|
||||||
|
---
|
||||||
|
|
||||||
|
## Introduction
|
||||||
|
|
||||||
The *Nim Waku Node*, *nwaku*, has the capability of archiving messages until a certain limit (e.g. 30 days) so that other nodes can synchronize their message history throughout the *Store* protocol.
|
The *Nim Waku Node*, *nwaku*, has the capability of archiving messages until a certain limit (e.g. 30 days) so that other nodes can synchronize their message history throughout the *Store* protocol.
|
||||||
|
|
||||||
@ -8,7 +13,7 @@ Therefore, the *Postgres* adoption is needed to enhance that.
|
|||||||
|
|
||||||
[https://github.com/waku-org/nwaku/issues/1888](https://github.com/waku-org/nwaku/issues/1888)
|
[https://github.com/waku-org/nwaku/issues/1888](https://github.com/waku-org/nwaku/issues/1888)
|
||||||
|
|
||||||
# How to connect the *nwaku* to *Postgres*
|
## How to connect the *nwaku* to *Postgres*
|
||||||
|
|
||||||
Simply pass the next parameter to *nwaku*
|
Simply pass the next parameter to *nwaku*
|
||||||
|
|
||||||
@ -23,21 +28,21 @@ Notice that this only makes sense if the _nwaku_ has the _Store_ protocol mounte
|
|||||||
|
|
||||||
(start the _nwaku_ node with `--help` parameter for more _Store_ options)
|
(start the _nwaku_ node with `--help` parameter for more _Store_ options)
|
||||||
|
|
||||||
# Examples of *nwaku* using *Postgres*
|
## Examples of *nwaku* using *Postgres*
|
||||||
|
|
||||||
[https://github.com/waku-org/nwaku-compose](https://github.com/waku-org/nwaku-compose)
|
[https://github.com/waku-org/nwaku-compose](https://github.com/waku-org/nwaku-compose)
|
||||||
|
|
||||||
[https://github.com/waku-org/test-waku-query](https://github.com/waku-org/test-waku-query)
|
[https://github.com/waku-org/test-waku-query](https://github.com/waku-org/test-waku-query)
|
||||||
|
|
||||||
# Stress tests
|
## Stress tests
|
||||||
|
|
||||||
The following repository was created as a tool to stress and compare performance between *nwaku*+*Postgres* and *nwaku*+*SQLite*:
|
The following repository was created as a tool to stress and compare performance between *nwaku*+*Postgres* and *nwaku*+*SQLite*:
|
||||||
|
|
||||||
[https://github.com/waku-org/test-waku-query](https://github.com/waku-org/test-waku-query)
|
[https://github.com/waku-org/test-waku-query](https://github.com/waku-org/test-waku-query)
|
||||||
|
|
||||||
## Insert test results
|
### Insert test results
|
||||||
|
|
||||||
### Maximum insert throughput
|
#### Maximum insert throughput
|
||||||
|
|
||||||
**Scenario**
|
**Scenario**
|
||||||
|
|
||||||
@ -61,7 +66,7 @@ The reason why few messages were lost is because the message rate was higher tha
|
|||||||
|
|
||||||
As a conclusion, the bottleneck is within the *Relay* protocol itself and not the underlying databases. Or, in other words, both *SQLite* and *Postgres* can support the maximum insert rate a Waku node will operate within normal conditions.
|
As a conclusion, the bottleneck is within the *Relay* protocol itself and not the underlying databases. Or, in other words, both *SQLite* and *Postgres* can support the maximum insert rate a Waku node will operate within normal conditions.
|
||||||
|
|
||||||
## Query test results (jmeter)
|
### Query test results (jmeter)
|
||||||
|
|
||||||
In this case, we are comparing *Store* performance by means of Rest service.
|
In this case, we are comparing *Store* performance by means of Rest service.
|
||||||
|
|
||||||
@ -85,7 +90,7 @@ With this, the *node_b* brings a higher throughput than the *node_a* and that in
|
|||||||
|
|
||||||
![jmeter results](imgs/jmeter-results.png)
|
![jmeter results](imgs/jmeter-results.png)
|
||||||
|
|
||||||
## Query test results (only Store protocol)
|
### Query test results (only Store protocol)
|
||||||
|
|
||||||
In this test suite, only the Store protocol is being analyzed, i.e. without REST. For that, a go-waku node is used, which acts as *Store* client. On the other hand, we have another go-waku app that publishes random *Relay* messages periodically. Therefore, this can be considered a more realistic approach.
|
In this test suite, only the Store protocol is being analyzed, i.e. without REST. For that, a go-waku node is used, which acts as *Store* client. On the other hand, we have another go-waku app that publishes random *Relay* messages periodically. Therefore, this can be considered a more realistic approach.
|
||||||
|
|
||||||
@ -102,7 +107,7 @@ That topology is defined in [this](https://github.com/waku-org/test-waku-query/b
|
|||||||
|
|
||||||
Notice that the two `nwaku` nodes run the very same version, which is compiled locally.
|
Notice that the two `nwaku` nodes run the very same version, which is compiled locally.
|
||||||
|
|
||||||
### Comparing archive SQLite & Postgres performance in [nwaku-b6dd6899](https://github.com/waku-org/nwaku/tree/b6dd6899030ee628813dfd60ad1ad024345e7b41)
|
#### Comparing archive SQLite & Postgres performance in [nwaku-b6dd6899](https://github.com/waku-org/nwaku/tree/b6dd6899030ee628813dfd60ad1ad024345e7b41)
|
||||||
|
|
||||||
The next results were obtained by running the docker-compose-manual-binaries.yml from [test-waku-queryc078075](https://github.com/waku-org/test-waku-query/tree/c07807597faa781ae6c8c32eefdf48ecac03a7ba) in the sandbox machine (metal-01.he-eu-hel1.wakudev.misc.statusim.net.)
|
The next results were obtained by running the docker-compose-manual-binaries.yml from [test-waku-queryc078075](https://github.com/waku-org/test-waku-query/tree/c07807597faa781ae6c8c32eefdf48ecac03a7ba) in the sandbox machine (metal-01.he-eu-hel1.wakudev.misc.statusim.net.)
|
||||||
|
|
||||||
@ -146,7 +151,7 @@ In this case, the performance is similar regarding the timings. The store rate i
|
|||||||
|
|
||||||
![Query time distribution](imgs/query-time-dist-3.png)
|
![Query time distribution](imgs/query-time-dist-3.png)
|
||||||
|
|
||||||
### Comparing archive SQLite & Postgres performance in [nwaku-b452ed8](https://github.com/waku-org/nwaku/tree/b452ed865466a33b7f5b87fa937a8471b28e466e)
|
#### Comparing archive SQLite & Postgres performance in [nwaku-b452ed8](https://github.com/waku-org/nwaku/tree/b452ed865466a33b7f5b87fa937a8471b28e466e)
|
||||||
|
|
||||||
This nwaku commit is after a few **Postgres** optimizations were applied.
|
This nwaku commit is after a few **Postgres** optimizations were applied.
|
||||||
|
|
||||||
@ -184,7 +189,7 @@ It cannot be appreciated but the average *****Store***** time was 11ms.
|
|||||||
|
|
||||||
![Query time distribution](imgs/query-time-dist-6.png)
|
![Query time distribution](imgs/query-time-dist-6.png)
|
||||||
|
|
||||||
### Conclusions
|
#### Conclusions
|
||||||
|
|
||||||
After comparing both systems, *SQLite* performs much better than *Postgres* However, a benefit of using *Postgres* is that it performs asynchronous operations, and therefore doesn’t consume CPU time that would be better invested in *Relay* for example.
|
After comparing both systems, *SQLite* performs much better than *Postgres* However, a benefit of using *Postgres* is that it performs asynchronous operations, and therefore doesn’t consume CPU time that would be better invested in *Relay* for example.
|
||||||
|
|
||||||
@ -196,7 +201,7 @@ Notice that we usually have a rate below 1100 req/minute in _status.prod_ fleet
|
|||||||
|
|
||||||
-----------------------------
|
-----------------------------
|
||||||
|
|
||||||
## Multiple nodes & one single database
|
### Multiple nodes & one single database
|
||||||
|
|
||||||
This study aims to look for possible issues when having only one single database while several Waku nodes insert or retrieve data from it.
|
This study aims to look for possible issues when having only one single database while several Waku nodes insert or retrieve data from it.
|
||||||
The following diagram shows the scenery used for such analysis.
|
The following diagram shows the scenery used for such analysis.
|
||||||
@ -210,7 +215,7 @@ ERR 2023-11-27 13:18:07.575+00:00 failed to insert message top
|
|||||||
|
|
||||||
The `db-postgres-hammer` is aimed to stress the database from the `select` point of view. It performs `N` concurrent `select` queries with a certain rate.
|
The `db-postgres-hammer` is aimed to stress the database from the `select` point of view. It performs `N` concurrent `select` queries with a certain rate.
|
||||||
|
|
||||||
### Results
|
#### Results
|
||||||
|
|
||||||
The following results were obtained by using the sandbox machine (metal-01.he-eu-hel1.wakudev.misc) and running nim-waku nodes from https://github.com/waku-org/nwaku/tree/b452ed865466a33b7f5b87fa937a8471b28e466e and using the `test-waku-query` project from https://github.com/waku-org/test-waku-query/tree/fef29cea182cc744c7940abc6c96d38a68739356
|
The following results were obtained by using the sandbox machine (metal-01.he-eu-hel1.wakudev.misc) and running nim-waku nodes from https://github.com/waku-org/nwaku/tree/b452ed865466a33b7f5b87fa937a8471b28e466e and using the `test-waku-query` project from https://github.com/waku-org/test-waku-query/tree/fef29cea182cc744c7940abc6c96d38a68739356
|
||||||
|
|
||||||
|
Loading…
x
Reference in New Issue
Block a user