docs: postgres-adoption.md add metadata title, description, and better first-readable-title (#2346)

This commit is contained in:
Ivan FB 2024-01-12 16:37:32 +01:00 committed by GitHub
parent 988580e001
commit 2f8e8bcb52
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 18 additions and 13 deletions

View File

@ -1,4 +1,9 @@
# Epic
---
title: PostgreSQL
description: Document that describes why Nim-Waku started to use Postgres and shows some benchmark and comparison results.
---
## Introduction
The *Nim Waku Node*, *nwaku*, has the capability of archiving messages until a certain limit (e.g. 30 days) so that other nodes can synchronize their message history throughout the *Store* protocol.
@ -8,7 +13,7 @@ Therefore, the *Postgres* adoption is needed to enhance that.
[https://github.com/waku-org/nwaku/issues/1888](https://github.com/waku-org/nwaku/issues/1888)
# How to connect the *nwaku* to *Postgres*
## How to connect the *nwaku* to *Postgres*
Simply pass the next parameter to *nwaku*
@ -23,21 +28,21 @@ Notice that this only makes sense if the _nwaku_ has the _Store_ protocol mounte
(start the _nwaku_ node with `--help` parameter for more _Store_ options)
# Examples of *nwaku* using *Postgres*
## Examples of *nwaku* using *Postgres*
[https://github.com/waku-org/nwaku-compose](https://github.com/waku-org/nwaku-compose)
[https://github.com/waku-org/test-waku-query](https://github.com/waku-org/test-waku-query)
# Stress tests
## Stress tests
The following repository was created as a tool to stress and compare performance between *nwaku*+*Postgres* and *nwaku*+*SQLite*:
[https://github.com/waku-org/test-waku-query](https://github.com/waku-org/test-waku-query)
## Insert test results
### Insert test results
### Maximum insert throughput
#### Maximum insert throughput
**Scenario**
@ -61,7 +66,7 @@ The reason why few messages were lost is because the message rate was higher tha
As a conclusion, the bottleneck is within the *Relay* protocol itself and not the underlying databases. Or, in other words, both *SQLite* and *Postgres* can support the maximum insert rate a Waku node will operate within normal conditions.
## Query test results (jmeter)
### Query test results (jmeter)
In this case, we are comparing *Store* performance by means of Rest service.
@ -85,7 +90,7 @@ With this, the *node_b* brings a higher throughput than the *node_a* and that in
![jmeter results](imgs/jmeter-results.png)
## Query test results (only Store protocol)
### Query test results (only Store protocol)
In this test suite, only the Store protocol is being analyzed, i.e. without REST. For that, a go-waku node is used, which acts as *Store* client. On the other hand, we have another go-waku app that publishes random *Relay* messages periodically. Therefore, this can be considered a more realistic approach.
@ -102,7 +107,7 @@ That topology is defined in [this](https://github.com/waku-org/test-waku-query/b
Notice that the two `nwaku` nodes run the very same version, which is compiled locally.
### Comparing archive SQLite & Postgres performance in [nwaku-b6dd6899](https://github.com/waku-org/nwaku/tree/b6dd6899030ee628813dfd60ad1ad024345e7b41)
#### Comparing archive SQLite & Postgres performance in [nwaku-b6dd6899](https://github.com/waku-org/nwaku/tree/b6dd6899030ee628813dfd60ad1ad024345e7b41)
The next results were obtained by running the docker-compose-manual-binaries.yml from [test-waku-queryc078075](https://github.com/waku-org/test-waku-query/tree/c07807597faa781ae6c8c32eefdf48ecac03a7ba) in the sandbox machine (metal-01.he-eu-hel1.wakudev.misc.statusim.net.)
@ -146,7 +151,7 @@ In this case, the performance is similar regarding the timings. The store rate i
![Query time distribution](imgs/query-time-dist-3.png)
### Comparing archive SQLite & Postgres performance in [nwaku-b452ed8](https://github.com/waku-org/nwaku/tree/b452ed865466a33b7f5b87fa937a8471b28e466e)
#### Comparing archive SQLite & Postgres performance in [nwaku-b452ed8](https://github.com/waku-org/nwaku/tree/b452ed865466a33b7f5b87fa937a8471b28e466e)
This nwaku commit is after a few **Postgres** optimizations were applied.
@ -184,7 +189,7 @@ It cannot be appreciated but the average *****Store***** time was 11ms.
![Query time distribution](imgs/query-time-dist-6.png)
### Conclusions
#### Conclusions
After comparing both systems, *SQLite* performs much better than *Postgres* However, a benefit of using *Postgres* is that it performs asynchronous operations, and therefore doesnt consume CPU time that would be better invested in *Relay* for example.
@ -196,7 +201,7 @@ Notice that we usually have a rate below 1100 req/minute in _status.prod_ fleet
-----------------------------
## Multiple nodes & one single database
### Multiple nodes & one single database
This study aims to look for possible issues when having only one single database while several Waku nodes insert or retrieve data from it.
The following diagram shows the scenery used for such analysis.
@ -210,7 +215,7 @@ ERR 2023-11-27 13:18:07.575+00:00 failed to insert message top
The `db-postgres-hammer` is aimed to stress the database from the `select` point of view. It performs `N` concurrent `select` queries with a certain rate.
### Results
#### Results
The following results were obtained by using the sandbox machine (metal-01.he-eu-hel1.wakudev.misc) and running nim-waku nodes from https://github.com/waku-org/nwaku/tree/b452ed865466a33b7f5b87fa937a8471b28e466e and using the `test-waku-query` project from https://github.com/waku-org/test-waku-query/tree/fef29cea182cc744c7940abc6c96d38a68739356