mirror of
https://github.com/logos-messaging/logos-messaging-nim-compose.git
synced 2026-01-03 22:43:07 +00:00
feat: adding helper script for storage size
This commit is contained in:
fryorcraken
parent
b35e9cc16f
commit
b0f16df6f1
38
README.md
38
README.md
@ -6,8 +6,9 @@ Ready to use docker-compose to run your own [nwaku](https://github.com/waku-org/
|
||||
* Grafana dashboard for advanced users or node operators.
|
||||
* Requires `docker-compose` and `git`.
|
||||
|
||||
## Setup and Run
|
||||
|
||||
**📝 0. Prerequisites**
|
||||
### 📝 0. Prerequisites
|
||||
|
||||
You need:
|
||||
* Ethereum Sepolia HTTP endpoint. Get one free from [Infura](https://www.infura.io/).
|
||||
@ -24,7 +25,7 @@ ${EDITOR} .env
|
||||
Make sure to **NOT** place any secrets into `.env.example`, as they might be unintentionally published in the Git repository.
|
||||
|
||||
|
||||
**🔑 1. Register RLN membership**
|
||||
### 🔑 1. Register RLN membership
|
||||
|
||||
The RLN membership is your access key to The Waku Network. Its registration is done onchain, and allows your nwaku node to publish messages in a decentralized and private way, respecting some [rate limits](https://rfc.vac.dev/spec/64/#rate-limit-exceeded).
|
||||
Messages exceeding the rate limit won't be relayed by other peers.
|
||||
@ -36,15 +37,32 @@ Note that if you just want to relay traffic (not publish), you don't need one.
|
||||
./register_rln.sh
|
||||
```
|
||||
|
||||
**🖥️ 2. Start your node**
|
||||
### 💽 2. Select storage size
|
||||
|
||||
Waku runs a PostgreSQL Database to store messages from the network and serve them to other peers.
|
||||
To prevent the database to grow indefinitely, you need to select how much disk space to allocate.
|
||||
You can either run a script that will estimate and set a good value:
|
||||
|
||||
```
|
||||
./set_storage_retention.sh
|
||||
```
|
||||
|
||||
Or select your own value. For example, `50GB`:
|
||||
|
||||
```shell
|
||||
echo "export STORAGE_SIZE=50GB" >> .env
|
||||
```
|
||||
|
||||
### 🖥️ 3. Start your node
|
||||
|
||||
Start all processes: nwaku node, database and grafana for metrics. Your [RLN](https://rate-limiting-nullifier.github.io/rln-docs/what_is_rln.html) membership is loaded into nwaku under the hood.
|
||||
```console
|
||||
docker-compose up -d
|
||||
```
|
||||
⚠️ The node might take ~5' the very first time it runs because it needs to build locally the RLN community membership tree.
|
||||
⚠️ The node might take ~5s the very first time it runs because it needs to build locally the RLN community membership tree.
|
||||
|
||||
###🏄🏼♂️ 4. Interact with your nwaku node
|
||||
|
||||
**🏄🏼♂️ 3. Interact with your nwaku node**
|
||||
* See [localhost:3000](http://localhost:3000/d/yns_4vFVk/nwaku-monitoring) for node metrics.
|
||||
* See [localhost:4000](http://localhost:4000) for a nice frontend to chat with other users.
|
||||
|
||||
@ -76,11 +94,11 @@ curl -X GET "http://127.0.0.1:8645/store/v1/messages?contentTopics=%2Fmy-app%2F2
|
||||
For advanced documentation, refer to [ADVANCED.md](https://github.com/waku-org/nwaku-compose/blob/master/ADVANCED.md).
|
||||
|
||||
-----
|
||||
# How to update to latest version
|
||||
## How to update to latest version
|
||||
|
||||
We regularly announce new available versions in our [Discord](https://discord.waku.org/) server.
|
||||
|
||||
## From `v0.29` or older
|
||||
### From `v0.29` or older
|
||||
|
||||
You will need to delete both the `keystore` and `rln_tree` folders, and register your membership again before using the new version by running the following commands:
|
||||
|
||||
@ -91,7 +109,7 @@ You will need to delete both the `keystore` and `rln_tree` folders, and register
|
||||
5. `./register_rln.sh`
|
||||
6. `docker-compose up -d`
|
||||
|
||||
## From `v0.30` or newer
|
||||
### From `v0.30` or newer
|
||||
|
||||
Updating the node is as simple as running the following:
|
||||
1. `cd nwaku-compose` ( go into the root's repository folder )
|
||||
@ -99,7 +117,7 @@ Updating the node is as simple as running the following:
|
||||
3. `git pull origin master`
|
||||
4. `docker-compose up -d`
|
||||
|
||||
## Check
|
||||
### Check
|
||||
|
||||
Once done, check your node is healthy:
|
||||
|
||||
@ -131,7 +149,7 @@ If the `./chkhealth.sh` script is hanging or returns the following, wait a few m
|
||||
}
|
||||
```
|
||||
|
||||
## Clean-up
|
||||
### Clean-up
|
||||
|
||||
Docker artefact can take some precious disk space, run the following commands to free space **while your node is running**.
|
||||
|
||||
|
||||
83
set_storage_retention.sh
Executable file
83
set_storage_retention.sh
Executable file
@ -0,0 +1,83 @@
|
||||
#!/bin/sh
|
||||
|
||||
# args <available space on disk MB> <size of existing postgresql dir MB> <leeway MB> <min storage size MB> <max storage size MB>
|
||||
# set SELECTED_STORAGE_SIZE_MB
|
||||
select_store_size()
|
||||
{
|
||||
_AVAIL_SPACE_MB=$1
|
||||
_PGSQL_SIZE_MB=$2
|
||||
_LEEWAY_MB=$3
|
||||
_MIN_SIZE_MB=$4
|
||||
_MAX_SIZE_MB=$5
|
||||
unset SELECTED_STORAGE_SIZE_MB
|
||||
if [ -z "$_AVAIL_SPACE_MB" ] || [ -z "$_PGSQL_SIZE_MB" ] || [ -z "$_LEEWAY_MB" ] || [ -z "$_MIN_SIZE_MB" ] || [ -z "$_MAX_SIZE_MB" ]; then
|
||||
echo "Internal error, not enough arguments passed to select_store_size"
|
||||
fi
|
||||
|
||||
_USABLE_SPACE_MB=$(( _AVAIL_SPACE_MB + _PGSQL_SIZE_MB ))
|
||||
|
||||
# Give all the available space to the DB, minus what is already used and 3GB for logs etc
|
||||
_TARGET_MB=$(( _USABLE_SPACE_MB - _LEEWAY_MB))
|
||||
|
||||
if [ $_TARGET_MB -lt $_MIN_SIZE_MB ]; then
|
||||
echo "Flooring storage retention to ${_MIN_SIZE_MB}MB"
|
||||
SELECTED_STORAGE_SIZE_MB=$_MIN_SIZE_MB
|
||||
elif [ $_TARGET_MB -gt $_MAX_SIZE_MB ]; then
|
||||
echo "Capping storage retention to ${_MAX_SIZE_MB}MB"
|
||||
SELECTED_STORAGE_SIZE_MB=$_MAX_SIZE_MB
|
||||
else
|
||||
echo "Storage retention set to ${_TARGET_MB}"
|
||||
SELECTED_STORAGE_SIZE_MB=$_TARGET_MB
|
||||
fi
|
||||
}
|
||||
|
||||
if [ "$1" = "test" ]; then
|
||||
echo "Running tests"
|
||||
i=0
|
||||
# avail pgdir leew min max
|
||||
echo "test $i"; i=$(( i + 1)); select_store_size 10000 0 1000 1000 20000; [ "$SELECTED_STORAGE_SIZE_MB" -eq 9000 ] || echo "fail: $SELECTED_STORAGE_SIZE_MB"
|
||||
echo "test $i"; i=$(( i + 1)); select_store_size 5000 5000 1000 1000 20000; [ "$SELECTED_STORAGE_SIZE_MB" -eq 9000 ] || echo "fail: $SELECTED_STORAGE_SIZE_MB"
|
||||
echo "test $i"; i=$(( i + 1)); select_store_size 30000 10000 1000 1000 20000; [ "$SELECTED_STORAGE_SIZE_MB" -eq 20000 ] || echo "fail: $SELECTED_STORAGE_SIZE_MB"
|
||||
echo "test $i"; i=$(( i + 1)); select_store_size 30000 0 1000 1000 20000; [ "$SELECTED_STORAGE_SIZE_MB" -eq 20000 ] || echo "fail: $SELECTED_STORAGE_SIZE_MB"
|
||||
echo "test $i"; i=$(( i + 1)); select_store_size 1000 2000 1000 1000 20000; [ "$SELECTED_STORAGE_SIZE_MB" -eq 2000 ] || echo "fail: $SELECTED_STORAGE_SIZE_MB"
|
||||
echo "test $i"; i=$(( i + 1)); select_store_size 1000 0 1000 1000 20000; [ "$SELECTED_STORAGE_SIZE_MB" -eq 1000 ] || echo "fail: $SELECTED_STORAGE_SIZE_MB"
|
||||
echo "test $i"; i=$(( i + 1)); select_store_size 500 0 1000 1000 20000; [ "$SELECTED_STORAGE_SIZE_MB" -eq 1000 ] || echo "fail: $SELECTED_STORAGE_SIZE_MB"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Check we are in right folder
|
||||
if ! [ -f "./run_node.sh" ]; then
|
||||
echo "This script must be run from inside the 'nwaku-compose' folder"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# check if STORAGE_SIZE is already specified
|
||||
if [ -f "./.env" ]; then
|
||||
. ./.env
|
||||
if [ -n "$STORAGE_SIZE" ]; then
|
||||
echo "STORAGE_SIZE is specified in .env file, doing nothing"
|
||||
exit 0
|
||||
fi
|
||||
fi
|
||||
|
||||
SUDO=""
|
||||
PGSQL_SIZE_MB=0
|
||||
if [ -d "./postgresql" ]; then
|
||||
# Check if we need to use SUDO moving forward
|
||||
if [ "$(ls postgresql/* 2>&1 | grep -c "Permission denied")" ]; then
|
||||
SUDO="sudo"
|
||||
fi
|
||||
|
||||
PGSQL_SIZE_MB=$($SUDO du -sBM "./postgresql" | awk '{ print $1 }' | sed 's/^\([0-9]\+\)M$/\1/')
|
||||
fi
|
||||
|
||||
AVAIL_SPACE_MB=$(df -BM . | tail -1 | awk '{ print $4 }' | sed 's/^\([0-9]\+\)M$/\1/')
|
||||
|
||||
select_store_size $AVAIL_SPACE_MB $PGSQL_SIZE_MB 1024 1024 $(( 30 * 1024 ))
|
||||
|
||||
if [ -z "$SELECTED_STORAGE_SIZE_MB" ]; then
|
||||
echo "Failed to selected a storage size"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "export STORAGE_SIZE=${SELECTED_STORAGE_SIZE_MB}MB" >> .env
|
||||
Loading…
x
Reference in New Issue
Block a user