Python script for pushing data about peers into PostgreSQL

Go to file

Jakub Sokołowski 5d9e59ef61 show warning about no entries found Signed-off-by: Jakub Sokołowski <jakub@status.im>		2020-07-14 12:15:14 +02:00
.gitignore	add gitignore	2020-07-14 10:54:17 +02:00
README.md	add get_present_days() and use it to skip indices	2020-07-14 12:04:10 +02:00
log.py	add logging	2020-07-14 12:11:31 +02:00
main.py	show warning about no entries found	2020-07-14 12:15:14 +02:00
postgres.py	skip current day	2020-07-14 12:08:17 +02:00
query.py	add get_present_days() and use it to skip indices	2020-07-14 12:04:10 +02:00
requirements.txt	add requirements	2020-07-14 10:54:03 +02:00

README.md

Description

This Python script queries for unique Status peers and pushes them to an SQL database.

Details

The script queries an ElasticSearch endpoint for logstash-* indices and aggregates counts of instances of log messages with set peer_id field.

This data is pushed to a PostgreSQL database in the following format:

peers=> \d peers;
                       Table "public.peers"
┌────────┬───────────────────────┬───────────┬──────────┬─────────┐
│ Column │         Type          │ Collation │ Nullable │ Default │
├────────┼───────────────────────┼───────────┼──────────┼─────────┤
│ date   │ date                  │           │          │         │
│ peer   │ character varying(64) │           │          │         │
│ count  │ integer               │           │          │         │
└────────┴───────────────────────┴───────────┴──────────┴─────────┘

Example

peers=> select * from peers limit 3;
┌────────────┬──────────────────────────────────────────────────────────────────┬───────┐
│    date    │                               peer                               │ count │
├────────────┼──────────────────────────────────────────────────────────────────┼───────┤
│ 2020-06-01 │ a18d4417b1d2fbddd7f9474250f703ba20472be5e1131bc09e35e9b18c1a5bf7 │  1300 │
│ 2020-06-01 │ 7dba96249159cef53fbb5ec010c2d7799fec7dcaf8b1d9754559ce9fbd463328 │   652 │
│ 2020-06-01 │ 3a13adfa4799f9505c83fab18d49a47f6de09344db3d96e18678c5d3c92f717e │   632 │
└────────────┴──────────────────────────────────────────────────────────────────┴───────┘
(3 rows)