status-im
/
airflow-dags
mirror of https://github.com/status-im/airflow-dags.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
97 Commits 4 Branches 0 Tags 317 KiB
Commit Graph

97 Commits

Author SHA1 Message Date
Alexis Pentori b273f250d2
comm_extraction: add status, ift, and operator accounts 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-10-24 11:38:00 +02:00
Alexis Pentori 11c93275a1
temporal: run dbt model to generate temporal data 
Required for dbt models: twitter, status, social, discord, podcast

Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-10-24 09:48:44 +02:00
Alexis Pentori 0114baa445
com: add twitter dbt models 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-10-22 11:02:38 +02:00
Alexis Pentori 15cb13a1b3
coms: update dag for new twitter setup 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-10-21 20:11:51 +02:00
Alexis Pentori 3d32930ad4
com: adding twitter connection for exit_operator and institute_ft 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-09-10 14:40:43 +02:00
Alexis Pentori 5fb6c83693
organization_sync: init dag 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-08-13 19:01:21 +02:00
Alexis Pentori b18d4a7b34
logos-org-map: remove discourse sync 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-06-26 15:32:01 +02:00
Alexis Pentori 568bcaa3ce
comm_extraction: make it run even in case of previous failure 
Create task_group to simplify dag

Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-06-20 10:39:58 +02:00
Alexis Pentori 7df57ea7ed
forecast_sync: init 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-06-13 14:00:38 +02:00
Alexis Pentori 6d1a7596bc
com: fix airbyte_connections list 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-06-13 10:07:24 +02:00
José Anaya d30a7f6f6f Update for spiff models and comms extractio 2024-05-29 09:19:20 -06:00
Alexis Pentori b195f4ac86
infra: fix naming dbt task 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-05-23 13:27:30 +02:00
Alexis Pentori 4a082ba9bb
comm_extraction: fix dag 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-04-30 18:21:23 +02:00
Alexis Pentori a9bdfd38f7
infra_sync: init dag 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-04-30 18:21:22 +02:00
José Anaya 061ecbabb5
Fixed typo and schedule for Treasure Sync and created Mod Prod Spiff dag sync (#6) 
* comm_extraction: fetching data from twitter account

Signed-off-by: Alexis Pentori <alexis@status.im>

* Fixed typo and updated schedule to avoid API rate limits

- Had a typo when declaring the bash operator for blockchain models.

- Changed the schedule to every 6 hours since the API is hitting rate limits for the wallet data.

* Created dag to sync mod prod spiff data and models

---------

Signed-off-by: Alexis Pentori <alexis@status.im>
Co-authored-by: Alexis Pentori <alexis@status.im>
 2024-04-23 10:58:07 +02:00
José Anaya 99ab33846e Changed Treasure Dashboard scheduling dag 
Changed scheduling to every hour for treasure dashboard dag
 2024-04-14 16:19:06 +01:00
José Anaya 1fed8c5cfa Update treasure_dashboard.py 
Changed names of connections to adhere to <Group of connection>-<type-of-data>
 2024-04-14 16:13:00 +01:00
José Anaya 72dc5b4678 Update treasure_dashboard.py 
Modified the schedule interval to run at minute 30 past every hour.

Added xe and coingecko airbyte connections to the flow.
 2024-04-08 12:10:22 +03:00
Alexis Pentori ab020c8341
comm_extraction: init dag 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-03-15 10:39:37 +01:00
Alexis Pentori 0b00621424
treasure_dashboard: adding dbt run 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-03-06 10:04:02 +01:00
Alexis Pentori 7fa0c7bf30
forums_sync: creation of separate DAG 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-03-06 09:18:47 +01:00
Alexis Pentori 91f469f0a8
logos-org-map: adding discourse connection 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-02-28 17:38:38 +01:00
Alexis Pentori 4d0a8f1b29
website_sync: fix execution order 
set schedule for every 30min

Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-02-22 14:36:27 +01:00
Alexis Pentori be4fbe7427
logos-org-map: adding order in tasks 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-02-22 14:36:15 +01:00
Alexis Pentori ab7d57499c
logos_org_map: init dag 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-02-20 11:16:57 +01:00
Alexis Pentori 7424b1f1cd
treasure_dashboard: fix update config 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-02-13 17:49:49 +01:00
Alexis Pentori 51f2ff2acb Merge branch 'test' into prod 2024-02-07 18:20:15 +01:00
Alexis Pentori 225db506fa
dbt: test models selector 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-02-02 10:36:19 +01:00
Alexis Pentori d2d13c0a22
website_sync: fix dbt 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-02-02 10:02:00 +01:00
Alexis Pentori 9cc14e7b4c
github_website_sync: fix dbt 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-02-02 09:40:14 +01:00
Alexis Pentori 8722b38d82
treasure_dashboard: adding bank balance fetching and renaming airbyte connection 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-01-26 10:01:02 +01:00
Alexis Pentori d03bbb2a45
Merge branch 'prod' into test 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-01-18 20:01:27 +01:00
Alexis Pentori 499a61fd54
treasure_dashboard: add Airbyte Trigger to sync market value 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-01-16 14:37:21 +01:00
Alexis Pentori c93d86aaac
treasure_dashboard: Updating airbyte source config 
Calling Config Server for wallets list
  Updating Airbyte source

Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-01-09 14:04:52 +01:00
Alexis Pentori e4c121dca0
dbt: Move dbt dags 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2024-01-08 10:31:04 +01:00
Alexis Pentori 39cc101b0c
Refactoring DAG for spiff data extractions 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2023-12-14 14:13:47 +01:00
Alexis Pentori e750b74d5c
Refactorying DAG for github data extractions 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2023-12-14 13:43:55 +01:00
Alexis Pentori d54d0dabc9
refactorying DAG for website data ELT 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2023-12-14 13:43:42 +01:00
Alexis Pentori c8860ce6ad
renaming treasure dashboard 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2023-12-14 13:31:26 +01:00
Alexis Pentori e7ecbe01e4
Adding DAG for Treasure Dashboard 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2023-12-14 13:29:29 +01:00
Alexis Pentori 4967a87e85
Renaming tasks_group to tasks 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2023-12-14 13:28:16 +01:00
Alexis Pentori 0781264ac4
Provider overriding Airbyte providers 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2023-12-14 13:25:21 +01:00
Alexis Pentori 98ff616e1f
Adding group Tasks 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2023-12-14 13:24:31 +01:00
Alexis Pentori 0cbf3b9415
README: update 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2023-11-23 11:14:44 +01:00
Alexis Pentori bd03f3313e
README: update 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2023-11-23 11:13:32 +01:00
Alexis Pentori f692d438cf
update README 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2023-10-31 14:46:14 +01:00
Alexis Pentori 7fcaabf62b
dbt: update dag name 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2023-10-26 15:55:19 +02:00
Alexis Pentori 266dce0d97
website_sync: update airbyte connection name 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2023-10-25 17:28:57 +02:00
Alexis Pentori fb75cec4cc
test: update sync time 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2023-10-25 16:03:57 +02:00
Alexis Pentori d23012c03d
update Readme 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2023-10-23 11:27:14 +02:00