status-im
/
airflow-dags
mirror of https://github.com/status-im/airflow-dags.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
Status BI python DAGs for Airflow
97 Commits 4 Branches 0 Tags 317 KiB
Python 100%
Go to file
Alexis Pentori b273f250d2
comm_extraction: add status, ift, and operator accounts 
Signed-off-by: Alexis Pentori <alexis@status.im>
 		2024-10-24 11:38:00 +02:00
providers/airbyte Provider overriding Airbyte providers 2023-12-14 13:25:21 +01:00
tasks treasure_dashboard: Updating airbyte source config 2024-01-09 14:04:52 +01:00
.gitignore gitignore: add __pycache__ 2021-03-15 19:08:48 +01:00
README.md README: update 2023-11-23 11:14:44 +01:00
comm_extraction.py comm_extraction: add status, ift, and operator accounts 2024-10-24 11:38:00 +02:00
dbt.py Merge branch 'test' into prod 2024-02-07 18:20:15 +01:00
dev_spiff.py Update for spiff models and comms extractio 2024-05-29 09:19:20 -06:00
forecast_sync.py forecast_sync: init 2024-06-13 14:00:38 +02:00
forums_sync.py forums_sync: creation of separate DAG 2024-03-06 09:18:47 +01:00
github_extraction.py Refactorying DAG for github data extractions 2023-12-14 13:43:55 +01:00
infra_sync.py infra: fix naming dbt task 2024-05-23 13:27:30 +02:00
logos-org-map.py logos-org-map: remove discourse sync 2024-06-26 15:32:01 +02:00
organisation_sync.py organization_sync: init dag 2024-08-13 19:01:21 +02:00
prod_spiff.py Update for spiff models and comms extractio 2024-05-29 09:19:20 -06:00
spiff_extraction.py Refactoring DAG for spiff data extractions 2023-12-14 14:13:47 +01:00
temporal.py temporal: run dbt model to generate temporal data 2024-10-24 09:48:44 +02:00
test_spiff.py Update for spiff models and comms extractio 2024-05-29 09:19:20 -06:00
treasure_dashboard.py Fixed typo and schedule for Treasure Sync and created Mod Prod Spiff dag sync (#6) 2024-04-23 10:58:07 +02:00
website_sync.py website_sync: fix execution order 2024-02-22 14:36:27 +01:00

README.md

Description

This repo contains implementations of Airflow workflows and tasks called respectively DAGs and Operators.

  • DAGs - Direct Acyclic Graphs - Python scripts defining workflows in a way that reflects their relationships.
  • Operators - Python functions which define the individual tasks that are executed as part of a DAG run.

To learn how to write DAGs and Operators read about core concepts and follow the official tutorial.

DAG

This repository contains:

  • website_sync: DAG to launch the Airbyte jobs for the status-website charts.
  • spiff_sync: DAG to synchronize Spiff workflows data
  • dbt: DAG to run all the dbt models,
  • gh_sync: DAG to synchronize data from repository (logos, waku, codex)

The DBT models run in some DAG are stored in dbt-models.

Continuous Integration

Changes pushed to master are automatically fetched to our Airflow instance by the airflow-webhook service.

Branches

This repos has 3 working branches:

Infrastructure

All Airflow infrastructure is managed in the infra-bi repository.