Jakub Sokołowski
e307d97bc6
Signed-off-by: Jakub Sokołowski <jakub@status.im> |
||
---|---|---|
airflow_xcom_redis | ||
LICENSE.txt | ||
Pipfile | ||
README.md | ||
setup.cfg | ||
setup.py |
README.md
Description
This module is an alternative Airflow XCom backend.
Details
Airflow XCom is a way Tasks exchange values in a DAG. By default the values are stored in the default Airflow database. This makes it unsuitable for sharing large volumes of data for two reasons:
- PostgreSQL has a limit of 1GB per XCom value
- Using the same database as other Airflow data may degrade performance
For these reasons Airflow allows for implementing a custom XCom backend. This module is one such implementation for use with Redis.
Requirements
Currently the module uses the Python Redis module which is already provided by Airflow.
The only requirements is for Airflow to have a Redis Connection defined called xcom_cache
.
TODO
Possible future improvements:
- Get Connection ID from configuration or environment variables
- Splitting of values into different keys to avoid size limits