status-im
/
airflow-xcom-redis
mirror of https://github.com/status-im/airflow-xcom-redis.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity

readme: add details and a todo list 

Signed-off-by: Jakub Sokołowski <jakub@status.im>
This commit is contained in:
Jakub Sokołowski 2021-03-30 18:09:07 +02:00
parent 85e4bbc6d1
commit 86c147e55b
No known key found for this signature in database
GPG Key ID: 4EF064D0E6D63020
1 changed files with 22 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

22
README.md
View File

@ -1,3 +1,25 @@
# Description
This module is an alternative [Airflow](https://github.com/apache/airflow) XCom backend.
# Details
Airflow [XCom](https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#xcoms) is a way Tasks exchange values in a DAG. By default the values are stored in the default Airflow database. This makes it unsuitable for sharing large volumes of data for two reasons:
* PostgreSQL has a limit of 1GB per XCom value
* Using the same database as other Airflow data may degrade performance
For these reasons Airflow [allows for implementing a custom XCom backend](https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#custom-xcom-backend). This module is one such implementation for use with Redis.
# Requirements
Currently the module uses the [Python Redis module](https://pypi.org/project/redis/) which is already provided by Airflow.
The only requirements is for Airflow to have a Redis [Connection](https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#connections) defined called `xcom_cache`.
# TODO
Possible future improvements:
* Get Connection ID from configuration or environment variables
* Splitting of values into different keys to avoid size limits