History

Alexis Pentori 101a18b1e6 twitter: adding separate Stream for promoted metrics Signed-off-by: Alexis Pentori <alexis@status.im>		2024-10-21 12:59:52 +02:00
..
sample_files	twitter: adding separate Stream for promoted metrics	2024-10-21 12:59:52 +02:00
source_twitter_fetcher	twitter: adding separate Stream for promoted metrics	2024-10-21 12:59:52 +02:00
Dockerfile	renaming source-social-network-fetcher into source-twitter-fetcher	2024-02-15 12:02:02 +01:00
README.md	twitter: separate fecthing of tweets and metrics from tweets	2024-10-11 17:25:11 +02:00
main.py	renaming source-social-network-fetcher into source-twitter-fetcher	2024-02-15 12:02:02 +01:00
metadata.yaml	twitter-fetcher: fix oauth refresh token issue	2024-10-17 12:15:50 +02:00
requirements.txt	renaming source-social-network-fetcher into source-twitter-fetcher	2024-02-15 12:02:02 +01:00
setup.py	renaming source-social-network-fetcher into source-twitter-fetcher	2024-02-15 12:02:02 +01:00

README.md

Twitter Fetcher Source

This is the repository for fetching Twitter information, written in Python.

Todos

Implements first version based on original script
- Fetch the Data users of each count.
- Fetch Tweets details
Improve version:
- limit the data fetching based on input date

Usage

This connector fetch information from Twitter based on their API: https://developer.twitter.com/en/docs/twitter-api

Configuration

The connector takes the following input:

twitter:
  credientials:
    client_id: "Id from the Twitter Developer Account"
    client_secret: "Secret from the Twitter Developer Account"
    access_token: "Token generated from the generated Twitter account"
    refresh_token: "Refresh token obtain from the Twitter Account" 
  account_id: "Id of the Twitter account"
  start_time: 'AAAA-MM-DD"  # Start of the period of tweets sync

To obtain the account_id, run the following command:

curl -X GET "https://api.x.com/2/users/me" \
    -H "Authorization: Bearer $access_token"

Output

The connector will return the following:

Local development

Prerequisites

Activate Virtual Environment and install dependencies

From this connector directory, create a virtual environment:

python -m venv .venv

source .venv/bin/activate
pip install -r requirements.txt

Locally running the connector

python main.py spec
python main.py check --config sample_files/config-example.json
python main.py discover --config sample_files/config-example.json
python main.py read --config sample_files/config-example.json --catalog sample_files/configured_catalog.json

Locally running the connector docker image

docker build -t airbyte/twitter-fetcher:dev .
# Running the spec command against your patched connector
docker run airbyte/twitter-fetcher:dev spec

Run

Then run any of the connector commands as follows:

docker run --rm airbyte/twitter-fetcher:dev spec
docker run --rm -v $(pwd)/sample_files:/sample_files airbyte/twitter-fetcher:dev check --config /sample_files/config-example.json
docker run --rm -v $(pwd)/sample_files:/sample_files airbyte/twitter-fetcher:dev discover --config /sample_files/config-example.json
docker run --rm -v $(pwd)/sample_files:/sample_files -v $(pwd)/sample_files:/sample_files airbyte/twitter-fetcher:dev read --config /sample_files/config-example.json --catalog /sample_files/configured_catalog.json