airbyte-custom-connector/source-discord-fetcher/README.md

91 lines
3.2 KiB
Markdown
Raw Permalink Normal View History

# Discord Fetcher Source
This is the repository for fetching data from Discord Server, written in Python.
> *Note*: In the rest of the document, the term guild design a discord server (API term)
## Usage
This connector fetches channels and members data from servers instances.
### Configuration
The connector takes the following input:
```yaml
api_key: 'Token of the bot used'
guilds_id:
- 123456789
```
The `guild_id` can be found in the `server setting` > `Widget` > `Server Id`
The `api-key` is the token of a bot account associated to the servers.
#### Bot Configuration
In order to access the API endpoints of the server, the connector must be authentified as a discord BOT with the priviledge intent `SERVER MEMBERS INTENT`.
For that:
1. Create a Discord Application at https://discord.com/developers/applications
2. Generate a BOT account:
* Go to the OAuth2 page on the application settings:
* https://discord.com/developers/applications/<app-id>/oauth2)
* Store the Client ID and Client Secret in a password manager
* Select `bot` in the `Oauth2 URL generator` scope and copy the url at the end of the page.
* Visite the URL and select the Discord Server you want to log into.
3. Configure the BOT token
* Go to the `Bot` page of the application settings
- https://discord.com/developers/applications/<app-id>/bot
* Store the bot token in a password manager (the `api-key`)
* Select `SERVER MEMBERS INTENT` in the `Privileged Gateway Intents` category. It will give the bot access to the `members` endpoint.
### Output
The connector will return the following:
- `guild`: List of server information based on the `guilds_id` values.
- `guild_channel`: List of channel for each discord server (contains partial data).
- `channel`: List of channel for each discord server.
- `members`: List of user on the Discord server.
## Local development
### Prerequisites
#### Activate Virtual Environment and install dependencies
From this connector directory, create a virtual environment:
```
python -m venv .venv
```
```
source .venv/bin/activate
pip install -r requirements.txt
```
### Locally running the connector
```
python main.py spec
python main.py check --config sample_files/config-example.json
python main.py discover --config sample_files/config-example.json
python main.py read --config sample_files/config-example.json --catalog sample_files/configured_catalog.json
```
### Locally running the connector docker image
```bash
docker build -t airbyte/twitter-fetcher:dev .
# Running the spec command against your patched connector
docker run airbyte/twitter-fetcher:dev spec
````
#### Run
Then run any of the connector commands as follows:
```
docker run --rm airbyte/twitter-fetcher:dev spec
docker run --rm -v $(pwd)/sample_files:/sample_files airbyte/twitter-fetcher:dev check --config /sample_files/config-example.json
docker run --rm -v $(pwd)/sample_files:/sample_files airbyte/twitter-fetcher:dev discover --config /sample_files/config-example.json
docker run --rm -v $(pwd)/sample_files:/sample_files -v $(pwd)/sample_files:/sample_files airbyte/twitter-fetcher:dev read --config /sample_files/config-example.json --catalog /sample_files/configured_catalog.json
```