bank-balance-connector: init connector
Signed-off-by: Alexis Pentori <alexis@status.im>
This commit is contained in:
parent
8ec37e115b
commit
f367ff8728
|
@ -0,0 +1,8 @@
|
||||||
|
FROM airbyte/python-connector-base:1.1.0
|
||||||
|
|
||||||
|
COPY . ./airbyte/integration_code
|
||||||
|
RUN pip install ./airbyte/integration_code
|
||||||
|
|
||||||
|
# The entrypoint and default env vars are already set in the base image
|
||||||
|
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
|
||||||
|
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
|
|
@ -0,0 +1,166 @@
|
||||||
|
# Bank Balance Fetcher Source
|
||||||
|
|
||||||
|
This is the repository for the Bank Balance Fetcher source connector, written in Python.
|
||||||
|
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/sources/bank-balance-fetcher).
|
||||||
|
|
||||||
|
## Local development
|
||||||
|
|
||||||
|
### Prerequisites
|
||||||
|
**To iterate on this connector, make sure to complete this prerequisites section.**
|
||||||
|
|
||||||
|
#### Minimum Python version required `= 3.9.0`
|
||||||
|
|
||||||
|
#### Activate Virtual Environment and install dependencies
|
||||||
|
From this connector directory, create a virtual environment:
|
||||||
|
```
|
||||||
|
python -m venv .venv
|
||||||
|
```
|
||||||
|
|
||||||
|
This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your
|
||||||
|
development environment of choice. To activate it from the terminal, run:
|
||||||
|
```
|
||||||
|
source .venv/bin/activate
|
||||||
|
pip install -r requirements.txt
|
||||||
|
pip install '.[tests]'
|
||||||
|
```
|
||||||
|
If you are in an IDE, follow your IDE's instructions to activate the virtualenv.
|
||||||
|
|
||||||
|
Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is
|
||||||
|
used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`.
|
||||||
|
If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything
|
||||||
|
should work as you expect.
|
||||||
|
|
||||||
|
#### Create credentials
|
||||||
|
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/bank-balance-fetcher)
|
||||||
|
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_bank_balance_fetcher/spec.yaml` file.
|
||||||
|
Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
|
||||||
|
See `integration_tests/sample_config.json` for a sample config file.
|
||||||
|
|
||||||
|
**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source bank-balance-fetcher test creds`
|
||||||
|
and place them into `secrets/config.json`.
|
||||||
|
|
||||||
|
### Locally running the connector
|
||||||
|
```
|
||||||
|
python main.py spec
|
||||||
|
python main.py check --config secrets/config.json
|
||||||
|
python main.py discover --config secrets/config.json
|
||||||
|
python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
|
||||||
|
```
|
||||||
|
|
||||||
|
### Locally running the connector docker image
|
||||||
|
|
||||||
|
#### Use `airbyte-ci` to build your connector
|
||||||
|
The Airbyte way of building this connector is to use our `airbyte-ci` tool.
|
||||||
|
You can follow install instructions [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#L1).
|
||||||
|
Then running the following command will build your connector:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
airbyte-ci connectors --name source-bank-balance-fetcher build
|
||||||
|
```
|
||||||
|
Once the command is done, you will find your connector image in your local docker registry: `airbyte/source-bank-balance-fetcher:dev`.
|
||||||
|
|
||||||
|
##### Customizing our build process
|
||||||
|
When contributing on our connector you might need to customize the build process to add a system dependency or set an env var.
|
||||||
|
You can customize our build process by adding a `build_customization.py` module to your connector.
|
||||||
|
This module should contain a `pre_connector_install` and `post_connector_install` async function that will mutate the base image and the connector container respectively.
|
||||||
|
It will be imported at runtime by our build process and the functions will be called if they exist.
|
||||||
|
|
||||||
|
Here is an example of a `build_customization.py` module:
|
||||||
|
```python
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
# Feel free to check the dagger documentation for more information on the Container object and its methods.
|
||||||
|
# https://dagger-io.readthedocs.io/en/sdk-python-v0.6.4/
|
||||||
|
from dagger import Container
|
||||||
|
|
||||||
|
|
||||||
|
async def pre_connector_install(base_image_container: Container) -> Container:
|
||||||
|
return await base_image_container.with_env_variable("MY_PRE_BUILD_ENV_VAR", "my_pre_build_env_var_value")
|
||||||
|
|
||||||
|
async def post_connector_install(connector_container: Container) -> Container:
|
||||||
|
return await connector_container.with_env_variable("MY_POST_BUILD_ENV_VAR", "my_post_build_env_var_value")
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Build your own connector image
|
||||||
|
This connector is built using our dynamic built process in `airbyte-ci`.
|
||||||
|
The base image used to build it is defined within the metadata.yaml file under the `connectorBuildOptions`.
|
||||||
|
The build logic is defined using [Dagger](https://dagger.io/) [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/pipelines/builds/python_connectors.py).
|
||||||
|
It does not rely on a Dockerfile.
|
||||||
|
|
||||||
|
If you would like to patch our connector and build your own a simple approach would be to:
|
||||||
|
|
||||||
|
1. Create your own Dockerfile based on the latest version of the connector image.
|
||||||
|
```Dockerfile
|
||||||
|
FROM airbyte/source-bank-balance-fetcher:latest
|
||||||
|
|
||||||
|
COPY . ./airbyte/integration_code
|
||||||
|
RUN pip install ./airbyte/integration_code
|
||||||
|
|
||||||
|
# The entrypoint and default env vars are already set in the base image
|
||||||
|
# ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
|
||||||
|
# ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
|
||||||
|
```
|
||||||
|
Please use this as an example. This is not optimized.
|
||||||
|
|
||||||
|
2. Build your image:
|
||||||
|
```bash
|
||||||
|
docker build -t airbyte/source-bank-balance-fetcher:dev .
|
||||||
|
# Running the spec command against your patched connector
|
||||||
|
docker run airbyte/source-bank-balance-fetcher:dev spec
|
||||||
|
````
|
||||||
|
|
||||||
|
#### Run
|
||||||
|
Then run any of the connector commands as follows:
|
||||||
|
```
|
||||||
|
docker run --rm airbyte/source-bank-balance-fetcher:dev spec
|
||||||
|
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-bank-balance-fetcher:dev check --config /secrets/config.json
|
||||||
|
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-bank-balance-fetcher:dev discover --config /secrets/config.json
|
||||||
|
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-bank-balance-fetcher:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
|
||||||
|
```
|
||||||
|
## Testing
|
||||||
|
Make sure to familiarize yourself with [pytest test discovery](https://docs.pytest.org/en/latest/goodpractices.html#test-discovery) to know how your test files and methods should be named.
|
||||||
|
First install test dependencies into your virtual environment:
|
||||||
|
```
|
||||||
|
pip install .[tests]
|
||||||
|
```
|
||||||
|
### Unit Tests
|
||||||
|
To run unit tests locally, from the connector directory run:
|
||||||
|
```
|
||||||
|
python -m pytest unit_tests
|
||||||
|
```
|
||||||
|
|
||||||
|
### Integration Tests
|
||||||
|
There are two types of integration tests: Acceptance Tests (Airbyte's test suite for all source connectors) and custom integration tests (which are specific to this connector).
|
||||||
|
#### Custom Integration tests
|
||||||
|
Place custom tests inside `integration_tests/` folder, then, from the connector root, run
|
||||||
|
```
|
||||||
|
python -m pytest integration_tests
|
||||||
|
```
|
||||||
|
|
||||||
|
### Acceptance Tests
|
||||||
|
Customize `acceptance-test-config.yml` file to configure tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
|
||||||
|
If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
|
||||||
|
Please run acceptance tests via [airbyte-ci](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#connectors-test-command):
|
||||||
|
```bash
|
||||||
|
airbyte-ci connectors --name source-bank-balance-fetcher test
|
||||||
|
```
|
||||||
|
|
||||||
|
## Dependency Management
|
||||||
|
All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.
|
||||||
|
We split dependencies between two groups, dependencies that are:
|
||||||
|
* required for your connector to work need to go to `MAIN_REQUIREMENTS` list.
|
||||||
|
* required for the testing need to go to `TEST_REQUIREMENTS` list
|
||||||
|
|
||||||
|
### Publishing a new version of the connector
|
||||||
|
You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
|
||||||
|
1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-bank-balance-fetcher test`
|
||||||
|
2. Bump the connector version in `metadata.yaml`: increment the `dockerImageTag` value. Please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors).
|
||||||
|
3. Make sure the `metadata.yaml` content is up to date.
|
||||||
|
4. Make the connector documentation and its changelog is up to date (`docs/integrations/sources/bank-balance-fetcher.md`).
|
||||||
|
5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
|
||||||
|
6. Pat yourself on the back for being an awesome contributor.
|
||||||
|
7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
|
||||||
|
|
|
@ -0,0 +1,8 @@
|
||||||
|
#
|
||||||
|
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||||
|
#
|
||||||
|
|
||||||
|
from source_bank_balance_fetcher.run import run
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
run()
|
|
@ -0,0 +1,25 @@
|
||||||
|
data:
|
||||||
|
allowedHosts:
|
||||||
|
registries:
|
||||||
|
oss:
|
||||||
|
enabled: false
|
||||||
|
cloud:
|
||||||
|
enabled: false
|
||||||
|
connectorBuildOptions:
|
||||||
|
baseImage: docker.io/airbyte/python-connector-base:1.0.0@sha256:dd17e347fbda94f7c3abff539be298a65af2d7fc27a307d89297df1081a45c27
|
||||||
|
connectorSubtype: api
|
||||||
|
connectorType: source
|
||||||
|
definitionId: 1c448bfb-8950-478c-9ae0-f03aaaf4e920
|
||||||
|
dockerImageTag: 0.0.2
|
||||||
|
dockerRepository: harbor.status.im/status-im/airbyte/source-bank-balance-fetcher
|
||||||
|
githubIssueLabel: source-bank-balance-fetcher
|
||||||
|
icon: bank-balance-fetcher.svg
|
||||||
|
license: MIT
|
||||||
|
name: Bank Balance Fetcher
|
||||||
|
releaseDate: TODO
|
||||||
|
supportLevel: community
|
||||||
|
releaseStage: alpha
|
||||||
|
documentationUrl: https://docs.airbyte.com/integrations/sources/bank-balance-fetcher
|
||||||
|
tags:
|
||||||
|
- language:python
|
||||||
|
metadataSpecVersion: "1.0"
|
|
@ -0,0 +1 @@
|
||||||
|
-e .
|
|
@ -0,0 +1,4 @@
|
||||||
|
{
|
||||||
|
"url": "https://config.test.bi.status.im/api/bank",
|
||||||
|
"api_key": "something"
|
||||||
|
}
|
|
@ -0,0 +1,18 @@
|
||||||
|
{
|
||||||
|
"streams": [
|
||||||
|
{
|
||||||
|
"stream": {
|
||||||
|
"name": "bank_balance",
|
||||||
|
"json_schema": {
|
||||||
|
"$schema": "http://json-schema.org/draft-04/schema#",
|
||||||
|
"type": "object"
|
||||||
|
},
|
||||||
|
"supported_sync_modes": [
|
||||||
|
"full_refresh", "incremental"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"sync_mode": "incremental",
|
||||||
|
"destination_sync_mode": "overwrite"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
|
@ -0,0 +1,35 @@
|
||||||
|
#
|
||||||
|
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||||
|
#
|
||||||
|
|
||||||
|
|
||||||
|
from setuptools import find_packages, setup
|
||||||
|
|
||||||
|
MAIN_REQUIREMENTS = [
|
||||||
|
"airbyte-cdk~=0.2",
|
||||||
|
]
|
||||||
|
|
||||||
|
TEST_REQUIREMENTS = [
|
||||||
|
"requests-mock~=1.9.3",
|
||||||
|
"pytest~=6.2",
|
||||||
|
"pytest-mock~=3.6.1",
|
||||||
|
"connector-acceptance-test",
|
||||||
|
]
|
||||||
|
|
||||||
|
setup(
|
||||||
|
name="source_bank_balance_fetcher",
|
||||||
|
description="Source implementation for Bank Balance.",
|
||||||
|
author="Status",
|
||||||
|
author_email="devops@status.im",
|
||||||
|
packages=find_packages(),
|
||||||
|
install_requires=MAIN_REQUIREMENTS,
|
||||||
|
package_data={"": ["*.json", "*.yaml", "schemas/*.json", "schemas/shared/*.json"]},
|
||||||
|
extras_require={
|
||||||
|
"tests": TEST_REQUIREMENTS,
|
||||||
|
},
|
||||||
|
entry_points={
|
||||||
|
"console_scripts": [
|
||||||
|
"source-bank-balance-fetcher=source_bank_balance_fetcher.run:run",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
)
|
|
@ -0,0 +1,8 @@
|
||||||
|
#
|
||||||
|
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||||
|
#
|
||||||
|
|
||||||
|
|
||||||
|
from .source import SourceBankBalanceFetcher
|
||||||
|
|
||||||
|
__all__ = ["SourceBankBalanceFetcher"]
|
|
@ -0,0 +1,13 @@
|
||||||
|
#
|
||||||
|
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||||
|
#
|
||||||
|
|
||||||
|
|
||||||
|
import sys
|
||||||
|
|
||||||
|
from airbyte_cdk.entrypoint import launch
|
||||||
|
from .source import SourceBankBalanceFetcher
|
||||||
|
|
||||||
|
def run():
|
||||||
|
source = SourceBankBalanceFetcher()
|
||||||
|
launch(source, sys.argv[1:])
|
|
@ -0,0 +1,67 @@
|
||||||
|
{
|
||||||
|
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"account_id": {
|
||||||
|
"type": [
|
||||||
|
"null",
|
||||||
|
"string"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"bank_name": {
|
||||||
|
"type": [
|
||||||
|
"null",
|
||||||
|
"string"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"ccy": {
|
||||||
|
"type": [
|
||||||
|
"null",
|
||||||
|
"string"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"tier": {
|
||||||
|
"type": [
|
||||||
|
"null",
|
||||||
|
"string"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"volatility": {
|
||||||
|
"type": [
|
||||||
|
"null",
|
||||||
|
"string"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"asset_type": {
|
||||||
|
"type": [
|
||||||
|
"null",
|
||||||
|
"string"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"account_owner": {
|
||||||
|
"type": [
|
||||||
|
"null",
|
||||||
|
"string"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"balance_date": {
|
||||||
|
"type": [
|
||||||
|
"null",
|
||||||
|
"number"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"balance": {
|
||||||
|
"type": [
|
||||||
|
"null",
|
||||||
|
"number"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"balance_usd": {
|
||||||
|
"type": [
|
||||||
|
"null",
|
||||||
|
"number"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
}
|
||||||
|
}
|
|
@ -0,0 +1,52 @@
|
||||||
|
#
|
||||||
|
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||||
|
#
|
||||||
|
|
||||||
|
|
||||||
|
from abc import ABC
|
||||||
|
from typing import Any, Iterable, List, Mapping, MutableMapping, Optional, Tuple
|
||||||
|
|
||||||
|
import requests
|
||||||
|
from airbyte_cdk.sources import AbstractSource
|
||||||
|
from airbyte_cdk.sources.streams import Stream
|
||||||
|
from airbyte_cdk.sources.streams.http import HttpStream
|
||||||
|
from airbyte_cdk.sources.streams.http.auth import TokenAuthenticator
|
||||||
|
|
||||||
|
# Basic full refresh stream
|
||||||
|
class BankBalance(HttpStream):
|
||||||
|
|
||||||
|
url_base = ""
|
||||||
|
primary_key = None
|
||||||
|
|
||||||
|
def __init__(self, api_key, url, **kwargs):
|
||||||
|
super().__init__(**kwargs)
|
||||||
|
self.api_key = api_key
|
||||||
|
self.url = url
|
||||||
|
|
||||||
|
def path(
|
||||||
|
self, stream_state: Mapping[str, Any] = None, stream_slice: Mapping[str, Any] = None, next_page_token: Mapping[str, Any] = None
|
||||||
|
) -> str:
|
||||||
|
return f"{self.url}"
|
||||||
|
|
||||||
|
def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]:
|
||||||
|
return None
|
||||||
|
|
||||||
|
def request_headers(
|
||||||
|
self, stream_state: Mapping[str, Any], stream_slice: Mapping[str, any] = None, next_page_token: Mapping[str, Any] = None
|
||||||
|
) -> MutableMapping[str, Any]:
|
||||||
|
return { "API-Key" : f"{self.api_key}"}
|
||||||
|
|
||||||
|
def parse_response(self, response: requests.Response, **kwargs) -> Iterable[Mapping]:
|
||||||
|
bank_balances = response.json()
|
||||||
|
for b in bank_balances:
|
||||||
|
yield b
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# Source
|
||||||
|
class SourceBankBalanceFetcher(AbstractSource):
|
||||||
|
def check_connection(self, logger, config) -> Tuple[bool, any]:
|
||||||
|
return True, None
|
||||||
|
|
||||||
|
def streams(self, config: Mapping[str, Any]) -> List[Stream]:
|
||||||
|
return [BankBalance(config['api_key'], config['url'])]
|
|
@ -0,0 +1,16 @@
|
||||||
|
documentationUrl: https://docsurl.com
|
||||||
|
connectionSpecification:
|
||||||
|
$schema: http://json-schema.org/draft-07/schema#
|
||||||
|
title: Bank Balance Fetcher Spec
|
||||||
|
type: object
|
||||||
|
required:
|
||||||
|
- url
|
||||||
|
- api_key
|
||||||
|
properties:
|
||||||
|
url:
|
||||||
|
type: string
|
||||||
|
description: URL of the data source containing the bank balances informations
|
||||||
|
api_key:
|
||||||
|
type: string
|
||||||
|
description: API Key to use the API
|
||||||
|
airbyte_secret: true
|
Loading…
Reference in New Issue