bank-balance-connector: init connector

Signed-off-by: Alexis Pentori <alexis@status.im>

source-bank-balance-fetcher/Dockerfile

@@ -0,0 +1,8 @@
+FROM airbyte/python-connector-base:1.1.0
+
+COPY . ./airbyte/integration_code
+RUN pip install ./airbyte/integration_code
+
+# The entrypoint and default env vars are already set in the base image
+ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
+ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

source-bank-balance-fetcher/README.md

@@ -0,0 +1,166 @@
+# Bank Balance Fetcher Source
+
+This is the repository for the Bank Balance Fetcher source connector, written in Python.
+For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/sources/bank-balance-fetcher).
+
+## Local development
+
+### Prerequisites
+**To iterate on this connector, make sure to complete this prerequisites section.**
+
+#### Minimum Python version required `= 3.9.0`
+
+#### Activate Virtual Environment and install dependencies
+From this connector directory, create a virtual environment:
+```
+python -m venv .venv
+```
+
+This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your
+development environment of choice. To activate it from the terminal, run:
+```
+source .venv/bin/activate
+pip install -r requirements.txt
+pip install '.[tests]'
+```
+If you are in an IDE, follow your IDE's instructions to activate the virtualenv.
+
+Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is
+used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`.
+If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything
+should work as you expect.
+
+#### Create credentials
+**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/bank-balance-fetcher)
+to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_bank_balance_fetcher/spec.yaml` file.
+Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
+See `integration_tests/sample_config.json` for a sample config file.
+
+**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source bank-balance-fetcher test creds`
+and place them into `secrets/config.json`.
+
+### Locally running the connector
+```
+python main.py spec
+python main.py check --config secrets/config.json
+python main.py discover --config secrets/config.json
+python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
+```
+
+### Locally running the connector docker image
+
+#### Use `airbyte-ci` to build your connector
+The Airbyte way of building this connector is to use our `airbyte-ci` tool.
+You can follow install instructions [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#L1).
+Then running the following command will build your connector:
+
+```bash
+airbyte-ci connectors --name source-bank-balance-fetcher build
+```
+Once the command is done, you will find your connector image in your local docker registry: `airbyte/source-bank-balance-fetcher:dev`.
+
+##### Customizing our build process
+When contributing on our connector you might need to customize the build process to add a system dependency or set an env var.
+You can customize our build process by adding a `build_customization.py` module to your connector.
+This module should contain a `pre_connector_install` and `post_connector_install` async function that will mutate the base image and the connector container respectively.
+It will be imported at runtime by our build process and the functions will be called if they exist.
+
+Here is an example of a `build_customization.py` module:
+```python
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    # Feel free to check the dagger documentation for more information on the Container object and its methods.
+    # https://dagger-io.readthedocs.io/en/sdk-python-v0.6.4/
+    from dagger import Container
+
+
+async def pre_connector_install(base_image_container: Container) -> Container:
+    return await base_image_container.with_env_variable("MY_PRE_BUILD_ENV_VAR", "my_pre_build_env_var_value")
+
+async def post_connector_install(connector_container: Container) -> Container:
+    return await connector_container.with_env_variable("MY_POST_BUILD_ENV_VAR", "my_post_build_env_var_value")
+```
+
+#### Build your own connector image
+This connector is built using our dynamic built process in `airbyte-ci`.
+The base image used to build it is defined within the metadata.yaml file under the `connectorBuildOptions`.
+The build logic is defined using [Dagger](https://dagger.io/) [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/pipelines/builds/python_connectors.py).
+It does not rely on a Dockerfile.
+
+If you would like to patch our connector and build your own a simple approach would be to:
+
+1. Create your own Dockerfile based on the latest version of the connector image.
+```Dockerfile
+FROM airbyte/source-bank-balance-fetcher:latest
+
+COPY . ./airbyte/integration_code
+RUN pip install ./airbyte/integration_code
+
+# The entrypoint and default env vars are already set in the base image
+# ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
+# ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
+```
+Please use this as an example. This is not optimized.
+
+2. Build your image:
+```bash
+docker build -t airbyte/source-bank-balance-fetcher:dev .
+# Running the spec command against your patched connector
+docker run airbyte/source-bank-balance-fetcher:dev spec
+````
+
+#### Run
+Then run any of the connector commands as follows:
+```
+docker run --rm airbyte/source-bank-balance-fetcher:dev spec
+docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-bank-balance-fetcher:dev check --config /secrets/config.json
+docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-bank-balance-fetcher:dev discover --config /secrets/config.json
+docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-bank-balance-fetcher:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
+```
+## Testing
+Make sure to familiarize yourself with [pytest test discovery](https://docs.pytest.org/en/latest/goodpractices.html#test-discovery) to know how your test files and methods should be named.
+First install test dependencies into your virtual environment:
+```
+pip install .[tests]
+```
+### Unit Tests
+To run unit tests locally, from the connector directory run:
+```
+python -m pytest unit_tests
+```
+
+### Integration Tests
+There are two types of integration tests: Acceptance Tests (Airbyte's test suite for all source connectors) and custom integration tests (which are specific to this connector).
+#### Custom Integration tests
+Place custom tests inside `integration_tests/` folder, then, from the connector root, run
+```
+python -m pytest integration_tests
+```
+
+### Acceptance Tests
+Customize `acceptance-test-config.yml` file to configure tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
+If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
+Please run acceptance tests via [airbyte-ci](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#connectors-test-command):
+```bash
+airbyte-ci connectors --name source-bank-balance-fetcher test
+```
+
+## Dependency Management
+All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.
+We split dependencies between two groups, dependencies that are:
+* required for your connector to work need to go to `MAIN_REQUIREMENTS` list.
+* required for the testing need to go to `TEST_REQUIREMENTS` list
+
+### Publishing a new version of the connector
+You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
+1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-bank-balance-fetcher test`
+2. Bump the connector version in `metadata.yaml`: increment the `dockerImageTag` value. Please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors).
+3. Make sure the `metadata.yaml` content is up to date.
+4. Make the connector documentation and its changelog is up to date (`docs/integrations/sources/bank-balance-fetcher.md`).
+5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
+6. Pat yourself on the back for being an awesome contributor.
+7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
+

source-bank-balance-fetcher/main.py

@@ -0,0 +1,8 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+from source_bank_balance_fetcher.run import run
+
+if __name__ == "__main__":
+    run()

source-bank-balance-fetcher/metadata.yaml

@@ -0,0 +1,25 @@
+data:
+  allowedHosts:
+  registries:
+    oss:
+      enabled: false
+    cloud:
+      enabled: false
+  connectorBuildOptions:
+    baseImage: docker.io/airbyte/python-connector-base:1.0.0@sha256:dd17e347fbda94f7c3abff539be298a65af2d7fc27a307d89297df1081a45c27
+  connectorSubtype: api
+  connectorType: source
+  definitionId: 1c448bfb-8950-478c-9ae0-f03aaaf4e920
+  dockerImageTag: 0.0.2
+  dockerRepository: harbor.status.im/status-im/airbyte/source-bank-balance-fetcher
+  githubIssueLabel: source-bank-balance-fetcher
+  icon: bank-balance-fetcher.svg
+  license: MIT
+  name: Bank Balance Fetcher
+  releaseDate: TODO
+  supportLevel: community
+  releaseStage: alpha
+  documentationUrl: https://docs.airbyte.com/integrations/sources/bank-balance-fetcher
+  tags:
+    - language:python
+metadataSpecVersion: "1.0"

source-bank-balance-fetcher/requirements.txt

@@ -0,0 +1 @@
+-e .

source-bank-balance-fetcher/sample_files/config-example.json

@@ -0,0 +1,4 @@
+{
+  "url": "https://config.test.bi.status.im/api/bank",
+  "api_key": "something"
+}

source-bank-balance-fetcher/sample_files/configured_catalog.json

@@ -0,0 +1,18 @@
+{
+  "streams": [
+    {
+      "stream": {
+        "name": "bank_balance",
+        "json_schema": {
+          "$schema": "http://json-schema.org/draft-04/schema#",
+          "type": "object"
+        },
+        "supported_sync_modes": [
+          "full_refresh",  "incremental"
+        ]
+      },
+      "sync_mode":  "incremental",
+      "destination_sync_mode": "overwrite"
+    }
+  ]
+}

source-bank-balance-fetcher/setup.py

@@ -0,0 +1,35 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+
+from setuptools import find_packages, setup
+
+MAIN_REQUIREMENTS = [
+    "airbyte-cdk~=0.2",
+]
+
+TEST_REQUIREMENTS = [
+    "requests-mock~=1.9.3",
+    "pytest~=6.2",
+    "pytest-mock~=3.6.1",
+    "connector-acceptance-test",
+]
+
+setup(
+    name="source_bank_balance_fetcher",
+    description="Source implementation for Bank Balance.",
+    author="Status",
+    author_email="devops@status.im",
+    packages=find_packages(),
+    install_requires=MAIN_REQUIREMENTS,
+    package_data={"": ["*.json", "*.yaml", "schemas/*.json", "schemas/shared/*.json"]},
+    extras_require={
+        "tests": TEST_REQUIREMENTS,
+    },
+    entry_points={
+        "console_scripts": [
+            "source-bank-balance-fetcher=source_bank_balance_fetcher.run:run",
+        ],
+    },
+)

source-bank-balance-fetcher/source_bank_balance_fetcher/__init__.py

@@ -0,0 +1,8 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+
+from .source import SourceBankBalanceFetcher
+
+__all__ = ["SourceBankBalanceFetcher"]

source-bank-balance-fetcher/source_bank_balance_fetcher/run.py

@@ -0,0 +1,13 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+
+import sys
+
+from airbyte_cdk.entrypoint import launch
+from .source import SourceBankBalanceFetcher
+
+def run():
+    source = SourceBankBalanceFetcher()
+    launch(source, sys.argv[1:])

source-bank-balance-fetcher/source_bank_balance_fetcher/schemas/bank_balance.json

@@ -0,0 +1,67 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "type": "object",
+  "properties": {
+    "account_id": {
+      "type": [
+        "null",
+        "string"
+      ]
+    },
+    "bank_name": {
+      "type": [
+        "null",
+        "string"
+      ]
+    },
+    "ccy": {
+      "type": [
+        "null",
+        "string"
+      ]
+    },
+    "tier": {
+      "type": [
+        "null",
+        "string"
+      ]
+    },
+    "volatility": {
+      "type": [
+        "null",
+        "string"
+      ]
+    },
+    "asset_type": {
+      "type": [
+        "null",
+        "string"
+      ]
+    },
+    "account_owner": {
+      "type": [
+        "null",
+        "string"
+      ]
+    },
+    "balance_date": {
+      "type": [
+        "null",
+        "number"
+      ]
+    },
+    "balance": {
+      "type": [
+        "null",
+        "number"
+      ]
+    },
+    "balance_usd": {
+      "type": [
+        "null",
+        "number"
+      ]
+    }
+
+  }
+}

source-bank-balance-fetcher/source_bank_balance_fetcher/source.py

@@ -0,0 +1,52 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+
+from abc import ABC
+from typing import Any, Iterable, List, Mapping, MutableMapping, Optional, Tuple
+
+import requests
+from airbyte_cdk.sources import AbstractSource
+from airbyte_cdk.sources.streams import Stream
+from airbyte_cdk.sources.streams.http import HttpStream
+from airbyte_cdk.sources.streams.http.auth import TokenAuthenticator
+
+# Basic full refresh stream
+class BankBalance(HttpStream):
+
+    url_base = ""
+    primary_key = None
+
+    def __init__(self, api_key, url, **kwargs):
+        super().__init__(**kwargs)
+        self.api_key = api_key
+        self.url = url
+
+    def path(
+        self, stream_state: Mapping[str, Any] = None, stream_slice: Mapping[str, Any] = None, next_page_token: Mapping[str, Any] = None
+    ) -> str:
+        return f"{self.url}"
+
+    def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]:
+        return None
+
+    def request_headers(
+        self, stream_state: Mapping[str, Any], stream_slice: Mapping[str, any] = None, next_page_token: Mapping[str, Any] = None
+    ) -> MutableMapping[str, Any]:
+        return { "API-Key" : f"{self.api_key}"}
+
+    def parse_response(self, response: requests.Response, **kwargs) -> Iterable[Mapping]:
+        bank_balances = response.json()
+        for b in bank_balances:
+            yield b
+
+
+
+# Source
+class SourceBankBalanceFetcher(AbstractSource):
+    def check_connection(self, logger, config) -> Tuple[bool, any]:
+        return True, None
+
+    def streams(self, config: Mapping[str, Any]) -> List[Stream]:       
+        return [BankBalance(config['api_key'], config['url'])]

source-bank-balance-fetcher/source_bank_balance_fetcher/spec.yaml

@@ -0,0 +1,16 @@
+documentationUrl: https://docsurl.com
+connectionSpecification:
+  $schema: http://json-schema.org/draft-07/schema#
+  title: Bank Balance Fetcher Spec
+  type: object
+  required:
+    - url
+    - api_key
+  properties:
+    url:
+      type: string
+      description: URL of the data source containing the bank balances informations
+    api_key:
+      type: string
+      description: API Key to use the API
+      airbyte_secret: true