sartography
/
star-drive-dist
mirror of https://github.com/sartography/star-drive-dist.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
star-drive-dist/backend/README.md

11 KiB
Raw Permalink Blame History

Introduction

This is the API for Autism DRIVE, a platform to support Autism studies and participants.

Platform

This is a Python3 / Flask based api. It relies on a Relational Database for storing and organizing resources. It uses Elastic Search as a full text search engine for locating resources.

Prerequisites

Python 3 and python3-dev, and some cryptography libraries

MacOS:

brew install python

Debian:

sudo apt-get install python3 python3-dev
sudo apt-get install -y libssl-dev libffi-dev

Using Docker

You can use Docker to run the two primary dependencies for this project: Postgres, and ElasticSearch

  1. create a directory at ~/docker/volumes/stardrive
  2. chmod 777 ~/docker/volumes/stardrive/elasticsearch
  3. spin it all up, with docker-compose up
  • Optionally set up a run configuration in PyCharm

Not Using Docker

PostgreSQL
  • MacOS: Download and install Postgres.app. This will install postgres, along with the command-line tools, including psql, pg_ctl, and others. Then update your PATH variable to look in the Postgres.app bin directory for the relevant Postgres CLI tools.
export PATH="/Applications/Postgres.app/Contents/Versions/latest/bin:$PATH"
  • Debian/Ubuntu:
apt-get install postgresql postgresql-client libpq-dev
Database Setup

NOTE: Docker will do this automatically, only necissary if you are doing it locally. NOTE: The configuration is currently set up to use "ed_pass" as a password. You will be promoted to enter a password when you connect.

  • MacOS:
postgres -D /usr/local/var/postgres
createuser --no-createdb --no-superuser --pwprompt ed_user
createdb stardrive -O ed_user ed_platform
createdb stardrive_test -O ed_user ed_platform
  • Debian
sudo su postgres
createuser --no-createdb --no-superuser --pwprompt ed_user
createdb stardrive -O ed_user ed_platform WITH ENCODING = 'UTF8'
createdb stardrive_test -O ed_user ed_platform
exit

If you are using Ubuntu you will likely need to enable PSQL to manage its own users.

ElasticSearch

We are currently using version 6, and should look at upgrading this in the future when my hair isn't on fire.

Angular

You will need the angular command line utilities to run the front end.

npm install -g @angular/cli

Project Setup

  • Please use Python 3's virtual environment setup, and install the dependencies in requirements.txt
cd backend
python3 -m venv python-env
source python-env/bin/activate
pip3 install -r requirements.txt

Add a config file

In the backend directory, execute the following command:

mkdir instance && cp -r config instance/config && cp instance/config/default.py instance/config.py

Update the Database

You will need to update your database each time you return to do a pull to make sure all the migrations are run. In the backend directory, execute the following command:

flask db upgrade

Database Reset

If you encounter errors with the database, you can blow it away completely and reset with the following commands in PSQL:

DROP SCHEMA public CASCADE;
CREATE SCHEMA public;
GRANT ALL ON SCHEMA public TO postgres;
GRANT ALL ON SCHEMA public TO public;
COMMENT ON SCHEMA public IS 'standard public schema';

Update Data Models

Each time you modify your data models you will need to create new migrations. The following command will compare the database to the code and create new migrations as needed. You can edit this newly generated file - it will show up under migrations/versions

flask db migrate

Load in the seed data

This will pull in initial values into the database.

flask initdb

Run the app

  • Strongly suggest you set up run configurations for this in PyCharm and WebStorm

Start the backend app

In the backend directory, execute the following command:

flask run

Start the frontend app

In the frontend directory, execute the following commands:

npm install
ng serve

Setting up Mailtrap

To test email integration set up an account with Mailtrap In your instance config, set your sname and password. You can find these values in your Mailtrap account in the settings portion of your inbox; you will not find these values in your general account settings as the username and password are specific to each inbox you create.

MAIL_USERNAME = "numbersandletters" MAIL_PASSWORD = "lettersandnumbers"

Also note that mail is handled differently for tests. Make sure that your instance config has

TESTING = False

Maintenance

Clear out the database, indexes, and reseed the database

This will remove all data from the database, delete all information from the ElasticSearch Index, and remove all data and recreate it from the example data files. In the backend directory, execute the following command:

source python-env/bin/activate
export FLASK_APP=./app/__init__.py
flask cleardb
flask db upgrade
flask initdb

Migration Conflicts

If you find yourself with conflicting migrations, it might be appropriate to resolve with a merge:

flask db merge -m "merge cc4610a6ece3 and 2679ef53e0bd" cc4610a6ece3 2679ef53e0bd

This will auto-generate a new migration that ties the streams together.

Migrations with Enum columns

Alembic probably will not generate migrations that do everything that you need them to do when it comes to handling Enum values. Look at migration versions 2fd0ab60fe3a_.py and 5fb917adc751_.py to see some examples of handling enum additions and changes. 2fd0ab60fe3a_.py shows how to delete an existing column and replace it with an enum (I haven't found a way to simply alter the column and get the migrations to upgrade and downgrade successfully). 5fb917adc751_.py shows an example with adding a value to the enum list.

Best Practices

There are a few things I hope to do consistently for this project, to avoid some pitfalls from the Cadre Academy site. When adding code please follow these guidelines:

Write tests.

Don't commit code that doesn't have at least some basic testing of the new functionality. Don't commit code without all tests passing.

Database calls

  • Favor db.session.query over using the models for these calls.
db.session.query( ...

not

models.Resrouce.query( ...

Security / Authentication

This will become increasingly complicated, so check back here often. At present the system can handle single sign on (SSO) authentication through Shibboleth via a connector on the apache web server that looks for headers we know are provided by the University of Virginia. This will change in the future as we migrate to using a OnConnect which will allow connections with more institutions. We'll also need to offer direct log-ins for community users.

Once credentials are established, the front end (Angular) and backend (Flask/Python) will use a JWT token.

Develoment Mode

The SSO aspect is bypassed in Development mode. Clicking the log in button will immediately log you in as the user specified in your instance/config.py.

SSO_DEVELOPMENT_UID = 'dhf8r'

I've created users for primary developers in our example_data, and that information is loaded into the database automatically with a flask reset Add yourself there if needed.

Production Mode

In production, the redirect on the front end needs to point to the url that will direct us out to Shibboleth. The account we use will send the user back to an API endpoint that will generate a JWT token, and then redirect again to the front end, passing that token along as a GET parameter.

Testing

Run backend tests

Make sure you have set up your test database (see Database Setup above) You can use nose2 to execute all of tests, or you can run them individually using Pycharm or other IDE. In the backend directory, execute the following command:

source python-env/bin/activate
export FLASK_APP=./app/__init__.py
nose2

Run frontend tests

Make sure you have the database, backend, and frontend all running.

Execute the following at the top level of the repository, which will clear and re-seed the database, then run all e2e tests:

./test-e2e.sh

Alternatively, to run the e2e tests without reseeding first, execute the following command in the frontend directory:

ng e2e --dev-server-target=

Public/Private Configuration

For the most part, you don't need to run the second configuration in development unless you are specifically working on the handoff of data. The private mirror will pull data from the public instance and then potentially remove it.

To run the private mirror instance, you'll need a second database:

Run the following command (as postgres) from a bash prompt.

createdb stardrive_mirror -O ed_user ed_platform

Configuration

You will need to specify a different configuration file for the private mirror instance. A set of reasonable defaults for development is available under the "mirror.py" in the config directory. You can set an environment variable to specify this when you fire up the mirroring instance.

APP_CONFIG_FILE=/full/path/to/config/mirror.py

Note that it should be the full path. You'll be running both instances, so don't set this environment variable for all commands, just for running the instance. For me, I have it set as an environment variable under the Run Configuration within PyCharm. I copied by existing run command and added this environment variable there. You will also need to add a port (5001) argument so you aren't running on the same port as the primary server. Below are the settings in my Run configuration:

Parameters: 5001 Environment Variable: PYTHONUNBUFFERED=1;MIRRORING=true

You will need to build the basic data structures in the database in order to load data for this you will need to run the init_db flask command, but you will need to make that specific to the mirror instance. You'll need to provide ALL the environment settings with the flask command for it to work correctly.

FLASK_APP=app/__init__.py MIRRORING=true flask db upgrade

Production Deployment

You will need to install:

  • Python 3
  • Elastic Search 6
  • Apache Web Server
    • mod-wsgi (for running flask apps within apache)
  • Postgres

I've tended to set up the website under /var/www/star or /var/www/autismdrive (the new name and url) In the /var/www/autismdrive/ I create a python virtual enviroment with the command:

python3 -m venv python-env

When pushing to production please create a new 'Release' on gitHub describing the changes that were rolled out.

Currently we are using a separate repository for deployment: star-drive-dist. To prepare this for deployment, you should have a copy of star-drive-dist in the same directory as your local copy of star-drive. Your local star-drive should be on master, up to date with all the changes for the release. Once this is ready, run the prepare_for_deploy.sh script in star-drive-dist to prepare the release. Commit these changes referencing the release number and push them up.