sartography/cr-connect-workflow
1
0
Fork 0
mirror of https://github.com/sartography/cr-connect-workflow.git synced 2025-02-23 05:08:32 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,091 Commits 14 Branches 0 Tags 9 MiB
Commit Graph

331 Commits

Author SHA1 Message Date
Dan Funk
c79415a794 throw a sensible error when study is not found on get_study (don't 500) 
some ugly fixes in the file_service for improving panda output from spreadsheet processing that I need to revist.
now that the spiff-workflow handles multi-instance, we can't have random multi-instance tasks around.
Improved tests around study deletion.
 2020-04-08 13:28:43 -04:00
Dan Funk
c7a18709ee Merge branch 'master' of github.com:sartography/cr-connect-workflow 2020-04-07 14:09:28 -04:00
Dan Funk
697127660f Assure that all script tasks place data in a dictionary that is named exactly the same as the class - which is also the same as the Script tag. 2020-04-07 14:09:21 -04:00
Aaron Louie
d23b5e747b Replaces deprecated assertEquals with assertEqual 2020-04-06 18:15:32 -04:00
Dan Funk
c6b6ee5d70 Renamed the required_docs script to just "documents", and it returns all documented in the irb_documents look up table indexed on the "Code" - so details become available in the task data like "documents.IRB_INFOSEC_DOC.required". 
Updated the irb_documents with shorter code names, thanks to Alex. Re-worked the DMN models so they can properly read from this new datastructure.
 2020-04-06 16:56:00 -04:00
Dan Funk
e283b86466 Fixing a bug with deleting a study. 2020-04-06 13:08:17 -04:00
Dan Funk
a322801c91 Allow a study to be deleted, even if some statistics are laying around. 2020-04-03 16:41:16 -04:00
Dan Funk
60a10bb688 Marshmallow isn't the right tool when dealing with large models with lots of null values. Rather than fight the process of mamaging the Study Details, I'm letting that fall through, and we can test on an individual value or maybe set up a constants array when that becomes meaningful. 2020-04-03 16:24:38 -04:00
Dan Funk
c86f3321c5 reworking the test_study_service to run from a clean state, this is growing so complex. Tests really need to be isolated better. 2020-03-30 15:39:50 -04:00
Dan Funk
17796193de fixing a bug that was causing failing tests. 
Adding id and spec_version to the workflow metadata.
Refactoring the processing of the master_spec so that it doesn't polute the workflow database.
Adding tests to assure that the status and counts are updated on the workflow model as users make progress.
 2020-03-30 14:01:57 -04:00
Dan Funk
34b6ec92bf updating the API 
Removing the call for study/workflows - as workflow information is returned with the study by default.
Fixing a bug in the workflow spec model schema.
 2020-03-30 10:12:10 -04:00
Dan Funk
4a916c1ee3 Created a "StudyService" and moved all complex logic around study manipulation out of the study api, and this service, as things were getting complicated. The Workflow Processor no longer creates the WorkflowModel, the study object handles that, and only passes the model into the workflow processor when it is ready to start the workflow. 
Created a Study object (seperate from the StudyModel) that can cronstructed on request, and contains a different data structure than we store in the DB. This allows us to return underlying Categories and Workflows in a clean way.

Added a new status to workflows called "not_started", meaning we have not yet instantiated a processor or created a BPMN, they have no version yet and no stored data, just the possiblity of being started.

The Top Level Workflow or "Master" workflow is now a part of the sample data, and loaded at all times.

Removed the ability to "add a workflow to a study" and "remove a workflow from a study", a study contains all possible workflows by definition.

Example data no longer creates users or studies, it just creates the specs.
 2020-03-30 08:00:16 -04:00
Aaron Louie
6ebd4dce42 WIP: Adds default workflow spec categories 2020-03-27 15:32:07 -04:00
Dan Funk
b61a35f956 Merge remote-tracking branch 'origin/master' into feature/status_refactor 
Fixing adding a study so all workflows are again added, will add status on those workflows based on output from the master bpmn diagram, which is coming shortly.
 2020-03-27 11:55:36 -04:00
Dan Funk
c7d2c28178 Vastly more informative ApiError model that provides details on the underlying task where the error occured. 
Added a validate_workflow_specification endpoint that allows you to check if the workflow will execute from beginning to end using random data.
Minor fixes to existing bpmns to allow them to pass.
All scripts must include a "do_task_validate_only" that restricts external calls and database modifications, but performs as much logic as possible.
 2020-03-27 08:29:31 -04:00
Aaron Louie
f77ebe3177 Adds display order to categories 2020-03-26 15:29:52 -04:00
Dan Funk
e2c408b70d Removed all self-referential calls in the study_api. One api endpoint should never call another api endpoint. Moved the logic for updating a study to the study Model, rather than checking and setting dictionary values which will become very hard to maintain. 
The protocol builder service now returns real models, not dictionaries, forcing proper validation and fail-fast behavior.
Changed the name of the "status" spec, to "top_level_workflow" and removing any connection a workflow or study has with this specification.  It is only unused to determine status in real time, and is not reused or tracked.
Modified the required documents script to return a dictionary and not an array, making it easier to speak to specific values in the BPMN and DMN.
Working on new ways to test the top_level_workflow in the context of updates, this is still a work in progress.
Making use of several modifications to the Spiff library that enables more complex expressions in DMN models. This is evident in the new DMN models for the top_level_workflow
 2020-03-26 12:51:53 -04:00
Aaron Louie
4ed89eec5b Mocks Protocol Builder response 2020-03-26 10:30:15 -04:00
Aaron Louie
3fb9fb9b22 Simplifies status BPMN & DMN. 2020-03-26 10:06:12 -04:00
Dan Funk
37a8b69ffe Cleaning up a failing test, not sure why this isn't failing locally? 2020-03-24 14:40:24 -04:00
Dan Funk
f4342fc785 It became impossible to use the Swagger ui when we started adding authentication to all of the calls. I discovered Connexion and Swagger have a default way of handing JTW authentication and this cleans up our code quite a bit, moves the securing of endpoints into the API Definition, which is quite nice, and removes a whole library dependency (I never get to do that!) I added a SWAGGER_AUTH_KEY that can be used in non-production environments to allow users to quickly authenticate from the Swagger ui. I also removed all api calls to simple little happy api services, because that is just mean and pointless. 2020-03-24 14:15:21 -04:00
Dan Funk
b35427523d Merge remote-tracking branch 'origin/master' into feature/reference_files 
# Conflicts:
#	crc/models/file.py
#	crc/services/file_service.py
#	tests/data/reference/irb_documents.xlsx
#	tests/test_files_api.py
 2020-03-20 11:07:55 -04:00
Dan Funk
16c6ba9661 Removing unneeded files. 2020-03-20 08:33:37 -04:00
Dan Funk
3eb1167b8e Found a few errors in the sqlalchemy file definition that was causing failures, and had some consistency problems with the IRB Categories file name. The API was bailing out because we had restricted file types to bpmn,svg,or dmn in the connexion config file, I don't restrict this anymore we have plenty of checks elsewhere. Adding xlrd as a dependency - this didn't fail till a push to production. 2020-03-20 08:21:21 -04:00
Dan Funk
6e3b6c2635 Assure that files uploaded through web forms and files generated from templates can be cross-referenced to known document requirements from the protocol builder. Configurators can control this by managing an XLS Spreadsheet called "irb_documents.xslx". 
Required Documents is becoming complicated, so making this it's own script task, removing it from study_info.py
The file_service is now very aware of this irb_documents file, so it will always need to exist.  We seed this file
during setup, but it can be overwritten by the configurator.
 2020-03-19 17:14:20 -04:00
Dan Funk
dbe6701bb2 Removing the doc_types from the protocol builder, as these will eventually contradict what is coming from IRB and should not be used as an reference. Also fixing a failing test and assuring that only one reference file ever exists with a given name. 2020-03-19 10:40:07 -04:00
Dan Funk
560b8a8782 Mergers details from the irb_documents.xlsx into the values returned from the Protocol Builder to create a more complete picture of required document details. 2020-03-19 10:23:50 -04:00
Dan Funk
83d859fd3a Just merging stuff real quick. 2020-03-18 17:03:36 -04:00
Dan Funk
02be8ede75 Merge remote-tracking branch 'origin/master' into feature/reference_files 2020-03-18 15:16:34 -04:00
Aaron Louie
40e12f5ab5 Resolves marshmallow_sqlalchemy.ModelSchema deprecation warning 2020-03-16 13:37:31 -04:00
Aaron Louie
f742be3264 Replaces deprecated assertEquals with assertEqual 2020-03-16 13:05:15 -04:00
Aaron Louie
a0c35fe8cf Adds test for updating workflow spec 2020-03-16 12:32:39 -04:00
Aaron Louie
a98251237b Adds workflow spec category to API 2020-03-16 12:10:32 -04:00
Aaron Louie
0cc98616fd Merge branch 'master' into feature/workflow_spec_categories 2020-03-16 10:25:03 -04:00
Aaron Louie
358a485372 Fixes broken test 2020-03-16 09:07:39 -04:00
Aaron Louie
3885bc7624 Merge remote-tracking branch 'origin/master' into feature/delete_study 2020-03-16 08:49:44 -04:00
Aaron Louie
f9dbbefe89 Adds status spec to study when workflow is added. 2020-03-16 08:05:30 -04:00
Aaron Louie
bdd07685c6 Adds status spec when adding a study, and adds/removes workflows from study based on output data from status spec. 2020-03-15 15:54:13 -04:00
Dan Funk
779674ab60 Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks. 2020-03-13 15:03:57 -04:00
Aaron Louie
b1a6c9b6c7 Adds basic status-setting workflow spec and minimal test 2020-03-13 14:58:07 -04:00
Dan Funk
05b39df745 Fixes #12: Catching some specific common errors and re-raising as APIErrors with detailed codes and descriptions to improve debugging. In doing so, improving the error handling in the event a soft-reset causes an immediate error - and resetting to the original version of the specification in these events, to allow users the chance to try a hard reset instead. 2020-03-11 16:33:18 -04:00
Aaron Louie
3262833102 Requires login 2020-03-11 15:27:22 -04:00
Aaron Louie
bbfe9291e0 Adds endpoint to get workflow stats. Adds a test for the endpoint. 2020-03-11 14:28:53 -04:00
Aaron Louie
1119bb2b6c Tests that task counts and events are logged. 2020-03-11 12:35:37 -04:00
Dan Funk
e91d7aff2f Fixes #11: adding a delete endpoint for studies. It won't delete studies that have workflows, you have to delete those first) 
Removing the "default" error response from the api.yml, it was all noise.
 2020-03-09 15:12:40 -04:00
Aaron Louie
c928a6472c Edits typo 2020-03-06 10:57:24 -05:00
Aaron Louie
acad0023d2 Fixes a bug in get_study_workflows and adds a test covering it 2020-03-06 10:52:41 -05:00
Dan Funk
9fcd6f38f4 Merge remote-tracking branch 'origin/master' into feature/pb_services 2020-03-05 17:13:41 -05:00
Dan Funk
906bacff6a Expose a flag on the workflow model in the api to shown if it is using the latest spec. Added a soft_reset and hard_reset onto the workflow endpoint that will allow you to cause a hard or soft reset. 2020-03-05 16:45:44 -05:00
Dan Funk
7b21b78987 Workflow Processor will deserialize workflows using the version of the BPMN files used during creation, but allows for both a soft and hard reset - soft resets will use the new workflow without a restart and will retain full history. A hard-reset will restart the workflow from scratch, but will retain the data from the last completed task. Workflows have a complex version number associated with them that is used during the deserialization to find the correct files. 2020-03-05 15:38:30 -05:00