Commit Graph

48 Commits

Author SHA1 Message Date
Dan Funk 99503fb832 missed a file in the last commit. 2020-05-11 17:42:24 -04:00
Dan Funk b7c11fd893 Merge branch 'master' into feature/investigators_reference_file 2020-05-11 17:36:37 -04:00
Dan Funk 02f8764056 Updated to use the latest script engine / evaluation engine that creates a single location where all values used in BPMN/DMN are processed. Right now this is a python based interpreter, but we will eventually base this on FEEL expressions.
The validation process needs to take the api model into account so we catch errors with bad file names.
2020-05-11 17:04:05 -04:00
Dan Funk da7cae51b8 Adding a new reference file that provides greater details about the investigators related to a study.
Improving the study_info script documentation to provide detailed examples of values returned based on arguments.
Making the tests a little more targetted and less subject to breaking through better mocks.
Allow all tests to pass even when ther protocol builder mock isn't running locally.
Removing the duplication of reference files in tests and static, as this seems silly to me at the moment.
2020-05-07 13:57:24 -04:00
Dan Funk 07e58e923d Merge remote-tracking branch 'origin/chore/update_specs' into feature/previous_task
# Conflicts:
#	Pipfile.lock

Assuring that all documents from the xls spreadsheet are loaded when doing validations.
2020-05-06 11:25:50 -04:00
Dan Funk f1f8b91c9c Refactor the document details scripts. Now there is one script, it returns data in a consistent format, and has all the details required. The script is located in StudyInfo, with the argument documents. Make note that it returns a dictionary of ALL the documents, with a field to mark which ones are required according to the protocol builder. Others may become required if a workflow determines such, in which case the workflow will enforce this, and the document will have a count > 0, and additional details in a list of files within the document. I modified the XLS file to use lower case variable names, because it disturbed me, and we have to reference them frequently. Removed devious "as_object" variable on get_required_docs, so it behaves like the other methods all the time, and returns a dictionary. All the core business logic for finding the documents list now resides in the StudyService.
Because this changes the endpoint for all existing document details, I've modified all the test and static bpmn files to use the new format.
Shorting up the SponsorsList.xls file makes for slightly faster tests. seems senseless to load 5000 everytime we reset the data.
Tried to test all of this carefully in the test_study_details_documents.py test.
2020-04-29 15:08:11 -04:00
Aaron Louie beb86f0453 Adds protocol script to study service 2020-04-29 10:21:24 -04:00
Aaron Louie af1bb9f80d Adds more useful metadata to approvals and documents status scripts. Fleshes out and pretties up Documents & Approvals screen 2020-04-23 23:32:20 -04:00
Aaron Louie 47de010a88 Puts data from sequential calls to StudyInfo into the right place. Sets the required document flag correctly. 2020-04-23 21:02:08 -04:00
Aaron Louie d91f690388 Adds documents_status StudyInfo script. Adds Documents & Approvals workflow spec. 2020-04-23 19:25:01 -04:00
Aaron Louie 796c109611 Adds approvals to study service 2020-04-23 14:40:05 -04:00
Dan Funk ee999a0f15 fixing a bunch of stupid mistakes because I am tried. 2020-04-20 20:28:12 -04:00
Dan Funk edbd75bb75 Connect LDAP Requests to the StudyInfo service so we get back additional details. 2020-04-20 16:02:13 -04:00
Dan Funk d3dd9dcc25 Functional multi-instance - works with no changes to the front end - though I've added some attributes to task so we could give people a sense of how many iterations they will go through. 2020-04-19 15:14:10 -04:00
Dan Funk 678a45fa76 Merge branch 'master' into feature_personnel_multi_instance 2020-04-15 12:42:36 -04:00
Dan Funk dc2895cb05 Allow configurators to upload xls files into a workflow for defining enumrations of values for dropdown lists in forms. Fixing lots of tests.
Found a problem where the documentation for elements was being processed BEFORE data was loaded from a script.  There still may be some issues here.

Ran into an issue with circular dependencies - handling it with a new workflow_service, and pulling computational logic out of the api_models - it was the right thing to do.
2020-04-15 11:13:32 -04:00
Dan Funk aa17e30762 Updating libraries, modifying study_info script to save data in a consistent way, waiting on more fixes in spiffworkflow before continuing. 2020-04-13 16:23:31 -04:00
Dan Funk 316b9e6e8d Fixing a stupid mistake - must have typed a ' mark during commit.
Script should inherit from object.
Updating the ids so I could more easily test and assure things were working correctly.
2020-04-08 13:39:42 -04:00
Dan Funk 9bf5c9e727 Fixing an error that would prevent the correct counts from displaying if the record didn't come back from the protcol builder. 2020-04-07 14:14:43 -04:00
Dan Funk 697127660f Assure that all script tasks place data in a dictionary that is named exactly the same as the class - which is also the same as the Script tag. 2020-04-07 14:09:21 -04:00
Dan Funk c6b6ee5d70 Renamed the required_docs script to just "documents", and it returns all documented in the irb_documents look up table indexed on the "Code" - so details become available in the task data like "documents.IRB_INFOSEC_DOC.required".
Updated the irb_documents with shorter code names, thanks to Alex. Re-worked the DMN models so they can properly read from this new datastructure.
2020-04-06 16:56:00 -04:00
Dan Funk 60a10bb688 Marshmallow isn't the right tool when dealing with large models with lots of null values. Rather than fight the process of mamaging the Study Details, I'm letting that fall through, and we can test on an individual value or maybe set up a constants array when that becomes meaningful. 2020-04-03 16:24:38 -04:00
Dan Funk 4a916c1ee3 Created a "StudyService" and moved all complex logic around study manipulation out of the study api, and this service, as things were getting complicated. The Workflow Processor no longer creates the WorkflowModel, the study object handles that, and only passes the model into the workflow processor when it is ready to start the workflow.
Created a Study object (seperate from the StudyModel) that can cronstructed on request, and contains a different data structure than we store in the DB. This allows us to return underlying Categories and Workflows in a clean way.

Added a new status to workflows called "not_started", meaning we have not yet instantiated a processor or created a BPMN, they have no version yet and no stored data, just the possiblity of being started.

The Top Level Workflow or "Master" workflow is now a part of the sample data, and loaded at all times.

Removed the ability to "add a workflow to a study" and "remove a workflow from a study", a study contains all possible workflows by definition.

Example data no longer creates users or studies, it just creates the specs.
2020-03-30 08:00:16 -04:00
Dan Funk c9900d787e Every good deed goes punished. 2020-03-27 15:48:21 -04:00
Dan Funk c7d2c28178 Vastly more informative ApiError model that provides details on the underlying task where the error occured.
Added a validate_workflow_specification endpoint that allows you to check if the workflow will execute from beginning to end using random data.
Minor fixes to existing bpmns to allow them to pass.
All scripts must include a "do_task_validate_only" that restricts external calls and database modifications, but performs as much logic as possible.
2020-03-27 08:29:31 -04:00
Dan Funk e2c408b70d Removed all self-referential calls in the study_api. One api endpoint should never call another api endpoint. Moved the logic for updating a study to the study Model, rather than checking and setting dictionary values which will become very hard to maintain.
The protocol builder service now returns real models, not dictionaries, forcing proper validation and fail-fast behavior.
Changed the name of the "status" spec, to "top_level_workflow" and removing any connection a workflow or study has with this specification.  It is only unused to determine status in real time, and is not reused or tracked.
Modified the required documents script to return a dictionary and not an array, making it easier to speak to specific values in the BPMN and DMN.
Working on new ways to test the top_level_workflow in the context of updates, this is still a work in progress.
Making use of several modifications to the Spiff library that enables more complex expressions in DMN models. This is evident in the new DMN models for the top_level_workflow
2020-03-26 12:51:53 -04:00
Dan Funk f4342fc785 It became impossible to use the Swagger ui when we started adding authentication to all of the calls. I discovered Connexion and Swagger have a default way of handing JTW authentication and this cleans up our code quite a bit, moves the securing of endpoints into the API Definition, which is quite nice, and removes a whole library dependency (I never get to do that!) I added a SWAGGER_AUTH_KEY that can be used in non-production environments to allow users to quickly authenticate from the Swagger ui. I also removed all api calls to simple little happy api services, because that is just mean and pointless. 2020-03-24 14:15:21 -04:00
Dan Funk 3eb1167b8e Found a few errors in the sqlalchemy file definition that was causing failures, and had some consistency problems with the IRB Categories file name. The API was bailing out because we had restricted file types to bpmn,svg,or dmn in the connexion config file, I don't restrict this anymore we have plenty of checks elsewhere. Adding xlrd as a dependency - this didn't fail till a push to production. 2020-03-20 08:21:21 -04:00
Dan Funk 6e3b6c2635 Assure that files uploaded through web forms and files generated from templates can be cross-referenced to known document requirements from the protocol builder. Configurators can control this by managing an XLS Spreadsheet called "irb_documents.xslx".
Required Documents is becoming complicated, so making this it's own script task, removing it from study_info.py
The file_service is now very aware of this irb_documents file, so it will always need to exist.  We seed this file
during setup, but it can be overwritten by the configurator.
2020-03-19 17:14:20 -04:00
Dan Funk 560b8a8782 Mergers details from the irb_documents.xlsx into the values returned from the Protocol Builder to create a more complete picture of required document details. 2020-03-19 10:23:50 -04:00
Dan Funk 83d859fd3a Just merging stuff real quick. 2020-03-18 17:03:36 -04:00
Aaron Louie 6af1693bdc Fixes SonarCloud issue L23. Verifies that study ids match. 2020-03-16 10:37:06 -04:00
Aaron Louie cd6a70b747 Fixes code smell issues identified by SonarCloud 2020-03-16 08:31:19 -04:00
Dan Funk 05b39df745 Fixes #12: Catching some specific common errors and re-raising as APIErrors with detailed codes and descriptions to improve debugging. In doing so, improving the error handling in the event a soft-reset causes an immediate error - and resetting to the original version of the specification in these events, to allow users the chance to try a hard reset instead. 2020-03-11 16:33:18 -04:00
Dan Funk 3b39f3875d Fix data being over-written in the study_info. 2020-03-09 12:41:35 -04:00
Dan Funk c5cee4761e Improve version handling of files. Consolidate more of this logic in FileService. Place the version on the actual data model, not the file model, so the file model remains the same, and we just version the data associated with it. 2020-03-04 13:40:25 -05:00
Dan Funk 94f828dfd6 Adding a simple endpoint that describes what scripts are currently available, along with a brief description. 2020-03-03 15:30:42 -05:00
Dan Funk 7194d7d374 Standardizing the script tasks that can be executed on the server, adding tons of error messages for when things go wrong. All scripts must exist in side of the crc/scripts directory.
Adding a new script that script tasks can use to add in data about the study.

Moving all the test workflow specifications out of the main load.

fixing a pile of tests so they can find workflow specs that are now moved into the test directory.
2020-03-03 13:52:45 -05:00
Dan Funk 5e3fdaaa94 New set of "Tools" api endpoints, that provides a way to quickly render markdown or word documents by uploading json data and a template to populate.
Improved Error messages / Error processing.  You can now just throw an APIError anywhere, and it will be properly serialized and returned.
2020-02-29 17:22:38 -05:00
Aaron Louie 8611a23ad3 Renaming to snake case for consistency 2020-02-28 11:54:11 -05:00
Aaron Louie 27d7afb656 Adds Protocol Builder models and schemas. Reorganizes and cleans up some files. 2020-02-27 09:54:46 -05:00
Aaron Louie 5a53ddb88c Fixes vulnerability identified by SonarCloud 2020-02-12 11:07:01 -05:00
Aaron Louie 5f461d9e87 Fixes FactService 2020-02-11 22:13:46 -05:00
Aaron Louie 0ef1a178ed Merge branch 'master' into feature/crc_demo_workflows 2020-02-11 13:30:57 -05:00
Aaron Louie f128e7206b Adds placeholder scripts for testing 2020-02-10 17:18:51 -05:00
Dan Funk 1d24ebe382 Provide a script for generating word documents from template files. Refractored file managment into a service to make it easier to programatically add files. Modified the workflow_processor to inject the study_id and workflow_id into the running workflow so that this meta-information is avialable at the task level. 2020-02-10 16:19:23 -05:00
Dan Funk 91a91e9677 Read workflow configuration from the database, and not from a file.
And dropping the API call in the Fact Service so we aren't hitting external apis for the time being.
2019-12-30 11:07:26 -05:00
Dan Funk 9861f6baf3 Lots and lots of random stuff as I organize code and get things to make sense. 2019-12-18 14:02:17 -05:00