714b5f3be0
Merge branch 'feature/protocol_status' into feature/previous_task
...
# Conflicts:
# crc/services/study_service.py
2020-05-04 11:08:36 -04:00
2699f5c65c
Refactor the stats models, and assure they are very correct across all tests with the workflow api.
...
I noticed we were saving the workflow every time we loaded it up, rather than only when we were making changes to it. Refactored this to be a little more careful.
Centralized the saving of the workflow into one location in the processor, so we can make sure we update all the details about that workflow every time we save.
The workflow service has a method that will log any task action taken in a consistent way.
The stats models were removed from the API completely. Will wait for a use case for dealing with this later.
2020-05-04 10:57:09 -04:00
7c743d65ed
Adding tests around parallel which pass great, but ran into a bug down in Spiff that I'm passing back to Kelly with a test in that repo. More coming on this.
2020-04-30 15:39:11 -04:00
bec59a71d7
Deleteing stuff is a damn mess, but this is a little cleaner.
2020-04-29 16:07:39 -04:00
f1f8b91c9c
Refactor the document details scripts. Now there is one script, it returns data in a consistent format, and has all the details required. The script is located in StudyInfo, with the argument documents. Make note that it returns a dictionary of ALL the documents, with a field to mark which ones are required according to the protocol builder. Others may become required if a workflow determines such, in which case the workflow will enforce this, and the document will have a count > 0, and additional details in a list of files within the document. I modified the XLS file to use lower case variable names, because it disturbed me, and we have to reference them frequently. Removed devious "as_object" variable on get_required_docs, so it behaves like the other methods all the time, and returns a dictionary. All the core business logic for finding the documents list now resides in the StudyService.
...
Because this changes the endpoint for all existing document details, I've modified all the test and static bpmn files to use the new format.
Shorting up the SponsorsList.xls file makes for slightly faster tests. seems senseless to load 5000 everytime we reset the data.
Tried to test all of this carefully in the test_study_details_documents.py test.
2020-04-29 15:08:11 -04:00
13333fb306
Adding an API endpoint that allows setting the token on the workflow to a specific task.
...
Added error checking such that attempting to submit data for a task that is not in the "READY" state throws an error message.
For some reason I'm getting lots of errors in the tests as they try to hit API endpoints they were not hitting before, so adding a number of mocks to some of the study service tests.
2020-04-28 17:25:53 -04:00
0088364b1d
Merge branch 'master' of github.com:sartography/cr-connect-workflow
2020-04-28 13:48:52 -04:00
3e3a249e3c
Verifying Sub-Process works, and adding a field to expose a hint as to the sub-process in which the task occurs.
...
Because the name field is now used to expose workflow/sub-process information on tasks, we can't use it to store the workflow_version, so that is now just stored on the database model. Which is much cleaner and removes a duplication.
2020-04-28 13:48:44 -04:00
496e5b7719
Updates all workflow specs to match staging
2020-04-27 22:54:05 -04:00
447f4013f8
Assure that a hard-reset sticks, and the system is properly updated.
2020-04-27 16:08:23 -04:00
1b9743a4d1
Assure that if a form has an enumeration it errors out if that enumeration is empty.
2020-04-27 15:10:09 -04:00
c85173de88
Sorts approvals by display order
2020-04-24 08:54:14 -04:00
12eb039bc9
Server isn't erroring out, but can't find the lookup table id in the database, so trying to use the in-memory model instead, to give things time to get to the database. Really unsure what is happening here. Hard to see in the database.
2020-04-24 07:01:32 -04:00
08140eca17
Merge branch 'master' of github.com:sartography/cr-connect-workflow
2020-04-23 15:01:02 -04:00
3aeb7ad116
Server isn't erroring out, but can't find the lookup table id in the database, so trying to use the in-memory model instead, to give things time to get to the database. Really unsure what is happening here. Hard to see in the database.
2020-04-23 14:58:17 -04:00
796c109611
Adds approvals to study service
2020-04-23 14:40:05 -04:00
ae2313203c
fixing a failed test. Punished for doing a kind thing.
2020-04-23 13:06:51 -04:00
b5b46b7c2c
better overall search results for type ahead. Still dealing with stop words failing.
2020-04-23 12:05:08 -04:00
65b29e1a9d
Don't just bomb out as soon as someone types an empty string.
2020-04-23 09:44:11 -04:00
286af86f08
Forgot a missing bpmn file for running the tests.
2020-04-22 19:41:40 -04:00
7b085c9c9d
Adding an API Endpoint that will return a list of LookupValues that match a given query - can be used to populate an auto-complete table.
2020-04-22 19:40:40 -04:00
6de8c8b977
Create lookup tables for XSL files referenced in workflows so we can do full text searches and populate lists on the fly quickly.
2020-04-22 15:37:02 -04:00
fd0adb1d43
Updated the study status to use a different enumeration. Migration correctly handles modifying the enum.
...
INCOMPLETE = 'Incomplete in Protocol Builder',
ACTIVE = 'Active / Ready to roll',
HOLD = 'On Hold',
OPEN = 'Open - this study is in progress',
ABANDONED = 'Abandoned, it got deleted in Protocol Builder'
2020-04-21 17:13:30 -04:00
0a74bf8c44
We can now collect, and provide "extension properties" on a task as set in the camunda modeler.
...
These are provided as "properties" on a task, and are identical in structure to properties on a form field.
2020-04-21 12:07:59 -04:00
ec112f52be
Make use of cleaner data provided by Spiffworkflow about multi-instance settings.
2020-04-21 11:43:43 -04:00
ee999a0f15
fixing a bunch of stupid mistakes because I am tried.
2020-04-20 20:28:12 -04:00
59d400b058
Assure DMN can pick up name rather than label outputs.
2020-04-20 19:26:42 -04:00
edbd75bb75
Connect LDAP Requests to the StudyInfo service so we get back additional details.
2020-04-20 16:02:13 -04:00
2d3402a719
Ldap Service with Test and mocks.
...
LDAP_URL can be set in an environment variable.
2020-04-20 15:16:33 -04:00
d3dd9dcc25
Functional multi-instance - works with no changes to the front end - though I've added some attributes to task so we could give people a sense of how many iterations they will go through.
2020-04-19 15:14:10 -04:00
815f40a539
forgot a critical file.
2020-04-17 15:18:03 -04:00
d7bd2383a1
Fixing tests.
2020-04-17 15:12:48 -04:00
b099cac174
Merge branch 'master' into feature_personnel_multi_instance
2020-04-17 14:26:23 -04:00
241980f98f
If you name add a file to a workflow that has the exact same name as a Task Spec's ID, and an extension of "md", it wll use that file as the markdown content, and ignore the markdown in the documentation on the task spec.
...
Moving the primary process id from the workflow model to the file model, and assuring it is updated properly. This was causing a bug that would "lose" the workflow.
2020-04-17 13:30:32 -04:00
3d820fcb21
Inital work on manual instance.
2020-04-16 11:07:29 -04:00
d38f5f31d6
Improving tests to see what works with Multi-Instance
2020-04-15 16:32:29 -04:00
678a45fa76
Merge branch 'master' into feature_personnel_multi_instance
2020-04-15 12:42:36 -04:00
a5f5468e9a
Merge branch 'master' into feature/customer_lookup
2020-04-15 12:41:35 -04:00
c485ff5be6
fixing a test.
2020-04-15 11:59:02 -04:00
dc2895cb05
Allow configurators to upload xls files into a workflow for defining enumrations of values for dropdown lists in forms. Fixing lots of tests.
...
Found a problem where the documentation for elements was being processed BEFORE data was loaded from a script. There still may be some issues here.
Ran into an issue with circular dependencies - handling it with a new workflow_service, and pulling computational logic out of the api_models - it was the right thing to do.
2020-04-15 11:13:32 -04:00
faf4c0df97
Updates BPMN and DMN files
2020-04-15 10:58:13 -04:00
aa17e30762
Updating libraries, modifying study_info script to save data in a consistent way, waiting on more fixes in spiffworkflow before continuing.
2020-04-13 16:23:31 -04:00
c79415a794
throw a sensible error when study is not found on get_study (don't 500)
...
some ugly fixes in the file_service for improving panda output from spreadsheet processing that I need to revist.
now that the spiff-workflow handles multi-instance, we can't have random multi-instance tasks around.
Improved tests around study deletion.
2020-04-08 13:28:43 -04:00
c7a18709ee
Merge branch 'master' of github.com:sartography/cr-connect-workflow
2020-04-07 14:09:28 -04:00
697127660f
Assure that all script tasks place data in a dictionary that is named exactly the same as the class - which is also the same as the Script tag.
2020-04-07 14:09:21 -04:00
d23b5e747b
Replaces deprecated assertEquals with assertEqual
2020-04-06 18:15:32 -04:00
c6b6ee5d70
Renamed the required_docs script to just "documents", and it returns all documented in the irb_documents look up table indexed on the "Code" - so details become available in the task data like "documents.IRB_INFOSEC_DOC.required".
...
Updated the irb_documents with shorter code names, thanks to Alex. Re-worked the DMN models so they can properly read from this new datastructure.
2020-04-06 16:56:00 -04:00
e283b86466
Fixing a bug with deleting a study.
2020-04-06 13:08:17 -04:00
a322801c91
Allow a study to be deleted, even if some statistics are laying around.
2020-04-03 16:41:16 -04:00
60a10bb688
Marshmallow isn't the right tool when dealing with large models with lots of null values. Rather than fight the process of mamaging the Study Details, I'm letting that fall through, and we can test on an individual value or maybe set up a constants array when that becomes meaningful.
2020-04-03 16:24:38 -04:00
c86f3321c5
reworking the test_study_service to run from a clean state, this is growing so complex. Tests really need to be isolated better.
2020-03-30 15:39:50 -04:00
17796193de
fixing a bug that was causing failing tests.
...
Adding id and spec_version to the workflow metadata.
Refactoring the processing of the master_spec so that it doesn't polute the workflow database.
Adding tests to assure that the status and counts are updated on the workflow model as users make progress.
2020-03-30 14:01:57 -04:00
34b6ec92bf
updating the API
...
Removing the call for study/workflows - as workflow information is returned with the study by default.
Fixing a bug in the workflow spec model schema.
2020-03-30 10:12:10 -04:00
4a916c1ee3
Created a "StudyService" and moved all complex logic around study manipulation out of the study api, and this service, as things were getting complicated. The Workflow Processor no longer creates the WorkflowModel, the study object handles that, and only passes the model into the workflow processor when it is ready to start the workflow.
...
Created a Study object (seperate from the StudyModel) that can cronstructed on request, and contains a different data structure than we store in the DB. This allows us to return underlying Categories and Workflows in a clean way.
Added a new status to workflows called "not_started", meaning we have not yet instantiated a processor or created a BPMN, they have no version yet and no stored data, just the possiblity of being started.
The Top Level Workflow or "Master" workflow is now a part of the sample data, and loaded at all times.
Removed the ability to "add a workflow to a study" and "remove a workflow from a study", a study contains all possible workflows by definition.
Example data no longer creates users or studies, it just creates the specs.
2020-03-30 08:00:16 -04:00
6ebd4dce42
WIP: Adds default workflow spec categories
2020-03-27 15:32:07 -04:00
b61a35f956
Merge remote-tracking branch 'origin/master' into feature/status_refactor
...
Fixing adding a study so all workflows are again added, will add status on those workflows based on output from the master bpmn diagram, which is coming shortly.
2020-03-27 11:55:36 -04:00
c7d2c28178
Vastly more informative ApiError model that provides details on the underlying task where the error occured.
...
Added a validate_workflow_specification endpoint that allows you to check if the workflow will execute from beginning to end using random data.
Minor fixes to existing bpmns to allow them to pass.
All scripts must include a "do_task_validate_only" that restricts external calls and database modifications, but performs as much logic as possible.
2020-03-27 08:29:31 -04:00
f77ebe3177
Adds display order to categories
2020-03-26 15:29:52 -04:00
e2c408b70d
Removed all self-referential calls in the study_api. One api endpoint should never call another api endpoint. Moved the logic for updating a study to the study Model, rather than checking and setting dictionary values which will become very hard to maintain.
...
The protocol builder service now returns real models, not dictionaries, forcing proper validation and fail-fast behavior.
Changed the name of the "status" spec, to "top_level_workflow" and removing any connection a workflow or study has with this specification. It is only unused to determine status in real time, and is not reused or tracked.
Modified the required documents script to return a dictionary and not an array, making it easier to speak to specific values in the BPMN and DMN.
Working on new ways to test the top_level_workflow in the context of updates, this is still a work in progress.
Making use of several modifications to the Spiff library that enables more complex expressions in DMN models. This is evident in the new DMN models for the top_level_workflow
2020-03-26 12:51:53 -04:00
4ed89eec5b
Mocks Protocol Builder response
2020-03-26 10:30:15 -04:00
3fb9fb9b22
Simplifies status BPMN & DMN.
2020-03-26 10:06:12 -04:00
37a8b69ffe
Cleaning up a failing test, not sure why this isn't failing locally?
2020-03-24 14:40:24 -04:00
f4342fc785
It became impossible to use the Swagger ui when we started adding authentication to all of the calls. I discovered Connexion and Swagger have a default way of handing JTW authentication and this cleans up our code quite a bit, moves the securing of endpoints into the API Definition, which is quite nice, and removes a whole library dependency (I never get to do that!) I added a SWAGGER_AUTH_KEY that can be used in non-production environments to allow users to quickly authenticate from the Swagger ui. I also removed all api calls to simple little happy api services, because that is just mean and pointless.
2020-03-24 14:15:21 -04:00
b35427523d
Merge remote-tracking branch 'origin/master' into feature/reference_files
...
# Conflicts:
# crc/models/file.py
# crc/services/file_service.py
# tests/data/reference/irb_documents.xlsx
# tests/test_files_api.py
2020-03-20 11:07:55 -04:00
16c6ba9661
Removing unneeded files.
2020-03-20 08:33:37 -04:00
3eb1167b8e
Found a few errors in the sqlalchemy file definition that was causing failures, and had some consistency problems with the IRB Categories file name. The API was bailing out because we had restricted file types to bpmn,svg,or dmn in the connexion config file, I don't restrict this anymore we have plenty of checks elsewhere. Adding xlrd as a dependency - this didn't fail till a push to production.
2020-03-20 08:21:21 -04:00
6e3b6c2635
Assure that files uploaded through web forms and files generated from templates can be cross-referenced to known document requirements from the protocol builder. Configurators can control this by managing an XLS Spreadsheet called "irb_documents.xslx".
...
Required Documents is becoming complicated, so making this it's own script task, removing it from study_info.py
The file_service is now very aware of this irb_documents file, so it will always need to exist. We seed this file
during setup, but it can be overwritten by the configurator.
2020-03-19 17:14:20 -04:00
dbe6701bb2
Removing the doc_types from the protocol builder, as these will eventually contradict what is coming from IRB and should not be used as an reference. Also fixing a failing test and assuring that only one reference file ever exists with a given name.
2020-03-19 10:40:07 -04:00
560b8a8782
Mergers details from the irb_documents.xlsx into the values returned from the Protocol Builder to create a more complete picture of required document details.
2020-03-19 10:23:50 -04:00
83d859fd3a
Just merging stuff real quick.
2020-03-18 17:03:36 -04:00
02be8ede75
Merge remote-tracking branch 'origin/master' into feature/reference_files
2020-03-18 15:16:34 -04:00
40e12f5ab5
Resolves marshmallow_sqlalchemy.ModelSchema deprecation warning
2020-03-16 13:37:31 -04:00
f742be3264
Replaces deprecated assertEquals with assertEqual
2020-03-16 13:05:15 -04:00
a0c35fe8cf
Adds test for updating workflow spec
2020-03-16 12:32:39 -04:00
a98251237b
Adds workflow spec category to API
2020-03-16 12:10:32 -04:00
0cc98616fd
Merge branch 'master' into feature/workflow_spec_categories
2020-03-16 10:25:03 -04:00
358a485372
Fixes broken test
2020-03-16 09:07:39 -04:00
3885bc7624
Merge remote-tracking branch 'origin/master' into feature/delete_study
2020-03-16 08:49:44 -04:00
f9dbbefe89
Adds status spec to study when workflow is added.
2020-03-16 08:05:30 -04:00
bdd07685c6
Adds status spec when adding a study, and adds/removes workflows from study based on output data from status spec.
2020-03-15 15:54:13 -04:00
779674ab60
Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks.
2020-03-13 15:03:57 -04:00
b1a6c9b6c7
Adds basic status-setting workflow spec and minimal test
2020-03-13 14:58:07 -04:00
05b39df745
Fixes #12 : Catching some specific common errors and re-raising as APIErrors with detailed codes and descriptions to improve debugging. In doing so, improving the error handling in the event a soft-reset causes an immediate error - and resetting to the original version of the specification in these events, to allow users the chance to try a hard reset instead.
2020-03-11 16:33:18 -04:00
3262833102
Requires login
2020-03-11 15:27:22 -04:00
bbfe9291e0
Adds endpoint to get workflow stats. Adds a test for the endpoint.
2020-03-11 14:28:53 -04:00
1119bb2b6c
Tests that task counts and events are logged.
2020-03-11 12:35:37 -04:00
e91d7aff2f
Fixes #11 : adding a delete endpoint for studies. It won't delete studies that have workflows, you have to delete those first)
...
Removing the "default" error response from the api.yml, it was all noise.
2020-03-09 15:12:40 -04:00
c928a6472c
Edits typo
2020-03-06 10:57:24 -05:00
acad0023d2
Fixes a bug in get_study_workflows and adds a test covering it
2020-03-06 10:52:41 -05:00
9fcd6f38f4
Merge remote-tracking branch 'origin/master' into feature/pb_services
2020-03-05 17:13:41 -05:00
906bacff6a
Expose a flag on the workflow model in the api to shown if it is using the latest spec. Added a soft_reset and hard_reset onto the workflow endpoint that will allow you to cause a hard or soft reset.
2020-03-05 16:45:44 -05:00
7b21b78987
Workflow Processor will deserialize workflows using the version of the BPMN files used during creation, but allows for both a soft and hard reset - soft resets will use the new workflow without a restart and will retain full history. A hard-reset will restart the workflow from scratch, but will retain the data from the last completed task. Workflows have a complex version number associated with them that is used during the deserialization to find the correct files.
2020-03-05 15:38:30 -05:00
e07ad8d21c
Updates training session BPMN files. Fixes PB status enums.
2020-03-05 14:21:03 -05:00
697d930eab
Modify the workflow processor to accept a workflow model - so it can take on more of the responsibilities of updating this model and managing versions.
...
Changing the version information so that it includes the numbers of the files used to generate the serialized workflow.
2020-03-05 13:25:28 -05:00
c0a3bac4f4
missed some test errors in the last push. Just fixing the tests.
2020-03-05 11:23:22 -05:00
70611e2c1d
Adding the version of the specification used to create a workflow to the workflow api endpoint. Though the exact content of this version is likely to change.
...
Split the API specific models out from the workflow models to help me keep this straight.
Added tests to help me understand the errors thrown the and resolution path when a workflow specification changes in the midst of a running workflow.
2020-03-05 11:18:20 -05:00
78b6f040eb
Add the ability to forcibly restart a workflow, while retaining that workflows data.
...
A workflow specification knows it's version number, which is generated by the version of the files that make it up.
A workflow specification version number is the primary file (the lead BPMN) followed by a consistency ordered version each extra file associated with the workflow. A change in any file modifies the specifications version.
2020-03-04 17:08:45 -05:00
c5cee4761e
Improve version handling of files. Consolidate more of this logic in FileService. Place the version on the actual data model, not the file model, so the file model remains the same, and we just version the data associated with it.
2020-03-04 13:40:25 -05:00
f258090a0b
Fixes broken unit test
2020-03-04 09:54:00 -05:00
8cbd6f892f
WIP: Mocks study details PB endpoint response
2020-03-03 20:56:46 -05:00
Dan Funk
Aaron Louie