79 Commits

Author SHA1 Message Date
alicia pritchett
5b7e71d42d meta field for categories 2022-03-16 12:49:35 -04:00
alicia pritchett
37d92c6a5f WIP ~ adding a category metadata object 2022-03-15 10:49:41 -04:00
Dan
dde8873c9e 1. defer the loading of the bpmn_json when loading the workflow model. It can be stupidly larger. (see models/workflow.py)
2. Shave a little more time off the reset by not re-creating the whole WorkflowProcessor

cleaning up all the firsttime/sincetime statements.
2022-02-24 20:56:57 -05:00
Dan
1e4ec6f395 fixes to the workflow spec schema, so we can load null things without stuff blowing up. 2022-02-10 16:53:15 -05:00
Dan
f12c4aba52 Refactor the workflow_spec_service so it doesn't cache anything, it always reads what it needs from the file system. 2022-02-09 21:06:55 -05:00
Dan
8e7c8c87c3 down to just 38 failed tests 2022-02-09 11:37:54 -05:00
alicia pritchett
148796d311 fixes add_category 2022-02-09 11:24:41 -05:00
mike cullerton
9576089e84 Reorder categories 2022-02-09 10:55:02 -05:00
Dan
cc915ac25a refactoring the study service. 2022-02-09 08:50:00 -05:00
mike cullerton
420f50f44a Defaults for WorkflowSpecInfo 2022-02-08 14:40:33 -05:00
Dan
dc040f190e got one more test to work. 2022-02-08 11:30:13 -05:00
alicia pritchett
df67572549 puts specs into categories 2022-02-07 16:27:45 -05:00
mike cullerton
80322b392f Get category_name from fields 2022-02-07 16:09:51 -05:00
Dan
4b8d193fb9 a few more tests passing 2022-02-07 14:58:25 -05:00
Dan
abc17a8aff load the workflow_spec_service from app to avoid frequent file scans. 2022-02-07 12:30:20 -05:00
Dan
1c384a78e9 the tests kind of try and run now ... 2022-02-07 12:18:32 -05:00
Dan
ed07907ee2 at least allow the tests to spin up, even if they all fail. 2022-02-07 11:00:19 -05:00
Dan
4ba122fff3 WIP - completely broken. 2022-02-07 09:12:11 -05:00
Dan
6765afbd02 WIP - Move the categories and specifications out of the database. 2022-02-04 13:51:39 -05:00
Dan
4c00a5762f partial commit - new spec_file_service, and new spec_file_api endpoints that use spec and file name, not file id.
removed worklow_sync
cleaned up file and workflow models
most of the test are broken.
2022-01-28 06:42:37 -05:00
mike cullerton
68820c67cb Removed (almost) all references to WorkflowSpecDependencyFile
(There is still a call in the lookup service, but we need to decide how to fix that)
2022-01-19 16:12:54 -05:00
mike cullerton
acbb8898e4 Merge branch 'more-study-statuses-562' into waiting-workflow-errors-566 2021-12-10 16:52:07 -05:00
mike cullerton
f748fa3db8 Removed unused imports 2021-12-10 16:48:28 -05:00
mike cullerton
52baf25459 In WorkflowService.do_waiting, if a workflow causes an exception, change its status from waiting to erroring so we don't continue to call it in do_waiting. 2021-12-09 15:33:39 -05:00
mike cullerton
572ca1db6b Fixes for backend code after removing name column from workflow_spec and workflow_spec_category tables 2021-10-05 14:17:41 -04:00
mike cullerton
fcf538ca92 remove name column from workflow_spec and workflow_spec_category 2021-10-05 13:10:36 -04:00
alicia pritchett
d5d4496cd0 Admin flag on category + migration + updated test 2021-09-29 16:53:59 -04:00
Kelly McDonald
b4ecb0f97a Add backref cascade and make changes to the spot where we delete so it loads objects into memory and then uses the SQLAlchemy plumbing to make sure that the db object gets cleaned up. 2021-08-18 08:25:17 -04:00
Kelly McDonald
b857fddbad commit in intermediate changes to work on a bug 2021-07-26 13:00:11 -04:00
Kelly McDonald
48516e4009 Parent side working 2021-07-22 10:23:05 -04:00
Kelly McDonald
2455495457 got library side working - still need to have parent working 2021-07-22 10:20:34 -04:00
Kelly McDonald
dce1d86e93 commit changes before looking at something else 2021-07-06 10:34:57 -04:00
Dan
a1bb30e689 Switching from using "default" to "server_default" for all default time settings, and running migrations so that this is set at the database level rather than in python, to hopefully correct some issues with dates being 4 hours in the future. Having a very hard time replicating this issue locally. 2021-05-14 12:28:50 -04:00
Dan Funk
d6054a9846
Merge branch 'dev' into 310-task-event-timezone 2021-05-04 11:21:36 -04:00
Kelly McDonald
c029dad688 Convert some datetime columns to include timezone 2021-04-28 10:00:22 -04:00
mike cullerton
c0655c9d03 Added standalone column to workflow_spec
Added `user_id` column to workflow
2021-04-26 08:33:55 -04:00
Kelly McDonald
0e1aa59fa1 Make a change to make sure that if there is a new file locally that is not present remotely when we pull from the remote, the new local file gets deleted.
Also: add several things to the requirements.txt that should have been there in the first place.
2020-12-08 13:42:01 -05:00
Dan Funk
dba41f4759 Ludicrously stupid launch in a refactor of the way all files work in the system at a time where I crave sleep and peace above all other things.
Added a File class, that we wrap around the FileModel so the api endpoints don't change, but File no longer holds refences to versions or dates of the file_data model, we
figure this out based on a clean database structure.

The ApprovalFile is directly related to the file_data_model - so no chance that a reviewer would review the incorrect version of a file.py

Noticed that our FileType enum called "bpmn" "bpmm", hope this doesn't screw someone up.

Workflows are directly related to the data_models that create the workflow spec it needs.  So the files should always be there.  There are no more hashes, and thus no more hash errors where it can't find the files to rebuild the workflow.py

Not much to report here, other than I broke every single test in the system at one point.  So I'm super concerned about this, and will be testing it a lot before creating the pull request.
2020-05-28 20:03:50 -04:00
Dan Funk
d39ef658a2 Made some modifications to the Approval so that it knows exactly what versions of every file are being sent for approval
Added the following columns:
  * date_created - so we know when the file was created
  * renamed workflow_version to just "version", because everything has a version,  this is the version of the request.
  * workflow_hash - this is just a quick way to see what files and versions are associated with the request, it could be factored out.
  * study - a quick relationship link to the study, so that this model is easier to use.
  * workflow - ditto
  * approval_files - these is a list from a new link table that links an approval to specific files and versions.

The RequestApproval is logically sound, but still needs some additional pieces in place to be callable from a BPMN workflow diagram.

Altered the file service to pick up on changes to files vs adding new files, so that versions are picked up correctly as
users modify their submission - adding new files or replacing existing ones.  Deleting files worries me, and I will need to revisit this.

The damn base test keeps giving me a headache, so I made changes there to see if clearing and dropping the database each time won't allow the tests to pass more consistently.

Lots more tests around the file service to make sure it is versioning user uploaded files correctly.

The "Test Request Approval Script" tries to find to assure the correct behavior as this is likely to be called many times repeatedly and with little knowledge of the internal system.  So it should just "do the right thing".
2020-05-23 15:08:17 -04:00
Dan Funk
992a85e9a5 Rough idea of what the Approvals model will look like. 2020-05-22 11:56:43 -04:00
Carlos Lopez
abc90cdcb1 Adding serialiazer for study files 2020-05-20 15:10:22 -06:00
Dan Funk
2699f5c65c Refactor the stats models, and assure they are very correct across all tests with the workflow api.
I noticed we were saving the workflow every time we loaded it up, rather than only when we were making changes to it.  Refactored this to be a little more careful.
Centralized the saving of the workflow into one location in the processor, so we can make sure we update all the details about that workflow every time we save.
The workflow service has a method that will log any task action taken in a consistent way.
The stats models were removed from the API completely.  Will wait for a use case for dealing with this later.
2020-05-04 10:57:09 -04:00
Dan Funk
241980f98f If you name add a file to a workflow that has the exact same name as a Task Spec's ID, and an extension of "md", it wll use that file as the markdown content, and ignore the markdown in the documentation on the task spec.
Moving the primary process id from the workflow model to the file model, and assuring it is updated properly.  This was causing a bug that would "lose" the workflow.
2020-04-17 13:30:32 -04:00
Aaron Louie
9a9922c473 Adds display order to Workflow Spec 2020-04-09 14:25:14 -04:00
Dan Funk
34b6ec92bf updating the API
Removing the call for study/workflows - as workflow information is returned with the study by default.
Fixing a bug in the workflow spec model schema.
2020-03-30 10:12:10 -04:00
Dan Funk
4a916c1ee3 Created a "StudyService" and moved all complex logic around study manipulation out of the study api, and this service, as things were getting complicated. The Workflow Processor no longer creates the WorkflowModel, the study object handles that, and only passes the model into the workflow processor when it is ready to start the workflow.
Created a Study object (seperate from the StudyModel) that can cronstructed on request, and contains a different data structure than we store in the DB. This allows us to return underlying Categories and Workflows in a clean way.

Added a new status to workflows called "not_started", meaning we have not yet instantiated a processor or created a BPMN, they have no version yet and no stored data, just the possiblity of being started.

The Top Level Workflow or "Master" workflow is now a part of the sample data, and loaded at all times.

Removed the ability to "add a workflow to a study" and "remove a workflow from a study", a study contains all possible workflows by definition.

Example data no longer creates users or studies, it just creates the specs.
2020-03-30 08:00:16 -04:00
Dan Funk
b61a35f956 Merge remote-tracking branch 'origin/master' into feature/status_refactor
Fixing adding a study so all workflows are again added, will add status on those workflows based on output from the master bpmn diagram, which is coming shortly.
2020-03-27 11:55:36 -04:00
Aaron Louie
f77ebe3177 Adds display order to categories 2020-03-26 15:29:52 -04:00
Dan Funk
e2c408b70d Removed all self-referential calls in the study_api. One api endpoint should never call another api endpoint. Moved the logic for updating a study to the study Model, rather than checking and setting dictionary values which will become very hard to maintain.
The protocol builder service now returns real models, not dictionaries, forcing proper validation and fail-fast behavior.
Changed the name of the "status" spec, to "top_level_workflow" and removing any connection a workflow or study has with this specification.  It is only unused to determine status in real time, and is not reused or tracked.
Modified the required documents script to return a dictionary and not an array, making it easier to speak to specific values in the BPMN and DMN.
Working on new ways to test the top_level_workflow in the context of updates, this is still a work in progress.
Making use of several modifications to the Spiff library that enables more complex expressions in DMN models. This is evident in the new DMN models for the top_level_workflow
2020-03-26 12:51:53 -04:00
Dan Funk
13a688048c Fixing failed tests, setting unknown=EXCLUDE on the workflow spec model schema so that it can handle getting back the dump-only workflow_spec_category and just ignore it. 2020-03-20 07:54:01 -04:00