sartography
/
cr-connect-workflow
mirror of https://github.com/sartography/cr-connect-workflow.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,995 Commits 13 Branches 0 Tags 8.7 MiB
Commit Graph

524 Commits

Author SHA1 Message Date
Kelly McDonald 79c69037dd Filter out Library categories from the list returned 2021-07-08 08:57:40 -04:00
mike cullerton d6f9b013f7 Change `irb_documents` to `documents` 2021-07-07 12:54:03 -04:00
mike cullerton 23180908c8 Undo change. Wrong branch 2021-07-07 12:43:51 -04:00
mike cullerton c39e08972d Change irb_documents to documents 2021-07-07 12:42:01 -04:00
Dan Funk 3c52453c26
Merge branch 'dev' into add-name-error-hint-8 2021-07-07 11:35:24 -04:00
Dan Funk 2ed436327f
Merge pull request #333 from sartography/check-study-script-273 
Check study script #273
 2021-07-07 11:28:32 -04:00
Dan Funk db9b228735
Merge pull request #334 from sartography/WaitingEventsHotPatch 
Make a change so that anything that has a waiting event is labeled in…
 2021-07-07 11:27:10 -04:00
Dan 2a45f2fcda Merge branch 'dev' into add-name-error-hint-8 2021-07-07 11:24:00 -04:00
mike cullerton b0cf74fa3d Merge branch 'dev' into add-by-user-61 
# Conflicts:
#	tests/files/test_files_api.py
 2021-07-07 10:46:50 -04:00
Dan fb54edac1c Adding additional details to error messages, and cleaning up the cruft around these messages to keep them clear and succinct. 
Most noteable is the addition of the line on which the error occurs for script tasks.  It will report the line number and pass back the content of
the line that failed.
The validator only returns the first error it encounters, as it's clear that all we ever get right now is two of the same error.
Did a lot of work between this and spiffworkflow to remove all the places where we obfuscate or drop details as we converted between workflowExceptions and APIExceptions.
Dropped the python levenshtein dependency, in favor of just rolling a simple one ourselves in Spiffworkflow.
 2021-07-07 00:53:49 -04:00
mike cullerton 894377a607 Add user_uid to FileDataModel calls 2021-07-06 17:09:00 -04:00
Dan a9805ad40c Adding some documentation for clarity. 2021-07-06 14:40:20 -04:00
Dan 7aa4b0833c Merge remote-tracking branch 'origin/dev' into feature/documents_xls_refactor 2021-07-06 13:11:08 -04:00
Dan 1b1a994360 Refactoring Reference files to use the lookup table, rather than parsing the results directly out of the spreadsheet, or attempting to cache them. 
Adding a DocumentService to clean up the FileService, and get Documents well seperated, as it seems likely be pulled out or seperated in the future, there is now a Documents api file as well, for the same reason.
Some other minor changes are just fixing white space to assure our code is linting correctly.
I removed _create_study_workflow_approvals from the base test, as we don't use approvals like this anymore.
 2021-07-06 13:10:20 -04:00
Kelly McDonald 9803a04d6d Make a change so that anything that has a waiting event is labeled in the database as waiting, even if it is sitting around waiting on a user input task that is ready 2021-07-06 11:46:47 -04:00
mike cullerton 9690c69b6c added check_study method to protocol_builder service for new endpoint 2021-07-02 15:25:33 -04:00
mike cullerton db44882733 When we encounter a name error running a script, look in the task data and see if there is a variable with a similar name. 
If a variable with a similar name exists, add it to the ApiError as a hint
 2021-07-01 15:38:45 -04:00
Mike Cullerton 98d641f54d
Merge branch 'dev' into partial_testing 2021-06-30 16:13:31 -04:00
NWalker4483 508ef1ff2b Readded exit_at to workflow param 2021-06-30 11:15:48 -04:00
Dan 9a32fadc2f Merge branch 'dev' into 346-waiting-task-schedule 
# Conflicts:
#	Pipfile.lock
#	crc/services/workflow_service.py
 2021-06-30 10:14:37 -04:00
NWalker4483 2ba52a7095 Merge remote-tracking branch 'origin/HEAD' into partial_testing 2021-06-29 11:21:49 -04:00
Dan 187221537b fixing a corner case around file_data attributes. 2021-06-25 16:18:28 -04:00
Dan 628a587d97 We weren't committing to the database in the correct spot, so the delete wasn't sticking, and we were ending up with test workflows showing up in the main interface. 2021-06-23 16:39:58 -04:00
Dan 1f3febe2df Fix the panda warnings, which means we only support xlsx files now. 
Include the document details when reutnring a file from the file model.
Improve the validation of file form elements.
 2021-06-22 14:58:52 -04:00
NWalker4483 369b518384 Removed WorkflowService.delete_test_data() 2021-06-22 11:29:00 -04:00
Dan 07eb3f9ca8 Moving metrics into SpiffWorkflow so we can run the performance metrics deeply across both systems simultaniously. 
Upgrading libraries.
Fixing deprication issue with Pandas and numpy.
We can only process xlsx files now, plain oldschool xls is fully removed.
 2021-06-18 16:41:55 -04:00
NWalker4483 4ca94b39ce Merge branch 'dev' of https://github.com/sartography/cr-connect-workflow into partial_testing 2021-06-17 15:12:28 -04:00
Dan a4caae8d64 when validating, we need to take every step to remove the workflows we create during the validation. 2021-06-16 14:40:20 -04:00
Dan acae6030f5 The API for evaluating python expressions should not raise an error, it can simply return a result of false, and provide an error to explain the problem. In this way the front end doesn't error out when it's running the script tasks but doesn't have enough information to get a valid response back. 
The validation should take into account that repeating sections must be evaluated in the context of the data within the repeating section, not outside of it.
 2021-06-15 16:17:15 -04:00
NWalker4483 5c5b966bc4 Merge branch 'dev' of https://github.com/sartography/cr-connect-workflow into partial_testing 2021-06-15 09:11:18 -04:00
Dan 738a984dcd Merge branch 'dev' of github.com:sartography/cr-connect-workflow into dev 2021-06-11 17:21:47 -04:00
Dan 879d6e1df3 missed a critical change on the backend with the recent file updates, this assures that we properly handle repeating sections during the post process, 
and we update the file doc code in those repeating sections.
 2021-06-11 17:20:25 -04:00
Dan 7282419174 modified the tools api to return the expression and data with the result, making it easier to cash results on the front end, This should help drastically reduce the calls to the python_eval from the front end. 
The post processors for form submissions needs to take repeating sections into account, or it won't find the files it is looking for.
 2021-06-11 08:27:50 -04:00
Kelly McDonald 394906d840 Add refresh_waiting_tasks to the processor do_engine_steps so that if we have a waiting task that has met its condition prior to the scheduler running, it will go ahead and proceed 2021-06-10 08:33:24 -04:00
NWalker4483 e1a4a79b03 Added Partial Workflow spec 2021-06-09 15:21:21 -04:00
NWalker4483 2364c240b2 Error Text Changed 2021-06-09 14:54:43 -04:00
NWalker4483 338fe30984 Merge branch 'dev' of https://github.com/sartography/cr-connect-workflow into partial_testing 2021-06-09 14:32:01 -04:00
NWalker4483 17077be701 exit at task 2021-06-09 12:40:38 -04:00
Kelly McDonald 5f722d675f Add a function that gets runs via a background scheduler to look for any workflows that are in a 'waiting' state - it runs the update waiting tasks and does do_engine_steps 
We have a test for the function that runs, but an assumption was made that the scheduler module has its own unit tests and we do not need to test that.

fixes #346
 2021-06-09 10:42:34 -04:00
Kelly McDonald 0d8d251e8e Temp save to test dev 2021-06-09 09:24:37 -04:00
Dan Funk 586a0f2f1a
Merge pull request #325 from sartography/feature/file_refactor 
Prefer tasks that share a parent over just the the next available tas…
 2021-06-08 11:28:23 -04:00
Dan 8d79fe9d94 Fixing failing tests, and now asserting that we only perform the post_process_form if we actually have a form. 2021-06-08 11:16:10 -04:00
Dan Funk 9c4994581d
Merge pull request #324 from sartography/modify-infinite-loop-312 
Modify infinite loop #312
 2021-06-08 10:30:13 -04:00
Dan Funk a7b208f50a
Merge pull request #322 from sartography/data-store-refactor-330 
Data store refactor #330
 2021-06-08 08:58:53 -04:00
Dan 59f605c3df Prefer tasks that share a parent over just the the next available task when returning the next_task in the workflow processor. 2021-06-08 08:03:14 -04:00
mike cullerton c41657301a Modify `test_spec` and `make_test_workflow` to accept new optional `validate_study_id` parameter. 
Modify `make_test_workflow` to use study_id when creating test workflow for validation.
 2021-06-03 14:22:26 -04:00
mike cullerton e2e35b673d Moved data store code from script to service 2021-06-02 09:59:41 -04:00
Kelly McDonald a5d67bb245 337 partial fix 
if the user calls the file_data_set function for a valid file with the key 'irb_code' and a value of a valid IRB document code, then we should set the irb code on the file.
 2021-05-27 12:24:30 -04:00
Dan ef7ee284b2 Prefer tasks that share a parent over just the the next available task when returning the next_task in the workflow processor. 2021-05-19 16:26:16 -04:00
Mike Cullerton 668e2f4e3c
Merge pull request #313 from sartography/feature/add_file_data_store_properties_to_file 
Assure that any data store values associated with a file come back as…
 2021-05-14 16:30:35 -04:00
Dan 8e37f27399 Assure that any data store values associated with a file come back as a part of the get_study_data[documents 
] endooint.
 2021-05-14 15:52:25 -04:00
Dan Funk 4d7ce5eda9
Merge pull request #311 from sartography/deprecate-study-info-protocol-261 
Deprecate study info protocol #261
 2021-05-14 14:14:18 -04:00
Dan Funk ade7812540
Merge pull request #310 from sartography/get-primary-workflow-301 
Get primary workflow #301
 2021-05-14 12:41:48 -04:00
Dan Funk 6a34fc0cde
Merge pull request #309 from sartography/persist-read-only-data-324 
Persist read only data #324
 2021-05-14 12:38:48 -04:00
Dan Funk 99da502912
Merge branch 'dev' into 321-Markdown-Macro 2021-05-14 12:32:41 -04:00
mike cullerton 9b24ad319b Removed the `get_protocol` method from study_service. 2021-05-13 16:29:34 -04:00
mike cullerton 8a6bef5af4 New service to get the `primary workflow` for a workflow_spec, using a workflow_spec_id. 
This is necessary because `primary` is a file parameter, not a workflow_spec parameter.

- you can ask a workflow file whether it is primary, but
- you cannot ask a workflow_spec for its primary workflow file

Now, you can use `workflow_service.get_primary_workflow(workflow_spec_id)`
 2021-05-12 13:51:51 -04:00
mike cullerton 07f3d2c2ee Allow data from read-only fields to persist in task_data 2021-05-07 14:06:53 -04:00
Dan 5a79b80f32 fixes #322, do not error out deleting files that have assocaited data stores, just remove the data store. 2021-05-05 20:21:33 -04:00
Dan 7831bef050 Don't fail the sync completely when a remote file does not exist. 2021-05-05 15:59:00 -04:00
Kelly McDonald cbd1d01203 Add URL to the study_info('documents') script 
fixes #321 - I merged in branches that fix #320 and #297

320-add-default-for-file-data-get
297-filename-in-documents
 2021-05-05 11:30:08 -04:00
Dan Funk 8b0afa4437
Merge pull request #304 from sartography/irb-info-script-307 
Irb info script #307
 2021-05-04 11:40:52 -04:00
Dan Funk 34759a2f3d
Merge branch 'dev' into feature/performance-refactor 2021-05-04 11:30:13 -04:00
Dan Funk d6054a9846
Merge branch 'dev' into 310-task-event-timezone 2021-05-04 11:21:36 -04:00
Dan Funk 81467d6a8d
Merge pull request #299 from sartography/297-filename-in-documents 
Add filename to files listing.
 2021-05-04 11:19:59 -04:00
mike cullerton 804926dbac Fixed problem where WorkflowService.make_test_workflow did not add a study to the workflow_model 2021-05-01 15:55:20 -04:00
mike cullerton 22b3230243 Added service `get_irb_info` to `crc.services.protocol_builder.ProtocolBuilderService` 
Added environment variable `PB_IRB_INFO_URL` for new `get_irb_info` service
 2021-05-01 15:20:47 -04:00
Dan a719cf4bf9 When retrieving the study, only update the status of underlying workflows if specifically requested. 
Record the size of a file in the database for quick access (this helps with a frontend refactor, so it isn't downloading the file just to see it's size)
Cleaning up the timing/performance metric reporting to make it easier to read.
Fixing a bug that prevented non-admins for getting the document-directory
 2021-04-30 11:55:12 -04:00
mike cullerton 24c818bf31 Added study_id to workflow_api 2021-04-29 14:29:21 -04:00
Kelly McDonald 2b9cee6b89 Update database to include timezone and change all points where we set the time on an event to be utc time. If we get something in the database with a timezone, it will display properly on the front end, but by default everything will be put in the database in UTC 2021-04-29 10:25:28 -04:00
Kelly McDonald cc77a9182c Add filename to files listing. 
Fixes #297
 2021-04-28 10:37:14 -04:00
Mike Cullerton 39fcab3730
Merge branch 'dev' into launch-workflow-outside-study-204 2021-04-26 10:27:32 -04:00
mike cullerton 077b68c1e2 When creating a workflow for a study, set the user_id to None 2021-04-26 08:46:19 -04:00
mike cullerton f17a9dc0f4 Modified `get_users_assigned_to_task`. 
If we are running a standalone workflow, only return the current user.
 2021-04-26 08:45:08 -04:00
mike cullerton b6f5001687 Added methods for the 2 new api endpoints 2021-04-26 08:43:14 -04:00
Dan cf051a7d4f Merge branch 'dev' into 277-Document-pane 
# Conflicts:
#	crc/api/file.py
 2021-04-21 14:09:44 -04:00
Kelly McDonald e2197ddab9 Changes to support the frontend data pane 
fixes #277 and requires frontend branch 277-document-pane
 2021-04-20 08:12:27 -04:00
mike cullerton 91e80d01bc Added delete_files argument to the workflow_processor reset method. 
Added code to delete the files when delete_files is True.
 2021-04-15 11:25:58 -04:00
Dan 6427a54d74 Set the correct default value on enumerations - if it is a multi-select enum (checkbox is set) then it should return a default value of an empty array, and not a dictionary. 2021-04-09 08:40:58 -04:00
Dan dcb4e7b736 fixing an error by taking a bit of the magic out of one of Mike's regex hinting tools. I figure the hint is still good, and a little less bittle, and all the information is already there if people need to dig. 2021-04-02 17:58:27 -04:00
Kelly McDonald 116bf5e7aa Merge branch 'dev' into 263-optimize-dashboard 
# Conflicts:
#	crc/services/study_service.py
 2021-03-31 10:20:33 -04:00
Kelly McDonald 9084866015 Update error message - it may be that we should spend a bit and see if we can make the Box error 2021-03-31 10:17:08 -04:00
Kelly McDonald 60f5be1aef Check in - pending change from @cullerton 2021-03-30 12:10:49 -04:00
Dan Funk b5d4df595f
Merge pull request #271 from sartography/master-workflow-status-messages-222 
Master workflow status messages 222
 2021-03-26 20:52:24 -04:00
Dan ded11ddd82 fixing a code smell 2021-03-26 20:25:34 -04:00
Kelly McDonald f1ca44a0e1 Add primitive cache on the fileservice is_review, halves the time of loading the list on the primary dashboard 2021-03-23 10:17:48 -04:00
mike cullerton a201e49d21 The master workflow now generates a dictionary of dictionaries, rather than a dictionary of strings. 
Each entry in status is a dictionary with 'status' and 'message' keys.
We updated the StudyService._update_status_of_workflow_meta method to accommodate the master workflow changes.

Changed the order of some if statements so we move forward while we have good data, and run all the error states at the end.
Added some comments to explain the cascading if statements

We changed the names of private methods to begin with one underscore, so they work in the test environment.
Changed  some internal calls to accommodate the underscore change.
 2021-03-22 17:46:39 -04:00
Nile Walker a08c6b794f Updated Study Associate Endpoints 2021-03-22 17:30:49 -04:00
mike cullerton 938079cf59 Lowered count ceiling to 100, from 1000. This significantly increased validation time, from about 3 minutes to about 10 seconds. 
Changed ApiError call to from_task, instead of from_workflow_exception.
 2021-03-17 14:52:19 -04:00
mike cullerton 529bfc34b3 We now check for an infinite loop during validation. 
This happened when testing for a Department Chair in a workflow.
Our mocked data for study_info script did not have a Department Chair
 2021-03-16 13:35:51 -04:00
mike cullerton c05b879dd4 We want to modify the master workflow so that it returns both a status, and a message explaining the status for each workflow. 
I.e., from status = {'ids_full_submission': 'required',
                              'enter_core_info': 'disabled',
                              ...}
       to     status = {'ids_full_submission': {'status': 'required', 'message': 'required because...'},
                              'enter_core_info': {'status': 'disabled', 'message': 'disabled because...'},
                              ...}

Modified study_service.__update_status_of_workflow_meta to accommodate this change.
 2021-03-16 12:09:42 -04:00
Dan Funk 14386b8ba9
Merge pull request #264 from sartography/bug/243_complete_all 
Bug/243 complete all
 2021-03-14 12:34:55 -04:00
Dan Funk 0da07675b7
Merge pull request #258 from sartography/study-associated-email-207-223 
Study associated email 207 223
 2021-03-14 12:34:19 -04:00
Dan 6f80d816cd My privous efforts didn't take into account the fact that we use dot notation in field names. 
I've re-worked the workflow form endpoint, so that it only accepts the data that should be in the form, and ignores any other values that come back from the front end.  It seems Formly has some bugs that were introducing confusing information, and I want everything to behave consistently.
I had to re-work some of the tests, which were relying on an ability to set data through a form post without having a corresponding form to do so.
 2021-03-14 12:20:39 -04:00
Dan 15c64c862e Adding an argument to update task that will allow it to complete all remaining tasks through an "update_all" flag, 
this will allow us to remove some fairly complex logic from the front end that isn't behaving properly.
 2021-03-12 20:41:07 -05:00
Dan 1000b22a3b Don't raise errors on non-ready tasks that calculate a title, the information may not be avaiable yet. 
Also Allow autocomplete-num as a valid field property.
 2021-03-12 15:40:23 -05:00
Mike Cullerton e7c78c8d0e
Merge branch 'dev' into study-associated-email-207-223 2021-03-11 12:38:47 -05:00
mike cullerton e58f1dc484 Remove unused mails service 2021-03-11 11:56:18 -05:00
mike cullerton 67697b5076 Lots of changes! 
Email Script:
- The email script now accepts 3 keyword arguments; subject, recipients, and cc.
- Subject and recipients are required.
- Recipients (and cc) can be an email address or list of addresses.
- You can use the string 'associated' in place of an email address, in which case we look up the users associated with a study that have send_email set to True.
- Moved some helper methods to email_service.

Email Service:
- Added cc argument
- Added helper methods previously in email script.
- Modified get_rendered_content to accept message and data directly. Previously, we passed in the task and grabbed them from there. Now we can use the method outside a task.
 2021-03-11 11:32:21 -05:00
Kelly McDonald d250bac32e Merge remote-tracking branch 'origin/dev' into dev 2021-03-10 10:59:24 -05:00