154 Commits

Author SHA1 Message Date
Dan
513d5664c1 Getting all tests passing with latest SpiffWorkflow. 2022-11-16 12:29:18 -05:00
Dan
2249965ade Paginator arguments changed slightly in latest releases of SQLAlchemy
Spiffworkflow 1.2:  Top Level Imports moved to appropriate modules
   - replace 'from SpiffWorkflow import WorkflowException' to 'from SpiffWorkflow.exceptions import WorkflowException'
   - replace 'from SpiffWorkflow import TaskState' to 'from SpiffWorkflow.task import TaskState'
   - replace 'from SpiffWorkflow import Task' to 'from SpiffWorkflow.task import Task'

SpiffWorkflow 1.2: Navigation code removed completely.  Proved to be of little use to folks, was super complex and difficult to maintain.

SpiffWorkflow 1.2: When inserting custom functions into the PythonExecutionEngine - be aware that the task data will act as the full context for execution, and will contain global functions and methods during the exec call.

SpiffWorkflow 1.2: All Task Specs now have a spec_type attribute, containing a descriptive string of the type, such as "User Task", "Script Task", "Start Event" etc...
2022-10-07 14:58:08 -04:00
Dan
5c76c90d04 send_file's arguments changed in a 2.2 of Flask.
Spiffworkflow 1.2:  remove all references of timeit (no longer in SpiffWorkflow)
Spiffworkflow 1.2:  pythonScriptEngine._evaluate no longer accepts a task argument.
Spiffworkflow 1.2:  CancelEventDefinition was removed - please use SignalEventDefinition instead
 EX: replace bpmn_workflow.signal('cancel')  # generate a cancel signal.
              bpmn_workflow.catch(CancelEventDefinition())
 WITH: bpmn_workflow.catch(SignalEventDefinition('cancel'))
Spiffworkflow 1.2:  Task States are JUST integers and TaskSpecNames is now a public dictionary, and can be used to covert a state to human readable string
    EX:  REPLACE:  user_task.state.name
         WITH:     TaskStateNames[user_task.state]
2022-10-05 15:31:56 -04:00
Dan
4305b36b74 Removed total_tasks and completed_tasks - as they are no longer used by the front end, and were dubious as hell to begin with.
Upraded SpiffWorkflow and now use th new get_subprocess_specs
updated the calculate_stats in the workflow processor - as the serialization had changed drastically, and needed to debug some performance issues.
Added a get_navigation method that will calcuate a basic navigation list MUCH faster than using the get_flat_nav_list in Spiffworkflows Navigation object.
Modified a hellton of tests because we don't have total_task and completed_task counts, or a complex nested navigation list anymore.
2022-07-20 12:10:23 -04:00
Dan
9ab29cf1fe Upgrade spiffworkflow - the WorkflowTaskExecException was moved - so we have to update all import statements. 2022-07-14 13:37:16 -04:00
Dan
b15750d2e4 Cleaned up the Workflow Processor's init method
* removed all the performance metric code into a separate function.
* restructured the code so it is either creating a new workflow, or deserializing an old one.
* Added code to upgrade serialized objects from 1.0 to 1.1
* Using the new method of creating a bpmn_workflow object:
```python
                parser = self.get_spec_parser(self.spec_files, spec_info)
                top_level = parser.get_spec(spec_info.primary_process_id)
                subprocesses = parser.get_process_specs()
                self.bpmn_workflow = BpmnWorkflow(top_level, subprocesses, script_engine=self._script_engine)
```
Fixed a few minor bugs that stood out while testing
1. when updating a workflow, we should check for a valid task BEFORE calling cancel_notify, which requires a valid task.
2. get_localtime - quick fix on the date parser - for python 3.9.
3. the start_workflow script would error out in a way that made it unclear which workflow was having the problem.  Fixed the error.
2022-06-24 11:34:39 -04:00
mike cullerton
f74afc3e18 Remove current_user from task data 2022-05-31 16:51:03 -04:00
mike cullerton
1d8fc77cf3 Add impersonator to current_user, if we have one 2022-05-31 15:01:34 -04:00
Dan
e4627c472a rollback database transactions when we encounter an error resetting a workflow, or processing a waiting workflow. 2022-05-19 10:23:48 -04:00
Dan
063da419d0 updating for the last version of Spiffworkflow which deprecates the use of task.STATE values, in favor of TaskState.STATE, for example TaskState.READY 2022-05-10 12:28:55 -04:00
Dan
b1546d4ba1 Merge branch 'dev' into file-refactor-705 2022-04-27 12:12:47 -04:00
Dan
12d717b395 Bump version of SpiffWorkflow, and inclode some additional debugging tools for size of json data in BPMN. 2022-04-25 07:55:10 -04:00
Dan
c3e8793110 hot fix for issus with the new serializer. 2022-04-20 15:22:08 -04:00
Dan
18929580bc Hot fix for another issue with the new serializer. 2022-04-20 14:27:36 -04:00
Dan
02703b89d8 Don't bomb out trying to parse the old json format for creating warnings about the task size. 2022-04-20 12:35:34 -04:00
Dan
85d11d913f Merge branch 'dev' into feature/new_spiff_serializer 2022-04-20 10:59:58 -04:00
mike cullerton
8e6ca4c97f DocumentModel => FileModel
document_id => file_id
2022-04-19 14:56:34 -04:00
mike cullerton
fb5731ff23 Merge branch 'dev' into file-refactor-705
# Conflicts:
#	crc/scripts/delete_task_data.py
#	crc/services/workflow_processor.py
#	tests/base_test.py
#	tests/files/test_delete_task_data.py
2022-04-19 08:19:52 -04:00
mike cullerton
a1882c0615 delete_file no longer static method 2022-04-18 16:40:19 -04:00
Dan
5e54c90b47 "Reset" should not mean "re-start". Calling the reset_workflow script will now set the workflow to an "unstarted" state with no outstanding tasks, no json-state structure stored. The workflow is not yet running.
Also:
* Assured that arguments are consistent (we always seem to use workflow_spec_id, so I made sure we use that consistently.
* Don't require named parameters - so it's cool to call it like: reset_workflow('my_workflow_id')
* Task Actions (ie create, assign, etc...) are now an enumeration in the models, and not static variables on Workflow Service, so we can reference them consistently from anywhere.
* Removed some repetitive code
* Always try to validate as much as possible in the scripts to save folks time debugging.
*
2022-04-15 15:36:23 -04:00
mike cullerton
c2e39dfc95 Type is now a string, not an Enum 2022-04-15 11:45:59 -04:00
Dan
e68c95f03c New serialization code, with the ability to load from the old serlializer if needed.
pipenv insisted on updating some other libraries so slight modifications for markupsafe.
2022-04-15 10:26:23 -04:00
mike cullerton
b1dca3f212 FileModel => DocumentModel
Remove unused imports
2022-04-13 08:34:10 -04:00
Dan
0455506e2b Loading latest chages from Spiff whic refactored the way the PythonExpressionEngine works.
This cleans up the _evaluate method which previously accepted arbitrary args and kwargs, and now requires an expression, a context to which exectute it, and, optionally, the current task being executed if the DMN is being executed as a part of a BusinessRuleTask in a BPMN diagram.

This also cleans up several bits of duplicated code.

There is also a bit of code here to assure that the current user is included when running the master workflow.
2022-03-04 15:34:36 -05:00
Dan
3944d00e88 Removing an unused / rogue variable.
Adding a logger warning for very large workflow json data.
2022-02-25 09:59:41 -05:00
Dan
e61ca471db remove all the @timeit from method calls. 2022-02-24 21:06:22 -05:00
Dan
dde8873c9e 1. defer the loading of the bpmn_json when loading the workflow model. It can be stupidly larger. (see models/workflow.py)
2. Shave a little more time off the reset by not re-creating the whole WorkflowProcessor

cleaning up all the firsttime/sincetime statements.
2022-02-24 20:56:57 -05:00
Dan
3442655bd1 Performance Improvements:
1. Avoid ever re-generating the list of scripts that can be used in a script task.  Terribly expensive as we call eval constantly, and it never ever changes once the app starts. (see script.py changes, and comments)
2. Cache the DocumentStatus list in the flask session, so we calculate it at most once per API Call.  It's at least .25 seconds per call.  (see study_sevice)
3. We called UserFileService.get_files_for_study (which runs a db query EVERY time) for every possible document type.  Now we run the query once (study service line 321)
4. When returning a workflow, we looped through every single task in that workflow's navigation, and called the expensive spiff_task_to_api_task just to figure out it's proper display name. We run a much faster and more efficient method to calculate the display name naow (see workflow_service on lie 680, and 799)
5. A hellton of @timeit and sincetime() calls, that I want to leave in, to help debug any slowness on production.
2022-02-24 14:25:42 -05:00
Dan
c3e52f960b Minor stuff to help digging into validation of current workflows. 2022-02-18 17:20:04 -05:00
Dan
f2b6008e5f Fixes 619 - look up models were being built incorrectly, and repeatedly, and sometimes bombed out all together.
bonus: resond with a valid error message when an invalid task_id is requested (don't just 500)
2022-02-17 11:04:50 -05:00
Dan
0ad9843569 Dealing with some changes in SpiffWorkflow's latest release. I think this gets all tests passing again, and behaving as expected. 2022-02-16 10:46:46 -05:00
Dan
cc6f80bea9 fixing some paths. 2022-02-15 16:25:52 -05:00
Dan
093741fa90 By default the system will start up the scheduled tasks, which is usually what you want, but in development
it can be kind of irritating for this stuff to be spinning up when you are trying to debug something, so just set
PROCESS_WAITING_TASKS to false in instance/config.py and voila!!
2022-02-15 14:07:09 -05:00
Dan
648383f2ea for some reason jenkins doesn't like the spec_category_reorder test, so just doing it a different way. 2022-02-11 11:37:25 -05:00
Dan
f12c4aba52 Refactor the workflow_spec_service so it doesn't cache anything, it always reads what it needs from the file system. 2022-02-09 21:06:55 -05:00
Dan
89e02c6207 really need to avoid any kind of caching in the WorkflowSpecService 2022-02-09 13:37:57 -05:00
Dan
cc915ac25a refactoring the study service. 2022-02-09 08:50:00 -05:00
Dan
4b8d193fb9 a few more tests passing 2022-02-07 14:58:25 -05:00
Dan
2383c7d76d WIP - lots of random changes, trying to swtich over 2022-02-04 14:50:31 -05:00
Dan
4ec6e403f5 1. Created a UserFileService, so it's clear what we use when for File Services, UserFiles, SpecFiles, and ReferenceFiles each function differently.
2. Reference Files and Spec Files are written to disk, they do not exist in the database at all.
2022-02-02 12:59:56 -05:00
Dan
8529465322 Removed the method get_spec_data_files completly - using get_spec_files and get_spec_data to get this information instead.
Only load the spec data files if you are creating a new workflow, otherwise just deserialize the json.
Removed the stuff about calculaing the version of the spec, as we don't use it.
2022-01-25 16:10:54 -05:00
mike cullerton
68820c67cb Removed (almost) all references to WorkflowSpecDependencyFile
(There is still a call in the lookup service, but we need to decide how to fix that)
2022-01-19 16:12:54 -05:00
mike cullerton
b99ed73951 Remove unused imports 2022-01-12 15:00:26 -05:00
mike cullerton
9cc91f92c3 *** WIP ***
cleanup - removing commented code
2022-01-11 15:55:08 -05:00
mike cullerton
dc27f795c8 *** WIP ***
Committing because it is Friday afternoon, and my computer is acting flaky
2022-01-07 15:34:51 -05:00
mike cullerton
86a6039dc8 *** WIP ***
**Many** tests are failing!

Committing so I can merge dev into this branch
2022-01-06 11:46:54 -05:00
Dan
3480e1c8ab use a signal, in addition to firing off a cancel notify when a workflow is restarted. 2021-12-23 15:30:09 -05:00
Dan
bd3176eea6 Fixes #578 - issue with us displaying an incorrect end event in some cases. 2021-12-22 16:27:52 -05:00
Dan
ad93b5fae6 We were not correctly handing the possibility of there only being a waiting task (and no ready tasks).
We also had a problem with the validator throwing a 500 when it was trying tell us it would never finish validation.
2021-11-30 12:44:35 -05:00
Dan
4d1f4429ff Fixing a bug in enum_label that was not working when lookup up an select list from a spreadsheet.
Assure we raise more thoughtful error messages when running getting exceptions in engine tasks.
Field Options should always be available now due to a fix in Spiffworkflow.
2021-11-22 12:26:30 -05:00