Commit Graph

64 Commits

Author SHA1 Message Date
alicia pritchett b9b8210f8a rm pi id refs 2022-03-18 12:04:20 -04:00
Dan e61ca471db remove all the @timeit from method calls. 2022-02-24 21:06:22 -05:00
Dan 3442655bd1 Performance Improvements:
1. Avoid ever re-generating the list of scripts that can be used in a script task.  Terribly expensive as we call eval constantly, and it never ever changes once the app starts. (see script.py changes, and comments)
2. Cache the DocumentStatus list in the flask session, so we calculate it at most once per API Call.  It's at least .25 seconds per call.  (see study_sevice)
3. We called UserFileService.get_files_for_study (which runs a db query EVERY time) for every possible document type.  Now we run the query once (study service line 321)
4. When returning a workflow, we looped through every single task in that workflow's navigation, and called the expensive spiff_task_to_api_task just to figure out it's proper display name. We run a much faster and more efficient method to calculate the display name naow (see workflow_service on lie 680, and 799)
5. A hellton of @timeit and sincetime() calls, that I want to leave in, to help debug any slowness on production.
2022-02-24 14:25:42 -05:00
Dan 4ec6e403f5 1. Created a UserFileService, so it's clear what we use when for File Services, UserFiles, SpecFiles, and ReferenceFiles each function differently.
2. Reference Files and Spec Files are written to disk, they do not exist in the database at all.
2022-02-02 12:59:56 -05:00
Dan 4a1285b8e0 When validating a workflow, don't run the master workflow multiple times, its a huge waste of time.
Provide decent data when calling get_study_associate(s) scripts.
2021-11-22 17:17:19 -05:00
mike cullerton d186b24de3 UVA PB returns the info in a list. We want the dictionary in the list 2021-11-05 14:26:21 -04:00
mike cullerton 8df6c78925 Cleaned up what we return from the description, for the tools/list_scripts API endpoint 2021-09-22 12:12:26 -04:00
Dan 1b1a994360 Refactoring Reference files to use the lookup table, rather than parsing the results directly out of the spreadsheet, or attempting to cache them.
Adding a DocumentService to clean up the FileService, and get Documents well seperated, as it seems likely be pulled out or seperated in the future, there is now a Documents api file as well, for the same reason.
Some other minor changes are just fixing white space to assure our code is linting correctly.
I removed _create_study_workflow_approvals from the base test, as we don't use approvals like this anymore.
2021-07-06 13:10:20 -04:00
Dan 07eb3f9ca8 Moving metrics into SpiffWorkflow so we can run the performance metrics deeply across both systems simultaniously.
Upgrading libraries.
Fixing deprication issue with Pandas and numpy.
We can only process xlsx files now, plain oldschool xls is fully removed.
2021-06-18 16:41:55 -04:00
Dan Funk 9c4994581d
Merge pull request #324 from sartography/modify-infinite-loop-312
Modify infinite loop #312
2021-06-08 10:30:13 -04:00
mike cullerton 9a63ab9c4f Change `do_task_validate_only` in study_info script to call the real do_task.
We do this so we can seed settings into the workflow for testing different scenarios using the study from validate_study_id
2021-06-03 14:28:24 -04:00
mike cullerton 384da075ef Removed references to `protocol` from the description 2021-05-20 11:54:12 -04:00
mike cullerton d189b84ba1 Removed `protocol` as an option in the study_info script 2021-05-13 16:22:42 -04:00
Dan 4fd8fd13f8 Remove approvals from the study_info script. It was broken, and isn't really possible in this way anymore 2021-04-05 12:52:23 -04:00
Dan 501487b31c Assure we convert to box for validations as well, and test the bloodly snot out of this. 2021-04-05 12:47:50 -04:00
Dan 727aa72e18 Fix a bug from recent performance improvments - be sure that our study info script returns dicts that can be referenced immediately with dot notation, as this is a common pattern in our bpmn script tasks. 2021-04-05 11:31:06 -04:00
Kelly McDonald 116bf5e7aa Merge branch 'dev' into 263-optimize-dashboard
# Conflicts:
#	crc/services/study_service.py
2021-03-31 10:20:33 -04:00
Kelly McDonald 60f5be1aef Check in - pending change from @cullerton 2021-03-30 12:10:49 -04:00
mike cullerton e2ba19748d Added department chair to the mocked data for study_info script.
The missing department chair caused an infinite loop in a workflow that checked for a department chair.
2021-03-16 13:36:55 -04:00
Kelly McDonald 06dedd8a32 Add short_title to study, change update_study task to use kw arguments and add the short_title to the update_study script 2021-03-02 10:03:53 -05:00
Dan Funk 386feddeed
Merge pull request #214 from sartography/validate-hide-expression-103
Modified study_info do_task_validate_only so that we take the default…
2020-12-31 10:53:27 -05:00
Dan efdbbcbc72 StudyInfo sometimes returns arrays of items, each item in the array should be coverted to a Box dictionary if possible. 2020-12-30 14:51:51 -05:00
mike cullerton f29429f0d0 Modified study_info do_task_validate_only so that we take the default path in ind_update workflow. 2020-12-30 10:04:10 -05:00
Kelly McDonald e598994d34 Change the study_sponsors script so that we can actually call it :
var = study_info('sponsors')

to keep it more in line with other scripts that we have
2020-11-05 10:31:47 -05:00
Kelly McDonald 2931f45187 Add 'sponsors' to study_info call 2020-11-04 14:34:10 -05:00
Dan Funk b544334f45 1. Updating Personnel BPMN diagram to debug some issues.
2. Disabling the token timeout for now, to see if this corrects the issues Alex is having with lost work.
3. Raising more thoughtful error messages for unknown lookup options.
4. Providing better validation of default values and injecting the correct value for defaults related to enum lists of all types.
5. Bumping Spiffworkflow library which contains some better error messages and checks.
2020-09-01 15:58:50 -04:00
Dan Funk e8067fe34e adding additional data to the study details returned during validation/shield testing. 2020-08-12 17:04:59 -04:00
Aaron Louie c25536dcca Makes Sonar happy. 2020-08-04 16:42:35 -04:00
Aaron Louie 2846bf9451 Fleshes out fake PB response, so workflows that pull from PB actually trigger validation errors 2020-08-04 15:50:29 -04:00
Kelly McDonald d617af8565 All tests are passing - may need to refactor a bit, / remove comments 2020-07-28 11:02:49 -04:00
Kelly McDonald cc55aed89c Change exception name 2020-07-27 12:18:28 -04:00
Kelly McDonald a124e13c6a Replace all legacy style calls with new calls.
Still having issues where we try to eval an empty definition, not quite sure why there is a difference from what we had before. I may need to revert some of it and determine what is going on.
2020-07-24 14:33:24 -04:00
Kelly McDonald f5ab283538 Test of adding in the ability of augmenting the workflow to include internal scripts like StudyInfo
This is the first waypoint on a larger effort to make all of the 'special scripts' that currently require a shebang to be just another python function.
2020-07-24 12:08:46 -04:00
Dan Funk 6c6c5d315d Fixing a stupid mistake where I used an unquoted value in the mock data for validation, which created a dict that could not be serilized. 2020-07-16 13:28:00 -04:00
Dan Funk a5cef8775e * Modifying the StudyInfo script to return both "investigators" and "roles", the investigators argument will return only the current active investigators according to the protocol builder. The "roles" argument returns all possible roles - it is what "investigators" use to be.
* The Task.title returned to the front end will now attempt to process the "display_name" property for dot-notation syntax, making it possible to use this for multi-instance tasks,
  but will work in all cases where we want he title to change based on values in the data model.
* Fixing a bug the test_study_api where it wasn't updated when we made recent changes to the different states of a study.
*
2020-07-06 12:09:21 -04:00
Dan Funk 971d9a58e9 As we now have an approval_service.py, I moved all the business logic into this service and out of the request_approval.py script. And moved all tests for these features into a test file for the service. Will make it easier to cross reference what is happening, as everything all happens in one file.
As many of the scripts need to know the workflow, and it's down in a weird parameter, moved this so it is passed in each time.
2020-05-24 16:13:15 -04:00
Dan Funk 99503fb832 missed a file in the last commit. 2020-05-11 17:42:24 -04:00
Dan Funk b7c11fd893 Merge branch 'master' into feature/investigators_reference_file 2020-05-11 17:36:37 -04:00
Dan Funk 02f8764056 Updated to use the latest script engine / evaluation engine that creates a single location where all values used in BPMN/DMN are processed. Right now this is a python based interpreter, but we will eventually base this on FEEL expressions.
The validation process needs to take the api model into account so we catch errors with bad file names.
2020-05-11 17:04:05 -04:00
Dan Funk da7cae51b8 Adding a new reference file that provides greater details about the investigators related to a study.
Improving the study_info script documentation to provide detailed examples of values returned based on arguments.
Making the tests a little more targetted and less subject to breaking through better mocks.
Allow all tests to pass even when ther protocol builder mock isn't running locally.
Removing the duplication of reference files in tests and static, as this seems silly to me at the moment.
2020-05-07 13:57:24 -04:00
Dan Funk 07e58e923d Merge remote-tracking branch 'origin/chore/update_specs' into feature/previous_task
# Conflicts:
#	Pipfile.lock

Assuring that all documents from the xls spreadsheet are loaded when doing validations.
2020-05-06 11:25:50 -04:00
Dan Funk f1f8b91c9c Refactor the document details scripts. Now there is one script, it returns data in a consistent format, and has all the details required. The script is located in StudyInfo, with the argument documents. Make note that it returns a dictionary of ALL the documents, with a field to mark which ones are required according to the protocol builder. Others may become required if a workflow determines such, in which case the workflow will enforce this, and the document will have a count > 0, and additional details in a list of files within the document. I modified the XLS file to use lower case variable names, because it disturbed me, and we have to reference them frequently. Removed devious "as_object" variable on get_required_docs, so it behaves like the other methods all the time, and returns a dictionary. All the core business logic for finding the documents list now resides in the StudyService.
Because this changes the endpoint for all existing document details, I've modified all the test and static bpmn files to use the new format.
Shorting up the SponsorsList.xls file makes for slightly faster tests. seems senseless to load 5000 everytime we reset the data.
Tried to test all of this carefully in the test_study_details_documents.py test.
2020-04-29 15:08:11 -04:00
Aaron Louie beb86f0453 Adds protocol script to study service 2020-04-29 10:21:24 -04:00
Aaron Louie af1bb9f80d Adds more useful metadata to approvals and documents status scripts. Fleshes out and pretties up Documents & Approvals screen 2020-04-23 23:32:20 -04:00
Aaron Louie 47de010a88 Puts data from sequential calls to StudyInfo into the right place. Sets the required document flag correctly. 2020-04-23 21:02:08 -04:00
Aaron Louie d91f690388 Adds documents_status StudyInfo script. Adds Documents & Approvals workflow spec. 2020-04-23 19:25:01 -04:00
Aaron Louie 796c109611 Adds approvals to study service 2020-04-23 14:40:05 -04:00
Dan Funk ee999a0f15 fixing a bunch of stupid mistakes because I am tried. 2020-04-20 20:28:12 -04:00
Dan Funk edbd75bb75 Connect LDAP Requests to the StudyInfo service so we get back additional details. 2020-04-20 16:02:13 -04:00
Dan Funk aa17e30762 Updating libraries, modifying study_info script to save data in a consistent way, waiting on more fixes in spiffworkflow before continuing. 2020-04-13 16:23:31 -04:00