Commit Graph

136 Commits

Author SHA1 Message Date
Dan Funk 50d2acac9c Made a very stupid mistake with LDAP connections, pushing up quickly to production. 2020-06-04 11:57:00 -04:00
Dan Funk 68aeaf1273 BE VERY CAREFUL where you create a new LdapService() - construction is expensive.
Adding a few more details to the "csv" endpoint for RRT.
2020-06-04 10:33:17 -04:00
Dan Funk bbcbfef1ba Fixing the migrations so I don't break the universe. 2020-06-04 10:09:36 -04:00
Dan Funk 1324533865 Some additional cleanup - when a file is "archived" it is no longer returned for any endpoints about files, but it
is directly accessible via id, in the event some request is made for it at a later date.
2020-06-04 09:49:42 -04:00
Dan Funk 217ecfc911 When you can't delete a file, mark it as archived. Don't include archived files in new approval requests. 2020-06-03 17:34:27 -04:00
Dan Funk c179c7781b Do not process or return cancelled approvals via the API. 2020-06-03 16:50:47 -04:00
Dan Funk e102214809 minor cleanup of error codes. 2020-06-03 15:03:22 -04:00
Dan Funk 299ad4fc8b Adding more details to the csv output, and assuring we don't miss people with outstanding approvals that were cancelled. 2020-06-03 07:58:48 -04:00
Dan Funk c7484267e1 For the main approval endpoints - we now group the approvals by study. So you get one record back for each study, but it may have other approvals along with it as "related_approvals".
We now cache the LDAP records - so we look in our own database for the record before calling out to ldap for the details when given a straight up computing id like dhf8r.

Added "date_approved" to the approval model.

And moved the approver and primary investigator into real associated models to make it easier to dump.

Fixed a problem with the validation that was causing it to throw incorrect errors on valid workflows. Getting it to behave a little more like the front end behaves, and respecting the read-only fields.  But it was mainly to do with always returning all the data with each form submission.
2020-06-02 18:17:00 -04:00
Dan Funk b6bf843f6e used 'name' rather than 'value' in the lookup options during validation, causing a disconnect with how this is processed on the front end. 2020-06-01 11:00:56 -04:00
Aaron Louie 7c8b7829ea Merge branch 'rrt/dev' into feature/approvals_enhancements 2020-06-01 00:41:51 -04:00
Carlos Lopez bec11980eb Fixing broken test by using proper FileSchema 2020-05-31 22:00:52 -06:00
Carlos Lopez b2e56f797b Converting ApprovalModel to Approval in order to serialize properly the result 2020-05-31 21:02:47 -06:00
Aaron Louie f0bd8d4f9e Adds approvals to study schema. Adds approvals endpoint 2020-05-31 22:46:32 -04:00
Dan Funk 9c7de39b09 This adds additional file data details to the study model as well. 2020-05-31 21:15:40 -04:00
Carlos Lopez 26809d1470 Waiting status renaming 2020-05-31 13:35:42 -06:00
Dan Funk 2bc735a3f0 Kelly wrote a beautiful method for resetting the workflow that doesn't loose data when reset inside a parallel task, all I needed to do was use it.
Catching TypeErrors and reporting them back to the UI so we don't 500 in a bad way (but we still 500)
2020-05-31 13:48:00 -04:00
Dan Funk 98fb305868 Run the validation process twice, each time it is requested, first populating everything, and then a second time
using on the required form fields.
2020-05-30 18:43:20 -04:00
Dan Funk 1f0e8741ba Run the validation twice, once completing all of the data, and a second time, completing only the required fields.
Also, add a helper method to reduce boiler plate code around Workflow Exceptions.
2020-05-30 17:26:27 -04:00
Dan Funk 860c475b29 Fill out repeating sections during validation process.
Also, when returning error messages, attempt to include the task data for the task that caused the error.
Also, when attempting to delete any file, respond with an API error explaining the issue, and log the details.
2020-05-30 15:37:04 -04:00
Dan Funk 4e4cc7884c Better ldap searching. 2020-05-29 15:17:51 -04:00
Dan Funk afb6be7c60 re-working the way the redirects function, so we pass arguments as a get parameter. Just trying to get rid of the weird lag on production.
I noticed the validation sometimes looks ahead for files, so looking at all the tasks now, not just the ready tasks for the lookup field.
Ran into an issue with validation where a workflow model was required, so I create one and delete it.  Another refactor for another day.
2020-05-29 04:42:48 -04:00
Dan Funk 11413838a7 Faster lookup fields. We were parsing the spec each time to get details about how to search. We're just grabbing the workflow id and task id now and building that straight into the full text search index for faster lookups. Should be peppy.
Another speed improvement - data in the FileDataModel is deferred, and not queried until it is specifically used, as the new data structures need to use this model frequently.
2020-05-29 01:39:39 -04:00
Dan Funk aea78de066 Merge remote-tracking branch 'origin/dev' into feature/file_refactor_part2 2020-05-28 20:06:32 -04:00
Dan Funk dba41f4759 Ludicrously stupid launch in a refactor of the way all files work in the system at a time where I crave sleep and peace above all other things.
Added a File class, that we wrap around the FileModel so the api endpoints don't change, but File no longer holds refences to versions or dates of the file_data model, we
figure this out based on a clean database structure.

The ApprovalFile is directly related to the file_data_model - so no chance that a reviewer would review the incorrect version of a file.py

Noticed that our FileType enum called "bpmn" "bpmm", hope this doesn't screw someone up.

Workflows are directly related to the data_models that create the workflow spec it needs.  So the files should always be there.  There are no more hashes, and thus no more hash errors where it can't find the files to rebuild the workflow.py

Not much to report here, other than I broke every single test in the system at one point.  So I'm super concerned about this, and will be testing it a lot before creating the pull request.
2020-05-28 20:03:50 -04:00
Dan Funk cd7f67ab48 A major refactor of how we search and store files, as there was a lot of confusing bits in here.
From an API point of view you can do the following (and only the following)

/files?workflow_spec_id=x
* You can find all files associated with a workflow_spec_id, and add a file with a workflow_spec_id
/files?workflow_id=x
* You can find all files associated with a workflow_id, and add a file that is directly associated with the workflow
/files?workflow_id=x&form_field_key=y
* You can find all files associated with a form element on a running workflow, and add a new file.
   Note: you can add multiple files to the same form_field_key, IF they have different file names. If the same name, the original file is archived,
   and the new file takes its place.

The study endpoints always return a list of the file metadata associated with the study.  Removed /studies-files, but there is an
endpoint called

/studies/all  - that returns all the studies in the system, and does include their files.

On a deeper level:
 The File model no longer contains:
  - study_id,
  - task_id,
  - form_field_key

Instead, if the file is associated with workflow - then that is the one way it is connected to the study, and we use this relationship to find files for a study.
A file is never associated with a task_id, as these change when the workflow is reloaded.
The form_field_key must match the irb_doc_code, so when requesting files for a form field, we just look up the irb_doc_code.
2020-05-28 08:27:26 -04:00
Aaron Louie 97cdbfce94 Deletes extra line break 2020-05-27 23:48:48 -04:00
Dan Funk 77f72e408f Lookup Service now raises exact matches to the top. Very hackish, but it works. 2020-05-27 14:36:10 -04:00
Dan Funk d5e075db82 Order search results by relevancy in the lookup service. 2020-05-27 09:47:44 -04:00
Dan Funk 397cb23b52 Is true "true", yes it is true. So true, is "false", so true, it is true. 2020-05-26 23:38:57 -04:00
Dan Funk 0025931a2e Trying hard to figure out why the DCOS servers think the protocol builder is enabled. 2020-05-26 23:18:14 -04:00
Dan Funk 7869fa596e Protocol Builder isn't disabled on the dcos servers, trying to figure out why, and assure it isn't some sort of weird race condition. 2020-05-26 22:42:49 -04:00
Dan Funk ccbf374b40 Loads of bug fixes.
Modifed the request_approval to take a list of arguments, which works better for us... today.
UpdateStudy correctly handles validation.
WorkflowService correctly populates random values from lookup tables.
And several fixes down in Spiffworkflow, including a big bug where only the last item in a decision table made it through.
2020-05-26 20:06:50 -04:00
Dan Funk 13186176ba Improved LDAP searches, allow filtering on last name as well as uva id. 2020-05-25 16:00:36 -04:00
Dan Funk be057e8758 Adding an "UpdateStudy" task that is able to update the data on the study model, useful for setting core data points on the model, such as setting the Primary Investigator, or altering the Study Title.
Fixing a bug where the validation of forms did not correctly process auto-complete fields.
Fixing a bug where the approvals script and the update study script could not process dot notation correctly.

Moved populate_random_data into the WorkflowService where it makes more sense.
2020-05-25 15:30:06 -04:00
Dan Funk 6cd4ef64d1 Fixing add_study api endpoint, so you can actually add a new "Study" with just some basic information.
Using the LDAP service for checking user details in development mode - even if you are using the back door.
Added a new Flask fucntion load-example-rrt-data that loads the rrt workflow, and not the CRC wrokflows.
Modified the "load-example-data" in the tests to use some test data, rather than loading up all the workflows[
in CRC each time, with a parameter to load crc data if that is required - which is enabled for just a handful of tests.
(Tests run in 1/4 the time now)
2020-05-25 12:29:05 -04:00
Dan Funk 971d9a58e9 As we now have an approval_service.py, I moved all the business logic into this service and out of the request_approval.py script. And moved all tests for these features into a test file for the service. Will make it easier to cross reference what is happening, as everything all happens in one file.
As many of the scripts need to know the workflow, and it's down in a weird parameter, moved this so it is passed in each time.
2020-05-24 16:13:15 -04:00
Carlos Lopez 49eb4b3f98 Making working endpoints for approvals 2020-05-23 23:53:48 -06:00
Dan Funk d5c91e575f stuff that might be broken. 2020-05-23 15:21:30 -04:00
Dan Funk d39ef658a2 Made some modifications to the Approval so that it knows exactly what versions of every file are being sent for approval
Added the following columns:
  * date_created - so we know when the file was created
  * renamed workflow_version to just "version", because everything has a version,  this is the version of the request.
  * workflow_hash - this is just a quick way to see what files and versions are associated with the request, it could be factored out.
  * study - a quick relationship link to the study, so that this model is easier to use.
  * workflow - ditto
  * approval_files - these is a list from a new link table that links an approval to specific files and versions.

The RequestApproval is logically sound, but still needs some additional pieces in place to be callable from a BPMN workflow diagram.

Altered the file service to pick up on changes to files vs adding new files, so that versions are picked up correctly as
users modify their submission - adding new files or replacing existing ones.  Deleting files worries me, and I will need to revisit this.

The damn base test keeps giving me a headache, so I made changes there to see if clearing and dropping the database each time won't allow the tests to pass more consistently.

Lots more tests around the file service to make sure it is versioning user uploaded files correctly.

The "Test Request Approval Script" tries to find to assure the correct behavior as this is likely to be called many times repeatedly and with little knowledge of the internal system.  So it should just "do the right thing".
2020-05-23 15:08:17 -04:00
Dan Funk 571c1d7d24 Merge branch 'feature/rrp-endpoints' into feature/disable_protocol_builder 2020-05-22 16:18:33 -04:00
Dan Funk 503c1c8f18 Allow disabling the Protocol Builder
PB_ENABLED can be set to false in the configuration (either in a file called instance/config.py, or as an environment variable)

Added a check in the base_test, to assure that we are always running tests with the test configuration, and bail out otherwise.  Setting TESTING=true as an environment variable will get this, but so well the correct ordering of imports. Just be dead certain the first file every test file imports is base_test.py.

Aaron was right, and we call the Protocol Builder in all kinds of awful places.  But we don't do this now.  So Carlos, you should have the ability to reuse a lot of the logic in the study_service now.

I dropped the poorly named "study-update" endpoint completely.  We weren't using it. POST and PUT to Study still work just fine for doing exactly that.

All the tests now run and pass with the Protocol builder disabled. Tests that specifically check PB behavior turn it back on for the test, or mock it out.
2020-05-22 14:37:49 -04:00
Carlos Lopez a0c884499e Merge branch 'feature/rrp-endpoints' of github.com:sartography/cr-connect-workflow into feature/rrp-endpoints 2020-05-22 09:46:25 -06:00
Carlos Lopez 1ed7930aab Endpoint for studies with files 2020-05-22 09:46:03 -06:00
Dan Funk b490005af7 dropping the remaining config stuff for flask_sso.
updaing the user 'sso' endpoint to provide additional information for debugging.
Pulling information from ldap to stay super consistent on where we get our information.
2020-05-22 09:50:18 -04:00
Aaron Louie 48f9873548 Adding yet another flush, because Travis builds keep failing due to database race condition issues in this method. 2020-05-20 10:02:30 -04:00
Aaron Louie 58189285ad Cleans up 2020-05-20 00:12:48 -04:00
Aaron Louie 2b5687c3a3 Fixes pernicious bug where template document versions were not being updated properly, and template completion script was not honoring version numbers 2020-05-20 00:10:32 -04:00
Aaron Louie 481a9ed04c Gets image ids from task data and Injects images into jinja docx 2020-05-19 21:51:54 -04:00
Dan Funk c4f2bd8dc6 Quick cleanup, adding a space 2020-05-19 16:23:20 -04:00