sartography
/
cr-connect-workflow
mirror of https://github.com/sartography/cr-connect-workflow.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
cr-connect-workflow/tests/files/test_files_api.py

379 lines
19 KiB
Python
Raw Normal View History

Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
import io
import json
Refactoring Reference files to use the lookup table, rather than parsing the results directly out of the spreadsheet, or attempting to cache them. Adding a DocumentService to clean up the FileService, and get Documents well seperated, as it seems likely be pulled out or seperated in the future, there is now a Documents api file as well, for the same reason. Some other minor changes are just fixing white space to assure our code is linting correctly. I removed _create_study_workflow_approvals from the base test, as we don't use approvals like this anymore.
2021-07-06 17:10:20 +00:00
import os
A major refactor of how we search and store files, as there was a lot of confusing bits in here. From an API point of view you can do the following (and only the following) /files?workflow_spec_id=x * You can find all files associated with a workflow_spec_id, and add a file with a workflow_spec_id /files?workflow_id=x * You can find all files associated with a workflow_id, and add a file that is directly associated with the workflow /files?workflow_id=x&form_field_key=y * You can find all files associated with a form element on a running workflow, and add a new file. Note: you can add multiple files to the same form_field_key, IF they have different file names. If the same name, the original file is archived, and the new file takes its place. The study endpoints always return a list of the file metadata associated with the study. Removed /studies-files, but there is an endpoint called /studies/all - that returns all the studies in the system, and does include their files. On a deeper level: The File model no longer contains: - study_id, - task_id, - form_field_key Instead, if the file is associated with workflow - then that is the one way it is connected to the study, and we use this relationship to find files for a study. A file is never associated with a task_id, as these change when the workflow is reloaded. The form_field_key must match the irb_doc_code, so when requesting files for a form field, we just look up the irb_doc_code.
2020-05-28 12:27:26 +00:00
from tests.base_test import BaseTest
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
Refactoring Reference files to use the lookup table, rather than parsing the results directly out of the spreadsheet, or attempting to cache them. Adding a DocumentService to clean up the FileService, and get Documents well seperated, as it seems likely be pulled out or seperated in the future, there is now a Documents api file as well, for the same reason. Some other minor changes are just fixing white space to assure our code is linting correctly. I removed _create_study_workflow_approvals from the base test, as we don't use approvals like this anymore.
2021-07-06 17:10:20 +00:00
from crc import session, db, app
Ludicrously stupid launch in a refactor of the way all files work in the system at a time where I crave sleep and peace above all other things. Added a File class, that we wrap around the FileModel so the api endpoints don't change, but File no longer holds refences to versions or dates of the file_data model, we figure this out based on a clean database structure. The ApprovalFile is directly related to the file_data_model - so no chance that a reviewer would review the incorrect version of a file.py Noticed that our FileType enum called "bpmn" "bpmm", hope this doesn't screw someone up. Workflows are directly related to the data_models that create the workflow spec it needs. So the files should always be there. There are no more hashes, and thus no more hash errors where it can't find the files to rebuild the workflow.py Not much to report here, other than I broke every single test in the system at one point. So I'm super concerned about this, and will be testing it a lot before creating the pull request.
2020-05-29 00:03:50 +00:00
from crc.models.file import FileModel, FileType, FileSchema, FileModelSchema
Fixes import paths
2020-01-24 16:52:52 +00:00
from crc.models.workflow import WorkflowSpecModel
Found a few errors in the sqlalchemy file definition that was causing failures, and had some consistency problems with the IRB Categories file name. The API was bailing out because we had restricted file types to bpmn,svg,or dmn in the connexion config file, I don't restrict this anymore we have plenty of checks elsewhere. Adding xlrd as a dependency - this didn't fail till a push to production.
2020-03-20 12:21:21 +00:00
from crc.services.file_service import FileService
Assure that files uploaded through web forms and files generated from templates can be cross-referenced to known document requirements from the protocol builder. Configurators can control this by managing an XLS Spreadsheet called "irb_documents.xslx". Required Documents is becoming complicated, so making this it's own script task, removing it from study_info.py The file_service is now very aware of this irb_documents file, so it will always need to exist. We seed this file during setup, but it can be overwritten by the configurator.
2020-03-19 21:13:30 +00:00
from crc.services.workflow_processor import WorkflowProcessor
Prefer tasks that share a parent over just the the next available task when returning the next_task in the workflow processor.
2021-06-08 12:03:14 +00:00
from crc.models.data_store import DataStoreModel
Refactoring Reference files to use the lookup table, rather than parsing the results directly out of the spreadsheet, or attempting to cache them. Adding a DocumentService to clean up the FileService, and get Documents well seperated, as it seems likely be pulled out or seperated in the future, there is now a Documents api file as well, for the same reason. Some other minor changes are just fixing white space to assure our code is linting correctly. I removed _create_study_workflow_approvals from the base test, as we don't use approvals like this anymore.
2021-07-06 17:10:20 +00:00
from crc.services.document_service import DocumentService
Adding a new reference file that provides greater details about the investigators related to a study. Improving the study_info script documentation to provide detailed examples of values returned based on arguments. Making the tests a little more targetted and less subject to breaking through better mocks. Allow all tests to pass even when ther protocol builder mock isn't running locally. Removing the duplication of reference files in tests and static, as this seems silly to me at the moment.
2020-05-07 17:57:24 +00:00
from example_data import ExampleDataLoader
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
Protocol builder connections
2020-02-20 18:30:04 +00:00
class TestFilesApi(BaseTest):
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
def test_list_files_for_workflow_spec(self):
Fixing add_study api endpoint, so you can actually add a new "Study" with just some basic information. Using the LDAP service for checking user details in development mode - even if you are using the back door. Added a new Flask fucntion load-example-rrt-data that loads the rrt workflow, and not the CRC wrokflows. Modified the "load-example-data" in the tests to use some test data, rather than loading up all the workflows[ in CRC each time, with a parameter to load crc data if that is required - which is enabled for just a handful of tests. (Tests run in 1/4 the time now)
2020-05-25 16:29:05 +00:00
self.load_example_data(use_crc_data=True)
Updates BPMN and DMN files
2020-04-15 14:58:13 +00:00
spec_id = 'core_info'
spec = session.query(WorkflowSpecModel).filter_by(id=spec_id).first()
rv = self.app.get('/v1.0/file?workflow_spec_id=%s' % spec_id,
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
follow_redirects=True,
It became impossible to use the Swagger ui when we started adding authentication to all of the calls. I discovered Connexion and Swagger have a default way of handing JTW authentication and this cleans up our code quite a bit, moves the securing of endpoints into the API Definition, which is quite nice, and removes a whole library dependency (I never get to do that!) I added a SWAGGER_AUTH_KEY that can be used in non-production environments to allow users to quickly authenticate from the Swagger ui. I also removed all api calls to simple little happy api services, because that is just mean and pointless.
2020-03-24 18:15:21 +00:00
content_type="application/json", headers=self.logged_in_headers())
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
self.assert_success(rv)
json_data = json.loads(rv.get_data(as_text=True))
Updates all workflow specs to match staging
2020-04-28 02:54:05 +00:00
self.assertEqual(5, len(json_data))
files = FileModelSchema(many=True).load(json_data, session=session)
file_names = [f.name for f in files]
Changes to tests after removing `name` column from `workflow_spec` and `workflow_spec_category` tables.
2021-10-05 18:20:03 +00:00
self.assertTrue("%s.bpmn" % spec.id in file_names)
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
def test_list_multiple_files_for_workflow_spec(self):
self.load_example_data()
Created a "StudyService" and moved all complex logic around study manipulation out of the study api, and this service, as things were getting complicated. The Workflow Processor no longer creates the WorkflowModel, the study object handles that, and only passes the model into the workflow processor when it is ready to start the workflow. Created a Study object (seperate from the StudyModel) that can cronstructed on request, and contains a different data structure than we store in the DB. This allows us to return underlying Categories and Workflows in a clean way. Added a new status to workflows called "not_started", meaning we have not yet instantiated a processor or created a BPMN, they have no version yet and no stored data, just the possiblity of being started. The Top Level Workflow or "Master" workflow is now a part of the sample data, and loaded at all times. Removed the ability to "add a workflow to a study" and "remove a workflow from a study", a study contains all possible workflows by definition. Example data no longer creates users or studies, it just creates the specs.
2020-03-30 12:00:16 +00:00
spec = self.load_test_spec("random_fact")
Improve version handling of files. Consolidate more of this logic in FileService. Place the version on the actual data model, not the file model, so the file model remains the same, and we just version the data associated with it.
2020-03-04 18:40:25 +00:00
svgFile = FileModel(name="test.svg", type=FileType.svg,
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
primary=False, workflow_spec_id=spec.id)
Adds type hinting to db and session to fix Pycharm autocomplete. Replaces db.session with session.
2020-01-14 16:45:12 +00:00
session.add(svgFile)
session.flush()
Adds ability to get FileModel records by study and task
2020-02-04 02:56:18 +00:00
rv = self.app.get('/v1.0/file?workflow_spec_id=%s' % spec.id,
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
follow_redirects=True,
It became impossible to use the Swagger ui when we started adding authentication to all of the calls. I discovered Connexion and Swagger have a default way of handing JTW authentication and this cleans up our code quite a bit, moves the securing of endpoints into the API Definition, which is quite nice, and removes a whole library dependency (I never get to do that!) I added a SWAGGER_AUTH_KEY that can be used in non-production environments to allow users to quickly authenticate from the Swagger ui. I also removed all api calls to simple little happy api services, because that is just mean and pointless.
2020-03-24 18:15:21 +00:00
content_type="application/json", headers=self.logged_in_headers())
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
self.assert_success(rv)
json_data = json.loads(rv.get_data(as_text=True))
I added a second file to 'random_fact' test workflow, so another test was expecting 2 files in it after adding a new file, but there were 3 - Nothing to see here - move along
2020-12-14 15:37:16 +00:00
self.assertEqual(3, len(json_data))
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
Some additional cleanup - when a file is "archived" it is no longer returned for any endpoints about files, but it is directly accessible via id, in the event some request is made for it at a later date.
2020-06-04 13:49:42 +00:00
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
def test_create_file(self):
self.load_example_data()
Adds type hinting to db and session to fix Pycharm autocomplete. Replaces db.session with session.
2020-01-14 16:45:12 +00:00
spec = session.query(WorkflowSpecModel).first()
Adds ability to get FileModel records by study and task
2020-02-04 02:56:18 +00:00
data = {'file': (io.BytesIO(b"abcdef"), 'random_fact.svg')}
rv = self.app.post('/v1.0/file?workflow_spec_id=%s' % spec.id, data=data, follow_redirects=True,
It became impossible to use the Swagger ui when we started adding authentication to all of the calls. I discovered Connexion and Swagger have a default way of handing JTW authentication and this cleans up our code quite a bit, moves the securing of endpoints into the API Definition, which is quite nice, and removes a whole library dependency (I never get to do that!) I added a SWAGGER_AUTH_KEY that can be used in non-production environments to allow users to quickly authenticate from the Swagger ui. I also removed all api calls to simple little happy api services, because that is just mean and pointless.
2020-03-24 18:15:21 +00:00
content_type='multipart/form-data', headers=self.logged_in_headers())
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
self.assert_success(rv)
self.assertIsNotNone(rv.get_data())
json_data = json.loads(rv.get_data(as_text=True))
Adds type hinting to db and session to fix Pycharm autocomplete. Replaces db.session with session.
2020-01-14 16:45:12 +00:00
file = FileModelSchema().load(json_data, session=session)
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
self.assertEqual(FileType.svg, file.type)
self.assertFalse(file.primary)
self.assertEqual("image/svg+xml", file.content_type)
self.assertEqual(spec.id, file.workflow_spec_id)
It became impossible to use the Swagger ui when we started adding authentication to all of the calls. I discovered Connexion and Swagger have a default way of handing JTW authentication and this cleans up our code quite a bit, moves the securing of endpoints into the API Definition, which is quite nice, and removes a whole library dependency (I never get to do that!) I added a SWAGGER_AUTH_KEY that can be used in non-production environments to allow users to quickly authenticate from the Swagger ui. I also removed all api calls to simple little happy api services, because that is just mean and pointless.
2020-03-24 18:15:21 +00:00
rv = self.app.get('/v1.0/file/%i' % file.id, headers=self.logged_in_headers())
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
self.assert_success(rv)
json_data = json.loads(rv.get_data(as_text=True))
Adds type hinting to db and session to fix Pycharm autocomplete. Replaces db.session with session.
2020-01-14 16:45:12 +00:00
file2 = FileModelSchema().load(json_data, session=session)
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
self.assertEqual(file, file2)
Assure that files uploaded through web forms and files generated from templates can be cross-referenced to known document requirements from the protocol builder. Configurators can control this by managing an XLS Spreadsheet called "irb_documents.xslx". Required Documents is becoming complicated, so making this it's own script task, removing it from study_info.py The file_service is now very aware of this irb_documents file, so it will always need to exist. We seed this file during setup, but it can be overwritten by the configurator.
2020-03-19 21:13:30 +00:00
def test_add_file_from_task_and_form_errors_on_invalid_form_field_name(self):
self.create_reference_document()
workflow = self.create_workflow('file_upload_form')
processor = WorkflowProcessor(workflow)
Allow the workflow to be requested without making changes to the workflow - requires that you specify a read_only flag of true, otherwise it assumes that you want a fully prepared workflow with the next ready task set to run.
2020-07-28 17:33:38 +00:00
processor.do_engine_steps()
Assure that files uploaded through web forms and files generated from templates can be cross-referenced to known document requirements from the protocol builder. Configurators can control this by managing an XLS Spreadsheet called "irb_documents.xslx". Required Documents is becoming complicated, so making this it's own script task, removing it from study_info.py The file_service is now very aware of this irb_documents file, so it will always need to exist. We seed this file during setup, but it can be overwritten by the configurator.
2020-03-19 21:13:30 +00:00
task = processor.next_task()
data = {'file': (io.BytesIO(b"abcdef"), 'random_fact.svg')}
correct_name = task.task_spec.form.fields[0].id
data = {'file': (io.BytesIO(b"abcdef"), 'random_fact.svg')}
Add task_spec_name to a bunch of tests
2021-08-26 14:43:28 +00:00
rv = self.app.post('/v1.0/file?study_id=%i&workflow_id=%s&task_spec_name=%s&form_field_key=%s' %
Assure that the field type provided is supported. Catches errors such as adding a custom field type of 'text_area' rather than "textarea"
2021-09-29 18:05:45 +00:00
(workflow.study_id, workflow.id, task.get_name(), correct_name), data=data,
follow_redirects=True,
It became impossible to use the Swagger ui when we started adding authentication to all of the calls. I discovered Connexion and Swagger have a default way of handing JTW authentication and this cleans up our code quite a bit, moves the securing of endpoints into the API Definition, which is quite nice, and removes a whole library dependency (I never get to do that!) I added a SWAGGER_AUTH_KEY that can be used in non-production environments to allow users to quickly authenticate from the Swagger ui. I also removed all api calls to simple little happy api services, because that is just mean and pointless.
2020-03-24 18:15:21 +00:00
content_type='multipart/form-data', headers=self.logged_in_headers())
Assure that files uploaded through web forms and files generated from templates can be cross-referenced to known document requirements from the protocol builder. Configurators can control this by managing an XLS Spreadsheet called "irb_documents.xslx". Required Documents is becoming complicated, so making this it's own script task, removing it from study_info.py The file_service is now very aware of this irb_documents file, so it will always need to exist. We seed this file during setup, but it can be overwritten by the configurator.
2020-03-19 21:13:30 +00:00
self.assert_success(rv)
Some additional cleanup - when a file is "archived" it is no longer returned for any endpoints about files, but it is directly accessible via id, in the event some request is made for it at a later date.
2020-06-04 13:49:42 +00:00
def test_archive_file_no_longer_shows_up(self):
self.load_example_data()
self.create_reference_document()
workflow = self.create_workflow('file_upload_form')
processor = WorkflowProcessor(workflow)
Allow the workflow to be requested without making changes to the workflow - requires that you specify a read_only flag of true, otherwise it assumes that you want a fully prepared workflow with the next ready task set to run.
2020-07-28 17:33:38 +00:00
processor.do_engine_steps()
Some additional cleanup - when a file is "archived" it is no longer returned for any endpoints about files, but it is directly accessible via id, in the event some request is made for it at a later date.
2020-06-04 13:49:42 +00:00
task = processor.next_task()
data = {'file': (io.BytesIO(b"abcdef"), 'random_fact.svg')}
correct_name = task.task_spec.form.fields[0].id
data = {'file': (io.BytesIO(b"abcdef"), 'random_fact.svg')}
Add task_spec_name to a bunch of tests
2021-08-26 14:43:28 +00:00
rv = self.app.post('/v1.0/file?study_id=%i&workflow_id=%s&task_spec_name=%s&form_field_key=%s' %
Assure that the field type provided is supported. Catches errors such as adding a custom field type of 'text_area' rather than "textarea"
2021-09-29 18:05:45 +00:00
(workflow.study_id, workflow.id, task.get_name(), correct_name), data=data,
follow_redirects=True,
Some additional cleanup - when a file is "archived" it is no longer returned for any endpoints about files, but it is directly accessible via id, in the event some request is made for it at a later date.
2020-06-04 13:49:42 +00:00
content_type='multipart/form-data', headers=self.logged_in_headers())
self.assert_success(rv)
rv = self.app.get('/v1.0/file?workflow_id=%s' % workflow.id, headers=self.logged_in_headers())
self.assert_success(rv)
Handling incoming values from processor
2020-06-18 03:11:47 +00:00
self.assertEqual(1, len(json.loads(rv.get_data(as_text=True))))
Some additional cleanup - when a file is "archived" it is no longer returned for any endpoints about files, but it is directly accessible via id, in the event some request is made for it at a later date.
2020-06-04 13:49:42 +00:00
file_model = db.session.query(FileModel).filter(FileModel.workflow_id == workflow.id).all()
Handling incoming values from processor
2020-06-18 03:11:47 +00:00
self.assertEqual(1, len(file_model))
Some additional cleanup - when a file is "archived" it is no longer returned for any endpoints about files, but it is directly accessible via id, in the event some request is made for it at a later date.
2020-06-04 13:49:42 +00:00
file_model[0].archived = True
db.session.commit()
rv = self.app.get('/v1.0/file?workflow_id=%s' % workflow.id, headers=self.logged_in_headers())
self.assert_success(rv)
Handling incoming values from processor
2020-06-18 03:11:47 +00:00
self.assertEqual(0, len(json.loads(rv.get_data(as_text=True))))
Some additional cleanup - when a file is "archived" it is no longer returned for any endpoints about files, but it is directly accessible via id, in the event some request is made for it at a later date.
2020-06-04 13:49:42 +00:00
Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks.
2020-03-13 19:03:57 +00:00
def test_set_reference_file(self):
Fix the test
2021-07-07 16:54:16 +00:00
file_name = "documents.xlsx"
Refactoring Reference files to use the lookup table, rather than parsing the results directly out of the spreadsheet, or attempting to cache them. Adding a DocumentService to clean up the FileService, and get Documents well seperated, as it seems likely be pulled out or seperated in the future, there is now a Documents api file as well, for the same reason. Some other minor changes are just fixing white space to assure our code is linting correctly. I removed _create_study_workflow_approvals from the base test, as we don't use approvals like this anymore.
2021-07-06 17:10:20 +00:00
filepath = os.path.join(app.root_path, 'static', 'reference', 'irb_documents.xlsx')
with open(filepath, 'rb') as myfile:
file_data = myfile.read()
data = {'file': (io.BytesIO(file_data), file_name)}
Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks.
2020-03-13 19:03:57 +00:00
rv = self.app.put('/v1.0/reference_file/%s' % file_name, data=data, follow_redirects=True,
It became impossible to use the Swagger ui when we started adding authentication to all of the calls. I discovered Connexion and Swagger have a default way of handing JTW authentication and this cleans up our code quite a bit, moves the securing of endpoints into the API Definition, which is quite nice, and removes a whole library dependency (I never get to do that!) I added a SWAGGER_AUTH_KEY that can be used in non-production environments to allow users to quickly authenticate from the Swagger ui. I also removed all api calls to simple little happy api services, because that is just mean and pointless.
2020-03-24 18:15:21 +00:00
content_type='multipart/form-data', headers=self.logged_in_headers())
Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks.
2020-03-13 19:03:57 +00:00
self.assert_success(rv)
self.assertIsNotNone(rv.get_data())
json_data = json.loads(rv.get_data(as_text=True))
file = FileModelSchema().load(json_data, session=session)
Refactoring Reference files to use the lookup table, rather than parsing the results directly out of the spreadsheet, or attempting to cache them. Adding a DocumentService to clean up the FileService, and get Documents well seperated, as it seems likely be pulled out or seperated in the future, there is now a Documents api file as well, for the same reason. Some other minor changes are just fixing white space to assure our code is linting correctly. I removed _create_study_workflow_approvals from the base test, as we don't use approvals like this anymore.
2021-07-06 17:10:20 +00:00
self.assertEqual(FileType.xlsx, file.type)
Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks.
2020-03-13 19:03:57 +00:00
self.assertTrue(file.is_reference)
Refactoring Reference files to use the lookup table, rather than parsing the results directly out of the spreadsheet, or attempting to cache them. Adding a DocumentService to clean up the FileService, and get Documents well seperated, as it seems likely be pulled out or seperated in the future, there is now a Documents api file as well, for the same reason. Some other minor changes are just fixing white space to assure our code is linting correctly. I removed _create_study_workflow_approvals from the base test, as we don't use approvals like this anymore.
2021-07-06 17:10:20 +00:00
self.assertEqual("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", file.content_type)
Add test for user_uid
2021-07-07 13:39:01 +00:00
self.assertEqual('dhf8r', json_data['user_uid'])
Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks.
2020-03-13 19:03:57 +00:00
def test_set_reference_file_bad_extension(self):
Refactoring Reference files to use the lookup table, rather than parsing the results directly out of the spreadsheet, or attempting to cache them. Adding a DocumentService to clean up the FileService, and get Documents well seperated, as it seems likely be pulled out or seperated in the future, there is now a Documents api file as well, for the same reason. Some other minor changes are just fixing white space to assure our code is linting correctly. I removed _create_study_workflow_approvals from the base test, as we don't use approvals like this anymore.
2021-07-06 17:10:20 +00:00
file_name = DocumentService.DOCUMENT_LIST
Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks.
2020-03-13 19:03:57 +00:00
data = {'file': (io.BytesIO(b"abcdef"), "does_not_matter.ppt")}
rv = self.app.put('/v1.0/reference_file/%s' % file_name, data=data, follow_redirects=True,
It became impossible to use the Swagger ui when we started adding authentication to all of the calls. I discovered Connexion and Swagger have a default way of handing JTW authentication and this cleans up our code quite a bit, moves the securing of endpoints into the API Definition, which is quite nice, and removes a whole library dependency (I never get to do that!) I added a SWAGGER_AUTH_KEY that can be used in non-production environments to allow users to quickly authenticate from the Swagger ui. I also removed all api calls to simple little happy api services, because that is just mean and pointless.
2020-03-24 18:15:21 +00:00
content_type='multipart/form-data', headers=self.logged_in_headers())
Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks.
2020-03-13 19:03:57 +00:00
self.assert_failure(rv, error_code="invalid_file_type")
def test_get_reference_file(self):
file_name = "irb_document_types.xls"
Refactoring Reference files to use the lookup table, rather than parsing the results directly out of the spreadsheet, or attempting to cache them. Adding a DocumentService to clean up the FileService, and get Documents well seperated, as it seems likely be pulled out or seperated in the future, there is now a Documents api file as well, for the same reason. Some other minor changes are just fixing white space to assure our code is linting correctly. I removed _create_study_workflow_approvals from the base test, as we don't use approvals like this anymore.
2021-07-06 17:10:20 +00:00
filepath = os.path.join(app.root_path, 'static', 'reference', 'irb_documents.xlsx')
with open(filepath, 'rb') as myfile:
file_data = myfile.read()
data = {'file': (io.BytesIO(file_data), file_name)}
Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks.
2020-03-13 19:03:57 +00:00
rv = self.app.put('/v1.0/reference_file/%s' % file_name, data=data, follow_redirects=True,
It became impossible to use the Swagger ui when we started adding authentication to all of the calls. I discovered Connexion and Swagger have a default way of handing JTW authentication and this cleans up our code quite a bit, moves the securing of endpoints into the API Definition, which is quite nice, and removes a whole library dependency (I never get to do that!) I added a SWAGGER_AUTH_KEY that can be used in non-production environments to allow users to quickly authenticate from the Swagger ui. I also removed all api calls to simple little happy api services, because that is just mean and pointless.
2020-03-24 18:15:21 +00:00
content_type='multipart/form-data', headers=self.logged_in_headers())
rv = self.app.get('/v1.0/reference_file/%s' % file_name, headers=self.logged_in_headers())
Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks.
2020-03-13 19:03:57 +00:00
self.assert_success(rv)
data_out = rv.get_data()
Refactoring Reference files to use the lookup table, rather than parsing the results directly out of the spreadsheet, or attempting to cache them. Adding a DocumentService to clean up the FileService, and get Documents well seperated, as it seems likely be pulled out or seperated in the future, there is now a Documents api file as well, for the same reason. Some other minor changes are just fixing white space to assure our code is linting correctly. I removed _create_study_workflow_approvals from the base test, as we don't use approvals like this anymore.
2021-07-06 17:10:20 +00:00
self.assertEqual(file_data, data_out)
Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks.
2020-03-13 19:03:57 +00:00
Adds a POST for reference file. and a cool test
2021-10-12 18:29:15 +00:00
def test_add_reference_file(self):
ExampleDataLoader().load_reference_documents()
file_name = 'new.xlsx'
data = {'file': (io.BytesIO(b"abcdef"), file_name)}
rv = self.app.post('/v1.0/reference_file', data=data,
follow_redirects=True,
content_type='multipart/form-data', headers=self.logged_in_headers())
self.assertIsNotNone(rv.get_data())
json_data = json.loads(rv.get_data(as_text=True))
file = FileModelSchema().load(json_data, session=session)
self.assertEqual(FileType.xlsx, file.type)
self.assertFalse(file.primary)
self.assertEqual(True, file.is_reference)
Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks.
2020-03-13 19:03:57 +00:00
def test_list_reference_files(self):
Adding a new reference file that provides greater details about the investigators related to a study. Improving the study_info script documentation to provide detailed examples of values returned based on arguments. Making the tests a little more targetted and less subject to breaking through better mocks. Allow all tests to pass even when ther protocol builder mock isn't running locally. Removing the duplication of reference files in tests and static, as this seems silly to me at the moment.
2020-05-07 17:57:24 +00:00
ExampleDataLoader.clean_db()
Refactoring Reference files to use the lookup table, rather than parsing the results directly out of the spreadsheet, or attempting to cache them. Adding a DocumentService to clean up the FileService, and get Documents well seperated, as it seems likely be pulled out or seperated in the future, there is now a Documents api file as well, for the same reason. Some other minor changes are just fixing white space to assure our code is linting correctly. I removed _create_study_workflow_approvals from the base test, as we don't use approvals like this anymore.
2021-07-06 17:10:20 +00:00
file_name = DocumentService.DOCUMENT_LIST
filepath = os.path.join(app.root_path, 'static', 'reference', 'irb_documents.xlsx')
with open(filepath, 'rb') as myfile:
file_data = myfile.read()
data = {'file': (io.BytesIO(file_data), file_name)}
Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks.
2020-03-13 19:03:57 +00:00
rv = self.app.put('/v1.0/reference_file/%s' % file_name, data=data, follow_redirects=True,
It became impossible to use the Swagger ui when we started adding authentication to all of the calls. I discovered Connexion and Swagger have a default way of handing JTW authentication and this cleans up our code quite a bit, moves the securing of endpoints into the API Definition, which is quite nice, and removes a whole library dependency (I never get to do that!) I added a SWAGGER_AUTH_KEY that can be used in non-production environments to allow users to quickly authenticate from the Swagger ui. I also removed all api calls to simple little happy api services, because that is just mean and pointless.
2020-03-24 18:15:21 +00:00
content_type='multipart/form-data', headers=self.logged_in_headers())
Refactoring Reference files to use the lookup table, rather than parsing the results directly out of the spreadsheet, or attempting to cache them. Adding a DocumentService to clean up the FileService, and get Documents well seperated, as it seems likely be pulled out or seperated in the future, there is now a Documents api file as well, for the same reason. Some other minor changes are just fixing white space to assure our code is linting correctly. I removed _create_study_workflow_approvals from the base test, as we don't use approvals like this anymore.
2021-07-06 17:10:20 +00:00
self.assert_success(rv)
Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks.
2020-03-13 19:03:57 +00:00
rv = self.app.get('/v1.0/reference_file',
follow_redirects=True,
It became impossible to use the Swagger ui when we started adding authentication to all of the calls. I discovered Connexion and Swagger have a default way of handing JTW authentication and this cleans up our code quite a bit, moves the securing of endpoints into the API Definition, which is quite nice, and removes a whole library dependency (I never get to do that!) I added a SWAGGER_AUTH_KEY that can be used in non-production environments to allow users to quickly authenticate from the Swagger ui. I also removed all api calls to simple little happy api services, because that is just mean and pointless.
2020-03-24 18:15:21 +00:00
content_type="application/json", headers=self.logged_in_headers())
Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks.
2020-03-13 19:03:57 +00:00
self.assert_success(rv)
json_data = json.loads(rv.get_data(as_text=True))
self.assertEqual(1, len(json_data))
file = FileModelSchema(many=True).load(json_data, session=session)
self.assertEqual(file_name, file[0].name)
self.assertTrue(file[0].is_reference)
Fixes broken tests
2020-01-31 16:33:43 +00:00
def test_update_file_info(self):
self.load_example_data()
Refactoring Reference files to use the lookup table, rather than parsing the results directly out of the spreadsheet, or attempting to cache them. Adding a DocumentService to clean up the FileService, and get Documents well seperated, as it seems likely be pulled out or seperated in the future, there is now a Documents api file as well, for the same reason. Some other minor changes are just fixing white space to assure our code is linting correctly. I removed _create_study_workflow_approvals from the base test, as we don't use approvals like this anymore.
2021-07-06 17:10:20 +00:00
self.create_reference_document()
file: FileModel = session.query(FileModel).filter(FileModel.is_reference==False).first()
Fixes broken tests
2020-01-31 16:33:43 +00:00
file.name = "silly_new_name.bpmn"
rv = self.app.put('/v1.0/file/%i' % file.id,
Assure that the field type provided is supported. Catches errors such as adding a custom field type of 'text_area' rather than "textarea"
2021-09-29 18:05:45 +00:00
content_type="application/json",
data=json.dumps(FileModelSchema().dump(file)), headers=self.logged_in_headers())
Fixes broken tests
2020-01-31 16:33:43 +00:00
self.assert_success(rv)
Followup on Postgres - had some failing tests that needed cleaning up now that we are using a real database.
2020-02-04 20:44:06 +00:00
db_file = session.query(FileModel).filter_by(id=file.id).first()
Fixes broken tests
2020-01-31 16:33:43 +00:00
self.assertIsNotNone(db_file)
self.assertEqual(file.name, db_file.name)
Assure that the field type provided is supported. Catches errors such as adding a custom field type of 'text_area' rather than "textarea"
2021-09-29 18:05:45 +00:00
def test_load_valid_url_for_files(self):
self.load_example_data()
self.create_reference_document()
file: FileModel = session.query(FileModel).filter(FileModel.is_reference == False).first()
rv = self.app.get('/v1.0/file/%i' % file.id, content_type="application/json", headers=self.logged_in_headers())
self.assert_success(rv)
file_json = json.loads(rv.get_data(as_text=True))
print(file_json)
self.assertIsNotNone(file_json['url'])
file_data_rv = self.app.get(file_json['url'])
self.assert_success(file_data_rv)
Fixes broken tests
2020-01-31 16:33:43 +00:00
def test_update_file_data(self):
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
self.load_example_data()
Adds type hinting to db and session to fix Pycharm autocomplete. Replaces db.session with session.
2020-01-14 16:45:12 +00:00
spec = session.query(WorkflowSpecModel).first()
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
data = {}
If you name add a file to a workflow that has the exact same name as a Task Spec's ID, and an extension of "md", it wll use that file as the markdown content, and ignore the markdown in the documentation on the task spec. Moving the primary process id from the workflow model to the file model, and assuring it is updated properly. This was causing a bug that would "lose" the workflow.
2020-04-17 17:30:32 +00:00
data['file'] = io.BytesIO(self.minimal_bpmn("abcdef")), 'my_new_file.bpmn'
Followup on Postgres - had some failing tests that needed cleaning up now that we are using a real database.
2020-02-04 20:44:06 +00:00
rv = self.app.post('/v1.0/file?workflow_spec_id=%s' % spec.id, data=data, follow_redirects=True,
It became impossible to use the Swagger ui when we started adding authentication to all of the calls. I discovered Connexion and Swagger have a default way of handing JTW authentication and this cleans up our code quite a bit, moves the securing of endpoints into the API Definition, which is quite nice, and removes a whole library dependency (I never get to do that!) I added a SWAGGER_AUTH_KEY that can be used in non-production environments to allow users to quickly authenticate from the Swagger ui. I also removed all api calls to simple little happy api services, because that is just mean and pointless.
2020-03-24 18:15:21 +00:00
content_type='multipart/form-data', headers=self.logged_in_headers())
When retrieving the study, only update the status of underlying workflows if specifically requested. Record the size of a file in the database for quick access (this helps with a frontend refactor, so it isn't downloading the file just to see it's size) Cleaning up the timing/performance metric reporting to make it easier to read. Fixing a bug that prevented non-admins for getting the document-directory
2021-04-30 15:55:12 +00:00
file_json = json.loads(rv.get_data(as_text=True))
assertEquals -> assertEqual
2021-08-16 14:18:29 +00:00
self.assertEqual(80, file_json['size'])
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
If you name add a file to a workflow that has the exact same name as a Task Spec's ID, and an extension of "md", it wll use that file as the markdown content, and ignore the markdown in the documentation on the task spec. Moving the primary process id from the workflow model to the file model, and assuring it is updated properly. This was causing a bug that would "lose" the workflow.
2020-04-17 17:30:32 +00:00
data['file'] = io.BytesIO(self.minimal_bpmn("efghijk")), 'my_new_file.bpmn'
When retrieving the study, only update the status of underlying workflows if specifically requested. Record the size of a file in the database for quick access (this helps with a frontend refactor, so it isn't downloading the file just to see it's size) Cleaning up the timing/performance metric reporting to make it easier to read. Fixing a bug that prevented non-admins for getting the document-directory
2021-04-30 15:55:12 +00:00
rv = self.app.put('/v1.0/file/%i/data' % file_json['id'], data=data, follow_redirects=True,
It became impossible to use the Swagger ui when we started adding authentication to all of the calls. I discovered Connexion and Swagger have a default way of handing JTW authentication and this cleans up our code quite a bit, moves the securing of endpoints into the API Definition, which is quite nice, and removes a whole library dependency (I never get to do that!) I added a SWAGGER_AUTH_KEY that can be used in non-production environments to allow users to quickly authenticate from the Swagger ui. I also removed all api calls to simple little happy api services, because that is just mean and pointless.
2020-03-24 18:15:21 +00:00
content_type='multipart/form-data', headers=self.logged_in_headers())
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
self.assert_success(rv)
self.assertIsNotNone(rv.get_data())
Ludicrously stupid launch in a refactor of the way all files work in the system at a time where I crave sleep and peace above all other things. Added a File class, that we wrap around the FileModel so the api endpoints don't change, but File no longer holds refences to versions or dates of the file_data model, we figure this out based on a clean database structure. The ApprovalFile is directly related to the file_data_model - so no chance that a reviewer would review the incorrect version of a file.py Noticed that our FileType enum called "bpmn" "bpmm", hope this doesn't screw someone up. Workflows are directly related to the data_models that create the workflow spec it needs. So the files should always be there. There are no more hashes, and thus no more hash errors where it can't find the files to rebuild the workflow.py Not much to report here, other than I broke every single test in the system at one point. So I'm super concerned about this, and will be testing it a lot before creating the pull request.
2020-05-29 00:03:50 +00:00
file_json = json.loads(rv.get_data(as_text=True))
self.assertEqual(2, file_json['latest_version'])
self.assertEqual(FileType.bpmn.value, file_json['type'])
self.assertEqual("application/octet-stream", file_json['content_type'])
When retrieving the study, only update the status of underlying workflows if specifically requested. Record the size of a file in the database for quick access (this helps with a frontend refactor, so it isn't downloading the file just to see it's size) Cleaning up the timing/performance metric reporting to make it easier to read. Fixing a bug that prevented non-admins for getting the document-directory
2021-04-30 15:55:12 +00:00
self.assertEqual(spec.id, file_json['workflow_spec_id'])
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
Removed all self-referential calls in the study_api. One api endpoint should never call another api endpoint. Moved the logic for updating a study to the study Model, rather than checking and setting dictionary values which will become very hard to maintain. The protocol builder service now returns real models, not dictionaries, forcing proper validation and fail-fast behavior. Changed the name of the "status" spec, to "top_level_workflow" and removing any connection a workflow or study has with this specification. It is only unused to determine status in real time, and is not reused or tracked. Modified the required documents script to return a dictionary and not an array, making it easier to speak to specific values in the BPMN and DMN. Working on new ways to test the top_level_workflow in the context of updates, this is still a work in progress. Making use of several modifications to the Spiff library that enables more complex expressions in DMN models. This is evident in the new DMN models for the top_level_workflow
2020-03-26 16:51:53 +00:00
# Assure it is updated in the database and properly persisted.
When retrieving the study, only update the status of underlying workflows if specifically requested. Record the size of a file in the database for quick access (this helps with a frontend refactor, so it isn't downloading the file just to see it's size) Cleaning up the timing/performance metric reporting to make it easier to read. Fixing a bug that prevented non-admins for getting the document-directory
2021-04-30 15:55:12 +00:00
file_model = session.query(FileModel).filter(FileModel.id == file_json['id']).first()
Ludicrously stupid launch in a refactor of the way all files work in the system at a time where I crave sleep and peace above all other things. Added a File class, that we wrap around the FileModel so the api endpoints don't change, but File no longer holds refences to versions or dates of the file_data model, we figure this out based on a clean database structure. The ApprovalFile is directly related to the file_data_model - so no chance that a reviewer would review the incorrect version of a file.py Noticed that our FileType enum called "bpmn" "bpmm", hope this doesn't screw someone up. Workflows are directly related to the data_models that create the workflow spec it needs. So the files should always be there. There are no more hashes, and thus no more hash errors where it can't find the files to rebuild the workflow.py Not much to report here, other than I broke every single test in the system at one point. So I'm super concerned about this, and will be testing it a lot before creating the pull request.
2020-05-29 00:03:50 +00:00
file_data = FileService.get_file_data(file_model.id)
self.assertEqual(2, file_data.version)
Removed all self-referential calls in the study_api. One api endpoint should never call another api endpoint. Moved the logic for updating a study to the study Model, rather than checking and setting dictionary values which will become very hard to maintain. The protocol builder service now returns real models, not dictionaries, forcing proper validation and fail-fast behavior. Changed the name of the "status" spec, to "top_level_workflow" and removing any connection a workflow or study has with this specification. It is only unused to determine status in real time, and is not reused or tracked. Modified the required documents script to return a dictionary and not an array, making it easier to speak to specific values in the BPMN and DMN. Working on new ways to test the top_level_workflow in the context of updates, this is still a work in progress. Making use of several modifications to the Spiff library that enables more complex expressions in DMN models. This is evident in the new DMN models for the top_level_workflow
2020-03-26 16:51:53 +00:00
Assure that the field type provided is supported. Catches errors such as adding a custom field type of 'text_area' rather than "textarea"
2021-09-29 18:05:45 +00:00
rv = self.app.get('/v1.0/file/%i/data' % file_json['id'], headers=self.logged_in_headers())
Improve version handling of files. Consolidate more of this logic in FileService. Place the version on the actual data model, not the file model, so the file model remains the same, and we just version the data associated with it.
2020-03-04 18:40:25 +00:00
self.assert_success(rv)
data = rv.get_data()
self.assertIsNotNone(data)
If you name add a file to a workflow that has the exact same name as a Task Spec's ID, and an extension of "md", it wll use that file as the markdown content, and ignore the markdown in the documentation on the task spec. Moving the primary process id from the workflow model to the file model, and assuring it is updated properly. This was causing a bug that would "lose" the workflow.
2020-04-17 17:30:32 +00:00
self.assertEqual(self.minimal_bpmn("efghijk"), data)
Improve version handling of files. Consolidate more of this logic in FileService. Place the version on the actual data model, not the file model, so the file model remains the same, and we just version the data associated with it.
2020-03-04 18:40:25 +00:00
def test_update_with_same_exact_data_does_not_increment_version(self):
self.load_example_data()
spec = session.query(WorkflowSpecModel).first()
data = {}
If you name add a file to a workflow that has the exact same name as a Task Spec's ID, and an extension of "md", it wll use that file as the markdown content, and ignore the markdown in the documentation on the task spec. Moving the primary process id from the workflow model to the file model, and assuring it is updated properly. This was causing a bug that would "lose" the workflow.
2020-04-17 17:30:32 +00:00
data['file'] = io.BytesIO(self.minimal_bpmn("abcdef")), 'my_new_file.bpmn'
Improve version handling of files. Consolidate more of this logic in FileService. Place the version on the actual data model, not the file model, so the file model remains the same, and we just version the data associated with it.
2020-03-04 18:40:25 +00:00
rv = self.app.post('/v1.0/file?workflow_spec_id=%s' % spec.id, data=data, follow_redirects=True,
It became impossible to use the Swagger ui when we started adding authentication to all of the calls. I discovered Connexion and Swagger have a default way of handing JTW authentication and this cleans up our code quite a bit, moves the securing of endpoints into the API Definition, which is quite nice, and removes a whole library dependency (I never get to do that!) I added a SWAGGER_AUTH_KEY that can be used in non-production environments to allow users to quickly authenticate from the Swagger ui. I also removed all api calls to simple little happy api services, because that is just mean and pointless.
2020-03-24 18:15:21 +00:00
content_type='multipart/form-data', headers=self.logged_in_headers())
Improve version handling of files. Consolidate more of this logic in FileService. Place the version on the actual data model, not the file model, so the file model remains the same, and we just version the data associated with it.
2020-03-04 18:40:25 +00:00
self.assertIsNotNone(rv.get_data())
json_data = json.loads(rv.get_data(as_text=True))
Ludicrously stupid launch in a refactor of the way all files work in the system at a time where I crave sleep and peace above all other things. Added a File class, that we wrap around the FileModel so the api endpoints don't change, but File no longer holds refences to versions or dates of the file_data model, we figure this out based on a clean database structure. The ApprovalFile is directly related to the file_data_model - so no chance that a reviewer would review the incorrect version of a file.py Noticed that our FileType enum called "bpmn" "bpmm", hope this doesn't screw someone up. Workflows are directly related to the data_models that create the workflow spec it needs. So the files should always be there. There are no more hashes, and thus no more hash errors where it can't find the files to rebuild the workflow.py Not much to report here, other than I broke every single test in the system at one point. So I'm super concerned about this, and will be testing it a lot before creating the pull request.
2020-05-29 00:03:50 +00:00
self.assertEqual(1, json_data['latest_version'])
If you name add a file to a workflow that has the exact same name as a Task Spec's ID, and an extension of "md", it wll use that file as the markdown content, and ignore the markdown in the documentation on the task spec. Moving the primary process id from the workflow model to the file model, and assuring it is updated properly. This was causing a bug that would "lose" the workflow.
2020-04-17 17:30:32 +00:00
data['file'] = io.BytesIO(self.minimal_bpmn("abcdef")), 'my_new_file.bpmn'
Ludicrously stupid launch in a refactor of the way all files work in the system at a time where I crave sleep and peace above all other things. Added a File class, that we wrap around the FileModel so the api endpoints don't change, but File no longer holds refences to versions or dates of the file_data model, we figure this out based on a clean database structure. The ApprovalFile is directly related to the file_data_model - so no chance that a reviewer would review the incorrect version of a file.py Noticed that our FileType enum called "bpmn" "bpmm", hope this doesn't screw someone up. Workflows are directly related to the data_models that create the workflow spec it needs. So the files should always be there. There are no more hashes, and thus no more hash errors where it can't find the files to rebuild the workflow.py Not much to report here, other than I broke every single test in the system at one point. So I'm super concerned about this, and will be testing it a lot before creating the pull request.
2020-05-29 00:03:50 +00:00
rv = self.app.put('/v1.0/file/%i/data' % json_data['id'], data=data, follow_redirects=True,
It became impossible to use the Swagger ui when we started adding authentication to all of the calls. I discovered Connexion and Swagger have a default way of handing JTW authentication and this cleans up our code quite a bit, moves the securing of endpoints into the API Definition, which is quite nice, and removes a whole library dependency (I never get to do that!) I added a SWAGGER_AUTH_KEY that can be used in non-production environments to allow users to quickly authenticate from the Swagger ui. I also removed all api calls to simple little happy api services, because that is just mean and pointless.
2020-03-24 18:15:21 +00:00
content_type='multipart/form-data', headers=self.logged_in_headers())
Improve version handling of files. Consolidate more of this logic in FileService. Place the version on the actual data model, not the file model, so the file model remains the same, and we just version the data associated with it.
2020-03-04 18:40:25 +00:00
self.assertIsNotNone(rv.get_data())
json_data = json.loads(rv.get_data(as_text=True))
Ludicrously stupid launch in a refactor of the way all files work in the system at a time where I crave sleep and peace above all other things. Added a File class, that we wrap around the FileModel so the api endpoints don't change, but File no longer holds refences to versions or dates of the file_data model, we figure this out based on a clean database structure. The ApprovalFile is directly related to the file_data_model - so no chance that a reviewer would review the incorrect version of a file.py Noticed that our FileType enum called "bpmn" "bpmm", hope this doesn't screw someone up. Workflows are directly related to the data_models that create the workflow spec it needs. So the files should always be there. There are no more hashes, and thus no more hash errors where it can't find the files to rebuild the workflow.py Not much to report here, other than I broke every single test in the system at one point. So I'm super concerned about this, and will be testing it a lot before creating the pull request.
2020-05-29 00:03:50 +00:00
self.assertEqual(1, json_data['latest_version'])
Improve version handling of files. Consolidate more of this logic in FileService. Place the version on the actual data model, not the file model, so the file model remains the same, and we just version the data associated with it.
2020-03-04 18:40:25 +00:00
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
def test_get_file(self):
self.load_example_data()
Adds type hinting to db and session to fix Pycharm autocomplete. Replaces db.session with session.
2020-01-14 16:45:12 +00:00
spec = session.query(WorkflowSpecModel).first()
file = session.query(FileModel).filter_by(workflow_spec_id=spec.id).first()
It became impossible to use the Swagger ui when we started adding authentication to all of the calls. I discovered Connexion and Swagger have a default way of handing JTW authentication and this cleans up our code quite a bit, moves the securing of endpoints into the API Definition, which is quite nice, and removes a whole library dependency (I never get to do that!) I added a SWAGGER_AUTH_KEY that can be used in non-production environments to allow users to quickly authenticate from the Swagger ui. I also removed all api calls to simple little happy api services, because that is just mean and pointless.
2020-03-24 18:15:21 +00:00
rv = self.app.get('/v1.0/file/%i/data' % file.id, headers=self.logged_in_headers())
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
self.assert_success(rv)
Merge branch 'feature/update-task-data' of github.com:sartography/cr-connect-workflow into feature/update-task-data
2020-01-23 20:35:51 +00:00
self.assertEqual("text/xml; charset=utf-8", rv.content_type)
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
self.assertTrue(rv.content_length > 1)
Prefer tasks that share a parent over just the the next available task when returning the next_task in the workflow processor.
2021-06-08 12:03:14 +00:00
def test_get_file_contains_data_store_elements(self):
self.load_example_data()
spec = session.query(WorkflowSpecModel).first()
file = session.query(FileModel).filter_by(workflow_spec_id=spec.id).first()
ds = DataStoreModel(key="my_key", value="my_value", file_id=file.id);
db.session.add(ds)
rv = self.app.get('/v1.0/file/%i' % file.id, headers=self.logged_in_headers())
self.assert_success(rv)
json_data = json.loads(rv.get_data(as_text=True))
self.assertEqual("my_value", json_data['data_store']['my_key'])
Fixes #306 assure that we take the doc_code into account if passed in as an argument to the API for get_files.
2021-04-23 20:14:23 +00:00
def test_get_files_for_form_field_returns_only_those_files(self):
self.create_reference_document()
workflow = self.create_workflow('file_upload_form')
processor = WorkflowProcessor(workflow)
processor.do_engine_steps()
task = processor.next_task()
correct_name = task.task_spec.form.fields[0].id
data = {'file': (io.BytesIO(b"abcdef"), 'random_fact.svg')}
Add task_spec_name to a bunch of tests
2021-08-26 14:43:28 +00:00
rv = self.app.post('/v1.0/file?study_id=%i&workflow_id=%s&task_spec_name=%s&form_field_key=%s' %
Assure that the field type provided is supported. Catches errors such as adding a custom field type of 'text_area' rather than "textarea"
2021-09-29 18:05:45 +00:00
(workflow.study_id, workflow.id, task.get_name(), correct_name), data=data,
follow_redirects=True,
Fixes #306 assure that we take the doc_code into account if passed in as an argument to the API for get_files.
2021-04-23 20:14:23 +00:00
content_type='multipart/form-data', headers=self.logged_in_headers())
self.assert_success(rv)
# Note: this call can be made WITHOUT the task id.
rv = self.app.get('/v1.0/file?study_id=%i&workflow_id=%s&form_field_key=%s' %
(workflow.study_id, workflow.id, correct_name), follow_redirects=True,
content_type='multipart/form-data', headers=self.logged_in_headers())
self.assert_success(rv)
json_data = json.loads(rv.get_data(as_text=True))
self.assertEqual(len(json_data), 1)
# Add another file for a different document type
Add task_spec_name to a bunch of tests
2021-08-26 14:43:28 +00:00
FileService().add_workflow_file(workflow.id, 'Study_App_Doc', task.get_name(), 'otherdoc.docx',
Fixes #306 assure that we take the doc_code into account if passed in as an argument to the API for get_files.
2021-04-23 20:14:23 +00:00
'application/xcode', b"asdfasdf")
Add task_spec_name to a bunch of tests
2021-08-26 14:43:28 +00:00
# Note: this call can be made WITHOUT the task spec name.
Fixes #306 assure that we take the doc_code into account if passed in as an argument to the API for get_files.
2021-04-23 20:14:23 +00:00
rv = self.app.get('/v1.0/file?study_id=%i&workflow_id=%s&form_field_key=%s' %
(workflow.study_id, workflow.id, correct_name), follow_redirects=True,
content_type='multipart/form-data', headers=self.logged_in_headers())
self.assert_success(rv)
json_data = json.loads(rv.get_data(as_text=True))
self.assertEqual(len(json_data), 1)
Fix the panda warnings, which means we only support xlsx files now. Include the document details when reutnring a file from the file model. Improve the validation of file form elements.
2021-06-22 18:58:52 +00:00
def test_add_file_returns_document_metadata(self):
self.create_reference_document()
workflow = self.create_workflow('file_upload_form_single')
processor = WorkflowProcessor(workflow)
processor.do_engine_steps()
task = processor.next_task()
correct_name = task.task_spec.form.fields[0].id
data = {'file': (io.BytesIO(b"abcdef"), 'random_fact.svg')}
Add task_spec_name to a bunch of tests
2021-08-26 14:43:28 +00:00
rv = self.app.post('/v1.0/file?study_id=%i&workflow_id=%s&task_spec_name=%s&form_field_key=%s' %
Assure that the field type provided is supported. Catches errors such as adding a custom field type of 'text_area' rather than "textarea"
2021-09-29 18:05:45 +00:00
(workflow.study_id, workflow.id, task.get_name(), correct_name), data=data,
follow_redirects=True,
Fix the panda warnings, which means we only support xlsx files now. Include the document details when reutnring a file from the file model. Improve the validation of file form elements.
2021-06-22 18:58:52 +00:00
content_type='multipart/form-data', headers=self.logged_in_headers())
self.assert_success(rv)
json_data = json.loads(rv.get_data(as_text=True))
self.assertEqual('Ancillary Document', json_data['document']['category1'])
Fixed tests that broke from the new irb_documents spreadsheet
2021-08-26 12:51:52 +00:00
self.assertEqual('Study Team', json_data['document']['who_uploads?'])
Fixes #306 assure that we take the doc_code into account if passed in as an argument to the API for get_files.
2021-04-23 20:14:23 +00:00
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
def test_delete_file(self):
self.load_example_data()
Adds type hinting to db and session to fix Pycharm autocomplete. Replaces db.session with session.
2020-01-14 16:45:12 +00:00
spec = session.query(WorkflowSpecModel).first()
file = session.query(FileModel).filter_by(workflow_spec_id=spec.id).first()
Validation now assures that we won't encounter errors when generating navigation lists. Fixing workflow sync paths that were incorrect. Repairing a suddenly failing test in files, that just don't make no sense. Bumping spiffworkflow that contains a fix for issue #155
2020-12-28 22:33:38 +00:00
file_id = file.id
It became impossible to use the Swagger ui when we started adding authentication to all of the calls. I discovered Connexion and Swagger have a default way of handing JTW authentication and this cleans up our code quite a bit, moves the securing of endpoints into the API Definition, which is quite nice, and removes a whole library dependency (I never get to do that!) I added a SWAGGER_AUTH_KEY that can be used in non-production environments to allow users to quickly authenticate from the Swagger ui. I also removed all api calls to simple little happy api services, because that is just mean and pointless.
2020-03-24 18:15:21 +00:00
rv = self.app.get('/v1.0/file/%i' % file.id, headers=self.logged_in_headers())
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
self.assert_success(rv)
It became impossible to use the Swagger ui when we started adding authentication to all of the calls. I discovered Connexion and Swagger have a default way of handing JTW authentication and this cleans up our code quite a bit, moves the securing of endpoints into the API Definition, which is quite nice, and removes a whole library dependency (I never get to do that!) I added a SWAGGER_AUTH_KEY that can be used in non-production environments to allow users to quickly authenticate from the Swagger ui. I also removed all api calls to simple little happy api services, because that is just mean and pointless.
2020-03-24 18:15:21 +00:00
rv = self.app.delete('/v1.0/file/%i' % file.id, headers=self.logged_in_headers())
Validation now assures that we won't encounter errors when generating navigation lists. Fixing workflow sync paths that were incorrect. Repairing a suddenly failing test in files, that just don't make no sense. Bumping spiffworkflow that contains a fix for issue #155
2020-12-28 22:33:38 +00:00
db.session.flush()
rv = self.app.get('/v1.0/file/%i' % file_id, headers=self.logged_in_headers())
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
self.assertEqual(404, rv.status_code)
Removed all self-referential calls in the study_api. One api endpoint should never call another api endpoint. Moved the logic for updating a study to the study Model, rather than checking and setting dictionary values which will become very hard to maintain. The protocol builder service now returns real models, not dictionaries, forcing proper validation and fail-fast behavior. Changed the name of the "status" spec, to "top_level_workflow" and removing any connection a workflow or study has with this specification. It is only unused to determine status in real time, and is not reused or tracked. Modified the required documents script to return a dictionary and not an array, making it easier to speak to specific values in the BPMN and DMN. Working on new ways to test the top_level_workflow in the context of updates, this is still a work in progress. Making use of several modifications to the Spiff library that enables more complex expressions in DMN models. This is evident in the new DMN models for the top_level_workflow
2020-03-26 16:51:53 +00:00
If you name add a file to a workflow that has the exact same name as a Task Spec's ID, and an extension of "md", it wll use that file as the markdown content, and ignore the markdown in the documentation on the task spec. Moving the primary process id from the workflow model to the file model, and assuring it is updated properly. This was causing a bug that would "lose" the workflow.
2020-04-17 17:30:32 +00:00
def test_change_primary_bpmn(self):
self.load_example_data()
spec = session.query(WorkflowSpecModel).first()
data = {}
data['file'] = io.BytesIO(self.minimal_bpmn("abcdef")), 'my_new_file.bpmn'
# Add a new BPMN file to the specification
rv = self.app.post('/v1.0/file?workflow_spec_id=%s' % spec.id, data=data, follow_redirects=True,
content_type='multipart/form-data', headers=self.logged_in_headers())
self.assert_success(rv)
self.assertIsNotNone(rv.get_data())
json_data = json.loads(rv.get_data(as_text=True))
file = FileModelSchema().load(json_data, session=session)
# Delete the primary BPMN file for the workflow.
Assure that the field type provided is supported. Catches errors such as adding a custom field type of 'text_area' rather than "textarea"
2021-09-29 18:05:45 +00:00
orig_model = session.query(FileModel). \
filter(FileModel.primary == True). \
If you name add a file to a workflow that has the exact same name as a Task Spec's ID, and an extension of "md", it wll use that file as the markdown content, and ignore the markdown in the documentation on the task spec. Moving the primary process id from the workflow model to the file model, and assuring it is updated properly. This was causing a bug that would "lose" the workflow.
2020-04-17 17:30:32 +00:00
filter(FileModel.workflow_spec_id == spec.id).first()
rv = self.app.delete('/v1.0/file?file_id=%s' % orig_model.id, headers=self.logged_in_headers())
# Set that new file to be the primary BPMN, assure it has a primary_process_id
file.primary = True
rv = self.app.put('/v1.0/file/%i' % file.id,
Assure that the field type provided is supported. Catches errors such as adding a custom field type of 'text_area' rather than "textarea"
2021-09-29 18:05:45 +00:00
content_type="application/json",
data=json.dumps(FileModelSchema().dump(file)), headers=self.logged_in_headers())
If you name add a file to a workflow that has the exact same name as a Task Spec's ID, and an extension of "md", it wll use that file as the markdown content, and ignore the markdown in the documentation on the task spec. Moving the primary process id from the workflow model to the file model, and assuring it is updated properly. This was causing a bug that would "lose" the workflow.
2020-04-17 17:30:32 +00:00
self.assert_success(rv)
json_data = json.loads(rv.get_data(as_text=True))
self.assertTrue(json_data['primary'])
self.assertIsNotNone(json_data['primary_process_id'])