sartography
/
cr-connect-workflow
mirror of https://github.com/sartography/cr-connect-workflow.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
cr-connect-workflow/tests/files/test_files_api.py

218 lines
11 KiB
Python
Raw Normal View History

Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
import io
import json
Refactoring Reference files to use the lookup table, rather than parsing the results directly out of the spreadsheet, or attempting to cache them. Adding a DocumentService to clean up the FileService, and get Documents well seperated, as it seems likely be pulled out or seperated in the future, there is now a Documents api file as well, for the same reason. Some other minor changes are just fixing white space to assure our code is linting correctly. I removed _create_study_workflow_approvals from the base test, as we don't use approvals like this anymore.
2021-07-06 17:10:20 +00:00
import os
A major refactor of how we search and store files, as there was a lot of confusing bits in here. From an API point of view you can do the following (and only the following) /files?workflow_spec_id=x * You can find all files associated with a workflow_spec_id, and add a file with a workflow_spec_id /files?workflow_id=x * You can find all files associated with a workflow_id, and add a file that is directly associated with the workflow /files?workflow_id=x&form_field_key=y * You can find all files associated with a form element on a running workflow, and add a new file. Note: you can add multiple files to the same form_field_key, IF they have different file names. If the same name, the original file is archived, and the new file takes its place. The study endpoints always return a list of the file metadata associated with the study. Removed /studies-files, but there is an endpoint called /studies/all - that returns all the studies in the system, and does include their files. On a deeper level: The File model no longer contains: - study_id, - task_id, - form_field_key Instead, if the file is associated with workflow - then that is the one way it is connected to the study, and we use this relationship to find files for a study. A file is never associated with a task_id, as these change when the workflow is reloaded. The form_field_key must match the irb_doc_code, so when requesting files for a form field, we just look up the irb_doc_code.
2020-05-28 12:27:26 +00:00
from tests.base_test import BaseTest
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
Refactoring Reference files to use the lookup table, rather than parsing the results directly out of the spreadsheet, or attempting to cache them. Adding a DocumentService to clean up the FileService, and get Documents well seperated, as it seems likely be pulled out or seperated in the future, there is now a Documents api file as well, for the same reason. Some other minor changes are just fixing white space to assure our code is linting correctly. I removed _create_study_workflow_approvals from the base test, as we don't use approvals like this anymore.
2021-07-06 17:10:20 +00:00
from crc import session, db, app
1. Created a UserFileService, so it's clear what we use when for File Services, UserFiles, SpecFiles, and ReferenceFiles each function differently. 2. Reference Files and Spec Files are written to disk, they do not exist in the database at all.
2022-02-02 17:59:56 +00:00
from crc.models.file import FileModel, FileType, FileModelSchema, FileSchema
*** WIP *** Failing tests, and missing functionality. Committing to get stuff on Github.
2022-01-11 20:30:22 +00:00
from crc.services.spec_file_service import SpecFileService
Assure that files uploaded through web forms and files generated from templates can be cross-referenced to known document requirements from the protocol builder. Configurators can control this by managing an XLS Spreadsheet called "irb_documents.xslx". Required Documents is becoming complicated, so making this it's own script task, removing it from study_info.py The file_service is now very aware of this irb_documents file, so it will always need to exist. We seed this file during setup, but it can be overwritten by the configurator.
2020-03-19 21:13:30 +00:00
from crc.services.workflow_processor import WorkflowProcessor
Prefer tasks that share a parent over just the the next available task when returning the next_task in the workflow processor.
2021-06-08 12:03:14 +00:00
from crc.models.data_store import DataStoreModel
Refactoring Reference files to use the lookup table, rather than parsing the results directly out of the spreadsheet, or attempting to cache them. Adding a DocumentService to clean up the FileService, and get Documents well seperated, as it seems likely be pulled out or seperated in the future, there is now a Documents api file as well, for the same reason. Some other minor changes are just fixing white space to assure our code is linting correctly. I removed _create_study_workflow_approvals from the base test, as we don't use approvals like this anymore.
2021-07-06 17:10:20 +00:00
from crc.services.document_service import DocumentService
Adding a new reference file that provides greater details about the investigators related to a study. Improving the study_info script documentation to provide detailed examples of values returned based on arguments. Making the tests a little more targetted and less subject to breaking through better mocks. Allow all tests to pass even when ther protocol builder mock isn't running locally. Removing the duplication of reference files in tests and static, as this seems silly to me at the moment.
2020-05-07 17:57:24 +00:00
from example_data import ExampleDataLoader
1. Created a UserFileService, so it's clear what we use when for File Services, UserFiles, SpecFiles, and ReferenceFiles each function differently. 2. Reference Files and Spec Files are written to disk, they do not exist in the database at all.
2022-02-02 17:59:56 +00:00
from crc.services.user_file_service import UserFileService
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
*** WIP *** code for updating spec files some cleanup tests pass, but we don't have all the tests we need.
2022-01-13 20:24:29 +00:00
from sqlalchemy import column
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
Protocol builder connections
2020-02-20 18:30:04 +00:00
class TestFilesApi(BaseTest):
Adding file management. Files are stored in the database along with basic meta data.
2019-12-27 18:50:03 +00:00
Assure that files uploaded through web forms and files generated from templates can be cross-referenced to known document requirements from the protocol builder. Configurators can control this by managing an XLS Spreadsheet called "irb_documents.xslx". Required Documents is becoming complicated, so making this it's own script task, removing it from study_info.py The file_service is now very aware of this irb_documents file, so it will always need to exist. We seed this file during setup, but it can be overwritten by the configurator.
2020-03-19 21:13:30 +00:00
def test_add_file_from_task_and_form_errors_on_invalid_form_field_name(self):
self.create_reference_document()
workflow = self.create_workflow('file_upload_form')
processor = WorkflowProcessor(workflow)
Allow the workflow to be requested without making changes to the workflow - requires that you specify a read_only flag of true, otherwise it assumes that you want a fully prepared workflow with the next ready task set to run.
2020-07-28 17:33:38 +00:00
processor.do_engine_steps()
Assure that files uploaded through web forms and files generated from templates can be cross-referenced to known document requirements from the protocol builder. Configurators can control this by managing an XLS Spreadsheet called "irb_documents.xslx". Required Documents is becoming complicated, so making this it's own script task, removing it from study_info.py The file_service is now very aware of this irb_documents file, so it will always need to exist. We seed this file during setup, but it can be overwritten by the configurator.
2020-03-19 21:13:30 +00:00
task = processor.next_task()
data = {'file': (io.BytesIO(b"abcdef"), 'random_fact.svg')}
correct_name = task.task_spec.form.fields[0].id
data = {'file': (io.BytesIO(b"abcdef"), 'random_fact.svg')}
Add task_spec_name to a bunch of tests
2021-08-26 14:43:28 +00:00
rv = self.app.post('/v1.0/file?study_id=%i&workflow_id=%s&task_spec_name=%s&form_field_key=%s' %
Assure that the field type provided is supported. Catches errors such as adding a custom field type of 'text_area' rather than "textarea"
2021-09-29 18:05:45 +00:00
(workflow.study_id, workflow.id, task.get_name(), correct_name), data=data,
follow_redirects=True,
It became impossible to use the Swagger ui when we started adding authentication to all of the calls. I discovered Connexion and Swagger have a default way of handing JTW authentication and this cleans up our code quite a bit, moves the securing of endpoints into the API Definition, which is quite nice, and removes a whole library dependency (I never get to do that!) I added a SWAGGER_AUTH_KEY that can be used in non-production environments to allow users to quickly authenticate from the Swagger ui. I also removed all api calls to simple little happy api services, because that is just mean and pointless.
2020-03-24 18:15:21 +00:00
content_type='multipart/form-data', headers=self.logged_in_headers())
Assure that files uploaded through web forms and files generated from templates can be cross-referenced to known document requirements from the protocol builder. Configurators can control this by managing an XLS Spreadsheet called "irb_documents.xslx". Required Documents is becoming complicated, so making this it's own script task, removing it from study_info.py The file_service is now very aware of this irb_documents file, so it will always need to exist. We seed this file during setup, but it can be overwritten by the configurator.
2020-03-19 21:13:30 +00:00
self.assert_success(rv)
*** WIP *** Failing tests, and missing functionality. Committing to get stuff on Github.
2022-01-11 20:30:22 +00:00
def test_update_reference_file_data(self):
self.load_example_data()
Fix the test
2021-07-07 16:54:16 +00:00
file_name = "documents.xlsx"
1. Created a UserFileService, so it's clear what we use when for File Services, UserFiles, SpecFiles, and ReferenceFiles each function differently. 2. Reference Files and Spec Files are written to disk, they do not exist in the database at all.
2022-02-02 17:59:56 +00:00
filepath = os.path.join(app.root_path, 'static', 'reference', 'documents.xlsx')
Refactoring Reference files to use the lookup table, rather than parsing the results directly out of the spreadsheet, or attempting to cache them. Adding a DocumentService to clean up the FileService, and get Documents well seperated, as it seems likely be pulled out or seperated in the future, there is now a Documents api file as well, for the same reason. Some other minor changes are just fixing white space to assure our code is linting correctly. I removed _create_study_workflow_approvals from the base test, as we don't use approvals like this anymore.
2021-07-06 17:10:20 +00:00
with open(filepath, 'rb') as myfile:
file_data = myfile.read()
data = {'file': (io.BytesIO(file_data), file_name)}
Cleaned up code around differences between file info and file data Cleaned up some api code around differences between file, spec_file, and reference_file Cleaned up some api code around differences between file info and file data Fixed some tests for file api
2022-01-19 18:47:14 +00:00
rv = self.app.put('/v1.0/reference_file/%s/data' % file_name, data=data, follow_redirects=True,
It became impossible to use the Swagger ui when we started adding authentication to all of the calls. I discovered Connexion and Swagger have a default way of handing JTW authentication and this cleans up our code quite a bit, moves the securing of endpoints into the API Definition, which is quite nice, and removes a whole library dependency (I never get to do that!) I added a SWAGGER_AUTH_KEY that can be used in non-production environments to allow users to quickly authenticate from the Swagger ui. I also removed all api calls to simple little happy api services, because that is just mean and pointless.
2020-03-24 18:15:21 +00:00
content_type='multipart/form-data', headers=self.logged_in_headers())
Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks.
2020-03-13 19:03:57 +00:00
self.assert_success(rv)
self.assertIsNotNone(rv.get_data())
json_data = json.loads(rv.get_data(as_text=True))
file = FileModelSchema().load(json_data, session=session)
Refactoring Reference files to use the lookup table, rather than parsing the results directly out of the spreadsheet, or attempting to cache them. Adding a DocumentService to clean up the FileService, and get Documents well seperated, as it seems likely be pulled out or seperated in the future, there is now a Documents api file as well, for the same reason. Some other minor changes are just fixing white space to assure our code is linting correctly. I removed _create_study_workflow_approvals from the base test, as we don't use approvals like this anymore.
2021-07-06 17:10:20 +00:00
self.assertEqual(FileType.xlsx, file.type)
self.assertEqual("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", file.content_type)
*** WIP *** Failing tests, and missing functionality. Committing to get stuff on Github.
2022-01-11 20:30:22 +00:00
# self.assertEqual('dhf8r', json_data['user_uid'])
Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks.
2020-03-13 19:03:57 +00:00
def test_set_reference_file_bad_extension(self):
Refactoring Reference files to use the lookup table, rather than parsing the results directly out of the spreadsheet, or attempting to cache them. Adding a DocumentService to clean up the FileService, and get Documents well seperated, as it seems likely be pulled out or seperated in the future, there is now a Documents api file as well, for the same reason. Some other minor changes are just fixing white space to assure our code is linting correctly. I removed _create_study_workflow_approvals from the base test, as we don't use approvals like this anymore.
2021-07-06 17:10:20 +00:00
file_name = DocumentService.DOCUMENT_LIST
Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks.
2020-03-13 19:03:57 +00:00
data = {'file': (io.BytesIO(b"abcdef"), "does_not_matter.ppt")}
Cleaned up code around differences between file info and file data Cleaned up some api code around differences between file, spec_file, and reference_file Cleaned up some api code around differences between file info and file data Fixed some tests for file api
2022-01-19 18:47:14 +00:00
rv = self.app.put('/v1.0/reference_file/%s/data' % file_name, data=data, follow_redirects=True,
It became impossible to use the Swagger ui when we started adding authentication to all of the calls. I discovered Connexion and Swagger have a default way of handing JTW authentication and this cleans up our code quite a bit, moves the securing of endpoints into the API Definition, which is quite nice, and removes a whole library dependency (I never get to do that!) I added a SWAGGER_AUTH_KEY that can be used in non-production environments to allow users to quickly authenticate from the Swagger ui. I also removed all api calls to simple little happy api services, because that is just mean and pointless.
2020-03-24 18:15:21 +00:00
content_type='multipart/form-data', headers=self.logged_in_headers())
Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks.
2020-03-13 19:03:57 +00:00
self.assert_failure(rv, error_code="invalid_file_type")
Fixed get reference file info, and added test
2022-01-19 20:17:35 +00:00
def test_get_reference_file_data(self):
*** WIP *** Failing tests, and missing functionality. Committing to get stuff on Github.
2022-01-11 20:30:22 +00:00
ExampleDataLoader().load_reference_documents()
Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks.
2020-03-13 19:03:57 +00:00
file_name = "irb_document_types.xls"
1. Created a UserFileService, so it's clear what we use when for File Services, UserFiles, SpecFiles, and ReferenceFiles each function differently. 2. Reference Files and Spec Files are written to disk, they do not exist in the database at all.
2022-02-02 17:59:56 +00:00
filepath = os.path.join(app.root_path, 'static', 'reference', 'documents.xlsx')
Cleaned up code around differences between file info and file data Cleaned up some api code around differences between file, spec_file, and reference_file Cleaned up some api code around differences between file info and file data Fixed some tests for file api
2022-01-19 18:47:14 +00:00
with open(filepath, 'rb') as f_open:
file_data = f_open.read()
Refactoring Reference files to use the lookup table, rather than parsing the results directly out of the spreadsheet, or attempting to cache them. Adding a DocumentService to clean up the FileService, and get Documents well seperated, as it seems likely be pulled out or seperated in the future, there is now a Documents api file as well, for the same reason. Some other minor changes are just fixing white space to assure our code is linting correctly. I removed _create_study_workflow_approvals from the base test, as we don't use approvals like this anymore.
2021-07-06 17:10:20 +00:00
data = {'file': (io.BytesIO(file_data), file_name)}
Cleaned up code around differences between file info and file data Cleaned up some api code around differences between file, spec_file, and reference_file Cleaned up some api code around differences between file info and file data Fixed some tests for file api
2022-01-19 18:47:14 +00:00
self.app.post('/v1.0/reference_file', data=data, follow_redirects=True,
content_type='multipart/form-data', headers=self.logged_in_headers())
rv = self.app.get('/v1.0/reference_file/%s/data' % file_name, headers=self.logged_in_headers())
Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks.
2020-03-13 19:03:57 +00:00
self.assert_success(rv)
data_out = rv.get_data()
Refactoring Reference files to use the lookup table, rather than parsing the results directly out of the spreadsheet, or attempting to cache them. Adding a DocumentService to clean up the FileService, and get Documents well seperated, as it seems likely be pulled out or seperated in the future, there is now a Documents api file as well, for the same reason. Some other minor changes are just fixing white space to assure our code is linting correctly. I removed _create_study_workflow_approvals from the base test, as we don't use approvals like this anymore.
2021-07-06 17:10:20 +00:00
self.assertEqual(file_data, data_out)
Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks.
2020-03-13 19:03:57 +00:00
Fixed get reference file info, and added test
2022-01-19 20:17:35 +00:00
def test_get_reference_file_info(self):
self.load_example_data()
1. Created a UserFileService, so it's clear what we use when for File Services, UserFiles, SpecFiles, and ReferenceFiles each function differently. 2. Reference Files and Spec Files are written to disk, they do not exist in the database at all.
2022-02-02 17:59:56 +00:00
rv = self.app.get('/v1.0/reference_file/documents.xlsx', headers=self.logged_in_headers())
Fixed get reference file info, and added test
2022-01-19 20:17:35 +00:00
self.assert_success(rv)
self.assertIsNotNone(rv.get_data())
json_data = json.loads(rv.get_data(as_text=True))
1. Created a UserFileService, so it's clear what we use when for File Services, UserFiles, SpecFiles, and ReferenceFiles each function differently. 2. Reference Files and Spec Files are written to disk, they do not exist in the database at all.
2022-02-02 17:59:56 +00:00
self.assertEqual("documents.xlsx", json_data['name'])
self.assertEqual("xlsx", json_data['type'])
Fixed get reference file info, and added test
2022-01-19 20:17:35 +00:00
Adds a POST for reference file. and a cool test
2021-10-12 18:29:15 +00:00
def test_add_reference_file(self):
ExampleDataLoader().load_reference_documents()
file_name = 'new.xlsx'
data = {'file': (io.BytesIO(b"abcdef"), file_name)}
rv = self.app.post('/v1.0/reference_file', data=data,
follow_redirects=True,
content_type='multipart/form-data', headers=self.logged_in_headers())
self.assertIsNotNone(rv.get_data())
json_data = json.loads(rv.get_data(as_text=True))
file = FileModelSchema().load(json_data, session=session)
self.assertEqual(FileType.xlsx, file.type)
Cleaned up code around differences between file info and file data Cleaned up some api code around differences between file, spec_file, and reference_file Cleaned up some api code around differences between file info and file data Fixed some tests for file api
2022-01-19 18:47:14 +00:00
def test_delete_reference_file(self):
ExampleDataLoader().load_reference_documents()
1. Created a UserFileService, so it's clear what we use when for File Services, UserFiles, SpecFiles, and ReferenceFiles each function differently. 2. Reference Files and Spec Files are written to disk, they do not exist in the database at all.
2022-02-02 17:59:56 +00:00
name = "documents.xlsx"
rv = self.app.get('/v1.0/reference_file/%s' % name, headers=self.logged_in_headers())
Cleaned up code around differences between file info and file data Cleaned up some api code around differences between file, spec_file, and reference_file Cleaned up some api code around differences between file info and file data Fixed some tests for file api
2022-01-19 18:47:14 +00:00
self.assert_success(rv)
1. Created a UserFileService, so it's clear what we use when for File Services, UserFiles, SpecFiles, and ReferenceFiles each function differently. 2. Reference Files and Spec Files are written to disk, they do not exist in the database at all.
2022-02-02 17:59:56 +00:00
self.app.delete('/v1.0/reference_file/%s' % name, headers=self.logged_in_headers())
Cleaned up code around differences between file info and file data Cleaned up some api code around differences between file, spec_file, and reference_file Cleaned up some api code around differences between file info and file data Fixed some tests for file api
2022-01-19 18:47:14 +00:00
db.session.flush()
1. Created a UserFileService, so it's clear what we use when for File Services, UserFiles, SpecFiles, and ReferenceFiles each function differently. 2. Reference Files and Spec Files are written to disk, they do not exist in the database at all.
2022-02-02 17:59:56 +00:00
rv = self.app.get('/v1.0/reference_file/%s' % name, headers=self.logged_in_headers())
Fixed get reference file info, and added test
2022-01-19 20:17:35 +00:00
self.assertEqual(404, rv.status_code)
self.assertIsNotNone(rv.get_data())
json_data = json.loads(rv.get_data(as_text=True))
1. Created a UserFileService, so it's clear what we use when for File Services, UserFiles, SpecFiles, and ReferenceFiles each function differently. 2. Reference Files and Spec Files are written to disk, they do not exist in the database at all.
2022-02-02 17:59:56 +00:00
self.assertIn('No reference file found with the name documents.xlsx', json_data['message'])
Cleaned up code around differences between file info and file data Cleaned up some api code around differences between file, spec_file, and reference_file Cleaned up some api code around differences between file info and file data Fixed some tests for file api
2022-01-19 18:47:14 +00:00
Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks.
2020-03-13 19:03:57 +00:00
def test_list_reference_files(self):
Adding a new reference file that provides greater details about the investigators related to a study. Improving the study_info script documentation to provide detailed examples of values returned based on arguments. Making the tests a little more targetted and less subject to breaking through better mocks. Allow all tests to pass even when ther protocol builder mock isn't running locally. Removing the duplication of reference files in tests and static, as this seems silly to me at the moment.
2020-05-07 17:57:24 +00:00
ExampleDataLoader.clean_db()
Refactoring Reference files to use the lookup table, rather than parsing the results directly out of the spreadsheet, or attempting to cache them. Adding a DocumentService to clean up the FileService, and get Documents well seperated, as it seems likely be pulled out or seperated in the future, there is now a Documents api file as well, for the same reason. Some other minor changes are just fixing white space to assure our code is linting correctly. I removed _create_study_workflow_approvals from the base test, as we don't use approvals like this anymore.
2021-07-06 17:10:20 +00:00
file_name = DocumentService.DOCUMENT_LIST
1. Created a UserFileService, so it's clear what we use when for File Services, UserFiles, SpecFiles, and ReferenceFiles each function differently. 2. Reference Files and Spec Files are written to disk, they do not exist in the database at all.
2022-02-02 17:59:56 +00:00
filepath = os.path.join(app.root_path, 'static', 'reference', 'documents.xlsx')
Refactoring Reference files to use the lookup table, rather than parsing the results directly out of the spreadsheet, or attempting to cache them. Adding a DocumentService to clean up the FileService, and get Documents well seperated, as it seems likely be pulled out or seperated in the future, there is now a Documents api file as well, for the same reason. Some other minor changes are just fixing white space to assure our code is linting correctly. I removed _create_study_workflow_approvals from the base test, as we don't use approvals like this anymore.
2021-07-06 17:10:20 +00:00
with open(filepath, 'rb') as myfile:
file_data = myfile.read()
data = {'file': (io.BytesIO(file_data), file_name)}
*** WIP *** Failing tests, and missing functionality. Committing to get stuff on Github.
2022-01-11 20:30:22 +00:00
rv = self.app.post('/v1.0/reference_file', data=data, follow_redirects=True,
It became impossible to use the Swagger ui when we started adding authentication to all of the calls. I discovered Connexion and Swagger have a default way of handing JTW authentication and this cleans up our code quite a bit, moves the securing of endpoints into the API Definition, which is quite nice, and removes a whole library dependency (I never get to do that!) I added a SWAGGER_AUTH_KEY that can be used in non-production environments to allow users to quickly authenticate from the Swagger ui. I also removed all api calls to simple little happy api services, because that is just mean and pointless.
2020-03-24 18:15:21 +00:00
content_type='multipart/form-data', headers=self.logged_in_headers())
Refactoring Reference files to use the lookup table, rather than parsing the results directly out of the spreadsheet, or attempting to cache them. Adding a DocumentService to clean up the FileService, and get Documents well seperated, as it seems likely be pulled out or seperated in the future, there is now a Documents api file as well, for the same reason. Some other minor changes are just fixing white space to assure our code is linting correctly. I removed _create_study_workflow_approvals from the base test, as we don't use approvals like this anymore.
2021-07-06 17:10:20 +00:00
self.assert_success(rv)
Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks.
2020-03-13 19:03:57 +00:00
rv = self.app.get('/v1.0/reference_file',
follow_redirects=True,
It became impossible to use the Swagger ui when we started adding authentication to all of the calls. I discovered Connexion and Swagger have a default way of handing JTW authentication and this cleans up our code quite a bit, moves the securing of endpoints into the API Definition, which is quite nice, and removes a whole library dependency (I never get to do that!) I added a SWAGGER_AUTH_KEY that can be used in non-production environments to allow users to quickly authenticate from the Swagger ui. I also removed all api calls to simple little happy api services, because that is just mean and pointless.
2020-03-24 18:15:21 +00:00
content_type="application/json", headers=self.logged_in_headers())
Add the ability to upload and request general reference files by name. These will be used across workflows and will frequently contain lookup tables that can be referenced by various script tasks.
2020-03-13 19:03:57 +00:00
self.assert_success(rv)
json_data = json.loads(rv.get_data(as_text=True))
self.assertEqual(1, len(json_data))
file = FileModelSchema(many=True).load(json_data, session=session)
self.assertEqual(file_name, file[0].name)
1. Created a UserFileService, so it's clear what we use when for File Services, UserFiles, SpecFiles, and ReferenceFiles each function differently. 2. Reference Files and Spec Files are written to disk, they do not exist in the database at all.
2022-02-02 17:59:56 +00:00
def create_user_file(self):
self.create_reference_document()
workflow = self.create_workflow('file_upload_form')
processor = WorkflowProcessor(workflow)
processor.do_engine_steps()
task = processor.next_task()
correct_name = task.task_spec.form.fields[0].id
Fixes broken tests
2020-01-31 16:33:43 +00:00
1. Created a UserFileService, so it's clear what we use when for File Services, UserFiles, SpecFiles, and ReferenceFiles each function differently. 2. Reference Files and Spec Files are written to disk, they do not exist in the database at all.
2022-02-02 17:59:56 +00:00
data = {'file': (io.BytesIO(b"abcdef"), 'random_fact.svg')}
rv = self.app.post('/v1.0/file?study_id=%i&workflow_id=%s&task_spec_name=%s&form_field_key=%s' %
(workflow.study_id, workflow.id, task.get_name(), correct_name), data=data,
follow_redirects=True,
content_type='multipart/form-data', headers=self.logged_in_headers())
Fixes broken tests
2020-01-31 16:33:43 +00:00
self.assert_success(rv)
1. Created a UserFileService, so it's clear what we use when for File Services, UserFiles, SpecFiles, and ReferenceFiles each function differently. 2. Reference Files and Spec Files are written to disk, they do not exist in the database at all.
2022-02-02 17:59:56 +00:00
return json.loads(rv.get_data(as_text=True))
Fixes broken tests
2020-01-31 16:33:43 +00:00
Assure that the field type provided is supported. Catches errors such as adding a custom field type of 'text_area' rather than "textarea"
2021-09-29 18:05:45 +00:00
def test_load_valid_url_for_files(self):
1. Created a UserFileService, so it's clear what we use when for File Services, UserFiles, SpecFiles, and ReferenceFiles each function differently. 2. Reference Files and Spec Files are written to disk, they do not exist in the database at all.
2022-02-02 17:59:56 +00:00
file = self.create_user_file()
rv = self.app.get('/v1.0/file/%i' % file['id'], content_type="application/json", headers=self.logged_in_headers())
Assure that the field type provided is supported. Catches errors such as adding a custom field type of 'text_area' rather than "textarea"
2021-09-29 18:05:45 +00:00
self.assert_success(rv)
file_json = json.loads(rv.get_data(as_text=True))
print(file_json)
self.assertIsNotNone(file_json['url'])
file_data_rv = self.app.get(file_json['url'])
self.assert_success(file_data_rv)
Prefer tasks that share a parent over just the the next available task when returning the next_task in the workflow processor.
2021-06-08 12:03:14 +00:00
def test_get_file_contains_data_store_elements(self):
1. Created a UserFileService, so it's clear what we use when for File Services, UserFiles, SpecFiles, and ReferenceFiles each function differently. 2. Reference Files and Spec Files are written to disk, they do not exist in the database at all.
2022-02-02 17:59:56 +00:00
file = self.create_user_file()
ds = DataStoreModel(key="my_key", value="my_value", file_id=file['id'])
Prefer tasks that share a parent over just the the next available task when returning the next_task in the workflow processor.
2021-06-08 12:03:14 +00:00
db.session.add(ds)
1. Created a UserFileService, so it's clear what we use when for File Services, UserFiles, SpecFiles, and ReferenceFiles each function differently. 2. Reference Files and Spec Files are written to disk, they do not exist in the database at all.
2022-02-02 17:59:56 +00:00
rv = self.app.get('/v1.0/file/%i' % file['id'], headers=self.logged_in_headers())
Prefer tasks that share a parent over just the the next available task when returning the next_task in the workflow processor.
2021-06-08 12:03:14 +00:00
self.assert_success(rv)
json_data = json.loads(rv.get_data(as_text=True))
self.assertEqual("my_value", json_data['data_store']['my_key'])
Fixes #306 assure that we take the doc_code into account if passed in as an argument to the API for get_files.
2021-04-23 20:14:23 +00:00
def test_get_files_for_form_field_returns_only_those_files(self):
self.create_reference_document()
workflow = self.create_workflow('file_upload_form')
processor = WorkflowProcessor(workflow)
processor.do_engine_steps()
task = processor.next_task()
correct_name = task.task_spec.form.fields[0].id
data = {'file': (io.BytesIO(b"abcdef"), 'random_fact.svg')}
Add task_spec_name to a bunch of tests
2021-08-26 14:43:28 +00:00
rv = self.app.post('/v1.0/file?study_id=%i&workflow_id=%s&task_spec_name=%s&form_field_key=%s' %
Assure that the field type provided is supported. Catches errors such as adding a custom field type of 'text_area' rather than "textarea"
2021-09-29 18:05:45 +00:00
(workflow.study_id, workflow.id, task.get_name(), correct_name), data=data,
follow_redirects=True,
Fixes #306 assure that we take the doc_code into account if passed in as an argument to the API for get_files.
2021-04-23 20:14:23 +00:00
content_type='multipart/form-data', headers=self.logged_in_headers())
self.assert_success(rv)
# Note: this call can be made WITHOUT the task id.
rv = self.app.get('/v1.0/file?study_id=%i&workflow_id=%s&form_field_key=%s' %
(workflow.study_id, workflow.id, correct_name), follow_redirects=True,
content_type='multipart/form-data', headers=self.logged_in_headers())
self.assert_success(rv)
json_data = json.loads(rv.get_data(as_text=True))
self.assertEqual(len(json_data), 1)
# Add another file for a different document type
1. Created a UserFileService, so it's clear what we use when for File Services, UserFiles, SpecFiles, and ReferenceFiles each function differently. 2. Reference Files and Spec Files are written to disk, they do not exist in the database at all.
2022-02-02 17:59:56 +00:00
UserFileService().add_workflow_file(workflow.id, 'Study_App_Doc', task.get_name(), 'otherdoc.docx',
'application/xcode', b"asdfasdf")
Fixes #306 assure that we take the doc_code into account if passed in as an argument to the API for get_files.
2021-04-23 20:14:23 +00:00
Add task_spec_name to a bunch of tests
2021-08-26 14:43:28 +00:00
# Note: this call can be made WITHOUT the task spec name.
Fixes #306 assure that we take the doc_code into account if passed in as an argument to the API for get_files.
2021-04-23 20:14:23 +00:00
rv = self.app.get('/v1.0/file?study_id=%i&workflow_id=%s&form_field_key=%s' %
(workflow.study_id, workflow.id, correct_name), follow_redirects=True,
content_type='multipart/form-data', headers=self.logged_in_headers())
self.assert_success(rv)
json_data = json.loads(rv.get_data(as_text=True))
self.assertEqual(len(json_data), 1)
Fix the panda warnings, which means we only support xlsx files now. Include the document details when reutnring a file from the file model. Improve the validation of file form elements.
2021-06-22 18:58:52 +00:00
def test_add_file_returns_document_metadata(self):
self.create_reference_document()
workflow = self.create_workflow('file_upload_form_single')
processor = WorkflowProcessor(workflow)
processor.do_engine_steps()
task = processor.next_task()
correct_name = task.task_spec.form.fields[0].id
data = {'file': (io.BytesIO(b"abcdef"), 'random_fact.svg')}
Add task_spec_name to a bunch of tests
2021-08-26 14:43:28 +00:00
rv = self.app.post('/v1.0/file?study_id=%i&workflow_id=%s&task_spec_name=%s&form_field_key=%s' %
Assure that the field type provided is supported. Catches errors such as adding a custom field type of 'text_area' rather than "textarea"
2021-09-29 18:05:45 +00:00
(workflow.study_id, workflow.id, task.get_name(), correct_name), data=data,
follow_redirects=True,
Fix the panda warnings, which means we only support xlsx files now. Include the document details when reutnring a file from the file model. Improve the validation of file form elements.
2021-06-22 18:58:52 +00:00
content_type='multipart/form-data', headers=self.logged_in_headers())
self.assert_success(rv)
json_data = json.loads(rv.get_data(as_text=True))
self.assertEqual('Ancillary Document', json_data['document']['category1'])
Fixed tests that broke from the new irb_documents spreadsheet
2021-08-26 12:51:52 +00:00
self.assertEqual('Study Team', json_data['document']['who_uploads?'])
Fixes #306 assure that we take the doc_code into account if passed in as an argument to the API for get_files.
2021-04-23 20:14:23 +00:00