Commit Graph

229 Commits

Author SHA1 Message Date
jasquat c0e072a2e2
Mi large data debug bpm only (#1846)
* some debug code to help track down slow processing times for mi tasks with larget data sets w/ burnettk

* use mapping cache for task models and bpmn processes and some linting cleanup

* some minor bugfixes w/ burnettk

* assign the correct bpmn process to the given guid w/ burnettk

* fixed data migration tests w/ burnettk

* unit tests are passing w/ burnettk

* integration tets are passing w/ burnettk

* some cleanup while code reviewing w/ burnettk

* some more cleanup w/ burnettk

* pass new args to TaskService in data migration w/ burnettk

* fixed broken test w/ burnettk

* allow running acceptance tests without keycloak w/ burnettk

---------

Co-authored-by: jasquat <jasquat@users.noreply.github.com>
2024-07-01 16:05:51 -04:00
jasquat 1d79d77686
Celery avoid requeueing (#1828)
* some debug code and potential fixes to avoid requeueing future tasks w/ burnettk

* actually insert the queued time into the future task table w/ burnettk

* fixed tests w/ burnettk

* handle missing process instances better from celery worker and updated some comments w/ burnettk

* added comment w/ burnettk

* minor fixes w/ burnettk

* added test for should_schedule_waiting_timer_events w/ burnettk

* added test to ensure we do not queue recently queued tasks again w/ burnettk

* remove traceback code

---------

Co-authored-by: jasquat <jasquat@users.noreply.github.com>
Co-authored-by: burnettk <burnettk@users.noreply.github.com>
2024-06-26 10:48:28 -07:00
jasquat 5e3831f4d6
message-model-perm-check (#1805)
* check if user has permissions to messages before attempting call w/ burnettk

* fixed variable typo w/ burnettk

---------

Co-authored-by: jasquat <jasquat@users.noreply.github.com>
2024-06-24 12:36:07 -07:00
jasquat d2e005022b
use the greedy strategy when running messages from the background processor w/ burnettk (#1798)
Co-authored-by: jasquat <jasquat@users.noreply.github.com>
2024-06-21 16:22:42 -04:00
Kevin Burnett 6ab89dab40
default to local openid instead of keycloak container dependency (#1794)
Co-authored-by: burnettk <burnettk@users.noreply.github.com>
2024-06-21 08:38:52 -07:00
Kevin Burnett e4aac94a4e
Fix gunicorn and wait for schema issues (#1771)
* fix gunicorn and wait_for_db_schema_migrations

* avoid bad gunicorn version

* actually check that request succeeded

---------

Co-authored-by: burnettk <burnettk@users.noreply.github.com>
2024-06-19 10:59:27 -07:00
jasquat 95ed415bab
wait_for_db script executable and fixed package lock file w/ burnettk 2024-06-18 15:52:22 -04:00
jasquat 6a99762f70
better-db-migration-waiting (#1732)
* explicitly wait for db migrations to be completed instead of sleeping w/ burnettk

* code rabbit suggestion w/ burnettk

* get rid of the set x w/ burnettk

---------

Co-authored-by: jasquat <jasquat@users.noreply.github.com>
2024-06-13 13:34:32 -04:00
Kevin Burnett eae5f7dd2d
the big message improvements branch (#1549)
* imported patch from old message_improvements branch w/ burnettk

* wip.

* merging in changes from message_improvements

* remove patch files that were accidendetally added.

* Added a modal for editing a correlation.  Added ability to delete whole correlation keys.
A little css cleanup.

* * Removing migration - will add back in at the end.
* The Message models API should not require page and per_age parameters, it will return all.
* The Message model list should return a full json description of all messages/correlations for all containing groups.
*

* wip

* Add import, fix class name

* Getting ./bin/pyl to pass

* Getting ./bin/pyl to pass

* Some fe lint fixes

* Some ruff fixes

* Commands to nuke poetry dirs

* Temp skipping of a couple tests

* Getting ./bin/pyl to pass

* This needs to be back in

* Revert back to main

* Factored out data store handling

* Working on factoring out collecting messages, has test failure

* Formatting

* Fixed up test failures

* Remove commentted out lines

* Adding fields

* Fix merge issue

* Re-enable modal

* WIP

* Untested relationships

* Remove correlation key table

* Remove retrieval expression from uniqueness

* Remove commentted out lines

* WIP

* WIP

* WIP

* WIP

* WIP

* Make mypy pass

* Getting formatters to pass

* Add migration

* WIP fixing tests

* WIP fixing tests

* WIP fixing tests

* WIP fixing tests

* WIP fixing tests

* Getting ./bin/pyl to pass

* Fix skipped test

* Fix skipped test

* Getting ./bin/pyl to pass

* Remove unused method

* Remove unused methods

* Clean up unused code

* Refactor to support creating single messages from the UI

* Untested support for processing one process_group

* WIP test

* WIP test

* Filled out test

* Getting ./bin/pyl to pass

* Message Editor Modal Work

* Change migration and add in schemas.

* Swtich to using the associated branch of the process BPMN.io mods

* Get the backend returning messages created from the frontend to the drop down list in the BPMN.io editor.

* Merge main, fix up test

* Getting ./bin/pyl to pass

* Show path in location

* Rename var

* install packages from bpmn-js-spiffworkflow as well for local development

* process group api can add and update message models now w/ burnettk

* backend tests are passing now w/ burnettk

* the launch message edit button is loading the editor w/ burnettk

* updated bpmn-js-spiffworkflow

* pyl is passing w/ burnettk

* updated bpmn-js-spiffworkflow w/ burnettk

* updated bpmn-js-spiffworkflow w/ burnettk

* fixed console errors w/ burnettk

* a couple tweaks w/ burnettk

* save the message json in the new format from the mform w/ burnettk

* display the correlation props in the form w/ burnettk

* default to empty schema so the format is obvious

* allow removing correlation props from web ui w/ burnettk

* added save notification when saving a message on a process model w/ burnettk

* fixed broken test w/ burnettk

* Updating test cases to new message format, tests are failing

* support schema from messages in frontend

* Fixing tests

* Fixing tests

* Fixing tests

* removed references to correlation keys and removed unused components w/ burnettk

* removed temp mesasge model edit button w/ burnettk

* Make mypy pass

* Fixing tests

* Fixing tests

* Getting ./bin/pyl to pass

* save deleted messages before attempting to add new ones w/ burnettk

* set state for the message id so it can be changed w/ burnettk

* do not wait for the message id to be set since it is not necessary w/ burnettk

* updated bpmn-js-spiffworkflow w/ burnettk

* build images for this branch w/ burnettk

* put location in path of message-models so we can control permissions on it w/ burnettk

* fix black

* some coderabbit suggestions

* pull in spiff fix

* Default schema to {}

* Temp fix for invalid schema

* updated bpmn-js-spiffworkflow

* some updates for issue 1626

* minor name tweaks and attempts to update message dropdown in panel when message changes - does not work yet w/ burnettk

* updated bpmn-js-spiffworkflow w/ burnettk

* attempt to call add_message.returned event when message updates w/ burnettk

* treat formData as a state in the MesasgeEditor so it can be updated when the form contents is modified w/ burnettk

* updated bpmn-js-spiffworkflow w/ burnettk

* Feature/merge correlation properties (#1693)

* Merge XML Correlation properties with Process group properties

* updates for messages w/ burnettk

---------

Co-authored-by: theaubmov <ayoubaitlachgar98@gmail.com>
Co-authored-by: jasquat <jasquat@users.noreply.github.com>

* do not wait for message id state to be set to better support new messages w/ burnettk

* updated SpiffWorkflow w/ burnettk

* some cleanup from coderabbit and linting

* added index to message tables, run typecheck in ci, and other updates while code reviewing w/ burnettk

* updated bpmn-js-spiffworkflow w/ burnettk

* remove branch to build

---------

Co-authored-by: jasquat <jasquat@users.noreply.github.com>
Co-authored-by: danfunk <daniel.h.funk@gmail.com>
Co-authored-by: Jon Herron <jon.herron@yahoo.com>
Co-authored-by: burnettk <burnettk@users.noreply.github.com>
Co-authored-by: jasquat <2487833+jasquat@users.noreply.github.com>
Co-authored-by: theaubmov <ayoubaitlachgar98@gmail.com>
2024-06-10 09:15:54 -07:00
jasquat 586a088e21
update-spiffworkflow (#1694)
* updated SpiffWorkflow w/ burnettk

* added version 5 migration w/ burnettk

* pyl w/ burnettk

* added test for the version 5 migration w/ burnettk

* fixed the version 5 migration w/ burnettk

---------

Co-authored-by: jasquat <jasquat@users.noreply.github.com>
2024-06-07 13:23:19 -07:00
jasquat 59d19bb457
Rjsf typing is slow (#1663)
* updated rjsf packages w/ burnettk

* use performance improvement branch of validate js v8 w/ burnettk

* Update spiffworkflow-backend/bin/task_data_to_python_variables.py

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

---------

Co-authored-by: jasquat <jasquat@users.noreply.github.com>
Co-authored-by: Kevin Burnett <18027+burnettk@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2024-06-03 16:19:15 -07:00
jasquat ae6f3119bb
updated libs for safety issues w/ burnettk (#1662)
Co-authored-by: jasquat <jasquat@users.noreply.github.com>
2024-06-03 11:18:02 -04:00
Kevin Burnett 0329c11182
more flexible local dev args (#1613)
Co-authored-by: burnettk <burnettk@users.noreply.github.com>
2024-05-24 23:06:16 +00:00
jasquat c882a208fb
Hunter is me main (#1585)
* get token object for local development env

* print just token to stdout in case we want to shell out to this script for some reason

* migrated get_token bash script to python version w/ burnettk

* ruff should ignore print statements in bin script w/ burnettk

---------

Co-authored-by: unknown <hdl560@163.com>
Co-authored-by: HunterIsMe <36193162+HunterIsMe@users.noreply.github.com>
Co-authored-by: burnettk <burnettk@users.noreply.github.com>
Co-authored-by: jasquat <jasquat@users.noreply.github.com>
2024-05-21 15:40:25 -04:00
jasquat b6f50a65f3
save all bpmn files right before starting the server w/ burnettk 2024-05-14 11:29:09 -04:00
Kevin Burnett 8cb39ef214
allow large tokens to pass through nginx (#1550)
* allow large tokens to pass through nginx

* improve comment

---------

Co-authored-by: burnettk <burnettk@users.noreply.github.com>
2024-05-13 16:56:37 -07:00
Kevin Burnett c967af3d65
safety updates and ignore unfixable pip issue (#1544)
Co-authored-by: burnettk <burnettk@users.noreply.github.com>
2024-05-13 07:23:55 -07:00
burnettk 141a327627
make sure input line is set before doing things 2024-05-07 20:43:15 -04:00
jasquat 973e8fce72
added option to clone process models in boot in docker script w/ burnettk 2024-05-03 16:31:27 -04:00
jasquat d01e52d81f
fixed run process model script to work with celery w/ burnettk 2024-05-03 14:57:35 -04:00
jasquat 86f97d5937
New calling process table (#1480)
* added new table to handle called process relationships

* process caller api endpoint is working again w/ burnettk

* fixed more tests and mypy w/ burnettk

* unit tests are passing and commented out unrelated failing test w/ burnettk

* all tests are passing almost w/ burnettk

* uncommented flakey test w/ burnettk

* minor change while reviewing w/ burnettk

* some changes from coderabbit suggestions w/ burnettk

* commented out sampling change w/ burnettk

* name the foreign keys on the process caller table w/ burnettk

---------

Co-authored-by: jasquat <jasquat@users.noreply.github.com>
2024-05-03 10:09:32 -04:00
jasquat 00c2e85fee
Data store background fix (#1455)
* added test to demonstrate data store issue when run in certain contexts w/ burnettk

* added method to add kkv data store getters back onto tasks if appropriate w/ burnettk

* removed debug code w/ burnettk

---------

Co-authored-by: jasquat <jasquat@users.noreply.github.com>
2024-04-26 10:14:26 -04:00
burnettk c2856be3bf
remove flake8 because ruff and bandit because it is in backend, fix long line 2024-04-24 12:59:20 -04:00
burnettk d197bf00a2
couple fixes to get current user finder script 2024-04-24 12:57:11 -04:00
jasquat 32141a238e
Reset task in call activity fix (#1417)
* updated import pi test to set task data w/ burnettk

* fixed persist test so it loads the processor with data

* load data and completed tasks when getting bpmn json for an instance

* save start and end times on tasks on import w/ burnettk

* verify we are saving start and end times for imported process instances w/ burnettk

---------

Co-authored-by: jasquat <jasquat@users.noreply.github.com>
2024-04-18 14:11:53 -04:00
jasquat 6643db9f2f
Pi file data on fs (#1403)
* added basic support to store process instance file data on the file system w/ burnettk

* the files need to be in a hashed directory structure w/ burnettk

* fixed exception message w/ burnettk

* use the same entry method to the contents of data file w/ burnettk

* fixed hash method w/ burnettk

* make hash method easier to read w/ burnettk

* added a migrator and test for migration w/ burnettk

* fix a couple things while testing

---------

Co-authored-by: jasquat <jasquat@users.noreply.github.com>
Co-authored-by: burnettk <burnettk@users.noreply.github.com>
2024-04-18 11:24:59 -04:00
jasquat a99c36964d
mi-task-show-fix (#1412)
* check for mi tasks first before checking more newest in task show api w/ burnettk

* remove console

---------

Co-authored-by: jasquat <jasquat@users.noreply.github.com>
Co-authored-by: burnettk <burnettk@users.noreply.github.com>
2024-04-17 11:53:10 -07:00
Kevin Burnett f2c982a96c
Look for bad get_current_user scripts (#1370)
* initial script to find get_current_user calls

* handle pre and post scripts as well

* refactor

* omit absolute root dir path when printing

* avoid printing NS

* consolidate lxml parsing in one place

---------

Co-authored-by: burnettk <burnettk@users.noreply.github.com>
2024-04-10 09:44:13 -07:00
jasquat 293aa867a1
Auth for secrets (#1369)
* added new api to show secrets so we can use that in permissions

* updated frontend to use new secret show value api

* cleaned up secret_show method

---------

Co-authored-by: jasquat <jasquat@users.noreply.github.com>
2024-04-09 13:44:47 -04:00
Kevin Burnett 2b6640f24c
Prevent duplicated path prefix elements in generated URLs take two (#1350)
* Prevent duplicated path prefix elements in generated URLs

* add more debug info

* pure refactor

* let flask determine how to access site

* build this branch

* build image

* use more url_for, which includes a slash, and things are working locally

* fix hopefully the last missing url_for

* add some code that works for any openid provider

* one more url_for and remove more backend url config references

---------

Co-authored-by: Bret Mogilefsky <bmogilefsky@gmail.com>
Co-authored-by: burnettk <burnettk@users.noreply.github.com>
2024-04-08 08:50:55 -07:00
jasquat 01c749aa0b
Allow setting path prefix (#1344)
* allow setting the path prefix of the server w/ burnettk

* added error in case someone was successfully using SPIFFWORKFLOW_BACKEND_APPLICATION_ROOT w/ burnettk

* Update docs/DevOps_installation_integration/path_based_routing.md

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

---------

Co-authored-by: jasquat <jasquat@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2024-04-05 15:14:09 -04:00
jasquat b21a3b3ec7
Celery docs and debug (#1328)
* added script to import process instance and updated some docs for redis and celery w/ burnettk

* pyl and removed bad gitignore w/ burnettk

---------

Co-authored-by: jasquat <jasquat@users.noreply.github.com>
2024-04-02 16:13:18 -04:00
dependabot[bot] 4a373be939
Bump black from 23.1.0 to 24.3.0 (#1285)
* Bump black from 23.1.0 to 24.3.0

Bumps [black](https://github.com/psf/black) from 23.1.0 to 24.3.0.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/compare/23.1.0...24.3.0)

---
updated-dependencies:
- dependency-name: black
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* fixed pre commit black issues w/ burnettk

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: jasquat <jasquat@users.noreply.github.com>
2024-03-27 14:05:49 -04:00
jasquat bc3708bb00
async-support-on-message-start (#1173)
* added async and sync support for the message start api w/ burnettk

* removed debug dump call w/ burnettk

---------

Co-authored-by: jasquat <jasquat@users.noreply.github.com>
2024-03-06 13:43:06 -08:00
jasquat b2126e8f38
check if the dir looks like it is a git directory already before cloning to it w/ burnettk 2024-02-28 14:25:38 -05:00
jasquat 947c00670d
set the GIT_SSH_COMMAND so we can use the private key path 2024-02-28 11:07:29 -05:00
jasquat 1fbf237823
remove lost and found dir if it exists w/ burnettk 2024-02-28 10:35:35 -05:00
jasquat b210b84b3a
added script to clone the process model repo w/ burnettk 2024-02-26 16:34:06 -05:00
jasquat 4983d48f22
cherry-picked 81b6431c58 into main for task_guid backfill fix 2024-02-09 10:47:24 -05:00
jasquat 096fc7befc
added some logging when booting backend in docker w/ burnettk 2024-02-07 16:57:11 -05:00
jasquat 5ae59367cb
Feature/reset pi go button (#978)
* delete human task for task that is being reset to in a process instance w/ burnettk

* added script to remove duplicate human tasks from the database w/ burnettk

* test that human tasks are not duplicated during pi reset w/ burnettk

---------

Co-authored-by: jasquat <jasquat@users.noreply.github.com>
2024-02-06 15:50:00 -05:00
Dan Funk 03b21accec
allow looping back in a process to create more messages - minor bug fix - didn't add what would be a complex test to this, as it was a one line change that seems an oversight in the original code. (#924) 2024-01-29 06:40:39 -08:00
jasquat 80bc2f2e42
Feature/typeahead allow guest user (#897)
* decode tokens with jwt instead of with base64 w/ burnettk

* try to verify jwt token with keycloak when we decode it w/ burnettk

* make the token algorithm a constant w/ burnettk

* WIP: create more valid looking jwt from spiff w/ burnettk

* tests are passsing now w/ burnettk

* some pyl stuff w/ burnettk

* fixed mypy issues w/ burnettk

* fixed issues from mypy fixes w/ burnettk

* do not load openid blueprint if not using those configs w/ burnettk

* used the process instance to determine if guest user can use connector api w/ burnettk

* only check the db for process instance if the api call is for typeahead

* removed unused test code

* pyl

---------

Co-authored-by: jasquat <jasquat@users.noreply.github.com>
2024-01-16 14:47:25 -05:00
jasquat 0c8ff4ee45
updated get_token to work with internal openid as well w/ burnettk 2023-12-18 16:21:28 -05:00
jasquat c00d810704 Feature/data object category (#796)
* WIP: some initial code for category in data objects

* attempt to get the data object for the given bpmn process and instance of it w/ burnettk

* updates for data objects

* fixed tests

* made suggestions by code rabbit and moved logic to get process data file out of shared method since it has a completely different implentation

* remove commented out code

* updated SpiffWorkflow for data object category

---------

Co-authored-by: jasquat <jasquat@users.noreply.github.com>
2023-12-11 07:42:01 -08:00
jasquat 4d6358b0e7 queue the process instance AFTER unlocking it, do not set celery worker concurrency to 12, log already locked error in celery workers for debugging 2023-12-07 10:22:50 -05:00
jasquat 40b02ae488 check that connector-proxy-demo can run when releasing new tags w/ burnettk 2023-12-06 11:21:32 -05:00
jasquat 18600189c8 Feature/background proc with celery (#788)
* WIP: some initial test code to test out celery w/ burnettk

* some cleanup for celery and added base model to put tasks waiting on timers

* removed dup bpmn file

* some more cleanup and added strategy to queue instructions

* some minor code changes w/ burnettk

* remove the unused next_task key from api calls since nobody uses it w/ burnettk essweine

* added migration for future tasks and added test to make sure we are inserting into it w/ burnettk essweine

* ensure future task run at time can be updated w/ burnettk

* added table to queue instructions for end user in w/ burnettk

* added test to ensure we are storing instructions for end users w/ burnettk

* added progress page to display new instructions to user

* ignore dup instructions on db insert w/ burnettk

* some more updates for celery w/ burnettk

* some pyl and test fixes w/ burnettk

* fixed tests w/ burnettk

* WIP: added in page to show instructions on pi show page w/ burnettk

* pi show page is fully using not interstitial now w/ burnettk

* fixed broken test w/ burnettk

* moved background processing items to own module w/ burnettk

* fixed apscheduler start script

* updated celery task queue to handle future tasks and upgraded black and set its line-length to match ruff w/ burnettk

* added support to run future tasks using countdown w/ burnettk

* build image for celery branch w/ burnettk

* poet does not exist in the image w/ burnettk

* start blocking scheduler should always start the scheduler w/ burnettk

* add init and stuff for this branch

* make this work not just on my mac

* send other args to only

* added running status for process instance and use that on fe to go to show page and added additional identifier to locking system to isolate celery workers better w/ burnettk

* fixed typing error that typeguard found, not sure why mypy did not w/ burnettk

* do not check for no instructions on interstitial page for cypress tests on frontend w/ burnettk

* do not queue process instances twice w/ burnettk

* removed bad file w/ burnettk

* queue tasks using strings to avoid circular imports when attmepting to queue w/ burnettk

* only queue imminent new timer events and mock celery

* some keyboard shortcut support on frontend and added ability to force run a process instance over the api w/ burnettk

* some styles added for the shortcut menu w/ burnettk

* pyl w/ burnettk

* fixed test w/ burnettk

* removed temporary celery script and added support for celery worker in run server locally w/ burnettk

* cleaned up migrations w/ burnettk

* created new migration to clean up old migrations

---------

Co-authored-by: jasquat <jasquat@users.noreply.github.com>
Co-authored-by: burnettk <burnettk@users.noreply.github.com>
2023-12-05 11:41:59 -05:00
Kevin Burnett 353ba4cd31 avoid print statements, generally (#784)
* avoid print statements, generally

* remove test for empty main script

---------

Co-authored-by: burnettk <burnettk@users.noreply.github.com>
2023-12-01 06:24:11 -08:00
jasquat 1627f5dd07 bugfix/guest-login-multiple-auths (#782)
* This fixes guest login with using multiple auths, removes empty items from ApiError, and raises if redirect_url given to login does not match expected frontend host w/ burnettk

* get body for debug

* try to get the logs from the correct place to upload w/ burnettk

* mock the openid call instead of actually calling it w/ burnettk

---------

Co-authored-by: jasquat <jasquat@users.noreply.github.com>
Co-authored-by: burnettk <burnettk@users.noreply.github.com>
2023-11-30 13:51:01 -05:00