Improve output validation/upload logging

Contents (#691)

Enhancements

• Improve output validation/upload logging

Fixes

• Avoid attempting to upload output manifest if it's None

Log warning when runtime timeout is near

Contents (#690)

Enhancements

• Log warning when running on on Cloud Run and runtime timeout (1 hour) is near

Dependencies

• Add pydash dependency

Refactoring

• Replace custom nested attribute functions with pydash usage

Allow optional strands

Contents (#688)

New features

• Allow optional strands

Dependencies

• Use twined=0.6.0

Fixes

• Skip dataset validation for missing optional manifests

Add documentation on updating Octue services

Contents (#683)

Operations

• Use latest ruff pre-commit check

Dependencies

• Add ruff to dev dependencies

Other

• Add doc on updating an Octue service

Switch to ruff developer tooling

Contents (#682)

Operations

• Switch from flake8, black, and isort to ruff

Dependencies

• Remove old formatters/linters and add ruff config

Refactoring

• Apply ruff to all files

Check for service revision existence

Contents (#680)

IMPORTANT: There is 1 breaking change.

Enhancements

• 💥 BREAKING CHANGE: Use cloud URIs by default for datasets in output manifests
• Add comments around checking for service revision existence
• Improve error when octue.services topic doesn't exist

Fixes

• Raise error if service revision subscription doesn't exist when no service registry is in use
• Remove octue.services prefix from subscription names

Refactoring

• Avoid repeated conversion to Pub/Sub ID for a service

---

Upgrade instructions

💥 Use cloud URIs by default for datasets in output manifests

Set use_signed_urls_for_output_datasets to True in the app configuration to keep using signed URLs for datasets in output manifests.

Revert analysis output location removal

Contents (#677)

Fixes

• Pass output arguments into Analysis and use them

Reversions

• Revert "REF: Stop storing output_location in Analysis"

Make signed URLs for output datasets optional

Contents (#676)

IMPORTANT: There is 1 breaking change.

Enhancements

• Allow using non-signed URLs for datasets in output manifest (controllable via the app configuration file)
• Handle all requests errors while:
  • Getting cloud metadata for datafiles and datasets
  • Downloading datafiles

Fixes

• Avoid trying to access buckets for URL datasets

Refactoring

• 💥 BREAKING CHANGE: Stop storing output_location in Analysis
• Remove unnecessary finalisation from template apps

---

Upgrade instructions

💥 Stop storing `output_location` in `Analysis`

If calling Analysis.finalise manually, either stop doing this and rely on the output_location field of the app configuration or explicitly pass in the upload_output_datasets_to argument.

Improve event filtering

Contents (#673)

IMPORTANT: There are 2 breaking changes.

New features

• Add dictionary_product utility function

Enhancements

• 💥 BREAKING CHANGE: Disable event validation in EventReplayer by default
• 💥 BREAKING CHANGE: Enable filtering by multiple event kinds in get_events
• Enable excluding multiple event kinds in get_events
• Use all non-question events for question redelivery check in flask app
• Add ability to skip handling logs containing certain text in AbstractEventHandler and subclasses
• Return outside of ThreadPoolExecutor context managers

---

Upgrade instructions

💥 Disable event validation in `EventReplayer` by default

Set validate_events=True in the EventReplayer constructor to retain the previous behaviour.

💥 Enable filtering by multiple event kinds in `get_events`

To filter by one event kind as before, use kinds=[event_kind] instead of kind=event_kind.

Enable question retries on single questions

Summary

This release adds to the question retry capability already available on concurrent questions by allowing retries of single questions.

Contents (#671)

Enhancements

• Enable question retries on single questions with Child.ask
• Log when retries are prevented for an exception type in Child.ask
• Remove PYTHONUNBUFFERED warning

Refactoring

• Move retry logic from Child.ask_multiple into Child.ask