v0.0.59

• Allow Magda API endpoints to be served at non-root path
• Fixed: Distribution Page may show a blank page when the distribution has no downloadUrl #3039
• Fixed: Date Extractor for metadata creation tool will fail to process CSV or excel with incomplete rows #3040
• Other Date Extractor extration accuracy improvement
• Admin Panel User Admin feature should set/unset both isAdmin field & admin role for compatibility purpose #3033
• Make GET /auth/users/all API return user role info as well
• Make PUT /auth/users/:userId API accept other user field rather then isAdmin field only
• Release acs-cmd & org-tree as separate NPM packages
• Add set admin function to acs-cmd commandline utility package
• #3024 Gateway will automatically proxy all GET requests received at /api/v0/registry to registry read-only nodes (excepts Hooks APIs)
• For #3010:
  • Set autovacuum_freeze_max_age & autovacuum_multixact_freeze_max_age default value to lower value (100k & 200K) for events table only
  • Added daily DB VACUUM ANALYZE script
  • Allow akka.http.server requestTimeout & idleTimeout to be configured via registry api helm chart
  • Allow gateway proxy timeout setting to be configured via gateway helm chart
• #3057 Fixed: API /auth/plugins will skip the failed loading config item and return other config items without breaking request processing
• #3060 Upgrade urijs to 1.19.5
• #3051 Make command line tools DB PORT is configurable
• Change openfaas docker image repository & re-enable faas-idler using re-build image from version 0.2.1

v0.0.59-alpha.3

• Allow Magda API endpoints to be served at non-root path
• Fixed: Distribution Page may show a blank page when the distribution has no downloadUrl #3039
• Fixed: Date Extractor for metadata creation tool will fail to process CSV or excel with incomplete rows #3040
• Other Date Extractor extration accuracy improvement
• Admin Panel User Admin feature should set/unset both isAdmin field & admin role for compatibility purpose #3033
• Make GET /auth/users/all API return user role info as well
• Make PUT /auth/users/:userId API accept other user field rather then isAdmin field only
• Release acs-cmd & org-tree as separate NPM packages
• Add set admin function to acs-cmd commandline utility package
• #3024 Gateway will automatically proxy all GET requests received at /api/v0/registry to registry read-only nodes (excepts Hooks APIs)
• For #3010:
  • Set autovacuum_freeze_max_age & autovacuum_multixact_freeze_max_age default value to lower value (100k & 200K) for events table only
  • Added daily DB VACUUM ANALYZE script
  • Allow akka.http.server requestTimeout & idleTimeout to be configured via registry api helm chart
  • Allow gateway proxy timeout setting to be configured via gateway helm chart
• #3057 Fixed: API /auth/plugins will skip the failed loading config item and return other config items without breaking request processing
• #3060 Upgrade urijs to 1.19.5
• #3051 Make command line tools DB PORT is configurable
• Change openfaas docker image repository & re-enable faas-idler using re-build image from version 0.2.1

v0.0.59-alpha.2

#3053 Disabled faas-idler by default as it's now only available via Openfaas Pro

v0.0.59-alpha.1

0.0.59

• Allow Magda API endpoints to be served at non-root path
• Fixed: Distribution Page may show a blank page when the distribution has no downloadUrl #3039
• Fixed: Date Extractor for metadata creation tool will fail to process CSV or excel with incomplete rows #3040
• Other Date Extractor extration accuracy improvement
• Admin Panel User Admin feature should set/unset both isAdmin field & admin role for compatibility purpose #3033
• Make GET /auth/users/all API return user role info as well
• Make PUT /auth/users/:userId API accept other user field rather then isAdmin field only
• Release acs-cmd & org-tree as separate NPM packages
• Add set admin function to acs-cmd commandline utility package
• Make Gateway should proxy all GET requests to registry-read-only pods

v0.0.59-alpha.0

• Allow Magda API endpoints to be served at non-root path
• Fixed: Distribution Page may show a blank page when the distribution has no downloadUrl #3039
• Fixed: Date Extractor for metadata creation tool will fail to process CSV or excel with incomplete rows #3040
• Other Date Extractor extration accuracy improvement
• Admin Panel User Admin feature should set/unset both isAdmin field & admin role for compatibility purpose #3033
• Make GET /auth/users/all API return user role info as well
• Make PUT /auth/users/:userId API accept other user field rather then isAdmin field only
• Release acs-cmd & org-tree as separate NPM packages
• Add set admin function to acs-cmd commandline utility package

v0.0.58

• Added Documentation for Magda Helm Charts (generated using helm-docs)
• Upgrade papaParse to 5.3.0 for the Regular Expression Denial of Service (ReDos) vulnerability
• Fixed papaParse's chunk data loading HTTP 416 Error (on our forked repo)
• On dataset page, display an alert box when the dataset is superseded or retired
• Added Helm chart option to disable "use internal storage to store file" option in metadata creation tool UI
• Fixed: elasticsearch helm chart incorrect selector
• Updated cluster-issuer & certificate api version to be compatible with cert-manager v1.0
• Fixed: some helm chart (gateway & web) deployment doesn't allow setting replicas value properly
• Added an opt-in for chart/map/table previews if files are large.
• Upgraded helm-docs to v1.2.1
• Upgraded minio to 7.1.2
• Allowed region to be specified for storage API
• Made storage API process BucketAlreadyExists error code correctly when create bucket
• Fixed magda-builder-scala docker image failed to build due to expired resource link
• Fixed: Registry should not create an event when failed to delete the record #2976
• Fixed: Time Travel API response 500 sometimes #2977
• All registry APIs that create / changes / updates the data will now return the eventId of the event created for the operation #2983
• When submit a dataset, system will try to tag current version with relevant EventId #2980
• Keep tracking superseded data files in versions to avoid superseded data files being deleted #2981
• UI Changes that allow user to access the distribution page of different versions #2982
• Use pseudo url for data file stored in storage api #3000
• Turn off Open In National Map Button for Internal Storage data file Link
• Use magda-preview-map v0.0.58
• Fixed: Dataset Creation Tool may not always capture format properly #3001
• Add Authentication Plugin Support to Gateway & UI
• Authentication Plugin Documentation
• Publish AuthApiClient as a NPM package
• Publish Authentication Plugin SDK as a NPM package
• Publish create-secrets tool as a NPM package
• Replace google auth provider with auth plugin magda-auth-google
• Replace ckan auth provider with auth plugin magda-auth-ckan
• Replace internal auth provider with auth plugin magda-auth-internal
• Remove Google, CKAN & internal auth provider code from Gateway codebase
• Publish docker image utils as a seperate NPM package @magda/docker-utils
• Use magda-auth-arcgis auth plugin instead
• Make Add a dataset link on Homepage (admin user only) go to metadata creation tool directly
• #2985 Allow default web route to be configured like all other route (allow specify method, auth etc.)
• Removed createGenericProxy from Gateway codebase

v0.0.57

General:

• CSV Connector can now process ampersand character properly
• Fixed broken link minion causes json schema validation error
• Upgraded typescript to 3.7.2 & Use Project References to organize typescript project
• Upgraded prettier to 1.19.1 to support typescript 3.7 better
• Moved out connectors & minions from main magda repostory
• Broke helm chart to magda-core (core magda components excluding connectors & minions) & magda (full magda chart) and adjusted CI pipeline accordingly
• Release npm packages for building connectors & minions without depending on main repo
• Upgrade charts to use with Helm 3, Kubernetes 1.17 and Minikube 1.7.2
• Make recompiling and updating create secrets scripts an action in the CI
• Move magda-preview-map out of the core repo
• Upgraded everything cert-manager related to work with v0.13
• Update build & run document to provide more information regarding running local build connector & minion docker images
• Update building and running documents: replace npx with yarn in all commands & replace keywords npm dependecies with dependecies
• Update building and running documents: remove lerna & pancake from prerequisites list
• New: Extract metadata from data portal URLs
• Fixed function is incorrectly deployed for UI only deployment
• Use helm3 in inttest:registryAuth step in CI
• Make Magda charts helm v3 charts
• Add migration document for migrating Magda release v0.0.56-RC6 or eariler to v0.0.57-0
• Use a feature flag to turn on/off the dataset approval workflow
• Add internal authentication provider allow user to be authenticated locally
• Upgrade to typescript 3.9.5
• Add missing cloud-sql-proxy dependecy to magda-core chart
• Add more operators to Registry /records & records/count API aspectOrQuery
• Add aspectOrQuery, orderBy, orderByDir & orderNullFirst parameters to Registry API /records & records/count
• Upgrade bcrypt to 5.0.0
• Make the constructor of AuthorizedRegistryClient also accepts a customised jwt token.
• Set pwgen to 0.1.6 (was ^0.1.6)
• Remove ESRI-portal related code from deployment charts
• Allow admin users to download history report of a dataset
• Fixed building-and-running doc: instruct users to update helm charts with yarn update-all-charts
• Added Docs of supplying custom policy files
• Openfaas should be turn on / of via condition field global.openfaas.enabled instead of tags.
• When global.openfaas.enabled is false, all openfaas dependent objects will either report error to users or be skiped for creation.
• Largely Reduced the web-server docker image size
• Fixed incorrect path in API docs #2964

UI:

• Fixed the issue of modifying date string in text input using backspaces to an empty string will cause text input to reset text input
• Added pre-specified options for themes on Add dataset page
• User can't input a future date to date of last modification on add dataset page
• Allow to config whether keywords / themes input can accept manual inputs (or only pre-defined phrases)
• Only check for validity in temporal filters (in the search dataset page)
• Allow a blacklist of strings to be specified for automatic keyword generation
• Make the global notification banner configurable
• Removed all references to the DTA Design System react components, so that all styles come through our SCSS compilation, which should make SCSS smaller and more consistent.
• Swap the order of custodian & team dropdown on add dataset page
• Updated text & margin of add files page to match the new design
• Updated Add Dataset Welcome screen options UI design
• Added UI for "Link to dataset already hosted online" box
• Added UI for "Link to an API or web service" box
• Added UI for storing files
• Added UI for "Publish as Open Data to data.gov.au"
• Refactor view dataset page & remove the edit function from the page
• Clarify tooltip text
• Use a new "All done!" screen at the end of the dataset flow
• Added Error Screen for adding distributions from URL feature & allow manually create distributions
• Disable edit button on harvested datasets
• Added the support of processing ESRI REST API URL
• Use IBM Plex Sans font
• Fixed: error notification no longer has a white background
• Added add dataset review page
• Make add dataset page save state data to registry (unless in preview mode) rather than localStorage
• Change search text to say "Search for data" instead of "Search for open data"
• Set flag previewAddDataset to false by default
• Made files that are dropped get uploaded, if this is specified by the user
• Bugfix: Read raw value from CSV files
• Add a flag placeholderWorkflowsOn to hide UI components that are not complete yet
• Add MyDatasets Section to Homepage (after logged in)
• Fix magda & ckan login options are missing
• Always set dataset publishing state to "published" no matter the approval flow is on / off
• Fix a sheet file contains date would cause blank screen during processing
• Set an error if security classification is at PROTECTED or above
• Add revised edit flow for adding/deleting/superceding existing distributions
• Show extra metadata on the dataset page
• Fixed ckan-export aspect's fields should be only changed via JSON patch from the frontend (#2887)
• Fixed patch ckan-export request trigger 400 error when the aspect doesn't exist at all
• Allowed UI to be served at a different URL path
• Fixed: incompatible papaparse version makes CSV file download fail (set to version 5.1.0 now)
• Fixed: Upgrade dompurify to 2.0.3 (#2952)
• Fixed: Borken SSO links (#2951)

Storage:

• Add an API for storing and streaming content
• Add a DELETE endpoint
• Improves error handling (returns 404 from GET if the file doesn't exist)
• Add apidocs
• Add endpoint to create a bucket
• Restrict file upload to admins only
• Add authorization to GET endpoint
• Support multipart upload
• Fixed: minio chart will not be deployed if storage-api is not turned on
• Make MinIO a dependency of storage's helm chart
• Make minio, by default, deploy with Recreate strategy

Gateway:

• Add /data to ckan URL, remove the came_from param
• Allow cookie option to be configured via helm chart
• Set cookie secure default value to auto
• Set dev site cookie sameSite attribute to none
• Allow to configure webRoutes in a way of similar to routes (i.e. specify method, auth etc)
• Make preview-map route configurable (through web route)
• Gateway will auto add headers to turn off cache if the incoming request carry a Cache-Control header that contains no-cache keyword
• Upgrade arcgis oAuth plugin to latest version used by Terria team #2959

Authorization:

• Made integration tests for authorisation run automatically as part of CI.
• Added ability to set per-record authorization policies in the registry (for getting a single record)
• Added ability to set per-record authorization policies in the registry (for getting multiple records)
• Added ability to use OPA policies that use data types other than strings in the registry
• Added authorization inside links with dereferencing on or off, for the /records/<id> endpoint
• Added authorization inside links with dereferencing on or off, for the /records endpoint
• Added per-record authorization around the /records/summary/<recordid> endpoint matching the /records/<id> one.
• Added per-record authorization around the /records/summary endpoint matching the /records one.
• Added per-record authorization around the /records/<recordid>/history endpoint
• Added per-record authorization around the /records/<recordid>/history/<eventId> endpoint
• Fixed a bug where the registry would attempt to query for policies against every single record id in the registry all at once.
• Added API Key authentication support
• Add auth policy to distribution records
• Allowed custom opa policy files to be supplied during the deployment
• Fixed: process unconditional true properly #2956
• Remove ESRI Policy from built-in OPA policy list (but still make them available for unit tests) #2958

Registry:

• Made events record the correct user id
• Fixed Registry History API Performance Issue when limit=1 & Updated Registry History API Document
• Fixed not support Neg operator in OPA policy
• Fixed: when authnreadpolicyid is set to empty string, registry will throw 500 error #2955

Search:

• Began work on a NodeJS-based search API to replace the scala one
• Allow fetch region record by region id
• Use a list of acronyms for publishers that have a 'Department of' in their name

Minions:

• Added a minion for publishing to CKAN

CI/CD:

• Use Directed Acyclic Graph in Gitlab CI
• Moved data.gov.au specific connector config out of the magda helm chart

Others:

• Use a "Year" column from a CSV file to extract a temporal extent
• Added tentative documentation/scripting for building and running on microk8s.
• Added tentative documentation/scripting for building and running on k3d.

v0.0.55

UI:

• Updated design system components
• Integrate sentry release notification and source map upload
• Added "role" to account page
• Fixed clicking out of search filter causes the results to refetch
• Provide access to CKAN Data API for enabled resources
• Changed label of email validation error message in suggest dataset page to be consistent with rest of the error messages in that page

Correspondence:

• Broadened search for a valid dataset email contact address

Others:

• Upgraded JDK version for magda-builder-scala to 8u201
• Added craco to allow for some Create React App overrides for a faster build and to allow use of PDFjs without warnings.
• Fixed Unable to use Google / Facebook Login on Preview Site
• Fixed warnings: as props is compulsory for AUpageAlert & boolean value was sent to id props
• Fixed docker build cache issue that causes DB image not build

v0.0.53

Connectors

We’ve added the ability for the CKAN connector to only pull datasets from a certain organisation — this should be helpful for internal instances of Magda, where an organisation has published open datasets on a CKAN portal already and wants the ability to pull them back in so they can be searched alongside its private data.

allowedOrganisationName: "doee"

We’ve also added the ability to add default parameters to connectors, so you can fill in things you know that we might not be able to get from the connector directly — e.g. if you’re connecting to an open data portal you might want to default the creation.isOpenData field in dcat-dataset-strings to true:

presetRecordAspects:
  - id: "dcat-dataset-strings"
    recordType: "dataset"
    data:
      creation:
        isOpenData: true

We’ve also updated the UI so that this field is shown if present:

We’ve also made some adjustments to the CSW connector, in order to allow it to pick up open licenses more reliably, particularly those from Geoscience Australia’s CSW connector as used by data.gov.au.
Customisation

In this release we’ve added the ability to change the feature (homepage background) images and the favicon via the admin ui, replacing the eye-searing default blue background and the unremovable data.gov.au logo that were there before.

Operations

This release we added snyk.io to our builds as a dependency scanner, and updated a large number of dependencies with vulnerabilities in them. This isn’t noticable directly, but makes it safer to use and deploy Magda.

We’ve also added a magda.reservedFor=statefulsets:NoSchedule toleration to our statefulsets in the helm charts. This makes it easier to run statefulsets (databases, elasticsearch) on certain nodes (with that taint set) and the rest of the system on cheap, less reliable infrastructure like GKE preemptible nodes (which we run our dev infrastructure on), without the databases and elasticsearch continually going up and down.
Migration

This version involves a reindex of the datasets and publishers index in ElasticSearch. If you’re not OK with the datasets index dropping to 0 datasets and taking a few minutes to rebuild itself, set this in the config for search-api before running helm upgrade.

image:
  tag: 0.0.52

Then remove it and run helm upgrade again once the indexer has finished reindexing.

How to get it

The new version is available as a helm chart at https://charts.magda.io, and you can try it at https://data.gov.au. Full changelog is at https://github.com/magda-io/magda/blob/master/CHANGES.md

v0.0.53-RC1

Bumped to 0.0.53-RC1