Releases: wireapp/wire-server
2022-07-05
Release notes
-
For users of the (currently alpha) coturn Helm chart:
manual intervention may be required when upgrading to
this version of the chart from a prior version, due to a bug in
Kubernetes which
may interfere with applying changes to pod and service port configuration
correctly.If, after updating this chart, the coturn pods do not have both a
coturn-udp
port and acoturn-tcp
port, then the coturnStatefulSet
must be manually
deleted from the cluster, and then recreated by re-running Helm. Similarly, if
the coturnService
does not have both acoturn-udp
port and acoturn-tcp
port, thisService
must also be deleted and recreated. (#2500) -
The
nginz{-tcp,-http}
services have been unified into anginz
service, and
moved into the nginz chart.The nginz-ingress-services chart simply targets the
nginz
service, so there's
no need to set matchingservice.nginz.external{Http,Tcp}Port
inside the
nginx-ingress-services
chart anymore.The
config.http.httpPort
andconfig.ws.wsPort
values in thenginz
chart
still configure the ports thenginz
service is listening on.Metrics were moved from
config.http.httpPort
to a newhttp-metrics
port.The
nginz
chart also gained support formetrics.serviceMonitor.enabled
,
creating aServiceMonitor
resource to scrape metrics, like for other wire
services.(#2476)
-
Upgrade webapp version to 2022-06-30-production.0-v0.30.5-0-3e2aaf6 (#2302)
-
In the helm charts, the
wireService
label has been removed.In some cases, we were already setting the
app
label too.Now we consistently use the
app
label to label different wire services.The
wireService
label was also used in thespec.selector.matchLabels
field
on existingDeployment
/StatefulSet
resources.
As these fields being immutable, changing them isn't possible without recreation.If you encounter an issue like
field is immutable && cannot patch "*" with kind *
you need to manually delete these StatefulSet and Deployment resources, and apply helm again, which will recreate them.
This means downtime, so plan a maintenance window for it.
The
wire-server-metrics
chart was previously running some custom
configuration to automatically add all payloads with awireService
label into
metrics scraping.With the removal of the
wireService
label, this custom configuration has been
removed.Instead, all services that expose metrics will now create
ServiceMonitor
resources, if their helm chart is applied withmetrics.serviceMonitor.enable
set to true.This prevents scraping agents from querying services that don't expose metrics
at /i/metrics unnecessarily.Additionally, makes it easier to run other metric scraping operators, like
grafana-agent-operator
, without the need to also create some custom
wireService
label config there.Generally, if you have any monitoring solution installed in your cluster that
uses the Prometheus CRDs, setmetrics.serviceMonitor.enable
for the following charts:- brig
- cannon
- cargohold
- galley
- gundeck
- proxy
- spar (#2413)
API changes
-
The request body of
POST /conversations
endpoint can now contain an optionalcreator_client
field. Thecreator_client
field is only relevant for MLS conversations, in which case it must be set to the ID of the client making the request. (#2486) -
Retire deprecated feature config API endpoints for API version V2 (#2492)
Features
-
Prevent race conditions in concurrent MLS commit requests. (#2525)
-
charts/wire-server: Optionally include backoffice (#2490)
-
The coturn chart has new functionality to enable graceful pod termination, by
waiting for all active allocations on a coturn instance to drain first. When
combined with a suitable external service discovery mechanism which can steer
client traffic away from terminating coturn pods, this can be used to implement
graceful rolling restarts of clusters of coturn instances. (#2456) -
./deploy/services-demo/create_team_members.sh
creates users with given roles now (#2137) -
MLS implementation progress:
- Remote users can be added to MLS conversations
- MLS messages (both handshake and application) are now propagates to remote
conversation participants. (#2415)
-
charts/nginz: Serve swagger-ui for viewing swagger-1.2 docs (#2466)
-
GET teams/:tid
response now contains an optional fieldsplash_screen
which contains the asset key of the team's splash screen.PUT teams/:tid
now supports updating the splash screen asset key. (#2474) -
Missing feature config mapping added (#2494)
-
Add MLS team feature configuration (#2499)
-
Team feature API now includes endpoints to get and set the
searchVisibilityInbound
feature (#2503)
Bug fixes and other updates
-
charts/backoffice: Fix version of frontend and auto-bump version of stern on every release (#2490)
-
The service definitions in the coturn Helm chart were missing the control plane
UDP port used by coturn. (#2500) -
In nginx-ingress-services chart, when enabling useCertManager, now correctly creates the required issuer by default. (#2532)
-
Fix handling of creator client in MLS conversations (#2486)
-
Fix all clients having the same MLS public key (#2501)
-
A user now cannot delete an identity provider that they are authenticated with any more (#2519)
Internal changes
-
brig-types: remove all re-exports (#2505)
-
Fixed flakiness of email update test, related to the test user account being suspended, causing subsequent runs of the test to fail. (#2497)
-
galley-types: remove all re-exports (#2504)
-
Enforce some IdP invariants (#2533)
-
Switch to new MLS test CLI (https://github.com/wireapp/mls-test-cli) (#2508)
-
Forward /i/users/:uid/features/:feature to brig (#2468)
-
charts/nginz: Forward
/i/legalhold/whitelisted-teams
to galley instead of brig (#2460) -
make the ldap-scim-bridge chart deployable once per team, and improve docs. (#1843)
-
Refactored and simplified the feature config API (#2435)
-
Removed deprecated internal feature config API endpoints (#2496)
-
Deactivated gundeck's integration tests for local steps (
make ci
). (#2510) -
retry gundeck's Redis connection in case of network errors such as IP changes or network outages (#2512)
-
Add AWS security token metrics to all services (#2473)
2022-06-14
Release notes
-
Upgrade team-settings version to 4.10.0-v0.29.7-0-3be8ca3 (#2180)
-
Upgrade webapp version to 2022-06-13-production.0-v0.29.7-0-2819b90 (#2302)
Documentation
- Docs for guest links server and team feature settings added (#2480)
Internal changes
- All feature configs like guest links e.g. can now be overridden in the helm configuration, so that they can be disabled/enabled and configured server wide (#2479)
2022-06-08
Release notes
-
The
.cannon.drainTimeout
setting on the wire-server helm chart has been
removed and replaced with.cannon.config.drainOpts
. (#2416) -
Note for wire.com operators: deploy nginz (#2439)
API changes
-
The back-office (aka stern) team feature API now accenpts an optional TTL parameter (in days), so features can be activated for a limited period. (#2417)
-
Disable rate limiting for /api-version (#2439)
Features
-
Drain websockets in a controlled fashion when cannon receives a SIGTERM or
SIGINT. Instead of waiting for connections to close on their own, the websockets
are now severed at a controlled pace. This allows for quicker rollouts of new
versions. (#2416) -
Optionally allow to run cannon with its own nginz inside the same pod; and connect to a load balancer directly.
This allows the cannon-slow-drain behaviour implemented in #2416 to take effect by not having other intermediate network hops which could break websocket connections all at once.
Some (internal) context: https://wearezeta.atlassian.net/wiki/spaces/PS/pages/585564424/How+to+gracefully+drain+cannon+but+not+so+slowly
For details on how to configure this, see docs/src/how-to/install/configuration-options.rst (#2421) -
Support running brig with GeoIP database when using helm charts (#2406)
-
charts/nginz: Add upstream configuration for galeb (#2444)
-
charts/nginz: Allow upstreams to be in other namespaces (#2444)
-
CSV export in team management now includes the number of devices per user (#2407)
Bug fixes and other updates
-
charts/nginz: Resolve collision between brig and galeb endpoints. Ensure
/self/consent and /signatures endpoints are configured in all environments (#2457) -
When an IdP issuer (aka entity ID) is updated, the old issuer was still marked as "in use". (#2400)
-
On actions that require re-authentication a password is not required if the user has SAML credentials (#2430, #2434, #2437)
-
Use SCIM's preferred language as a fallback when privisioning users without a locale. (#2445)
Documentation
- Feature configs should have different swagger schema names (#2425)
Internal changes
-
AllFeatureConfigs
is now typed (#2403) -
Type class for default team feature status (#2404)
-
charts/{redis-ephemeral,legalhold}: Use old index for bitnami repo as the new index doesn't have old versions of postgresql and redis helm charts (#2448)
-
Bump haskell/zlib version to 0.6.3.0 (#2431)
-
New internal brig endpoints for MLS KeyPackage -> Conversation association query/update (#2375)
-
galley: refactor withSettingsOverrides (#2381)
-
charts/{nginz,cannon}: Increase map_hash_bucket_size for nginx to 128 (#2443)
-
charts/{cannon,nginz}: values listed in
nginx_conf.randomport_allowlisted_origins
must be full hostnames. Hostnames
listed here will be allowlisted with and without TLS. (#2438) -
Remove binding of users to saml idps using saml (this has never been picked up by clients; use scim instead) (#2441)
-
Remove golden test case generator
(#2442)
-
Convert Team CSV endpoint to Servant (#2419)
Federation changes
- Send only the raw welcome message in the Galley "mls-welcome" federation endpoint (#2412)
2022-05-18
Release notes
-
If using cert-manager, you need to have least version 1.0.0 (1.8.0 works at the time of writing) installed. Older cert-manager 0.15.X will no longer work. (#2401)
-
Upgrade team-settings version to 4.9.0-v0.29.7-0-142a76f (#2180)
API changes
-
Start version 2 of the public API. Main changes:
- Asset endpoints have lost their
v3
andv4
suffixes. So for example
/assets/v3
has been replaced by/assets
. GET /conversations/:conv/assets/:id
andGET /conversations/:conv/otr/assets/:id
have been removed.GET /assets/:key/v3
has been removed. Use the qualified endpointGET /assets/:domain/:key
instead.DELETE /assets/:key/v3
has been removed. Use the qualified endpoint
DELETE /assets/:domain/:key
instead.GET /connections
has been removed. UsePOST /list-connections
instead.POST /connections
has been removed. UsePOST /connections/:domain/:user
instead.PUT /connections/:domain/:user
has been removed: usePOST
instead.GET /conversations
has been removed. UsePOST /conversations/list-ids
followed byPOST /conversations/list
instead.POST /conversations/list/v2
has been replaced byPOST /conversations/list
.POST /conversations/:domain/:conv/members/v2
has lost itsv2
suffix, so
it is nowPOST /conversations/:domain/:conv/members
.GET /users
,GET /users/by-handle
andGET /users/handles
have been
removed. UsePOST /search/contacts
instead.GET /users/:id
has been removed. Use the qualified endpointGET /users/:domain/:id
instead.GET /users/:id/clients
has been removed. Use the qualified endpointGET /users/:domain/:id/clients
instead.GET /users/:id/clients/:client
has been removed. Use the qualified
endpointGET /users/:domain/:id/clients/:client
instead.
Swagger documentation for the previous version of the API can be accessed at
/v1/api/swagger-ui
. (#2297) - Asset endpoints have lost their
-
A new field
development
has been added to the object returned byGET /api-version
. Versions listed there are considered in flux, meaning that the
corresponding API contracts can change arbitrarily over time. Clients are free
to use development versions, as long as they are also listed insupported
,
and failures due to incompatibilities are acceptable (e.g. in testing
environments). Backends are the authoritative source on whether a development
version can be used at all. If a development version should not be used, the
backend will not list it among the supported versions at all. (#2297)
Features
-
charts: Various new values can now be configured and some got changed
Allow new configurations in the brig chart:
config.emailSMS.user.invitationUrl
config.emailSMS.team.tInvitationUrl
config.emailSMS.team.tActivationUrl
config.emailSMS.team.tCreatorWelcomeUrl
config.emailSMS.team.tMemberWelcomeUrl
config.setProviderSearchFilter
config.setWhitelist
config.setFeatureFlags
config.setCustomerExtensions
If any values in config.emailSMS.team are specified, all must be specified.
Allow new configurations in the gundeck chart:
config.perNativePushConcurrency
config.maxConcurrentNativePushes.soft
config.maxConcurrentNativePushes.hard
Other changes:
- Default
maxTeamSize
changed to 10000 from 500. (#2347)
-
charts/nginx-ingress-services: Allow more fine-grained control over what services are installed. Upgrade Certificate/Issuer resources to 'cert-manager.io/v1' (#2401)
-
MLS implementation progress:
- remote key package claim is now supported (#2353)
-
charts/{brig,cargohold,galley,gundeck}: Allow not configuring AWS credentials and allow using a special service account.
This way, when operating wire in AWS cloud either instance profiles or IAM role attached to a service account can be used to communicate with AWS. (#2347) -
Implement TURN service discovery using SRV records (#2389)
Bug fixes and other updates
-
When
config.enablePayment
andFEATURE_ENABLE_PAYMENT
(envVars
) were set,
the team-settings feature flagFEATURE_ENABLE_PAYMENT
was rendered two times.
The new behavior is to give theenvVars
entry priority. I.e. when it's set,
it's used instead of theconfig.enablePayment
value. (#2332) -
Modify the nginz access control configuration to prevent clients connecting
to listeners with PROXY protocol enabled (such as the websocket listener) from
accessing a private metrics endpoint. (#2307) -
Verification email is sent when external id is updated via SCIM (#2374)
Documentation
-
Move old /docs to /docs/legacy (leaving references). (#2328)
-
Add pagination docs to
POST /list-connections
(#2369) -
Documentation for the 2nd factor password challenge feature (#2329)
-
Documentation on how to enforce desktop application only for web app (#2334)
-
Documentation on how to enforce constant bit rate for all calls (#2336)
-
Documentation on how to disable media plugins for the web app (#2337)
-
Documentation on how to extra entropy in the web app (#2338)
-
Documentation on how to set the instance connection parameters and proxy settings (#2340)
-
Merged SAML/SCIM docs with its main documentation (#2356)
Internal changes
-
View and change team feature permissions apply to all features now (#2402)
-
Add sed to direnv (#2319)
-
Add python3 to nix development environment. It's needed by hack/bin/serve-charts.sh . (#2333)
-
Add a target to the Makefile to run ShellCheck. I.e. to run a linter on shell scripts. This will be used in the CI. For now, all scripts with linter issues are excluded from this check. (#2361)
-
Drop snappy support from bonanza (#2350)
-
Use cabal in buildah-based builds (#2341)
-
Fix flakyness of path traversal test (#2387)
-
Github Actions: disable mac builds (#2355)
-
Apply
versionMiddleware
last. This makes sure that every other middleware sees
the rewritten (unversioned) path. In particular, the prometheus middleware will
now only see paths it knows about, which prevents it from reporting "N/A" as the
path. (#2316) -
Upgrade version of libzauth dependencies, notably sodiumoxide bindings to libsodium, and fix resulting errors and warnings. (#2327)
-
libzauth: Update sha256 for source in nix expression (#2354)
-
Log IO exceptions in Galley and Brig (#2385)
-
Generalise and move the Logger effect (#2306)
-
Fix a comment in a Makefile target (#2330)
-
Fix flaky MLS conversation creation test (#2386)
-
Fix flaky key package test (#2384)
-
Fix locale variables in Nix and .envrc (#2393)
-
Team Member API has been migrated to Servant (#2309)
-
Integration test for edge case: change external id before account registration (#2396)
-
Allow specifying 'redisAdditionalWrite' for a secondary redis to which gundeck will write in the context of a redis migration without downtime. (#2304)
-
Start TURN discovery only when the app starts and not when the Env is created (#2376)
-
Avoid using IN queries for fetching multiple conversations (#2397)
-
Remove oromolu GH action (has been moved to concourse https://github.com/zinfra/cailleach/pull/1033) (#2320)
-
Remove unused data type AllowedUserSearch (#2373)
-
docs: add latex to docs and publish pdf if exists (#2321)
Federation changes
2022-05-04
Release notes
- Upgrade webapp version to 2022-05-04-production.0-v0.29.7-0-a6f2ded (#2302)
2022-04-25
Release notes
-
Note for wire.com operators: deploy nginz (#2270)
-
Wire cloud operators: Update brig's ES index mapping before deploying. After deploying, run a re-index (#2213, #2220)
-
Upgrade webapp version to 2022-04-21-production.0 (#2302)
-
Upgrade team-settings version to 4.7.0-v0.29.7-0-74b81b8 (#2180)
Features
-
[helm-charts] Allow filtering cassandra nodes by datacenter (#2273)
-
MLS implementation progress:
- commit messages containing add proposals are now processed (#2247)
- do initial validation and forwarding of all types of messages via POST /mls/messages (#2253)
- fixed bug where users could not be added to MLS conversations if they had non-MLS clients (#2290)
- MLS/Proteus mismatches (e.g. sending a proteus message to an MLS conversation) are now handled (#2278)
- the
POST /mls/key-packages/claim
endpoint gained askip_own
query parameter, which can be used to avoid claiming a key package for the requesting client itself (#2287)
-
The user profiles that are returned by a team admin search now contain the additional fields SAML NameID, IdP Issuer, and SCIM externalId (#2213), and unvalidated email address (#2220)
-
- Avoid dropping messages when redis is down. (#2295)
Bug fixes and other updates
-
Add missing helm chart mapping for inbound search visibility (#2265)
-
Fix bug: User search endpoint hides exact handle results in SearchVisibilityNoNameOutsideTeam setting (#2280)
-
backoffice app (aka stern):
Documentation
- Import wire-docs to docs/ (see also #2258)
Internal changes
-
Migrate API routes from wai-route to servant for better Swagger (#2284, #2277, #2266, #2286, #2294, #2244)
-
Update nginx to latest stable: v1.20.2 (#2289)
-
Allow additional origins at random ports in nginz Helm chart. This is useful for
testing with an HTTP proxy. It should not be used in production. (#2283) -
makdeb and bonanza: remove stack-based Makefiles (#2311)
-
Add
skip_reauth
param to internal API for creating clients. This is intended to be used in test. (#2260) -
Removes an unused function in Brig and relocates another one (#2305)
-
Print more logs while migrating data in Elasticsearch (#2279)
-
Replace the base monad in Brig with the Polysemy Sem monad (#2264, #2288)
-
Move the Random effect from Spar to the polysemy-wire-zoo library (#2303)
-
Move the Now effect from Spar to a library (#2292)
-
Improve readability of user search test cases (#2276)
-
Chart/gundeck's 'bulkpush' optimization is now activated by default (after using it in production for some time) (#2293)
-
Add an alpha version of a Helm chart for coturn. (#2209)
-
Document error handling and simplify error logging (#2274)
-
Improve speed of reindexing by increasing the batch size of processing users. (#2200)
-
Fix federator integration tests (#2298)
-
Switch the Haskell driver used in Gundeck to connect to Redis from 'redis-io' to 'hedis', which now supports cluster mode. (#2151)
-
Various Galley MLS test improvements and cleanups (#2278)
-
Flag for sending a validation email when updating a user's email address via backoffice/stern (#2301)
-
Remove stack from all builder docker images (#2312)
-
Make internal search-visibility endpoint available to staging environments (#2282)
-
Remove TemplateHaskell as a global default extension (#2291)
2022-04-04
Release notes
-
Note for wire.com operators: deploy nginz (#2175)
-
Deploy galley before brig (#2248)
-
Wire cloud operators: Update brig's ES index mapping before deploying. After deploying run a reindex. (#2241)
-
Upgrade webapp version to 2022-03-30-production.0-v0.29.2-0-d144552 (#2246)
API changes
- New endpoint to get the status of the guest links feature for a conversation that potentially has been created by someone from another team. (#2231)
Features
-
Cross-team user search (#2208)
-
restund chart: add dtls support (#2227)
-
MLS implementation progress:
- welcome messages are now being propagated (#2175)
-
The bot API will be blocked if the 2nd factor authentication team feature is enabled. Please refer to /docs/reference/config-options.md#2nd-factor-password-challenge. (#2207)
-
Translations for 2nd factor authentication email templates (#2235)
-
Script for creating a team with owner via the public API (#2218)
Bug fixes and other updates
-
Conversation rename endpoints now return 204 instead of 404 when the conversation name is unchanged (#2239)
-
Revert temporary sftd bump (#2230)
Internal changes
-
Remove the MonadMask instance for AppT in Brig (#2259)
-
Remove the MonadUnliftIO instance for the app monad in Brig (#2233)
-
Bump hsaml2 version (#2221)
-
Fix: cabal-install-artefacts.sh fails if not run from root of wire-server (#2236)
-
Fix: pushing to cachix not working (#2257)
-
Cannon has been fully migrated to Servant (#2243)
-
Refactor conversation record and conversation creation functions. This removes a lot of duplication and makes the types of protocol-specific data in a conversation tighter. (#2234)
- Move conversation name size check to
NewConv
- Make the
NewConversation
record (used as input to the data
function creating a conversation) contain aConversationMetadata
. - Implement all "special" conversation creation in terms of a general
createConversation
- Move protocol field from metadata to Conversation
- Restructure MLS fields in Conversation record
- Factor out metadata fields from Data.Conversation
- Move conversation name size check to
-
Fix Docs: real-world domain used in examples (#2238)
-
The
CanThrow
combinator can now be used to set the corresponding error effects in polysemy handlers. (#2239) -
Most error effects in Galley are now defined at the granularity of single error values. For example, a handler throwing
ConvNotFound
will now directly declareConvNotFound
(as a promoted constructor) among its error effects, instead of the genericConversationError
that was used before. Correspondingly, all such fine-grained Galley errors have been moved to wire-api as constructors of a single enumerated typeGalleyError
, and similarly for Brig, Cannon and Cargohold. (#2239) -
Add a column for MLS clients to the Galley member table (#2245)
-
Pin direnv version in nix-hls.sh script (#2232)
-
nginx-ingress-services chart: allow for custom challenge solvers (#2222, #2229)
-
Remove unused debian Makefile targets (#2237)
-
Use local serial consistency for Cassandra lightweight transactions (#2251)
2022-03-30
Release notes
- Upgrade webapp version to 2022-03-30-production.0-v0.29.2-0-d144552 (#2246)
2022-03-18
Release notes
- Deploy Brig before Spar. (#2149)
- If you are in a federated network of backends, you need to update all participating instances at the same time. (#2173)
API changes
-
The
client
JSON object now has an additional fieldmls_public_keys
, containing an object mapping signature schemes to public keys, e.g.{ ... "mls_public_keys": { "ed25519": "GY+t1EQu0Zsm0r/zrm6zz9UpjPcAPyT5i8L1iaY3ypM=" } ... }
At the moment,
ed25519
is the only supported signature scheme, corresponding to MLS ciphersuite 1.When creating a new client with
POST /clients
, the fieldmls_public_keys
can be set, and the corresponding public keys are bound to the device identity on the backend, and will be used to verify uploaded key packages with a matching signature scheme.When updating a client with
PUT /clients/:client
, the fieldmls_public_keys
can also be set, with a similar effect. If a given signature scheme already has a public key set for that device, the request will fail. (#2147) -
Introduce an endpoint for creating an MLS conversation (#2150)
-
The
/billing
and/teams/.*/billing
endpoints are now available on a versioned path (e.g./v1/billing
)(#2167)
Features
-
MLS implementation progress:
- key package refs are now mapped after being claimed (#2192)
-
2nd factor authentication via 6 digit code, sent by email:
- for login, sent by email. The feature is disabled per default and can be enabled server or team wide. (#2142)
- for "create SCIM token". The feature is disabled per default and can be enabled server or team wide. (#2149)
- for "add new client" via 6 digit code, sent by email. This only happens inside the login flow (in particular, when logging in from a new device). The code obtained for logging in is used a second time for adding the device. (#2186)
- 2nd factor authentication for "delete team" via 6 digit code, sent by email. (#2193)
- The
SndFactorPasswordChallenge
team feature is locked by default. (#2205) - Details: /docs/reference/config-options.md#2nd-factor-password-challenge
Bug fixes and other updates
-
Fix data consistency issue in import of users from TM invitation to SCIM-managed (#2201)
-
Use the same context string as openmls for key package ref calculation (#2216)
-
Ensure that only conversation admins can create invite links. (Until now we have relied on clients to enforce this.) (#2211)
Internal changes
-
account-pages Helm chart: Add a "digest" image option (#2194)
-
Add more test mappings (#2185)
-
Internal endpoint for re-authentication (
GET "/i/users/:uid/reauthenticate"
) in brig has changed in a backwards compatible way. Spar depends on this change for creating a SCIM token with 2nd password challenge. (#2149) -
Asset keys are now internally validated. (#2162)
-
Spar debugging; better internal combinators (#2214)
-
Remove the MonadClient instance of the Brig monad
- Lots of functions were generalized to run in a monad constrained by
MonadClient instead of running directly in Brig'sAppIO r
monad. (#2187)
- Lots of functions were generalized to run in a monad constrained by
Federation changes
- Refactor conversation actions to an existential type consisting of a singleton tag (identifying the action) and a dedicated type for the action itself. Previously, actions were represented by a big sum type. The new approach enables us to describe the needed effects of an action much more precisely. The existential type is initialized by the Servant endpoints in a way to mimic the previous behavior. However, the messages between services changed. Thus, all federated backends need to run the same (new) version. The deployment order itself does not matter. (#2173)