Migrations

Migration to RELEASE v2.1.0

jupyterhub UI is now accessable through same port as other components
Enabling jhub ingress

Enabling host and ingressClassName
- host and ingressClassName should be added in values
```
jupyterhub:
  ingress:
    ingressClassName: nginx
    hosts:
    - <hostname>
```

cassandra dashboard has been corrected by th2 team

Replacing dashboard

changing the name of the dashboard

cassandra dashboard name should be updated in grafana values

grafana:
  dashboards:
    default:
      Cassandra-dashboard:
        url: http://infra-repo.service.svc.cluster.local:8080/dashboards/Cassandra-Dashboard(corrected-by-th2-team).json

Migration to RELEASE v2.0.0

jupyterhub is now included as a dependency and should not be deployed separately. All its values are under jhub parent value.

Update secrets.yaml

Adding Jupyterhub users

Adding Credentials

Values for admin and regular user credentials should be added into secrets.yaml

jupyterhub:
  hub:
    config:
      Authenticator:
        admin_users:
          - <admin-username>
        allowed_users:
          - <username>
      DummyAuthenticator:
        password: <password>

Migrated to new Kubernetes API versions. Now th2-infra supports Kubernetes 1.22-1.26 releases
NGINX Ingress Controller chart must be upgraded 4.1.2 > 4.3.0.
Prometheus-stack should be upgraded 21.0.5 > 41.4.0.
Loki-stack should be upgraded 2.6.5 > 2.8.3.

Infra, Diagnostic and JVM dashboards should be added in grafana during deployment

Adding dashboards

Adding Urls in Prometheus-stack

Infra, Diagnostic and JVM dashboards should be added in grafana, monitoring-old should be removed and monitoring-new should be renamed during deployment

  grafana:
    dashboards:
      default:
        Infra-dashboard:
          url: http://infra-repo.service.svc.cluster.local:8080/dashboards/infra-dashboard.json
        JVM-dashboard:
          url: http://infra-repo.service.svc.cluster.local:8080/dashboards/JVM-dashboard(for-Prometheus-Operator).json
        Diagnostic-dashboard:
          url: http://infra-repo.service.svc.cluster.local:8080/dashboards/diagnostic-dashboard-v1.0.2.json

Editing monitoring dashboards

Edit Urls in Prometheus-stack

The section with monitoring-old and monitoring-new dashboards should be replaced with the following.

  grafana:
    dashboards:
      default:
        Monitoring:
          url: http://infra-repo.service.svc.cluster.local:8080/dashboards/Monitoring.json

converter has been added to infra
Include converter in installation

Adding converter values
- schema repo should be passed to converter.git.repository
```
  converter:
    git:
      repository: <repository>
```
Adding converter key in secrets
- secrets.yaml should convain value from converter-ed25519.key
```
  converter:
    git:
      privateKey: <privateKey>
```

Migration to RELEASE v1.8.0

Migrated to new Kubernetes API versions. Now th2-infra supports Kubernetes 1.19-1.23 releases
Prometheus stack must be upgraded 15.0.0 > 21.0.5
Loki-stack must be upgraded 2.4.1 > 2.6.5. Remove a release before upgrade. Set new values for loki with ones from ../example-values/loki.values.yaml.
NGINX Ingress Controller chart must be upgraded 3.31.0 > 4.1.2.

$ helm install -n service --version=4.1.2 ingress ingress-nginx/ingress-nginx -f ./ingress.values.yaml

HelmOperator dependency upgraded 1.2.0 > 1.4.2. HelmRelease CRD must be removed before infra installation

$ kubectl delete customresourcedefinitions helmreleases.helm.fluxcd.io

infra-mgr secrets are now created automatically.
Infra-mgr secrets should be removed from service namespace

Delete secret infra-mgr
- since infra-mgr secret is automatically created old secret should be deleted (if present)
```
$ kubectl -n service delete secret infra-mgr
```
Delete secret th2-git-access-schemas
- since th2-git-access-schemas secret is automatically created old secret should be deleted (if present)
```
$ kubectl -n service delete secret th2-git-access-schemas
```
secrets.yaml file should be updated
- secrets.yaml should contain value from infra-mgr-rsa.key
```
  infraMgr:
    git:
      privateKey: <privateKey>
```

Dashboards, Dashboard Provider and grafana plugins should be added in grafana during deployment.

Adding Dashboard Provider, Dashboards and plugins in Prometheus-stack

Adding Dashboard Provider in Prometheus-stack

Dashboard Provider should be added in grafana by dashboardProviders.dashboardproviders.yaml.

  grafana:
    dashboardProviders:
      dashboardproviders.yaml:
        apiVersion: 1
        providers:
          - name: 'default'
            orgId: 1
            folder: ''
            type: file
            disableDeletion: false
            editable: true
            options:
              path: /var/lib/grafana/dashboards/default

Adding Dashboard Urls in Prometheus-stack

Dashboards should be added in grafana from infra-repo by Url.

  grafana:
    dashboards:
      default:
        Cassandra-dashboard:
          url: http://infra-repo.service.svc.cluster.local:8080/dashboards/cassandra-dashboard_rev2.json
        Rabbitmq-dashboard:
          url: http://infra-repo.service.svc.cluster.local:8080/dashboards/rabbitmq-overview_rev11.json
        Node-monitoring:
          url: http://infra-repo.service.svc.cluster.local:8080/dashboards/nodes-monitoring-v1.0.0.json
        Namespace-health:
          url: http://infra-repo.service.svc.cluster.local:8080/dashboards/namespace_health-v1.0.2.json
        Components-logs:
          url: http://infra-repo.service.svc.cluster.local:8080/dashboards/components-logs.json
        Monitoring-1:
          url: http://infra-repo.service.svc.cluster.local:8080/dashboards/Monitoring-old.json
        Monitoring-3:
          url: http://infra-repo.service.svc.cluster.local:8080/dashboards/Monitoring-new.json

Adding Plugin Urls in Prometheus-stack

Plugins should be added in grafana from infra-repo by plugins.

  grafana:
    plugins:
      - http://infra-repo.service.svc.cluster.local:8080/plugins/flant-statusmap-panel-0.4.2.zip;flant-statusmap-panel
      - http://infra-repo.service.svc.cluster.local:8080/plugins/vonage-status-panel-1.0.11.zip;vonage-status-panel
      - http://infra-repo.service.svc.cluster.local:8080/plugins/blackmirror1-statusbygroup-panel-1.1.2.zip;blackmirror1-statusbygroup-panel
      - http://infra-repo.service.svc.cluster.local:8080/plugins/grafana-polystat-panel-1.2.8.any.zip;grafana-polystat-panel
      - http://infra-repo.service.svc.cluster.local:8080/plugins/briangann-gauge-panel-0.0.9.zip;briangann-gauge-panel
      - http://infra-repo.service.svc.cluster.local:8080/plugins/yesoreyeram-boomtable-panel-1.4.1.zip;yesoreyeram-boomtable-panel

Rabbitmq values should be updated.
Updating rabbitmq values

Rabbitmq should be adjusted to bitnami chart
- Rabbitmq values should be changed in service.values.
```
  rabbitmq:
    persistence:
      storageClass: local-storage
    ingress:
      hostname: <hostname>
```
Authentication format should be updated
- rabbitmqUsername and rabbitmqPassword should be replaced in secrets.yaml
```
  rabbitmq:
    auth:
      username: th2
      password: rab-pass
      # must be random string
      erlangCookie: cookie
```
Note: Persistence data from previous rabbitmq should be deleted before instalation
Prometheus and Alert Manager should be added in Prometheus-stack.
Update Alert Manager and Prometheus Operator

Add ingress rules to Alert Manager
- ingress rules should be added in alertmanager.
```
  alertmanager:
    alertmanagerSpec:
      externalUrl: http://localhost:9093/alertmanager
    ingress:
      hosts: []
```
Add ingress rules to Prometheus Operator
- ingress rules should be added in prometheusOperator.
```
  prometheus:
    prometheusSpec:
      externalUrl: http://localhost:9090/prometheus
    ingress:
      hosts: []
```
Kube-state-metrics values should be updated
Update kube-state-metrics

Add values to kube-state-metrics
- metricLabelsAllowlist should be added in kube-state-metrics
```
  kube-state-metrics:
    metricLabelsAllowlist: ['pods=[*]','deployments=[*]']
```

InfraGit values have to be be updated.

new persistence configuration

Changing persistence structure

persistence has to be updated to new format.

  infraGit:
    internal: true
    nodePort: 32600
    image:
      repository: ghcr.io/th2-net/git-ssh
      tag: v0.1.0
    persistence:
      enabled: true
      # -- "repos-volume" claim will be created and mounted if empty
      existingClaim: ""

secrets.yaml has to be updated.

Registries have new structure

Changing Registries structure

registries have to updated to new format.

  registries:
    registry1.name.com:8080:
      username: <username>
      password: <password>
    registry2.name.com:8080:
      username: <username>
      password: <password>

ingress.annotations structure changed

Update custom annotations if required

  ingress:
    annotations:
      th2Namespace: {...}
      infraNamespace: {...}
      root: {...}

Migration to RELEASE v1.7.2

Loki must be added in grafana during deployment.
Adding Loki datasource in Prometheus-stack

Adding Loki datasource in Prometheus-stack
- Loki datasource can be added in Prometheus-stack by grafana.additionalDataSources.
```
  grafana:
    additionalDataSources:
    - name: Loki
      access: proxy
      type: loki
      url: http://loki:3100
      jsonData:
        maxLines: "5000"
```
Delete old infra-mgr secret and replace it with new one.
Add New Secret

Add New Secret
- Delete old infra-mgr secret
```
$ kubectl -n service delete secret infra-mgr
```
- Create infra-mgr secret from the private key:
```
$ kubectl -n service create secret generic infra-mgr --from-file=id_rsa=./infra-mgr-rsa.key
```

Migration to RELEASE v1.7.1

Loki stack 2.4.1 chart version must be used during deployment.

Upgrade-loki

Upgrade loki

Loki can be upgraded without additional configuration only if new version uses the same schema version. Current config can be retrieved from cluster:

kubectl get secret -n monitoring loki -o jsonpath="{.data.loki\.yaml}"|base64 -d; echo

schema version is defined in schema_config.schema parameter. Schema of new version loki can be found in chart default values for loki

If schema versions are different should be used transition config for loki. Example of this config:

    schema_config:
      configs:
      - from: "2018-04-15"
        index:
          period: 168h
          prefix: index_
        object_store: filesystem
        schema: v9
        store: boltdb
      - from: "2022-01-22"
        store: boltdb-shipper
        object_store: filesystem
        schema: v11
        index:
          prefix: index_
          period: 24h
    storage_config:
    # because boltdb is used in old schema we need to define this storage
      boltdb:
        directory: /data/loki/index

Helm operator chart now is included as dependency and should not be deployed separately. All its values are under helmoperator parent value.
Kubernetes dashboard chart now is included as dependency and should not be deployed separately. Previous deployment must be uninstalled from "monitoring" namespace. All its values are under dashboard parent value.

More information about seamless migration between schemas: https://grafana.com/docs/loki/v2.2.0/storage/#schema-configs https://grafana.com/docs/loki/v2.2.0/configuration/#schema_config

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MIGRATION.md

MIGRATION.md

Migrations

Migration to RELEASE v2.1.0

Enabling host and ingressClassName

changing the name of the dashboard

Migration to RELEASE v2.0.0

Adding Credentials

Adding Urls in Prometheus-stack

Edit Urls in Prometheus-stack

Adding converter values

Adding converter key in secrets

Migration to RELEASE v1.8.0

Delete secret infra-mgr

Delete secret th2-git-access-schemas

secrets.yaml file should be updated

Adding Dashboard Provider in Prometheus-stack

Adding Dashboard Urls in Prometheus-stack

Adding Plugin Urls in Prometheus-stack

Rabbitmq should be adjusted to bitnami chart

Authentication format should be updated

Add ingress rules to Alert Manager

Add ingress rules to Prometheus Operator

Add values to kube-state-metrics

Changing persistence structure

Changing Registries structure

Migration to RELEASE v1.7.2

Adding Loki datasource in Prometheus-stack

Add New Secret

Migration to RELEASE v1.7.1

Upgrade loki

Files

MIGRATION.md

Latest commit

History

MIGRATION.md

File metadata and controls

Migrations

Migration to RELEASE v2.1.0

Enabling host and ingressClassName

changing the name of the dashboard

Migration to RELEASE v2.0.0

Adding Credentials

Adding Urls in Prometheus-stack

Edit Urls in Prometheus-stack

Adding converter values

Adding converter key in secrets

Migration to RELEASE v1.8.0

Delete secret infra-mgr

Delete secret th2-git-access-schemas

secrets.yaml file should be updated

Adding Dashboard Provider in Prometheus-stack

Adding Dashboard Urls in Prometheus-stack

Adding Plugin Urls in Prometheus-stack

Rabbitmq should be adjusted to bitnami chart

Authentication format should be updated

Add ingress rules to Alert Manager

Add ingress rules to Prometheus Operator

Add values to kube-state-metrics

Changing persistence structure

Changing Registries structure

Migration to RELEASE v1.7.2

Adding Loki datasource in Prometheus-stack

Add New Secret

Migration to RELEASE v1.7.1

Upgrade loki