How to iterate overy values in array's

[EmaLinuxawy]

Hello Community

I am using YTT to generate multiple pipeline templates, so i want to iterate over data values in an array to get the values based on it's name to the respective job.

i would like to iterate over this job to get an output with 3 build jobs with the name of the job based on each one inside stages array, then iterate over the repo from it's array and just get the value name without the key, and the same thing with the params

#@ def build(name, repo, params):
name: #@ name 
plan:
  - in_parallel:
       - get: #@ repo
         trigger: true
  - task: build
     file: path/to/file.yaml
     params: #@ params
#@ end

jobs:
#@ for/end stage in [s for s in data.values.build.stages]:
  - #@ build()

and this is my data/values:

build:
  stages: ["dev", "stage", "prod"]
  repo:  ["repo-dev", "repo-stage", "repo-prod"]
  params: ["dev", "stage", "prod"]

lastly, I have data-values-file with the following:

repo-dev: dev-repo-app
repo-stage: dev-stage-app
repo-prod dev-prod-app

dev: 
  key: value
  key2: value2

stage: 
  key: value
  key2: value2

prod:
  key: value
  key2: value2

[mamachanko]

@EmaLinuxawy if you are able to restructure you data values, then you can make templating a little easier.

Consider the following:

#! values.yaml
#@data/values
---
builds:
  - name: dev
    repo: dev-repo
    params:
      dev-param1: dev-value1
  - name: prod
    repo: prod-repo
    params:
      prod-param1: prod-value1

#! config.yaml
#@ load("@ytt:data", "data")

#@ def job(build):
name: #@ build.name
plan:
  - in_parallel:
      - get: #@ build.repo
        trigger: true
  - task: build
    file: path/to/file.yaml
    params: #@ build.params
#@ end

jobs:
  #@ for/end build in [s for s in data.values.builds]:
  - #@ job(build)

Running ytt -f config.yaml -f values.yaml yields:

jobs:
- name: dev
  plan:
  - in_parallel:
    - get: dev-repo
      trigger: true
  - task: build
    file: path/to/file.yaml
    params:
      dev-param1: dev-value1
- name: prod
  plan:
  - in_parallel:
    - get: prod-repo
      trigger: true
  - task: build
    file: path/to/file.yaml
    params:
      prod-param1: prod-value1

[EmaLinuxawy]

@mamachanko Thanks a lot for your answer, the idea is some pipeline that i need to generate they have more than 14 different jobs, and each job has around 7 lines of params, so I would like to make the data/values file is simple as possible, and the other file that contains only the parameters

[mamachanko]

and the other file that contains only the parameters

Just to avoid a misunderstanding, but they would all be data values files.

I understand that you would want to normalize params across files. With ytt there are many ways to achieve your desired outcome. I am going to show you how to do it with a schema. When using a values schema you can provide multiple data values files and they get merged.

Consider the following:

#! config.yaml
#@ load("@ytt:data", "data")

#@ def job(name, repo, params):
name: #@ name
plan:
  - in_parallel:
      - get: #@ repo
        trigger: true
  - task: build
    file: path/to/file.yaml
    params: #@ params
#@ end

---
jobs:
  #@ for/end build in [s for s in data.values.builds]:
  - #@ job(build.name, build.repo, data.values.params[build.name])

#! value-schema.yaml
#@data/values-schema
---
builds:
  - name: ""
    repo: ""
params:
  #@schema/type any=True
  dev: null
  #@schema/type any=True
  prod: null

#! values-builds.yaml
#@data/values
---
builds:
  - name: dev
    repo: dev-repo
  - name: prod
    repo: prod-repo

#! values-dev.yaml
#@data/values
---
params:
  dev:
    dev-param1: dev-value1

#! values-prod.yaml
#@data/values
---
params:
  prod:
    prod-param1: prod-value1

Yields:

❯ ytt -f .
jobs:
- name: dev
  plan:
  - in_parallel:
    - get: dev-repo
      trigger: true
  - task: build
    file: path/to/file.yaml
    params:
      dev-param1: dev-value1
- name: prod
  plan:
  - in_parallel:
    - get: prod-repo
      trigger: true
  - task: build
    file: path/to/file.yaml
    params:
      prod-param1: prod-value1

[vrabbi]

this is another way you can do it
https://carvel.dev/ytt/#gist:https://gist.github.com/vrabbi/642414d8ea5a13c7f4cb070bc5d025c1

[EmaLinuxawy]

That was actually what i'm looking for, Thanks a lot @vrabbi