manifer

Manifer uses yaml processors and variable interpolation to generate yaml documents from a starting template and a set of reusable yaml snippets.

To reduce the size and complexity of yaml documents, named sets of ops files or yq scripts can be organized into a 'library'. The 'template' is now only responsible for simplified high level definitions. Running manifer will combine the library and template to compose the final document.

glossary

template: an arbitrary yaml file to be modified
snippet: a file used to modify the template (e.g. BOSH ops files or yq scripts)
scenario: a named collection of snippets that should be used together
library: manifer's yaml file format that defines a collection of scenarios and dependencies between scenarios
interpolate: replace variable placeholders with values from the CLI, files, or environment variables
process: apply structural changes to a template as defined by a snippet
compose: process and interpolate all snippets defined in a set of scenarios

getting started

1a) generate a library from your collection of opsfiles and yq scripts
manifer import -r -p ./ops-dir -o ./new-lib.yml
or
1b) generate a library from your yaml file
manifer generate -t ./template.yml -y opsfile -o ./new-lib.yml -d ./opsdir
2) view the generated scenarios
manifer list -l new-lib.yml
3) add scenarios for your common use cases, which can define variables or invoke other scenarios
manifer add -l new-lib.yml -n use_case -d "thing I need frequently" -s "dependency_name" -- -v foo=bar -o extra-op.yml
4) inspect the scenario you created
manifer inspect -l new-lib.yml -s use_case
5) use your new scenario to modify a template
manifer compose -l new-lib.yml -t base.yml -s use_case

global flags and environment variables

For convenience there are different ways to specify which libraries to read.
In order of precedence:

a local flag
manifer list -l mylib.yml
a global flag
manifer -l mylib.yml list
specific libraries via MANIFER_LIBS and the system path separator
MANIFER_LIBS=mylib.yml:myotherlib.yml manifer list
directories to search via MANIFER_LIB_PATH and the system path separator
MANIFER_LIB_PATH=./mylibs/:./sharedlibs manifer list

subcommands

import

./manifer import [--recursive] --path <import path> --out <library path>:
  create a library from a directory of snippets.

Usage:
  manifer import [flags]

Flags:
  -h, --help               help for import
  -o, --out string         Path to save generated library file
  -p, --path string        Directory or snippet to import
  -r, --recursive          Import snippets from subdirectories

Global Flags:
  -l, --library strings   Path to library file

generate

./manifer generate --template <yaml path> --out <library path> [--directory <snippet path>]:
  create a library based on the structure of a yaml file.

Usage:
  manifer generate [flags]

Flags:
  -d, --directory string   Directory to save generated snippets (default out/snippets)
  -h, --help               help for generate
  -o, --out string         Path to save generated library file
  -y, --processor string   Yaml backend for this library (opsfile or yq)
  -t, --template string    Template to generate from

Global Flags:
  -l, --library strings   Path to library file

add

./manifer add --library <library path> --name <scenario name> [--description <text>] [--scenario <dependency>...] [-- passthrough flags ...]:
  add a new scenario to a library.

Usage:
  manifer add [flags]

Flags:
  -d, --description string   Informative description of the new scenario
  -h, --help                 help for add
  -n, --name string          Name to identify the new scenario
  -s, --scenario strings     Dependency of the new scenario

Global Flags:
  -l, --library strings   Path to library file

list

./manifer list [--all] (--library <library path>...):
  list scenarios in selected libraries.

Usage:
  manifer list [flags]

Flags:
  -a, --allScenarios   Include all referenced libraries
  -h, --help           help for list
  -j, --json           Print output in json format

Global Flags:
  -l, --library strings   Path to library file

search

./manifer search (--library <library path>...) (query...):
  search scenarios in selected libraries by name and description.

Usage:
  manifer search [flags]

Flags:
  -h, --help   help for search
  -j, --json   Print output in json format

Global Flags:
  -l, --library strings   Path to library file

inspect

./manifer inspect (--library <library path>...) [--tree|--plan] (-s <scenario name>...) [-- passthrough flags ...]:
  inspect scenarios as a dependency tree or execution plan.

Usage:
  manifer inspect [flags]

Flags:
  -h, --help               help for inspect
  -j, --json               Print output in json format
  -p, --plan               Print execution plan
  -s, --scenario strings   Scenario name in library
  -t, --tree               Print dependency tree (default)

Global Flags:
  -l, --library strings   Path to library file

compose

./manifer compose --template <template path> (--library <library path>...) (--scenario <scenario>...) [--print] [--diff] [-- passthrough flags ...] [\;] :
  compose a yml file from snippets. Use '\;' as a separator when reusing a scenario with different variables.

Usage:
  manifer compose [flags]

Flags:
  -d, --diff               Show diff after each snippet is applied
  -h, --help               help for compose
  -p, --print              Show snippets and arguments being applied
  -s, --scenario strings   Scenario name in library
  -t, --template string    Path to initial template file

Global Flags:
  -l, --library strings   Path to library file

appending additional compositions

Additional compositions can be appended using \; as a separator. For each additional composition:

the output of the last composition is used as the template
the list of libraries will be preserved
new libraries can be referenced
new scenarios and passthrough arguments can be specified
global variables cleared

This allows the value of a variable to be changed, without having to re-enter file paths for the libraries or template. The following invocations are equivalent:

./manifer compose -t my-template -l my-library -s my-scenario -- -v arg=foo > temp
./manifer compose -t temp -l my-library -s my-scenario -- -v arg=bar > final

./manifer compose -t my-template -l my-library -s my-scenario -- -v arg=foo \; \
  -s my-scenario -- -v arg=bar > final

schemas

template

Any valid yaml document you would like to modify with snippets and implicit bosh variables

e.g. foo-template.yml

foo:
  bar: bizz
  buzz: ((bazz))
  extra: redundant

snippets

Yaml snippets to compose into the template:

opsfile snippets use go-patch format, Also known as BOSH Ops Files.
yq snippets use yq script format

e.g. base-case.yml

- path: /foo/bar
  type: replace
  value: ((newbar))

- path: /foo/extra
  type: remove

- path: /foo/((sub))? # note: bosh/go-patch do not natively support variables in opsfile paths
  type: replace
  value:
    new: struct

library

library.go

Library files are used to organize sets of snippets, interpolation variables, and processor options to allow easy dynamic composition of large yaml files.

Libraries consist of:

a default processor type for all snippets
type: opsfile

aliases to other libraries

libraries:
- alias: common
  path: ./commonlib.yml

a list of scenarios, consisting of:
- a unique name
- a user-friendly description
- references to other scenarios this scenario depends on:
  - by name, prefixed with . delimited library aliases
  - interpolator variables to apply to the referenced scenario
- snippets to transform the template yaml:
  - path to the snippet file
  - interpolator variables for this snippet
  - processor options for this snippet
- interpolator variables to use with all snippets in this scenario and referenced scenarios
- global interpolator variables to use with all snippets in this composition
```
scenarios:
- name: first
  description: my first scenario
  scenarios:
  - name: second
    interpolator:
      vars:
        foo: bar
  - name: common.setup
  snippets:
  - path: ./opsfile.yml
    interpolator:
      vars:
        bizz: bazz
    processor:
      type: opsfile
      options:
        path: /buzz
- name: second
  snippets:
  - path: ./secondop.yml
  - path: ./thirdop.yml
  global_interpolator:
    vars_store: ./generated.yml
```

migrating from v1

In v1 libraries interpolator variables were specified as CLI args. In v2 args is replaced by the interpolator struct.

When upgrading from manifer v1=>v2 you can either:

move each args element to interpolator.raw_args or
replace args with the appropriate interpolator.var* field

interpolator variables

Variables can be defined by adding an interpolator block to a snippet, scenario reference, scenario, or via passthrough flags from the CLI

interpolator:
  vars: {} # map variable names to static values [--var=key=val (-v)]
  var_files: {} # map variable names to files that contain the value [--var-file=key=path]
  vars_files: [] # file paths that contain a map of variable names to static values [--vars-file=path (-l)]
  vars_env: [] # environment variables with the given prefixes [--vars-env=prefix]
  vars_store: "" # a vars-file that can lazily generate random passwords or certificates [--vars-store=path]
  raw_args: [] # insert CLI flags into the scenario definition (for internal use)

See bosh interpolate and variable types for more details

processor options

A snippet can override the library's default processor type, or provide options

opsfile processor

type: opsfile
options:  
  path: "/foo" # return a section of the composed yaml instead of the full document

See the go-patch and ops-file docs for more details

Differentiating features: field matching and index selection

yq processor

type: yq
options:  
  command: # write, read, delete, prefix, merge (default write)
  path: # yaml element to read or delete
  prefix: # key to nest the current yaml structure under
  overwrite: # boolean for merges to replace existing elements
  append: # boolean for merges to append new array elements

See the yq docs for more details

Differentiating features: wildcards, prefix, and merge

Invocation

Running manifer compose --library mainlib.yml --template foo-template.yml --scenario my-use-case should produce:

foo:
  bar: trendy
  buzz: 123
  nested:
    new: struct

interpolation and variables

There are four variable scopes:

snippet vars
scenario vars
scenario global vars
CLI global vars

Every snippet is used in two interpolations:

the snippet itself is interpolated with snippet vars, scenario vars, and global vars
then the interpolated snippet is applied to the template, interpolated with global vars

multiple libraries

Libraries can include other libraries by specifying the path and an alias under libraries:. If a scenario needs to include a scenario from a referenced library the name should be prefixed with <library alias>..

If multiple independant libraries are provided to the CLI all scenario names should be unambiguous.

build

./scripts/build.sh [all]

all will build manifer_darwin and manifer_linux

test

./scripts/test.sh [unit|integration|go test flags]

-count=1 can be used to disable test caching of integration tests in cmd/manifer

use manifer in your project

lib.Manifer can be imported to list scenarios or compose yaml

package main

import (
  "os"
  "fmt"
  "github.com/cjnosal/manifer/v2/lib"
)

func main() {
  logger := os.Stderr
  output := os.Stdout
  manifer := lib.NewManifer(logger)

  // starting yaml file
  template := "test/data/v2/template.yml"

  // collection of scenarios
  libraries := []string{"test/data/v2/library.yml"}

  // sets of snippets to apply
  scenarios := []string{"placeholder"}

  // arguments to pass through to `bosh interpolate`
  interpolationVars := []string{"-vpath3=/foo", "-vvalue3=tweaks"}

  // list scenario names with descriptions
  scenarioSummary, err := manifer.ListScenarios(libraries, false)
  logger.Write([]byte(fmt.Sprintf("%v\n", scenarioSummary)))
  logger.Write([]byte(fmt.Sprintf("%v\n", err)))

  // apply snippets from the selected scenarios to the provided template
  composedYaml, err := manifer.Compose(template, libraries, scenarios, interpolationVars, false, false)
  output.Write(composedYaml)
  logger.Write([]byte(fmt.Sprintf("%v\n", err)))
}

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

manifer

glossary

getting started

global flags and environment variables

subcommands

import

generate

add

list

search

inspect

compose

appending additional compositions

schemas

template

snippets

library

migrating from v1

interpolator variables

processor options

opsfile processor

yq processor

Invocation

interpolation and variables

multiple libraries

build

test

use manifer in your project

Files

README.md

Latest commit

History

README.md

File metadata and controls

manifer

glossary

getting started

global flags and environment variables

subcommands

import

generate

add

list

search

inspect

compose

appending additional compositions

schemas

template

snippets

library

migrating from v1

interpolator variables

processor options

opsfile processor

yq processor

Invocation

interpolation and variables

multiple libraries

build

test

use manifer in your project