An important aside: Currently running acceptance tests that contain a specific change is challenging unless you have access to infrastructure internal to the Puppet, Inc. network. This is a known issue, and we are working to make this a better experience for our community.
- git
- ruby
- bundler
- a local clone of the puppet repo
All command examples in this readme assume you are working in the same directory
this README is in, puppet/acceptance.
All of the dependencies you need to run and develop tests are defined in
Gemfile. To install them, run bundle install --path .bundle/gems. This
command, as well all the command examples in this README, assume you are working
in the acceptance directory. If you ever have issues with your runtime
dependencies, you can update them with bundle update or start over fresh with
rm -rf .bundle/gems; bundle install --path .bundle/gems.
To ensure installation was successful, you can run bundle exec rake -T. This
should return something along these lines:
$ bundle exec rake -T
rake ci:help # Print usage information
rake ci:test:aio # Run the acceptance tests using puppet-agent (AI...
rake ci:test:gem # Run the acceptance tests against puppet gem on ...
rake ci:test:git # Run the acceptance tests against a git checkout
rake ci:test:quick # Run a limited but representative subset of acce...
rake clean # Remove any temporary products
rake clobber # Remove any generated files
To get a detailed description of all of these tasks, run bundle exec rake -D.
Currently, there isn't a good way for community members to run acceptance tests. This is a known problem. We currently have multiple avenues we are exploring to make running puppet acceptance tests easier for our community. In the meantime, we apologize for the inconvenience.
If you have access to infrastructure internal to the Puppet, Inc. network, then the quickest way to get acceptance tests running is with vmpooler.
To test changes that are available on a branch on github.com:
bundle exec rake ci:test:git OPTIONS='--preserve-hosts=always' SHA=ticket/6.0.x/ticketed-work-description RUNTIME_BRANCH=6.0.x FORK=melissa TESTS='tests/path/to/test.rb,tests/other/test.rb'
Where SHA is the branch name, RUNTIME_BRANCH is the agent version stream,
and FORK is the github fork where the branch lives.
To test changes that are available in a puppet-agent package on builds.delivery.puppetlabs.net:
bundle exec rake ci:test:aio OPTIONS='--preserve-hosts=always' SHA=9124b4e81ec0ac6394d3edc67d4ab71866869fd7 TESTS='tests/path/to/test.rb,tests/other/test.rb'
SHA is a sha or tag that exists on builds.delivery.puppetlabs.net/puppet-agent
To rerun a test on the hosts that have already been provisioned, use beaker subcommands:
bundle exec beaker exec tests/path/to/test.rb,tests/other/test.rb
Always clean up after yourself when you are done:
bundle exec beaker destroy
This will remove any provisioned hosts. Only run this once you are done with the hosts that have been checked out and provisioned for a given run.
A detailed description of the available environment variables can be found by
running bundle exec rake ci:help. This will print a list of both required and
optional environment variables with short descriptions on how they are used.
Please review all of these options as they will impact how your test servers
are provisioned. This rake task is the most up to date source for this
information. Please read through the available variables, their defaults, and
what they do. They may impact your acceptance run in ways you do not expect.
If you are using the vmpooler hypervisor internal to Puppet, Inc. infrastructure,
you can customize the platforms to test on using the HOSTS environment variable.
You'll set the HOSTS environment variable to the host string you want to test,
such as HOSTS=redhat7-64ma-windows2012r2-64a.
For a list of available HOSTS platforms and their exact naming structures,
check the keys listed in beaker hostgenerator. Generally, this string will be in the format
{platform}{version}-{architecture}{role/s}. You will most often use either the
agent (a) or master (m) role, but you can find a list of available roles in
beaker hostgenerator.
Multiple hosts in the string are separated with a dash(-). You must have at
least one agent and at least one master.
Be careful not to confuse the different host string formats. We have different
tools that expect the host string to be in different forms. For example,
packaging_platform is specific to how Vanagon
parses that string.
The rake tasks that run acceptance will by default create a hosts file and
populate it using beaker-hostgenerator using either the HOSTS environment
variable or the default host string (currently redhat7-64ma-windows2012r2-64a).
The automation assumes you are using the vmpooler hypervisor and a vmpooler
instance that is only available to Puppet, Inc. employees. If you want to
customize the hypervisor or the vmpooler instance, you'll need to generate your own
hosts file. You must pass in a valid host string to the beaker-hostgenerator
command. See Customizing Test Targets for more
information on how to construct a valid host string.
To customize the hypervisor, pass in --hypervisor {hypervisor name}. To set
the vmpooler instance, use --global-config pooling_api={vmpooler uri}. Only the
vmpooler hypervisor uses the pooling_api key.
The host string that is passed in is the same that you would use with the
HOSTS environment variable. See Customizing Test Targets
on how to format this string.
To have the automation recognize and use your custom hosts file, you'll need to
set the HOSTS environment variable to the hosts file. In the above example, we
called this file hosts.yaml, so we will set HOSTS to hosts.yaml when running
all future beaker commands or rake tasks to run acceptance tests.
For example, if you were to run this command:
bundle exec beaker-hostgenerator redhat7-64ma-windows2012r2-64a --disable-default-role --osinfo-version 1 --hypervisor vmpooler --global-config pooling_api=http://customvmpooler/ > hosts.yaml
You would generate a file called hosts.yaml that contains something like this:
---
HOSTS:
redhat7-64-1:
platform: el-7-x86_64
packaging_platform: el-7-x86_64
template: redhat-7-x86_64
hypervisor: vmpooler
roles:
- master
- agent
windows2012r2-64-1:
platform: windows-2012r2-64
packaging_platform: windows-2012-x64
ruby_arch: x64
template: win-2012r2-x86_64
hypervisor: vmpooler
roles:
- agent
CONFIG:
nfs_server: none
consoleport: 443
pooling_api: http://customvmpooler/
We can then run the acceptance tests with:
bundle exec rake ci:test:aio HOSTS=hosts.yaml SHA={sha}
The hypervisor dictates where you will be running the acceptance tests. The beaker hypervisors take care of basic host setup so that you will have a consistent host environment across every test run. You can find more details on the different hypervisor options in the beaker repo.
Here, we will focus on vmpooler and docker, as those are the two we use most
often internally. If you use a hypervisor other than abs, vagrant, vmpooler, or
docker, you'll have to add the gem to that hypervisor to Gemfile.local and run
bundle update to install the new gems. You also have the ability to run tests
on a static host.
VMPooler is the default hypervisor we
use. This is only available to Puppet, Inc. employees as it uses internal
infrastructure. If you have access to a similar setup, then you are welcome to
use this option with a few values changed. If you are using the Puppet internal
vmpooler, then you can simply run the acceptance rake tasks. See
Customizing Test Targets about how to use the
HOSTS environment variable to customize the platforms you are running tests on.
To use a different vmpooler instance, use
--global-config pooling_api=http://customvmpooler/ when you use
beaker-hostgenerator to generate hosts.yaml. Make sure you set HOSTS to
the hosts file you just generated so the automation can find that file. See
The Hosts File for more detail on the hosts file.
To test with the docker hypervisor, you will want to generate a custom hosts file. You will also mostly likely need to manually edit the file. See The Hosts File for more detail on the hosts file.
To create a hosts file with a centos 7 master and a centos 7 agent, we can use
the following beaker-hostgenerator command
bundle exec beaker-hostgenerator centos7-64m-centos7-64a --disable-default-role --osinfo-version 1 --hypervisor docker > hosts.yaml
Which will produce a file called hosts.yaml that contains the following:
---
HOSTS:
centos7-64-1:
docker_cmd:
- "/sbin/init"
image: centos:7
platform: centos-7-x86_64
packaging_platform: el-7-x86_64
docker_image_commands:
- cp /bin/true /sbin/agetty
- yum install -y crontabs initscripts iproute openssl sysvinit-tools tar wget
which ss
hypervisor: docker
roles:
- master
centos7-64-2:
docker_cmd:
- "/sbin/init"
image: centos:7
platform: centos-7-x86_64
packaging_platform: el-7-x86_64
docker_image_commands:
- cp /bin/true /sbin/agetty
- yum install -y crontabs initscripts iproute openssl sysvinit-tools tar wget
which ss
hypervisor: docker
roles:
- agent
CONFIG:
nfs_server: none
consoleport: 443
Run acceptance tests against pre-built puppet-agent packages with
bundle exec rake ci:test:aio SHA={sha|tag} TESTS=path/to/test.rb HOSTS=hosts.yaml
Note that if you are not running tests against the master branch and you are
installing the latest puppetserver package, you will likely need to set RELEASE_STREAM
to pick up the correct server version. Please see the section on environment variables
for more information.
When you generate your hosts file, beaker-hostgenerator does its best to populate the values as logically as possible. You will likely want to update or modify them to suite your needs.
With image, beaker-hostgenerator does its best to guess the most logical
image string based on the platform you are building. For the most part, this
should work without interference, but if you are using a custom docker image or
do not want the default, then you will have to manually update this string. Not
every string beaker-hostgenerator uses to populate this variable will be valid.
docker_image_commands is automatically populated when generating the hosts
file with beaker-hostgenerator. This has already been set for a handful of
host types, but may not be set for all.
- TODO I only tried this once using a docker image that already had puppetserver installed as the master host. The image I used took forever to provision, so I gave up. If we want to continue down this route, we need to make sure the setup steps can check if puppetserver has already been installed so that we don't try to install it agian.
- TODO There's something odd with
docker_mage_entrypointversusdocker_cmd. We should clarify the difference between these two values. I don't quite understand what the difference is between them. - TODO These docker containers have to run in privileged mode (or systemd, among possibly other things, won't function as we need them to). This is not ideal if you're testing code that affects your OS (ie running docker on linux without a docker machine in between the container and your laptop). BE CAREFUL
This is not recommended unless you are familiar with how beaker and beaker-puppet provision hosts.
To test on a server that's already been spun up or doesn't require a hypervisor, you should set the name of the host to the FQDN of the server you want to use, then remove the hypervisor and template settings. This is not recommended, and you may run into issues with failures or overwritten configuration due to either beaker provision steps or test provisioning steps.
---
HOSTS:
azeqdqmk14mvu3g.delivery.puppetlabs.net:
platform: el-7-x86_64
packaging_platform: el-7-x86_64
roles:
- master
bundle exec rake ci:test: SHA={sha|tag}
This is the primary method that we use to run puppet acceptance tests. It
requires puppet-agent packages that have been built with the version of the
puppet code that you want to test. As building packages usually takes quite a
bit of time, this method requires some patience. You are required to set SHA
when running acceptance tests against pre-built packages.
If you are testing a specific version, SHA must be set to a value that exists
on the path #{ENV['DEV_BUILDS_URL']}/puppet-agent/#{ENV['SHA']}. Note that
this value corresponds to the puppet-agent package, not to puppet.
DEV_BUILDS_URL defaults to the internal build server that is only accessible
to Puppet, Inc. employees. The method called here depends on information written
to a yaml file in that directory. Though you can override DEV_BUILDS_URL, the
automation here is very specific and likely will not work as you are expecting
it to.
bundle exec rake ci:test:aio SHA=3cfbac6857c10efc5b1e02262cfd7b849bb9c4b2
bundle exec rake ci:test:aio SHA=6.0.5
If you do not have access to internal infrastructure, you can test against
packages that have been made available on nightlies.puppet.com. Currently, you
cannot specify a specific version. Instead, you have to use the latest shipped
package for the release stream you are interested in. To do this, SHA must be
set to latest. If you want to modify the release stream you are testing,
RELEASE_STREAM can be modified. It defaults to puppet which should
correspond to the latest stream available. If you want to modify
RELEASE_STREAM, set it to an available repo, such as puppet5.
bundle exec rake ci:test:aio SHA=latest RELEASE_STREAM=puppet5
bundle exec rake ci:test:git SHA={sha|tag|branch}
Though we primarily run acceptance tests against a built package, it is possible to run these tests with a git checkout. This is most useful when testing locally to speed up the feedback cycle.
When testing from a github repo we need to unpack the appropriate runtime archive for the platform we are testing on. These pre-built archives are stored on an internal server, and are currently only available to Puppet, Inc. employees. With these archives, we get all of the runtime dependencies that are usually provided as a part of the puppet agent package. This allows us to replicate the runtime environment produced via a package install for the purpose of running acceptance tests.
When testing with git, SHA can be set to any git artifact: a long sha, a short
sha, a tag, a branch name, etc. What happens is that we write a gemfile with the
details of the puppet repo, pointing to the artifact referenced with SHA. Then
when we run bundle install on the testing host, bundler grabs puppet from
wherever the gemfile points. If the git artifact referenced is not from the
puppetlabs repo, you can use FORK to point to a different github namespace.
Likewise, if the artifact you want to access is not available on github.com
but a custom git server, you can set SERVER to customize the git uri bundler
pulls from. For more details on these environment variables, run
bundle exec rake ci:help.
As an example, if I have a development branch
(developent/master/major-feature) that I'm working on and it only exists in my
fork of puppet (github.com/joeschmoe/puppet), then I will run
bundle exec rake ci:test:git SHA=developent/master/major-feature FORK=joeschmoe
Please note that any changes you want to test must be pushed up to your github server. This is how we access the code to be tested.
If you are testing with git and using the docker hypervisor, you can run tests
against the puppet checkout on your local system. You need to update your hosts
file to add mount_folders to the docker host where you want the checkout of
puppet to be available. Here, host_path is the path to puppet on your local
machine. The container_path is where puppet will end up on the docker image,
so you can leave it as /build/puppet. Note that although SHA is required, it
is never used in this workflow. For consistency, I would recommend setting SHA
to your working branch name.
We still need access to our runtime dependencies when testing against a local
git checkout. When we are testing with the docker hypervisor, we assume that the
docker image you are using will have this. As of this writing (Jan. 2019), the
docker image you'll want to use for these tests is not public. The image is
called agent-runtime-{branch}, where {branch} is the branch of puppet you
are testing. This image includes everything we build as a part of the runtime
archive. These components are
normally provided as a part of the puppet agent package.
---
HOSTS:
debian8-64-1:
hypervisor: docker
docker_image_entrypoint: "/sbin/init"
image: pcr-internal.puppet.net/pe-and-platform/agent-runtime-master:201810110.17.gb5afc66
platform: debian-8-amd64
packaging_platform: debian-8-amd64
docker_image_commands:
- rm -f /usr/sbin/policy-rc.d
- systemctl mask [email protected] getty-static.service
- apt-get update && apt-get install -y cron locales-all net-tools wget
mount_folders:
puppet:
host_path: ~/puppet
container_path: /build/puppet
roles:
- agent
For more details on testing with docker, see the docker section.
Remember that HOSTS must be set to your hosts file for the automation to honor
it.
Currently, running acceptance tests with gems is not working.
bundle exec rake ci:test:gem
The rake tasks we use here take advantage of a newer feature in beaker that gives us quite a bit of flexibility. We take advantage of beaker subcommands. Subcommands are individual beaker invocations that are used to run the different stages of running tests: provisioning, pre-suite setup, tests, etc. We do this by writing state to the file .beaker/subcommand_options.yaml. With each new invocation of a subcommand, beaker will check for this file and load the contents if the file exists. The important thing about this feature is that you can rerun tests without going through the entire provisioning process every time.
To ensure your hosts aren't cleaned up after a run, set OPTIONS='--preserve-hosts=always'. With this set, we can rerun a failed test using the infrastructure beaker has already provisioned.
bundle exec rake ci:test:aio OPTIONS='--preserve-hosts=always' SHA=6.0.5
If this run fails because a small handful of tests fail, you can rerun only those tests that failed. For example, assume that tests/resource/package/yum.rb and tests/node/check_woy_cache_works.rb both had failing tests. you can run
bundle exec beaker exec tests/resource/package/yum.rb,tests/node/check_woy_cache_works.rb
This should work regardless of which hypervisor or testing method you are using.
Read more about writing beaker tests in beaker. Check out the tutorials section and how to write a quick test
- Slack (slack.puppet.com) #testing, #puppet-dev, #windows