Testing multiple Puppet versions with TravicCI (and allowing failures)
When it comes to running automated tests of my public Puppet code TravisCI has long been my favourite solution. It’s essentially a zero infrastructure, second pair of eyes, on all my changes. It also doesn’t have any of my local environment oddities and so provides a more realistic view of how my changes will impact users. I’ve had two Puppet testing scenarios pop up recently that were actually the same technical issue once you start exploring them, running tests against the Puppet version I use and support, and others I’m not so worried about.
This use case came up as I have code written for Puppet 3 that I need to
start migrating to Puppet 4 (and probably to Puppet 5 soon) and on the
other hand I have code on Puppet 4 that I’d like to continue
supporting on Puppet 3 until it becomes too much of burden. While I can
do the testing locally with overrides,
rvm and gemfiles, I wanted the
same behaviour on TravisCI.
It’s very easy to get started with TravisCI. Once you’ve signed up (probably with github auth) it only requires two quick steps to get going. The first step is to enable your repo on the TravisCI site.
You should then add a
.travis.yml file to the repo itself. This
contains the what and how of building and testing your code. You can see
a very minimal example, that just runs rake spec with a specific ruby
--- language: ruby rvm: - 2.1.0 script: "bundle exec rake spec"
This provides our basic safety net, but now we want to allow multiple versions
of puppet to be specified for testing. First we’ll modify our
to install a specific version of the puppet gem if an environment
variable is passed in via the TravisCI build config. If this is missing
we’ll just install the newest and run our tests using that. The lines
that implement this, the last five in our sample file, are the important
ones to note.
To support testing under multiple versions of Puppet we’ll modify our
Gemfile to install a specific version of the puppet gem if an environment
variable is passed in, otherwise we’ll just install the newest and run our tests
using that. The code that implements this, last five lines in our sample, are
the important ones to note.
#!ruby source 'https://rubygems.org' group :development, :test do gem 'json' gem 'puppetlabs_spec_helper', '~> 1.1.1' gem 'rake', '~> 11.2.0' gem 'rspec', '~> 3.5.0' gem 'rubocop', '~> 0.47.1', require: false end if puppetversion = ENV['PUPPET_GEM_VERSION'] gem 'puppet', puppetversion, :require => false else gem 'puppet', :require => false end
Now we’ve added this capability to the
Gemfile we’ll modify our
.travis.yml file to take advantage of it. Add an
with a version from each of the two major versions we want to test
under, with the same variable name as we use in our
--- language: ruby rvm: - 2.1.0 bundler_args: --without development script: "bundle exec rake spec SPEC_OPTS='--format documentation'" env: - PUPPET_GEM_VERSION="~> 3.8.0" - PUPPET_GEM_VERSION="~> 4.10.0" notifications: email: firstname.lastname@example.org
.travis.yml is getting a little mode complicated you might want to
lint it to confirm it’s valid. You can use the
online TravisCI linter or install the
TravisCI YAML gem and work offline.
The example file above will trigger two separate builds when TravisCI
receives the trigger from our change. If you want to explicitly test
under two versions of Puppet, and fail the tests if anything breaks under either
version, you are done. Congratulations!
If however you’d like to test against an older, best effort but
unsupported version or start testing a newer version that you’re
willing to accept failures from, assuming the main other version still
passes, while you migrate you’ll need to add another config option
.travis.yml file -
matrix: allow_failures: - env: PUPPET_GEM_VERSION="~> 3.8.0"
In this case (in combination with the config file above) failures under Puppet 4 fail the build, but we allow, and essentially ignore, failures against Puppet 3 as we no longer explicitly support it. If we were planning a move to Puppet 5 we’d add its version here and even on builds that failed we’d start to collect information on what needs to be investigated and fixed while still ensuring our code passes tests under Puppet 4.
I’d also recommend adding an explicit
fast_finish to your matrix
config if you allow_failures. This allows TravisCI to signal when
required tests are finished, even if the results of those allowed to
fail are not yet known, as you don’t need them to know if a run
has been successful or not.
matrix: fast_finish: true allow_failures: - env: PUPPET_GEM_VERSION="~> 3.8.0"
Here’s an example of a build with Allowed Failures in the UI: