Managing multiple puppet modules with modulesync
With the exception of children, puppies and medical compliance frameworks managing one of something is normally much easier than managing a lot of them. If you have a lot of puppet modules, and you’ll eventually always have a lot of puppet modules, you’ll get bitten by this and find yourself spending as much time managing supporting functionality as the puppet code itself.
Luckily you’re not the first person to have a horde of puppet modules that share a lot of common scaffolding. The fine people at Vox Pupuli had the same issue and maintain an excellent tool, modulesync that solves this very problem. With modulesync and a little YAML you’ll soon have a consistent, easy to iterate, on set of modules.
To get started with module sync you need three things, well four if you count the puppet module horde you want to manage.
- a file of repo names you want to manage
- a file of metadata containing how to issue the changes
- a directory of files to keep in sync
I’ve been using modulesync for some of my projects for a while but we recently adopted it for the GDS Operations Puppet Modules so there’s now a full, but nascent, example we can look at. You can find all the modulesync code in our public repo.
First we set up the basic module sync config in
--- git_base: 'email@example.com:' namespace: gds-operations branch: modulesync ... # vim: syntax=yaml
This YAML mostly controls how we interact with our upstream.
is the base of the URL to run git operations against. In our case we
explicitly specify GitHub (which is also the default) but this is easy
to change if you use bitbucket, gitlab or a local server. We treat
namespace as the GitHub organisation modules are under. As we never
push directly to master we specify a branch our changes should be pushed
to for later processing as a pull request.
The second config file,
managed_modules.yml, contains a list of all
the modules we want to manage:
--- - puppet-aptly - puppet-auditd - puppet-goenv
By default modulesync will perform any operations against every module in this file. It’s possible to filter this down to specific modules but there’s only really value in doing that as a simple test. After all keeping the modules in sync is pretty core to the tools purpose.
The last thing to configure is a little more abstract. Any files you
want to manage across the modules should be placed in the
directory and given a
.erb extension. At the moment we’re treating all
the files in this directory as basic, static, files modulesync does
expand them provides a
@configs hash, which contains any values you
specify in the base
config_defaults.yml file. These values can also be
overridden with more specific values stored along side the module itself
in the remote repository.
Once you’ve created the config files and added at least a basic file to
moduleroot, a LICENSE file is often a safe place to start, you can run
modulesync to see what will be changed. In this case I’m going to be working
bundle install # run the module sync against a single module and show potential changes bundle exec msync update -f puppet-rbenv --noop
This command will filter the managed modules (using the
-f flag to select them)
clone the remote git repo(s), placing them under
the branch to either master or the one specified in
modulesync.yml and then
present a diff of changes from the expanded templates contained in
against the cloned remote repo. None of the changes are actually made
thanks to the
--noop flag. If you’re happy with the diff you can add a
commit message (with
-m message), remove
--noop and then run the command again to
push the amended branch.
bundle exec msync update -m "Add LICENSE file" -f puppet-rbenv
Once the branch is pushed you can review and create a pull request as usual.
We’re at a very early stage of adoption so there is a large swathe of
functionality we’re not using so I’ve not mentioned. If you’re actually
moduleroot templates as actual templates you can have a local
override, in each remote module/github repo, that can localise the
configuration and be correctly merged with the main configuration. This
allows you to push settings out to where they’re needed while still
keeping most modules baselined. You can also customise the syncing
workflow to specify bumping the minor version, updating the CHANGELOG
and a number of other helpful shortcuts provided by modulesync.
Once you get above half-a-dozen modules it’s a good time to take a step back and think about how you’re going to manage dependencies, versions, spec_helpers and such in an ongoing, iterative way and modulesync presents one very helpful possible solution.