UNIXDAEMON Small Mosaic

Categories:

/books
/career
/cloud
/events
/firefox
/geekstuff
/linux
/meta
/misctech
/movies
/nottech
/perl
/programming
/security
/sites
/sysadmin
/tools
/tools/ansible
/tools/commandline
/tools/gui
/tools/network
/tools/online
/tools/puppet
/unixdaemon

Archives:

March 20146
February 20141
January 20145
December 20135
November 20132
October 20134
August 20134
July 20132
June 20131
May 20131
April 20131
March 20131
Full Archives


Tue, 25 Mar 2014

Managing CloudFormation Stacks with Ansible

Constructing a large, multiple application, virtual datacenter with CloudFormation can quickly lead to a sprawl of different stacks. The desire to split things sensibly, delegate control of separate tiers and loosely couple as many components as possible can lead to a large number of stacks, lots of which need values from stacks created earlier in the run order. While it's possible to do this with the native AWS CloudFormation command line tools, or even some clever bash (or Cumulus), having a strong, higher level tool can make life a lot easier and reproducible. In this post I'll show one possible way to manage interrelated stacks using Ansible.

We won't be delving into the individual templates used in this example. If you're having this kind of issue with CloudFormation then you probably have more than enough of your own to use as examples. Instead, I'll show a basic Ansible playbook for managing three related stacks.


---
- hosts: localhost
  connection: local
  gather_facts: False
  vars:
    stack_name: dswtest
    region: eu-west-1
    owner: dwilson
    ami_id: ami-n0tr34l
    keyname: key-64
    snsdest: test@example.org

The first part of our playbook should be familiar to most Ansible users. We set up where to run the playbook, how to connect and ensure we don't spend time gathering facts. We then define the variables that we'll be using as parameters to a number of stacks. The ability to specify literals in a single place was the first benefit I saw when converting a project to Ansible. This may not sound like a major win but being able to change the AMI ID in a single place, or even store it in an external file that our build system can automatically update, is something I'd find difficult to give up.

Now we'll move to the first of our Ansible tasks, a CloudFormation stack represented as a single Ansible resource. The underlying template creates a basic SNS resource we'll later use in all our auto-scaling groups.



  tasks:
  - name: Add SNS topic
    action: cloudformation
      stack_name={{ stack_name }}-sns-email-topic
      state=present
      region="{{region}}"
      template=sns-email-topic.json
    args:
      template_parameters:
        AutoScaleSNSTopic: "{{snsdest}}"
    register: asgsns

The 'args:' section contains the values we want to pass in to the template. Here we're only passing a single value that we defined earlier in the 'vars:' section. We'll see more complicated examples of this later. We also register the output from the CloudFormation action. This includes any values we specify as "Outputs" in the template and provides a nice way to deliberately define what we're exposing from our template. The alternative is to pull out arbitrary values from a given resource created in a previous stack but that's a hefty breach of encapsulation and will often bite you later when the templates change.

The Create Security Groups CloudFormation task doesn't really have anything interesting from an Ansible perspective, we run it, create the repos and gather the outputs using 'register' for use in our next template.



  - name: Create Security Groups
    action: cloudformation
      stack_name={{ stack_name }}-security-groups
      state=present
      region="{{region}}"
      template=security-groups.json
    register: secgrp

The 'Create Webapp' example below shows most of the basic CloudFormation resource features in a single task. We use variables defined at the start of the playbook to reduce duplication of literal strings. We prefix the stack names to allow multiple developers to each build full sets of stacks without duplicate stack name conflicts while keeping grouping simple in the AWS web dashboard.



  - name: Create Webapp
    action: cloudformation
      stack_name={{ stack_name }}-webapp
      state=present
      region="{{region}}"
      template=-webapp.json
    args:
      template_parameters:
        Owner: "{{ owner }}"
        AMIId: "{{ ami_id }}"
        KeyName: "{{ keyname }}"
        AppServerFleetSize: 1
        ASGSNSArn:             "{{ asgsns['stack_outputs']['EmailSNSTopicARN']      }}"
        WebappSGID:            "{{ secgrp['stack_outputs']['WebappSGID']            }}"
        ElasticacheClientSGID: "{{ secgrp['stack_outputs']['ElasticacheClientSGID'] }}"

In the args section we also use the return values from our previous stacks. The nested value access is a little verbose but it's easy to pickup and being able to see all the possible values when running Ansible under debug mode makes things a lot easier. We also had the need to pull down output values from stacks created outside of Ansible, so I wrote a simple Ansible CloudFormation lookup plugin.

So what does Ansible gain us as a stack management tool? In terms of raw CloudFormation it provides a nice way to remove boilerplate literals from each stack and define them once in the 'vars' section. The ability to register the output from a stack and then use it later on is an essential one for this kind of stack building and retrieving existing values as a pythonish hash is much easier than doing it on the command line. As for added power, it should be easier to implement AWS functionality that's currently missing from CloudFormation as an Ansible module than a CloudFormation external resource (although more on that when I actually write one) and performing other out of band tasks, letting your ticketing system know about a new stack for example, is a lot easier to integrate into Ansible than trying to wrap the cli tools manually.

I've been using Ansible for stack management in a project that involves over a dozen separate moving parts for the last month and so far it's been working fine with minimal pain.

Posted: 2014/03/25 23:38 | /tools | Permanent link to this entry


Sat, 22 Mar 2014

Project Book Pages

I've been doing my usual quarterly sweep of the always too full bookshelves and hit the usual dilemma of what to keep, what to donate to charity and what to recycle. Among the technical books in this batch is the 'Sendmail Cookbook', something I've always kept as a good luck charm to ward off the evil of needing to work with mail servers with m4 based configuration languages.

Sendmail is one of those projects that I've not kept up with over the years. I have no idea how much has changed since the book was published over a decade ago, 2003 in this case, so I don't know if this is a useful book to pass on or if it's dangerously out of date and should be removed from circulation. It'd be handy if the larger projects maintained a page of books related to the project and a table of how relevant the material is in relation to different versions.

This would not only help me prune my shelves of older, now out of date books, but would help people new to a project pick books that were still relevant for the versions they need to learn.

Posted: 2014/03/22 15:30 | /books | Permanent link to this entry


Mon, 17 Mar 2014

Managing CloudFormation Stacks With Cumulus

Working with multiple, related CloudFormation stacks can become quite taxing if you only use the native AWS command line tools. Commands start off gently -



cfn-create-stack dwilson-megavpc-sns-emails --parameters "AutoScaleSNSTopic=testy@example.org" \ 
  --template-file location/sns-email-topic.json


- but they quickly become painful. The two commands below each create stacks that depend on values from resources that have been defined in a previous stack. You can spot these values by their unfriendly appearance, such as 'rtb-9n0tr34lac55' and 'subnet-e4n0tr34la'.



# Add the bastion hosts template

cfn-create-stack dwilson-megavpc-bastionhosts --parameters \ 
"
VPC=vpc-n0tr34l;BastionServerSNSArn=arn:aws:sns:us-east-1:14204989:fooo;
PrivateNATRouteTableAZ1=rtb-9n0tr34lac55;PrivateNATRouteTableAZ2=rtb-6n0tr34l5e0a;
PublicSubnetAZ1=subnet-3n0tr34l5c;PublicSubnetAZ2=subnet-e4n0tr34la;
KeyName=dwilson; \
" --template-file bastion.json


# create the web/app servers
cfn-create-stack dwilson-megavpc-webapps --parameters \
"
VPC=vpc-65n0tr34lb;BastionserverSG=sg-n0tr34l;PrivateNATRouteTableAZ1=rtb-n0tr34l;
PrivateNATRouteTableAZ2=rtb-n0tr34l;PublicSubnetAZ1=subnet-n0tr34l;
PublicSubnetAZ2=subnet-n0tr34l;KeyName=dwilson;WebServerSNSArn=arn:aws:sns:us-east-1:14:fooo
" --template-file location/webapps.json


When building a large, multi-tier VPC you'll often find yourself needing to extract output values from existing stacks and pass them in as parameters to dependent stacks. This results in a lot of repeated literal strings and boilerplate in your commands and will soon cause you to start doubting your approach.

The real pain came for us when we started adding extra availability zones for resilience. A couple of my co-workers were keeping their stuff running with bash and python + boto but the code bases were starting to get a little creaky and complicated and this seemed like a problem that should have already been solved in a nice, declarative way. It was about the point when we decided to add an extra subnet to a number of tiers that I caved and went trawling through github for somebody else's solution. After some investigation I settled on Cumulus as the first project to experiment with as a replacement for our ever growing, hand hacked, creation scripts. To pay Cumulus the proper respect it did make life a lot easier at first.

The code snippets below show an example set of stacks that were converted over from raw command lines like the above to Cumulus yaml based configs. First up we have the base declaration and a simple stack definition.



locdsw:
  region: eu-west-1

  stacks:
    sns-email-topic:
      cf_template: sns-email-topic.json
      depends:
      params:
        AutoScaleSNSTopic:
          value: testymctest@example.org

Each of the keys under 'stacks:' will be created as a separate CloudFormation stack by cumulus. Their names will be prefixed with 'locdsw', taken from the first line of our example, and they'll be placed inside the 'eu-west-1' region. The configuration above will result in the creation of a stack called 'locdsw-sns-email-topic' appearing in the CloudFormation dashboard

The stacks resources are defined in the template specified in cf_template. Our example does not depend on existing stacks and takes a single parameter, AutoScaleSNSTopic, with a value of 'testymctest'. Cumulus has no support for variables so you'll find yourself repeating certain parameters, like ami id and key id, throughout the configuration.

For a while we had an internal branch that treated the CloudFormation templates as jinja2 templates. This enabled us to remove large amounts of duplication inside individual templates. These changes were submitted upstream but one of the goals of the Cumulus project is that the templates it manages can still be used by the native CloudFormation tools, so the patch was (quite fairly) rejected.

Let's move on to the second stack defined in our config. The point of interest here is the addition of an explicit dependency on the sns-email-topic stack. Note that it's not referred to using the prefixed name, which can be a point of confusion for new users.



    security-groups:
      cf_template: security-groups.json
      depends:
        - sns-email-topic

Finally we move on to an example declaration of a larger stack. The interesting parts of which are in the params section.



    webapp:
      cf_template: webapp.json
      depends:
        - sns-email-topic
        - security-groups
      params:
        AppServerFleetSize:
          value: 1
        Owner:
          value: dwilson
        AMIId:
          value: ami-n0tr34l
        KeyName:
          value: dwilson
        ASGSNSArn:
          source: sns-email-topic
          type: output
          variable: EmailSNSTopicARN
        WebappSGID:
          source: security-groups
          type: output
          variable: WebappSGID

The webapp params section contains two different types of values. Simple ones we've seen before, 'Owner' and 'AMIId' for example, and composite ones that reference values that other stacks define as outputs. Let's look at ASGSNSArn in a little more detail.



  ASGSNSArn:
    source: sns-email-topic
    type: output
    variable: EmailSNSTopicARN

Here, inside the webapp stack declaration, we look up a value defined in the output of the previously executed sns-email-topic template. From the CloudFormation Outputs for that template we retrieve the value of EmailSNSTopicARN. We then pass this to the webapp.json template as the ASGSNSArn parameter on stack creation. If you need to pull a parameter in from an existing stack that was created in some other way you can specify it as 'source: -fullstackname'. The '-' makes it an absolute name lookup, cumulus won't prefix the stackname with locdsw for example.

Cumulus met a number of my stack management needs, and I'm still using it for older, longer lived stacks such as monitoring, but because of its narrow focus it began to feel restricting quite quickly. I've started to investigate Ansible as a possible replacement as it's a more generic tool and I'm in need of flexibility that'd feel quite out of place in cumulus.

In terms of day to day operations the main issues we hit included the need to turn on ALL the debug, both cumulus and boto, to see why stack creations failed. A lot of the AWS returned errors were being caught and replaced by generic, unhelpful error messages at any filter level greater than debug. Running under debug results in a LOT of output, especially when boto is idle polling, waiting for the stack creation to complete so it can begin the next one. The lack of any variables or looping was also an early constraint. The answers to this seemed to include pushing the complexity down to the templates and writing large mapping sections, increasing duplication of literals between templates and a lot of FN::FindInMaps maps. The second approach was to have multiple configs. This was less than ideal due to the number of permutations, environment (dev, stage, live), region and in development which developer was using it. The third option, a small pre-processor that expanded embedded jinja2 in to a CloudFormation template, added another layer between writing and debugging and so didn't last very long.

If you're running a small number of simple templates then Cumulus might be the one tool you need. For us, Ansible seems to be a better fit, but more about that in the next post.

Posted: 2014/03/17 20:01 | /tools | Permanent link to this entry


Tue, 04 Mar 2014

Abstracting CloudFormation IAM with Nested Stacks

Once we started extracting applications into different logical CloudFormation stacks and physical templates, we began to notice quite a lot of duplication in our json when it came to declaring IAM rules. Some of our projects store their puppet, hiera and rpm files in restricted S3 buckets so allowing stacks access to them based upon environment, region, stack name and other criteria quickly becomes quite long-winded. After looking at a couple of dozen application templates and finding that over 30% of the json was IAM based it was time to find a different approach.

One of the CloudFormation techniques I'd seen mentioned but never used before was nested CloudFormation stacks. This allows you to define an entire stack as just another resource in your template. Here's some example json that does this:



  "Resources" : {

    "IAMRolesStack" : {
      "Type" : "AWS::CloudFormation::Stack",
      "Properties" : {
        "TemplateURL" : "https://s3-eu-west-1.amazonaws.com/my-iam-rules/projectname/iam-roles-20140301.json",
        "Parameters" : {
          "Stack": "testy-webapp",
          "Type":  "webapp",
          "App":   "tinyess",
          "Env":   { "Ref" : "DeploymentEnvironment" }
        }
      }
    }

  }

You can see that a stack is declared in the same manner as all other resources. The 'TemplateURL' property must point to a URL that hosts a complete, valid CloudFormation template. This allows you to develop the nested stack in the same way as you'd progress your actual application templates and test it in isolation. For my experiments I found it easiest to store them in S3 under a basic hierarchy with a little versioning to allow multiple versions of the IAM rules to be in use at once across the stacks. The other properties in the example are 'Parameters'. These are passed to the sub-stack at creation time as actual parameters and are what makes this approach so flexible and powerful.

Inside the nested stack template we add define a AWS::IAM::Role, an AWS::IAM::InstanceProfile and a number of AWS::IAM::Policy types that are abstracted to only allow access for one app/environment combination at a time. We do this using the parameters we pass in as values at different levels of the hierarchy. This way we can ensure that every application using a specific version of the IAM roles gets exactly the same permissions while not bulk pasting it into each applications json template or hard coding any of the application specific values. It's also worth noting that as stacks are given "CloudFormationed" IDs that include some randomness you can have multiple versions of the nested stack at once with no overlap or conflicts between apps.

You can see a small extract from our sample IAM template, with the parameters interpolated into the path, here -



  "SecretPolicy": {
    "Type": "AWS::IAM::Policy",
    "Properties": {
      "PolicyDocument": {
        "Statement": [ {
            "Effect": "Allow",
            "Action": [
              "S3:ListBucket"
            ],
            "Resource": [
              "arn:aws:s3:::org.example.test.secrets"
            ]
          },
          {
            "Effect": "Allow",
            "Action": [
              "S3:GetObject"
            ],
            "Resource": [
              "arn:aws:s3:::org.example.test.secrets/common.yaml",
              { "Fn::Join" : [ "", [
                "arn:aws:s3:::org.example.test.secrets/type/",
                { "Ref" : "App"   }, ".",
                { "Ref" : "Type"  }, ".",
                { "Ref" : "Env"   }, ".",
                { "Ref" : "Stack" }, ".yaml"
              ] ] },

              { "Fn::Join" : [ "", [
                "arn:aws:s3:::org.example.test.secrets/type/",
                { "Ref" : "App"  }, ".",
                { "Ref" : "Type" }, ".yaml"
              ] ] },

Now that we've declared and created the nested stack let's use the IamInstanceProfile it created in the auto scaling launch configuration that lives in the containing stack.



    "AppServerFleetLaunchConfig" : {
      "Type" : "AWS::AutoScaling::LaunchConfiguration",
      "Properties" : {
        ...
        "IamInstanceProfile": { "Fn::GetAtt" : [ "IAMRolesStack", "Outputs.InstanceProfile" ] },
        ...
      }
    }

Accessing nested stack outputs is as simple as a call to Fn::GetAtt with the resource name of the nested stack as the first argument (IAMRolesStack as seen in our first code snippet) and the outputs name as part of the second.


So what did we get from this? A few very worth while things. We removed a LOT of boilerplate from all our application templates. This also makes CloudFormation application templates easier to create as only a few people need in-depth knowledge of our IAM rules and bucketing scheme, application templates can focus on the application. It's easier to confirm that applications have the same access rights based on the S3 bucket used, rather than diffing through lots of subtly different IAM resources.

I'm using this technique on a couple of medium size projects at the moment and so far it seems like a good way to overcome IAM json spaghetti with no large drawbacks.

Posted: 2014/03/04 22:38 | /tools | Permanent link to this entry


Sat, 01 Mar 2014

Structured Facts with Facter 2

Structured facts in facter had become the Puppet communities version of 'Duke Nukem Forever', something that's always been just around the next corner. Now that the facter 2.0.1 release candidate is out you can finally get your hands on an early version and do some experimentation.

First we grab a version of facter 2 that supports structured facts from puppetlabs -



 # our play ground
 mkdir /tmp/facter && cd /tmp/facter

 # grab the code
 wget https://downloads.puppetlabs.com/facter/facter-2.0.1-rc1.tar.gz

 cd facter-2.0.1-rc1/

 # check facter runs from our expanded archive
 ruby -I lib bin/facter
  

This is the part where we can be underwhelmed, it's all still flat. Don't let the lack of nested facts dishearten you though. The Puppetlabs people have done all the hard work of implementing structured facts support, they've just not converted any showcase facts over yet. Instead of waiting for an official structured fact lets add our own and have a little play.

As we're experimenting with a throw away environment we'll drop the structured fact directly in to our expanded archive. In a real environment you'd never do this, you'd either use FACTERLIB or deploy your modules properly with puppet as Luke intended.



 # install the plugin
 wget https://raw.github.com/deanwilson/unixdaemon-puppet_facts/master/lib/facter/yumplugins.rb -O lib/facter/yumplugins.rb

 # and run it
 ruby -I lib bin/facter  yumplugins

 pluginblacklistlangpacksprestorefresh-packagekitwhiteoutdisabledblacklistwhiteoutenabledlangpacksprestorefresh-packagekit

Well, our first TODO will be to determine how to show structured facts as strings, but we'll defer that for now as we really want to see some deep nesting. Assuming you're on a RedHat osfamily host you can run facter with the yaml output, otherwise you'll have to settle for the sample outputs below:



 $ ruby -I lib bin/facter yumplugins --yaml

--- 
yumplugins: 
  plugin: 
  - blacklist
  - langpacks
  - presto
  - refresh-packagekit
  - whiteout
  enabled: 
  - langpacks
  - presto
  - refresh-packagekit
  disabled: 
  - blacklist
  - whiteout

  # and now try it as json

 $ ruby -I lib ./bin/facter yumplugins -j
{
  "yumplugins": {
    "plugin": [
      "blacklist",
      "langpacks",
      "presto",
      "refresh-packagekit",
      "whiteout"
    ],
    "enabled": [
      "langpacks",
      "presto",
      "refresh-packagekit"
    ],
    "disabled": [
      "blacklist",
      "whiteout"
    ]
  }
}

Success! Structured fact output! From (nearly) Puppet! Of course, this is only a release candidate for Facter 2 so we're not production ready yet but as a taster of what's coming and a way to get ahead and start converting your own facts it's a lovely, and amazingly overdue, gift.

As for writing structured facts, as you can see from my structured yumplugins fact example there's no difference between a structured and an unstructured one apart from the value it returns.

Posted: 2014/03/01 16:04 | /tools/puppet | Permanent link to this entry


Automatic CloudFormation Template Validation with Guard::CloudFormation

One of the nice little conveniences I've started to use in my daily work with Amazon Webservices CloudFormation is the Guard::CloudFormation ruby gem.

The Guard gem "is a command line tool to easily handle events on file system modifications" which, simply put, means "run a command when a file changes". While I've used a number of different little tools to do this in the past, Guard presents a promising base to build more specific test executors on so I've started to integrate it in to more aspects of my work flow. In this example I'm going to show you how to validate a CloudFormation template each time you save a change to it.

The example below assumes that you already have the AWS CloudFormation command line tools installed, configured and available on your path.



# our example, one resource, template.

$ cat example-sns-email-topic.json 
{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description" : "Creates an example SNS topic",

  "Parameters" : {
    "AutoScaleSNSTopic": {
      "Description": "Email address for notifications.",
      "Type": "String",
      "Default": "testymctest@example.org"
    }
  },

  "Resources" : {
    "EmailSNSTopic": {
      "Type" : "AWS::SNS::Topic",
      "Properties" : {
        "DisplayName" : "Autoscaling notifications for Location service",
        "Subscription": [ {
          "Endpoint" : { "Ref" : "AutoScaleSNSTopic" },
          "Protocol" : "email"
        } ]
      }
    }
  }
}

# now, do a manual run to ensure all the basics are working

$ cfn-validate-template --template-file example-sns-email-topic.json
PARAMETERS  AutoScaleSNSTopic  testymctest@example.org  false  Email address for notifications.


Now that we have confirmed our CloudFormation tools are working we can sprinkle some automation all over it.



# install the required gems

gem install guard guard-cloudformation

# and then create a basic Guardfile
# in this case we watch all .json files in the current directory

cat << 'EOC' > Guardfile

guard "cloudformation", :templates_path => ".", :all_on_start => false do
  watch(%r{.+\.json$})
end

EOC

# run guard

$ guard 
10:07:49 - INFO - Guard is using NotifySend to send notifications.
10:07:49 - INFO - Guard is using TerminalTitle to send notifications.
10:07:49 - INFO - Guard is now watching at '/home/cfntest/.../cloudformations/location'
[1] guard(main)> 


Now that guard is up and running open up a second terminal to the directory you've been working in. We'll now make a couple of changes and watch Guard in action. First we'll make a small change to the text that shouldn't break anything.



# run the sed command to change the email address - shouldn't break

$ sed -i -e 's/testymctest@example.org/test@example.org/' example-sns-email-topic.json

# in the term running Guard we see -

10:12:31 - INFO - Validating: example-sns-email-topic.json
Validating example-sns-email-topic.json...
PARAMETERS  AutoScaleSNSTopic  test@example.org  false  Email address for notifications.


On my desktop the validation output is a lovely terminal green and I also get a little pop-up in the corner telling me the validate was successful. Leaving Guard open, we'll run a breaking change.



# run the sed command to remove needed quotes - this will not end well

$ sed -i -e 's/"Ref" :/Ref :/' example-sns-email-topic.json

# in the term running Guard we see -

10:13:51 - INFO - Validating: example-sns-email-topic.json
Validating example-sns-email-topic.json...
cfn-validate-template:  Malformed input-Template format error: JSON not well-formed. (line 19, column
 27)
Usage:
cfn-validate-template
       [--template-file  value ] [--template-url  value ]  [General Options]
For more information and a full list of options, run "cfn-validate-template --help"
FAILED: example-sns-email-topic.json


The 'FAILED: example-sns-email-topic.json' line is displayed in less welcome red, the dialog box pops up again and we know that our last change was incorrect. While this isn't quite as nice as having vim running the validate in the background and taking you directly to the erroring line it's a lot easier to plumb in to your tool chain and gives you 80% of the benefit for very little effort. For completeness we'll reverse our last edit to fix the template.



# sed command to fix the template

$ sed -i -e 's/Ref :/"Ref" :/' example-sns-email-topic.json

# in the term running Guard we see -

10:22:42 - INFO - Validating: example-sns-email-topic.json
Validating example-sns-email-topic.json...
PARAMETERS  AutoScaleSNSTopic  test@example.org  false  Email address for notifications.


One last config option that's worth noting is ':all_on_start => false' from the Guardfile. If this is set to true then, as you'd expect by the name, all CloudFormation templates that match the watch will be validated when Guard starts. I find the validates to be quite slow and I often only dip in to a couple of templates so I set this to off. If you spend more focused time working on nothing but templates then having this set to 'true' gives you a nice early warning in case someone checked in a broken template. Although your git hooks shouldn't allow this anyway. But that's a different post.

After reading through the validate errors of a couple of days work it seems my most common issue is from continuation commas. It's just a shame that CloudFormation doesn't allow trailing commas everywhere.

Posted: 2014/03/01 10:44 | /tools | Permanent link to this entry


Wed, 12 Feb 2014

Config Management Camp: Juju Surprises

One of the biggest surprises of Config Management Camp 2014 for me was how interesting Canonicals orchestration management tool, Juju has become. Although I much preferred the name 'Ensemble'.

I attended the Juju session in an attempt to keep myself out of the Puppet room and was pleasantly surprised at how much Juju had progressed since I last looked at it. Rather than being another config management solution it allows you to model your systems using "charms", which can be implemented using anything from a bash script to a set of chef/puppet cookbooks/modules, and instead focuses on ensuring that they run across your fleet in a predictable way while enforcing dependencies, even over multiple tiers, no matter how many tools you choose to use underneath.

Listening to the presentations it seems Juju has some very well thought out parts. Multiple callback hooks, that are triggered on state changes, are used as an orchestration back channel between different hosts and the services they provide flowed nicely in the demos. The web dashboard was very polished and had some very shiny canvas magic that could be borrowed in other tools. I also liked the command line interface for linking different tiers and associating supporting roles, such as tying wordpress instances to a mysql back end. There is also some cloud provider performance abstraction code at work where you can request a certain amount of resources and Juju will map that to the closest instance type in which ever provider you're currently using.

I was only in the room for a couple of the talks but both the Canonical staff were a credit to their company. The material was well presented, they managed to answer all the audiences questions and you get the impression that they'd be a nice project to work with. Hopefully I'll have a chance to play with the platform some more in the future.

Posted: 2014/02/12 23:52 | /tools | Permanent link to this entry


Mon, 13 Jan 2014

Ansible Configuration Management Book - Short Review

I'm still new to Ansible and while it's been interesting seeing how people are starting to use the tool, picking up bits and pieces from different blog posts is a little too hit and miss for my learning needs. When I spotted Ansible Configuration Management (PacktPub) I decided to take the plunge and see if it could provide me with a more consistent introduction. And it did.

This book makes an ideal first stop for anyone wanting to learn Ansible. While it's a short book (92 pages and even less than that of actual content) it provides a very good introduction and overview of at least your first months experience of Ansible. While none of its coverage is going to be the only coverage of a subject you'll ever need it introduces enough of the concepts and features to be the best starting guide for Ansible I've seen so far. I found that each chapter filled in a number of gaps in my understanding of how Ansible should be used.

If you're looking at introducing Ansible to your team this book is far from the worst way to do it. Its coverage is broad enough that you'll probably get a few re-reads out of it as you bring more of your infrastructure under Ansible control and start to evolve your needs from basic playbooks to more advanced role composition. It's worth noting that this isn't a cookbook, it's not going to hand hold you through using each of the built in modules. For the more experienced sysadmins looking for a quick way to learn Ansible this is a boon as it keeps the page count down.

I'd liked to have seen more coverage of extending Ansible, the last chapter provides a basic introduction but it's not enough for what I need but this'd be a good subject for a second book once the testing tool chain and such as progressed to a more mature place. Score - 7/10

Posted: 2014/01/13 21:50 | /books | Permanent link to this entry


Sat, 11 Jan 2014

Ansible CloudFormation Lookup Plugin

As the Ansible/AWS investigations continue I had the need to lookup outputs from existing CloudFormation stacks. I spent ten minutes reading through the existing lookup plugins and came up with the Ansible CloudFormation Lookup Plugin.

I'm not sure this is going to be our final solution. Michael DeHaan suggested that moving to a fact plugin might be better in terms of cleaner usage and easier testing, so I'm at the least going to implement a trial version of that. I was quite surprised at how easy writing an Ansible lookup plugin was though, even for someone with my limited python skills.

Once you've downloaded and installed the plugin, using it in your templates is as simple as



    {{ lookup('cloudformation', 'region/stackname/output/outputname') }}

    # and an actual example:
    {{ lookup('cloudformation', 'eu-west-1/unixdaemon-natinstance/output/nat_group_id') }}



It uses boto under the covers and expects to find your credentials as environmental variables. This is only a tiny chunk of code but it's allowed us to continue on with the evaluations while gaining a little more comfort in our ability to extend Ansible to suit our needs.

Posted: 2014/01/11 12:49 | /tools/ansible | Permanent link to this entry


Mon, 06 Jan 2014

Learning AWS OpsWorks - Short Book Review

I picked up a copy of Learning AWS OpsWorks during the PacktPub holiday sale. It was cheap, short and covered a AWS product that I've never had need to dig in to and knew very little about.

The book takes you through creating a basic stack, the layers inside it and deploying an application to managed instances. Its coverage is very high level and doesn't really go beyond a cursory explanation of the services used. As you'd expect from the page count, it doesn't delve in to either the Amazon services you use or how to make chef do your bidding, instead sticking to its focus and giving you just enough information to get the example working and not much else. It's worth mentioning that the console screen shots are already out of date so you need to do a little exploring on your own as you follow the steps.

Learning AWS OpsWorks is a brief but informative high level overview of AWS OpsWorks and how you'd use it to create and manage basic stacks. I don't think it's worth the full price, a Safari account would be quite useful here. It's also very unlikely you'll need to read it more than once so it's not great value for money. It does however present the concepts in an easy to understand way, so if you're looking to pick up basic OpsWorks in a big rush it's the only competition to the official docs.

Posted: 2014/01/06 10:26 | /books | Permanent link to this entry


Fri, 03 Jan 2014

Introducing CloudFormation Conditionals

Back in November 2013 Amazon added a much requested feature to CloudFormation, the ability to conditionally include resources or their properties in to a stack. As an example I'm currently using this as a small cost saving measure to ensure only my production RDS instances have PIOPs applied to them while being able to build each environment from a single template.

CloudFormation Conditionals live in their own section of a CloudFormation template. In the example below we show two ways of setting values for use in a condition. A simple string comparison and a longer, composite comparison that includes an or. Each of these are based on a provided parameter.



{
  "AWSTemplateFormatVersion": "2010-09-09",

  "Parameters" : {
    "DeploymentEnvironment" : {
      "Default": "dev",
      "Description" : "The environment this stack is being deployed to.",
      "Type" : "String",
      "AllowedValues" : [ "dev", "stage", "prod" ],
      "ConstraintDescription" : "Must be one of the following environments - dev | stage | prod"
    }
  },

  "Conditions": {
    "InProd": { "Fn::Equals" : [ { "Ref" : "DeploymentEnvironment" }, "prod" ] },

    "NotInProd": {
      "Fn::Or" : [
        { "Fn::Equals" : [ { "Ref" : "DeploymentEnvironment"}, "stage" ] },
        { "Fn::Equals" : [ { "Ref" : "DeploymentEnvironment"}, "dev" ] }
      ]
    }
  }
}


Our first example is the conditional inclusion of an entire resource -



  "Resources" : {

    "WebappSG": {
      "Type": "AWS::EC2::SecurityGroup",
      "Condition" : "NotInProd",
      "Properties": {
        "GroupDescription": "Location webapp servers",
        "SecurityGroupIngress": [ {
          ... snip ...
        } ]
      }
    }
  }


The key part of this snippet is the 'Condition' line. If the value on the right hand evaluates to true the resource is created when the template runs. If the condition is false the entire resource is skipped. As a second example we'll show how to conditionally include a single property value. In this case the 'Iops' property of a AWS::RDS::DBInstance resource.



  "Resources" : {

    "MySQL" : { 
      "Type" : "AWS::RDS::DBInstance",
      "DeletionPolicy" : "Snapshot",
      "Properties" : { 
        "DBName"               : "testy",
        "AllocatedStorage"     : "5",
        "DBInstanceClass"      : "db.m2.4xlarge",
        "MasterUsername"       : "testy",
        "MasterUserPassword"   : "testy",
        "DBSecurityGroups"     : [ "rds" ],
        "Engine"               : "MySQL",
        "EngineVersion"        : "5.6",
        "MultiAZ"              : "true",
        "Iops" : {
          "Fn::If" : [ "InProd",
            "1000",
            { "Ref" : "AWS::NoValue" }
          ]
        }
      }
    }
  }


If InProd is true the Iops property is included and set to 1000. If the value of InProd is false then the special value of 'AWS::NoValue' is returned. This causes the property itself to be completely excluded.

CloudFormation Conditions are quite a new feature, and I was a little late in discovering it, so we've only just started to use them in our templates. They are however worth learning about as they provide a flexible new way to structure your templates.

Posted: 2014/01/03 12:51 | /cloud | Permanent link to this entry


Wed, 01 Jan 2014

Pragmatic Investment Plan - 2013 / 2014

I changed jobs midway through 2013 and quite quickly discovered that I'd been a little too buried in my role and not been keeping up other parts of my technical interests. As a first step I decided to put a very basic Pragmatic Investment Plan in place. Mostly as a simple way to ensure I actually started to get involved in non-work things again.

Firstly I set myself the task of recording which books I actually read. I only counted books I actually finished and ended up reading nearly 90 books in the last year. Totalling over 22000 pages. In 2014 I'm going to set myself a minimum goal of 15000 pages to ensure I actually make the time to both finish books, and read non-technical material.

In terms of technical reading I wanted to ensure I read an average of one a month so I set myself a goal of 12 books. I managed to finish 15 in the end so I'm happy with that. In the last six months I've also tech reviewed 4 books, all of which are due out in 2014. I've not done a proper tech review of a book for years and it made a nice change of pace.

I decided to try and add some content to my websites. And failed. Over the year I managed to publish a measly 29 blog posts to Unixdaemon.net, less than one a week on average. This year I'm hoping to make at least 52. As for Puppet Cookbook there's been a set of issues with the hosting and site generation that I've not had the time or energy to resolve so it's been sitting idle. Something I really need to fix in 2014 as parts of the sample code are not puppet-lint compliant, which annoys me. I did manage to get a mention in Ansible Weekly from one post so I'm slightly happier than I should be considering my output.

I think that having nothing to talk about for nearly months at a time is a sign that I'm approaching some tasks wrongly - especially considering I'm now working for a company that is a lot more open about discussing our code, techniques and projects. I didn't even mention a number of tools that I've published that might be useful to other people. At the very least a monthly release announcement would have kept better track of what I've been working on and where it lives.

In terms of getting out and about I had a very mixed bag last year. My attendance at user group events was very poor. A London Devops and the CloudCamp London SDN special were it for the year. I only managed to get to the latter as Greg Ferro was speaking and as a big fan of the Packet Pushers podcast the opportunity was too good to miss. My conference attendance was much better -

I was very fortunate to make it to both Devops Days in Mountain View and Velocity SF. I had a chance to meet a lot of the people I'd only spoken to online and meet some of my new co-workers, as well as catch up with old friends. The travel was a pain but it worked out to be well worth it. I was impressed by a number of the Netflix team I had a chance to speak with and finally got to see Andrew Shafer host the ignites.

Due to health issues I missed both Puppetcamps in London, for which I felt very slack, and donated my monitorarma EU ticket to the Centos Project. I also had to cancel a number of other conference visits. In 2014 I need to be a lot pickier about my travel so I'm resigned to missing a lot of great content. My first conferences of the year will be FOSDEM and Config Management Camp as I've been to Belgium enough times to feel safe(ish) with the travel.

I set myself the goal of getting at least two of my internal tools cleaned up and pushed to github. I managed to get a lot more code than this public but large parts of it were either very beta tools or small examples. I've started to deprecate a lot of my own code and adopt the functionally closest open source project instead to cut down my maintenance costs. I've sent a number of patches upstream and I assume I'll continue if not increase this in 2014.

I also had the lack of certifications on my CV pointed out to me a couple of times this year. I don't have a strong academic background (understatement warning!) and it was mentioned that I could balance this with some relevant certs. I'm not a keen test taker so I only managed to bag a couple of basic ones, the very intro level VCA-DCV and the newish CompTIA Cloud+. I'm unsure of the best direction to take with this in the future - maybe another RedHat specialisation would complement my role - so for now I'll just say I'd like another one or two as CV fodder.

In general I'll probably start the year with the similar basic categories in my 2014 PiP and refine them in to more focused monthly / quarterly batches as I discover what 2014 has planned for me. I don't feel comfortable trying to plan for a whole 12 months ahead but if I leave just the section headings I'll end up bouncing around too much. I've had some interesting chats with Paul Nasrat about this and while our approaches are quite different it's been great to have someone to talk this over with.

Well, that's my basic review of my 2013. I hope you all have a wonderful 2014 and thank ye kindly for visiting my site. Dean Wilson.

Posted: 2014/01/01 20:51 | /career | Permanent link to this entry


Fri, 20 Dec 2013

Package Install Options in Puppet

One of the new features released in Puppet 3.4.0 is the ability to add options to rpm package installs. This is a feature that's been discussed in a couple of tickets over the years and now we've got an official, in core, approach I've copied the code to the yum and apt providers github branch.



  class pkgoptions {

    package { 'strace':
      ensure          => 'installed',
      provider        => 'yum',
      install_options => [ '--noplugins', '--enablerepo=fedora' ],
      # or install_options => [ '-t', 'squeeze-backports' ], for Debian backports
    }

  }


The approach in this patch is not the final one I'd like to take so I've not submitted it upstream. It should be possible to extract some of the duplication between the providers up in to a util class. Of course they also need spec tests but as a replacement for some of the exec based workarounds I've seen, in yum to enable specific repos and in Debian for back-ported packages, this is much closer to where I'd want my manifests to be.

Posted: 2013/12/20 14:36 | /tools/puppet | Permanent link to this entry


Wed, 11 Dec 2013

Puppet External Resource - a Hidden gem

"a simple resource that blocks transactions until a check passes, theoretically indicating that a remote resource is in a desired state."
   -- Puppet Remote Resource Documentation

I stumbled over the Puppet Remote Resource module while looking around the Puppetlabs github account for something completely different and was surprised to find that I'd never seen this little gem mentioned anywhere else. A pre-built way to have a puppet resource skipped based on the result of an external command is a very flexible tool, especially when you couple it with all the available nagios checks.

Nagios checks have a wonderfully well known set of exit codes, in essence they return 0 if the check succeeds. Anything else is either a warning, a critical issue or an unhandled circumstance. In our case we only want puppet to apply our resource if everything is fine so this is ideal as external_resource treats anything but 0 as a reason to skip the resource.

First we install the nagios check -



  yum install nagios-plugins-tcp

  # to list the available plugins
  yum search nagios-plugins


then we grab a copy of the module for testing and add our small test script -



cd /tmp
git clone https://github.com/deanwilson/puppetlabs-external_resource.git external_resource

cat << 'EOC' > external_example.pp
class external_example(
  $tcp_check = '/usr/lib64/nagios/plugins/check_tcp -H 127.0.0.1 -p 8088 -t 30'
){

  external_resource { 'web server check':
    frequency => 1,
    timeout   => 3,
    check     => $tcp_check,
  }

  notify { 'Remote server check':
    require => External_resource['web server check'],
  }

  notify { 'i still run': }
}

include external_example

EOC


Our example has three resources, firstly the external_resource that controls what to check, how often to check for it and how long before it should be timed out. This timeout helps avoid long running but essentially stalled puppet runs if the remote resource is unavailable due to network issues. The second resource is a notify that requires external_resource and the last is a notify to show that only the 'Remote server check' notify is skipped, not the whole puppet run.

We then run the example, using the 'manifest' ordering ( excellently explained here)



  puppet apply --modulepath /tmp --ordering manifest -v external_example.pp

Info: /Stage[main]/External_example/External_resource[web server check]: Remote resource is not up; delaying for 1 seconds before next check
Info: /Stage[main]/External_example/External_resource[web server check]: Remote resource is not up; delaying for 1 seconds before next check
Info: /Stage[main]/External_example/External_resource[web server check]: Remote resource is not up; delaying for 1 seconds before next check

Error: /Stage[main]/External_example/External_resource[web server check]: Could not evaluate: Remote resource not up within timeout 3

Notice: /Stage[main]/External_example/Notify[Remote server check]: Dependency External_resource[web server check] has failures: true
Warning: /Stage[main]/External_example/Notify[Remote server check]: Skipping because of failed dependencies

Notice: i still run
Notice: /Stage[main]/External_example/Notify[i still run]/message: defined 'message' as 'i still run'


The output shows the failure of the external_resource and the consequent skipping of the 'Remote server check' notify. Now that we've seen the module in action and know how to use it let's make our scenario a little more complicated. Imagine we have a load balancer that has one config file per back end, we don't want puppet to add the new config until the back end node is up and on the network. We can add the new node config to puppet at any point but because the check_tcp plugin fails it won't add that resource to the loadbalancer.

This approach can easily be adapted to most network services. MySQL databases (don't restart the appserver on a config change if the remote mysql isn't present). Only deploy certain resources if a DNS entry resolves (check_dig) or the host has enough diskspace (check_disk). My current favourite is not to change any application resources on a host until the nagios check confirms the host's been cleanly removed from the load balancer and traffic has drained off (and then you can use a post_command to add it back in!)

I'd also like to say a quick thank you to Hunter from Puppetlabs who very quickly turned around a patch for this module.

Posted: 2013/12/11 13:07 | /tools/puppet | Permanent link to this entry


Wed, 04 Dec 2013

Introduction to Programming in Go - Short Review

I don't often impulse buy technical books. They cost too much and consume too much shelf space to be purchased frivolously but when it's 1.95 for a Kindle book on Go and I'm stuck miles from home it seemed like a good idea.

An Introduction to Programming in Go is a well written guide to your first hour in Go. While you can probably find coverage of the same material on the web, having it all nicely curated in one place is worth the money for someone like me who just wants a little taster and overview. The book covers all the language basics, types, variables, data structures, pointers, structs and, most interesting to me, the native concurrency methods.

While I personally don't have an immediate use for Go, at $WORK we have a couple of projects written in it so being able to understand the basics from such a quick read is a lovely thing. I can see myself re-reading this little book each time I'm called on to work with a Go based team as a terse refresher.

This book won't teach you everything you need to know about Go in its slender 166 pages but it does provide a nice, quick peek at Go code and its more immediately useful standard library. For people with no Go exposure I'd give this book 7/10

Posted: 2013/12/04 00:15 | /books | Permanent link to this entry


Tue, 03 Dec 2013

Puppet Resource Ordering Options

Over the years Puppet has handled resources ordering without explicit dependencies in different ways, with the release of Puppet 3.3.0 they've exposed this ordering logic to the admin with three interesting options.

To test these options out we'll use the 'ordering' test module shown below. We include three classes, ordering::beta, ordering::alpha and ordering::gamma (note that the includes are not in alphabetically sorted order). Each of these classes has three notify statements that show a number and the class they are contained in.



$ find

./modules
./modules/ordering
./modules/ordering/manifests
./modules/ordering/manifests/init.pp
./modules/ordering/manifests/alpha.pp
./modules/ordering/manifests/beta.pp
./modules/ordering/manifests/gamma.pp

$ find -type f | xargs -n 1 cat

class ordering {
  include ordering::beta
  include ordering::alpha
  include ordering::gamma
}

class ordering::alpha {
  notify { "Hello number 1 from ordering::alpha": }
  notify { "Hello number 2 from ordering::alpha": }
  notify { "Hello number 3 from ordering::alpha": }
}

class ordering::beta {
  notify { "Hello number 1 from ordering::beta": }
  notify { "Hello number 2 from ordering::beta": }
  notify { "Hello number 3 from ordering::beta": }
}

class ordering::gamma {
  notify { "Hello number 1 from ordering::gamma": }
  notify { "Hello number 2 from ordering::gamma": }
  notify { "Hello number 3 from ordering::gamma": }
}


First we'll try the default option - title-hash. This randomly orders resources, but will be consistent across runs and nodes. We'll do two runs of each option to see if anything changes.



  puppet apply --modulepath modules -e 'include ordering'  --ordering title-hash | grep -v Stage

Notice: Hello number 3 from ordering::alpha
Notice: Hello number 1 from ordering::alpha
Notice: Hello number 2 from ordering::alpha
Notice: Hello number 1 from ordering::beta
Notice: Hello number 2 from ordering::beta
Notice: Hello number 2 from ordering::gamma
Notice: Hello number 3 from ordering::gamma
Notice: Hello number 1 from ordering::gamma
Notice: Hello number 3 from ordering::beta

  puppet apply --modulepath modules -e 'include ordering'  --ordering title-hash | grep -v Stage

Notice: Hello number 3 from ordering::alpha
Notice: Hello number 1 from ordering::alpha
Notice: Hello number 2 from ordering::alpha
Notice: Hello number 1 from ordering::beta
Notice: Hello number 2 from ordering::beta
Notice: Hello number 2 from ordering::gamma
Notice: Hello number 3 from ordering::gamma
Notice: Hello number 1 from ordering::gamma
Notice: Hello number 3 from ordering::beta


This confirms that the ordering is consistent between runs. It starts off predictably with ordering::alpha running first but note the ordering::beta at the end, and the difference in ordering of the notifys within each class. Hopefully you can see that it's not an ordering you should rely upon within your own classes or assume you can predict.

Now we'll look at the manifest value, or 'chef mode' as I've heard it nicknamed. This option will cause the resources to be actioned in in exactly the order you've written.



  puppet apply --modulepath modules -e 'include ordering'  --ordering manifest | grep -v Stage

Notice: Hello number 1 from ordering::beta
Notice: Hello number 2 from ordering::beta
Notice: Hello number 3 from ordering::beta

Notice: Hello number 1 from ordering::alpha
Notice: Hello number 2 from ordering::alpha
Notice: Hello number 3 from ordering::alpha

Notice: Hello number 1 from ordering::gamma
Notice: Hello number 2 from ordering::gamma
Notice: Hello number 3 from ordering::gamma

  puppet apply --modulepath modules -e 'include ordering'  --ordering manifest | grep -v Stage

Notice: Hello number 1 from ordering::beta
Notice: Hello number 2 from ordering::beta
Notice: Hello number 3 from ordering::beta

Notice: Hello number 1 from ordering::alpha
Notice: Hello number 2 from ordering::alpha
Notice: Hello number 3 from ordering::alpha

Notice: Hello number 1 from ordering::gamma
Notice: Hello number 2 from ordering::gamma
Notice: Hello number 3 from ordering::gamma


I've spaced the output slightly you can see that the order is consistent between runs and grouped in order, by class. Looking at the notify output you can see it's been produced exactly as we wrote the classes.

The third and final option is my personal favourite and something I've already enabled on some of my dev puppet masters. Setting the ordering to random does exactly what the name implies, it orders resources, including includes!, randomly within a single run and across multiple runs.



  puppet apply --modulepath modules -e 'include ordering'  --ordering random | grep -v Stage

Notice: Hello number 3 from ordering::gamma
Notice: Hello number 2 from ordering::gamma
Notice: Hello number 2 from ordering::alpha
Notice: Hello number 3 from ordering::alpha
Notice: Hello number 2 from ordering::beta
Notice: Hello number 3 from ordering::beta
Notice: Hello number 1 from ordering::beta
Notice: Hello number 1 from ordering::alpha
Notice: Hello number 1 from ordering::gamma

  puppet apply --modulepath modules -e 'include ordering'  --ordering random | grep -v Stage

Notice: Hello number 1 from ordering::beta
Notice: Hello number 1 from ordering::alpha
Notice: Hello number 2 from ordering::beta
Notice: Hello number 3 from ordering::beta
Notice: Hello number 3 from ordering::alpha
Notice: Hello number 2 from ordering::alpha
Notice: Hello number 2 from ordering::gamma
Notice: Hello number 1 from ordering::gamma
Notice: Hello number 3 from ordering::gamma


This is a great way to find implicit relationships that have accidentally slipped in to modules before they become real problems. I'd recommend having development hosts and CI boxes running under random to help keep your code explicit and honest.

As a reminder these options -DO NOT- change explicit relationships in your classes and modules. If explicit ordering is required for your resources, and especially if you upload the modules to the puppetforge or github where you have no visibility of peoples puppet settings, you should continue to use requires / notify / before / after metaparameters even if you set ordering to manifest.

Posted: 2013/12/03 12:47 | /tools/puppet | Permanent link to this entry


Mon, 02 Dec 2013

Ansible and AWS Security Groups

We use Amazon CloudFormation for a number of our deployments at $WORK. Although it's nice to have security group creation inside the same template as the resources it will secure, CloudFormations 'helpful' addition of a unique string at the end of the resource names it creates can sometimes be a problem. A couple of tools assume security groups will have an absolute, unchanging name and lack a way to search for an appropriately tagged security group whose name can change on stack rebuild. In order to get around this I tried extracting some of our security group creation from CloudFormation and implemented it in Ansible instead.

For this example I'm going to assume you have a working Ansible install and an AWS account. While you can specify your AWS AWS_ACCESS_KEY and AWS_SECRET_KEY in your Ansible play books I set mine as environmental variables to avoid the risk of checking them in to a VCS. First we create our directory structure, group variables and site.yml -


export AWS_ACCESS_KEY=YOURKEY
export AWS_SECRET_KEY=YOURSECRET

# we'll do all our work under here
mkdir -p aws-sg aws-sg/{group_vars,roles/bastionhosts/tasks}
cd aws-sg

echo '127.0.0.1' > hosts


# add regions to create the security groups in here
cat << 'EOC' > group_vars/all
---
regions:
 - eu-west-1
 - us-east-1
 - us-east-2
EOC


# add the task to run
cat << 'EOC' > site.yml
---
- hosts: all
  roles:
   - role: bastionhosts
EOC


Now we've created our scaffolding we can actually write tasks to create our security group



cat << 'EOC' > ./roles/bastionhosts/tasks/main.yml
---
- name: "ssh ingress security group in region {{ item }}"
  local_action:
    module: ec2_group
    name: bastion-ingress
    description: Bastion host servers
    region: "{{ item }}"
    rules:
      - proto: tcp
        from_port: 22
        to_port: 22
        cidr_ip: 0.0.0.0/0
  register: bastion_ingress
  with_items: regions

EOC


The above code creates the 'bastion-ingress' security group in each region you specified in group_vars/all. with_items will run the task once for each region in regions. It also sets item to the current array element so it can be used within the task. The other line of interest is register. This stores meta-data about the newly created security group in an array that we can use later.

We now have our security group allowing ssh traffic in to the bastion hosts. We'll often want to allow ssh traffic from bastion hosts, those with the bastion-ingress group assign to them, to "internal" hosts that are not publicly reachable. To do this we add a second security group, that we assign to the internal hosts, that allows traffic in on port 22 from the existing bastion-ingress group. This reduces the number of places the internal hosts can be reached from and is often useful in default VPCs.



cat << 'EOC' >> ./roles/bastionhosts/tasks/main.yml

- name: "ssh from bastion hosts group in region {{ item }}"
  local_action:
    module: ec2_group
    name: bastion-clients
    description: Allows bastion servers to connect
    region: "{{ item.item }}"
    rules:
      - proto: tcp
        from_port: 22
        to_port: 22
        group_id: "{{ item.group_id }}"
  with_items: bastion_ingress.results

EOC


This new security group task is run for each entry in bastion_ingress.results, the array we stored the creation meta-data of the bastion-ingress group in earlier using register. We then use the group_id to limit where we allow incoming traffic from for this new group.

You can then run it with -



ansible-playbook -i hosts site.yml -v


As an approach, the pattern of storing with register and iterating with with_items: foo.results is a very handy one and I suspect it will be used in many of my playbooks. In terms of the ec2_group module itself it's inability to handle egress rules limits the number of places we can use it but it's a nice proof of concept for running under old, ec2 classic based deployments.

Posted: 2013/12/02 23:58 | /tools/ansible | Permanent link to this entry


Sat, 30 Nov 2013

Puppet Book Summary - 2013

Even though I don't spend as much time writing puppet code as I used to I try to stay relevant and as part of that I like to read all the Puppet books that come out. Below are the ones I've read this year, brief thoughts on them and the reading path I'd give to a new junior.

As the name implies the Puppet 3 Beginner's Guide is a decent place to start learning Puppet. It's an easy to read tour of the puppet language and how to use a number of the core resource types. I can see this being a very useful book for people just starting out and a handy reference for their first 3-6 months of writing manifests.

Pro Puppet 2nd Edition is the new book on the block at the time of writing. You can read my initial review but in general it's a good book for people that are beginner to intermediate level and a hands on bent. It introduces quite a few topics, how to structure and write modules, scale your puppet master, reporting and introduces you to extending puppet with facts, functions and types for example. I think it fits very well with the Beginner's Guide for new users and provides a lot of the 'why' things are done in a certain way above the 'how'.

I feel a little odd adding the Puppet 3 Cookbook to this post as I've never actually read it. As the maintainer of the Puppet Cookbook website I feel that reading the book would be a bit of a conflict of interest but from different community members I've spoken to the book comes very well regarded so I didn't want to exclude it. I've been told it provides very handy examples that will take you through a number of more intermediate use cases and introduce a number of useful approaches.

On to Managing Infrastructure with Puppet. I normally won't buy a book that falls outside of a certain page / price ratio but this being the first O'Reilly Puppet book overrode my caution. That was a mistake. There's nothing wrong with the actual content but there's not much of it, it's very jumpy in what it covers and I have trouble narrowing down which type of audience it's for. 90% of the topics are covered more comprehensively in Pro Puppet for example.

On the plus side it did have some of the best coverage of using Puppet with LDAP together that you could get until Pro Puppet (2nd Ed) came out. I can't recommend this book, and after a glance on the O'Reilly and Amazon review pages for it, I'm not alone in this view.

In terms of more advanced coverage, until we can convince Brice Figureau to write an extending puppet book, Puppet Types and Providers is the go to guide for people that want to implement their own types and providers. I've already written my Puppet Types and Providers review and I stand with what I said in the past, it's the best introduction and reference for people that want to write their own types.

What's missing from this list? Instant Puppet 3 Starter is the only other book I'm aware of but the price to page on that one means I won't be buying a copy.

Now we've covered the books what's my ideal reading path? I think I'd start someone with the Puppet 3 Beginner's Guide and somewhere around the halfway mark I'd get them to read the first 3 chapters of Pro Puppet and, the oddly placed at the end of the book, hiera chapter to give them some big picture views of how to apply the language to their problems. Once they've read those four chapters finishing off the Beginner's Guide should leave them self-sufficient enough to use the language and type sections of the Puppetlabs documentation site. After that I'd finish reading Pro Puppet for the breadth of coverage.

As for next steps, if you are interested in extending Puppet then Puppet Types and Providers is the best book on the market and provides a very solid introduction to the subject. If you are in the middle of bringing a number of different services under puppet management then from what I've heard the Puppet 3 Cookbook will expose you to a number of handy approaches and code snippets. After that? Well, you'll know more than most puppet users and it's time to get involved with the community through the forge and Planet Puppet.

Posted: 2013/11/30 14:27 | /tools/puppet | Permanent link to this entry


Pro Puppet 2nd Edition - Initial Impressions

The kind people at Apress provided me with an alpha review copy of Pro Puppet and while it's not the finished product you can already get a good feel for the books tone and coverage.

I quite liked the first edition of Pro Puppet and this update is more evolutionary than revolutionary. All chapters from the previous edition are present and the biggest addition is the very welcome chapter on using Hiera in your modules; even if it's oddly placed at the end of the book. The updates to the material are nicely woven in to the existing chapters and for a book with this many authors the writing style is nicely consistent.

Pro Puppet is quite wide in its coverage of puppet and it's ecosystem and provides a nicely hands on approach that encourages you to type along with the text, exposing you to a number of approaches and tools that will stay in your toolbox. The flip side of this is that the coverage is quite shallow in some areas, for example rspec-puppet is introduced in the 'Tools and Integration' chapter but it not used to test your new providers in 'Extending Facter and Puppet'.

I think Pro Puppet is an ideal first book for someone who has already gained some hands on puppet experience and is looking for a way to improve their abilities while learning more about the surrounding tools. After reading this you'll be in a good position to puppetise your systems while knowing enough to be able to dip in to the online docs as needed to supplement your reading.

In summary Pro Puppet 2nd Edition is a good book that covers a lot of ground. It's very well suited for people that learn by doing but it's not an essential upgrade if you've read and used the previous edition.

Posted: 2013/11/30 14:16 | /tools/puppet | Permanent link to this entry


Thu, 10 Oct 2013

Liquid Templates in Puppet - Initial Release

Puppet has always supported templating via ERB and while it's a powerful, flexible templating engine the ability to use any arbitrary ruby code inside a template that's run on the puppet master sometimes raises some eyebrows. As part of a security architecture review the concept of replacing the templating engine with something that still allows looping and text manipulation without allowing too much else was discussed and led to the idea of allowing templates to be written in Liquid.

Liquid is a 'Ruby library for rendering safe templates which cannot affect the security of the server they are rendered on.' That sounds like what we want so let's install the module and write a small test template



# required dependency
$ sudo gem install liquid

# install the function
$ puppet module install deanwilson/liquidtemplates

# create the test module structure
$ mkdir -p liquidtest/{manifests,templates}

# create the test class
$ cat << 'EOC' > liquidtest/manifests/init.pp
class liquidtest {

  file { '/tmp/liquidtest':
    ensure  => 'file',
    content => liquid_template('liquidtest/test.liquid'),
  }

}
EOC

# and the test template
$ cat << 'EOC' > liquidtest/templates/test.liquid
SELinux is {{ selinux_config_mode | upcase }} on {{ fqdn }}
EOC

$ puppet apply -v -e 'include liquidtest'
 ... snip pretty coloured output ...

$ cat /tmp/liquidtest
SELinux is ENFORCING on kvm06.udlabs


There is additional overhead in writing your templates in a language that's not puppets default but for situations where you have a number of different people writing your templates, changing to something like Liquid can provide another layer of protection for your puppetmasters. This is an initial proof of concept and while it's enough to keep our conversation going you may not want to move everything to it just yet.

Posted: 2013/10/10 11:24 | /tools/puppet | Permanent link to this entry


books career cloud events firefox geekstuff linux meta misctech movies nottech perl programming security sites sysadmin tools tools/ansible tools/commandline tools/gui tools/network tools/online tools/puppet unixdaemon

Copyright © 2000-2013 Dean Wilson :: RSS Feed