07 Jan 2016, 16:45 Continuous Delivery

Automatically Managing an Apple TV Build Radiator

I’ve recently joined IBM’s Bluemix Garage in London. The Garage is all about being lean and innovative. They have some really nice setups, including two-screen two-keyboard pairing stations, and Apple TVs instead of the more traditional data projectors. Whenever any of us want to present, we just mirror our screen to an enormous TV using AirPlay. No more fiddling with projector cables - bliss! When we’re not using the TV for a presentation, we have it showing our build radiator and our telepresence window. We commit tens of times a day, so having the build radiator always-on means the team notice almost instantly if one of our commits causes a build problem:

In my previous roles, I’ve always loved it when we brought fresh blood into my team. New people don’t know all our habits, and they don’t take stuff for granted, so they ask questions. I’ve now found myself on the other side of that relationship; I’ve been the one asking a lot of - mostly dumb - questions. One of those questions was “why isn’t the TV showing our build radiator anymore?” It turns out that every time we used our TV to share a presentation, we had to walk over and manually re-enable the sharing from the build monitor machine.

Well, I’ve never seen a problem I didn’t want to automate. When I heard the word ‘manual’ I had to drop what I was doing (“thud”) and run off and write a script.

Scripting mirroring a display to an Apple TV

There are some excellent resources on how to use AppleScript to drive the AirPlay menu, and it’s pretty easy (the applescript file I ended up with isn’t very long).

tell application "System Events"
    tell process "SystemUIServer"
        click (menu bar item 1 of menu bar 1 whose description contains "Displays")
        set displaymenu to menu 1 of result
        -- Tolerate numbers in brackets after the tv name --
        click ((menu item 1 where its name starts with tvname) of displaymenu)
    end tell
 end tell

One complexity is that the name of our Apple TV box sometimes gets a number in brackets tacked on to the name, so we can’t just hardcode the name. Using a “starts with” selector fixes that problem.

Driving the script automatically

I also set up a launch agent (a .plist file) to drive the display mirroring every five minutes. The exact plist file will depend on where the scripts are extracted, but copying our .plist file to ‘~/Library/LaunchAgents’ and updating the paths is a good start.

Detecting when an Apple TV is already showing something

This solution wasn’t quite good enough, though. If one of us was ~currently~ sharing our screen, we didn’t want the build radiator to grab control back every five minutes. How could we tell if the screen was currently being used? It turns out that this is harder. My initial assumption was that we’d get a ‘Garage TV is being used by someone else’ prompt if an AirPlay share was active, but we didn’t. I suspect it depends on the version of the Apple TV and also the type of share (photos, videos, mirroring … ). Looking at the AirPlay protocols, there doesn’t seem to be any HTTP or RTSP endpoint on the Apple TV which reports whether it’s currently active. That’s ok - we can figure it out ourselves.

AirPlay mirroring traffic is all handled on port 7000 of the Apple TV (other types of share uses different ports). We can sniff the network traffic to see if there are packets flying around, and only start AirPlay mirroring if there aren’t. Bonjour allows us to find an ip address if we know the device name, and tcpdump shows us traffic.

To get the IP address of an Apple TV, replace spaces with hyphens in the name, append a .local domain, and then ping it. The ping output will include the ip address, which can be parsed out. For example,

 # Substitute dashes for spaces to find the Bonjour name
 tv hostname=${tvname/ /-}.local
 ipaddress=$(ping -c 1 $tvhostname | awk -F'[()]' '/PING/{print $2}')

Then tcpdump can be used to monitor traffic to that host. My initial implementation was this:

 sudo tcpdump tcp port 7000 and host $ipaddress > /var/tmp/airplay-tcpdump-output

The tcpdump command will run indefinitely, so we let it run for ten seconds, then stop it:

# Get the PID of the tcpdump command
pid=$!
# Capture 10 seconds of output, then kill the job
sleep 10
sudo kill $pid

You may need to add tcpdump (and kill) to the sudoers file so that it doesn’t prompt for a password:

 buildmachine ALL=(ALL:ALL) NOPASSWD: /usr/sbin/tcpdump
 buildmachine ALL=(ALL:ALL) NOPASSWD: /bin/kill

This worked great when I tested locally, but when I tried it on a bigger network, I discovered that on a wireless network, traffic is point-to-point, and tcpdump can’t gather packets promiscuously. In other words, I could only see traffic to the Apple TV when it was coming from my machine. Useful, but not useful enough.

Switching to monitor mode allows all packets to be viewed, but since packets on a secure network are encrypted, tcpdump can’t filter on tcp-level details, like the port and IP address. Luckily, tcpdump does allow filtering by MAC address, and the arp -n command can be used to work out the MAC address from the IP address.

 arp -n $ipaddress &> /var/tmp/arp-output
 fieldindex='$4'
 macaddress=`awk -F"[ ]" "/($ipaddress)/{print $fieldindex}" /var/tmp/arp-output`

Finally, the output file can be parsed with awk to count how many packets have flown past.

# Process the output file to see how many packets are reported captured
packetcount=`awk -F'[ ]' '/captured/{print $1}' /var/tmp/airplay-tcpdump-output`

A heartbeat packet is sent every second, so I assumed that after listening for 10 seconds an in-use device will always generate some traffic. Listening to the tcp packets on port 7000 (my first attempt) doesn’t yield very many packets, so any packet count greater than 0 indicates the device is in use. Once I started listening to the lower-level packets, there was always some traffic, even when the tv isn’t in use, so I used a threshold of 20:

if [ $packetcount -gt 0 ]
then
    # Handle in-use case
else
    # Handle not-in-use case
fi

The pieces are connected together in the shell script.

Digging in the depths of Apple TV

If you want to do more hacking with AirPlay, the following command lists all devices on a network: dns-sd -B _airplay._tcp

To see the details for an individual device from the list, which includes version and configuration information, you can do (for example)

dns-sd -L "6C94F8CF0F3F@Garage TV" _raop._tcp local

That gives

Lookup 6C94F8CF0F3F@Garage TV._raop._tcp.local
DATE: ---Thu 05 Nov 2015---
16:38:28.679  ...STARTING...
16:38:28.680  6C94F8CF0F3F@Garage\032TV._raop._tcp.local. can be reached at Garage-TV.local.:7000 (interface 4)cn=0,1,2,3 da=true et=0,3,5 ft=0x5A7FFFF7,0x1E md=0,1,2 am=AppleTV3,2 pk=d8b7d729ee2cd56d19e1fe2c9396f63fa41b227727e1b510dc925396d2d86668 sf=0x204 tp=UDP vn=65537 vs=220.68 vv=2

10 Mar 2015, 15:52 Bluemix Continuous Delivery XP

Blue-Green Deployment

One of our core principles is to deliver stories with the highest business value from day one. To deliver something means to put it in front of our customer. Regardless of whether our customer decides to make it public or not, it is available in production from the very first cf push. We strongly believe that having a partially complete solution in production from day 1 is more valuable than having the full solution delivered later.

All Bluemix Garage projects are continuously and automatically deployed into production, regardless whether they are internal tools or software components that we develop for customers.

One of the goals with any production environment is maintaining high availability for your application. The CloudFoundry platform helps with this by giving you the ability to scale your application horizontally to prevent downtime. There is another availability challenge when continuously deploying applications - if you cf push an existing application in production, your instances must be restarted to update your code. Not only does this inevitably introduce downtime, but you are left without a fallback if something goes wrong.

Blue Green deployment is not a new concept by any means. Nor is it tied to any particular platform. In short the idea is:

  1. Deploy a new instance of your application (leaving your current “green” environment untouched).
  2. Run smoke tests to verify the new environment, before taking it live.
  3. Switch over production traffic, keeping the old environment as a hot backup.

As we are deploying straight to production, using blue-green is imperative.

The current status-quo

We discovered these two tools:

  • Buildpack Utilities from the IBM WAS team — An interactive Bash script that will take you through the process
  • Autopilot from Concourse — A plugin for cf (We really like that it’s built into the tooling)

Both tools in fact do roughly the same thing:

  1. Rename the existing app while keeping all routes in place
  2. Push the new instance, which will automatically map the correct routes to it
  3. Delete the old instance

There are three problems with this approach:

  • They do not support running tests against the new app before making it available on the production route.
  • They immediately delete the app, making it impossible to revert to the old environment if something goes wrong
  • There is a chance that the old app could be serving requests at the time it is deleted.

While Blue-Green is quite a simple concept, we discovered several nuances in how we wanted to approach the problem so we decided to build our own.

Our Take

Our continuous delivery environment already consisted of a couple of shell scripts that provide app lifecycle management on top of the cf cli. We decided to build an extension to these scripts to manage Blue-Green with this process:

  1. Push new app instance to Bluemix with a different name
  2. Run end-to-end integration tests against the new app instance using the auto-generated route
  3. Map production routes (including domain aliases) to the new app instance
  4. Unmap routes from previous app instance
  5. Clean up all older app instances, leaving only the previous instance running as a hot backup

If any step fails then the CI server immediately gives up. We don’t want to continue the deployment if anything goes wrong!

The crucial difference is that we do not touch the production app until we know we have a good instance to replace it with. This means using a new app name for every deploy — as easy as passing a specific application-name when calling cf push. We chose to append the date and time for every deploy, but you could also use the git hash or CI build number.

After the new instance has been tested — with simple smoke tests in our case, the tool will then cf map-route the production route to the application. Now the new instance is handling production traffic, cf unmap-route can be used to disconnect the old instance from the world, leaving it untouched until the next deploy.

Deploying with a new application name each time, means we always run in a clean environment. However, without cleaning up, we’ll be left with a lot of redundant, idle applications consuming resources. This is easy to deal with, we simply delete all previous instances except the new app and the instance we just unmapped.

Blue, Green, Cloud.

Arguably what we’re doing here isn’t strictly Blue-Green deployment. The name “Blue-Green” stems from there being two separate environments on physical hardware, each of which could run an instance of the application. It wasn’t possible to throw away the old hardware and start with a completely fresh system for each install so it was necessary to flip between them.

Developing for a Cloud environment such as IBM Bluemix, means that creating whole new application environments is a simple cf push away, and deleting them is just as easy. With such a cheap way to create whole new environments it just doesn’t make sense to keep flipping between two apps when you can always start afresh.

We’ve been running several applications in this configuration for months now and we’re really pleased with what we’ve built. Everyone has different needs, this works for us but it’s just a different way to approach the problem.

Update 07/09/2015

We have recently open sourced our Blue-Green deploy script. It is now available as a Cloud Foundry CLI plugin. The source code and documentation can be found here:

https://github.com/bluemixgaragelondon/cf-blue-green-deploy

06 Mar 2015, 11:15 Hugo XP Continuous Delivery

Continuously delivered to Bluemix, served by Hugo

This website runs on Bluemix and is built with Hugo, a static site generator written in Go. Pages are served directly by hugo server which thinly wraps the http package from Go’s stdlib.

The entire website setup is versioned by git including the Hugo binaries, continuous delivery scripts, content & assets.

Every local commit gets pushed to a Jazz Hub git repository. Our continuous delivery environment, which runs on Jenkins, triggers a new build every time there is a new commit to this repository. This translates to pulling the latest version from Jazz Hub, running cf push, checking that the new version of the website is up and running and finally making it available to the public. Widely known as zero downtime deployment, we call it Blue-Green deployment since it describes the process more accurately.

Why Hugo?

After having considered WordPress, KeystoneJS & Ghost, we have settled on Hugo for the following reasons:

  • a platform-specific binary file is the only dependency
  • no database required
  • fast feedback when creating new content
  • fits perfectly with our way of working (version control, instant feedback, Continuous Delivery)

Why Bluemix?

We could try other popular alternatives which needed a PHP & node.js runtime with no effort. Rather than spending time with programming language or database setup, we could focus on finding the solution which best met our needs. Bluemix Containers came in handy while evaluating Ghost & WordPress. As for KeystoneJS, a new instance was a simple a cf push away.

Once we have settled on Hugo, we had continuous delivery and blue green deployment set within minutes. We can scale to as many app instances as we need with a single command. We went with the Go buildpack initially, then settled on a generic CloudFoundry binary buildpack as it made our deployment 50% quicker.

Why Continuous Delivery?

All Bluemix Garage projects are continuously delivered, without exception. Whether it’s a utility such as our Bluemix Blue Green Deployment, cf cli plugins, our website or any projects which we deliver for our clients, they are all developed using Extreme Programming practices.

No code is written before there is a failing test. We keep the cost of change low by continuously refactoring. We always pair program. We keep our commits small and constantly push to a central repository. Our Continuous Delivery environment pushes directly to production. We keep all feedback loops as small as possible so that we can quickly learn what works and what doesn’t. There is no better way of consistently delivering working software of the highest quality.