11 Oct 2017, 20:02 Technologies Development Liberty Local

Using webpack watch with the Gradle and the Liberty Application Server

Using webpack watch with the Gradle and the Liberty Application Server

I recently realised that while I’m very comfortable coding in Java, and I’m become a lot more confident with Javascript, I ~had no idea how to use the two together~. When I was doing back-end code and implementing middleware, I was using Java, and when I switched to writing full-stack applications, I used Javascript for both front and back end. In the rare cases when I needed to do both a front end and a back end, I’d use webjars. Although webjars are handy, they’re also limiting for people used to npm.

Using gradle with modern client-side Javascript

I had some time in an airport (where all good development is done), so I decided to learn how to get a more powerful cross-language build. The heart of the application was Java, so I used gradle as my base build language, but I also wrote a package.json to capture some js-related content.

First, I defined a gradle task to wrap npm install.

task installJavascriptDependencies(type: Exec) { 
    inputs.file("package-lock.json") 
    inputs.file("package.json") 
    outputs.dir("node_modules")
    
    commandLine "npm", "install"
}

Although there are a few options, I find webpack works well for minification and bundling. I defined a gradle task which calls directly out to the webpack command line

task webpack(type: Exec) { 
    dependsOn installJavascriptDependencies
    inputs.file("webpack.config.js") 
    inputs.dir("src/main/webapp")
    outputs.dir("$buildDir/js")
    
    commandLine "$projectDir/node_modules/.bin/webpack", "-p", "--output-path", "$buildDir/js", "src/main/webapp/index.js" 
}

The webpack task depends on the install task. Notice also that the directory structure is gradle-style, rather than node-style; the client lives in src/main/webapp.

Finally, I modified the default war task to also copy in the generated javascript into the war:

war {
  dependsOn webpack
  from( "${buildDir}/js" )
}

So far so good, but every time I made a change, I had to re-run the whole build, including the full webpack processing, which was taking three minutes. Three minute builds do not go well with debugging.

Setting up a loose application

Liberty has the ability to run ‘loose applications’, that is, applications which haven’t actually been zipped up into a jar file. Instead, there is a virtual packaging structure defined by an xml file. Loose applications work well with Liberty’s application monitoring and dynamic application updates (hot reload). Every time a file in the virtual assembly is updated, Liberty will near-instantaneously update the application. This is perfect for development.

You can write a .war.xml file by hand and set up monitoring in the server.xml, but it’s simplest to let the IDE take care of both. For example, in Eclipse, right click on a web application and choose “Run on Server”:

If you look inside the server.xml, you should see something like the following:

    <applicationMonitor updateTrigger="mbean"/>

    <webApplication contextRoot="/" id="some-name" location="some-name.war" name="some-name"/>

It’s a good idea to have different server definition for the dynamic server and the production one, although they can include a common set of config. And, of course, the user directories should all be source-controlled (by storing them outside the runtime install tree.

In order to ensure all my runtime dependencies were included in the loose war, I had to edit the Deployment Assembly to include dependencies which should be included in the package:

Using webpack watch

All changes to my Java application would now be automatically reflected in the live app, but the client side still needed a full rebuild. To get the final piece of dynamism, I defined a script for dynamic webpack in my package.json:

"build:watch": "webpack --watch --progress --display-error-details --output-path build/js"

I then launched the webpack watch daemon from the command line and left it running:

npm run build:watch

With this, both client and server changes to my application were live almost immediately, and I was a happy debugger.

21 May 2017, 16:45 Technologies Cloud Foundry Bluemix Development

Configuring a Bluemix Cloud Foundry Application To Run Locally

During the course of a typical day, a Bluemix Garage programming pair will deploy code around twenty times. Each of those code pushes will be unit tested in the Build and Deploy pipeline, integration-tested, and then be subjected to a final smoke test before going live. All of this cloud-based testing is marvellous, but it’s not a substitute for pre-push local testing.

Every Cloud Foundry application has a VCAP_SERVICES environment variable automatically created for it, with details of bound platform services and user-provided services. For unit tests we always stub services, but for integration tests and general ‘exploring the application locally,’ it’s convenient to run against the same services as the live application.

This is trivial to do by creating a local .secrets file (and .gitignore-ing it), and sourcing the .secrets file before running. The .secrets file can be assembled manually by reading from the Bluemix console or inspecting the output of cf env, but it’s easiest to generate it automatically. The Garage has a handy script for generating them. Generating it automatically has an extra advantage; some Bluemix services generate a new set of credentials every time they’re bound to an application. Manually maintaining a .secrets file can be hard work if pushes are frequent. In those cases, it’s best to create an extra set of credentials in the Bluemix console, and name them PERMANENT_CREDENTIALS. If this naming convention is used, the script will prefer the permanent credentials to the transient ones.

For example, the complete workflow to run a Node.js application locally would be:

cf login
script/generateSecrets
. .secrets 
npm start

where the first three steps only have to be done once per terminal window.

Using generateSecrets will pick up a Bluemix setting which puts Node.js into production mode, so you may prefer to comment out the line for sharing environment variables, as follows:

//  write(environmentVariables.join(''));

11 Apr 2017, 16:45 Technologies Watson Conversation Chatbots

Chatbot Best Practices

One of the most frequent questions clients ask the Bluemix Garage is “can you build us a chatbot?”. This is reflective of an industry-wide trend towards more natural language in computerised interactions, and also more automation of interactions which are currently handled by humans. There are currently more than 33,000 chatbots on Facebook Messenger alone. Many businesses are turning to Watson Conversation to help take out cost and improve user satisfaction. Our Hursley Labs colleague Simon Burns has written an excellent series of articles on how to write great Watson chatbots, which you should definitely go read. Think of this as a supplement, with our experiences from the field.

Users will expect human-like behaviour

While you should make it clear to users they are not dealing with a person, that doesn’t mean you should ignore their expectation of human-like behaviour (this is the positive side of skeuomorphism). Building in some charm and humour to your chatbot will almost certainly improve the user experience. On one chatbot project we did, the chatbot would detect profanity and respond with a mild rebuke (“Oh, I don’t like that word” or something similar). When we analysed the conversation logs, we found many users would apologise to the chatbot after it asked them not to swear. I think it’s lovely that people are saying “sorry” to computers, and I wouldn’t want a world where everything was so sterile and transactional that those kinds of characterful interactions weren’t possible.

Conversational patterns

Acknowledge your limitations and don’t be afraid to say “I don’t know”

When users head beyond the scope you’ve defined for your chatbot, be open about it. Let them know this isn’t something the bot can handle, and give some suggestions for things they can ask. “I’m only a computer and I don’t know everything” can work wonders.

The Watson Conversation service returns a confidence for each detected intent. In general, users will be a lot more disappointed by an incorrect or nonsensical response than they will be by a message saying “I’m just a bot, so I don’t know that.” In order to maintain user trust in the system, you should check the confidence of the top intent, and if it’s below a threshold, return a response like “This is too hard for me.” How damaging a wrong answer is depends on the domain, which will affect the value you choose for the confidence threshold.

Users like concision


When we imagine a chatbot, we imagine something so human-like that users will interact with it exactly as they would a person. Users, however, may see the text input box and expect it to behave like a search engine. This is ok! Don’t require your users to type four words when one would suffice. A conversational interface shouldn’t be a burden.

Unfortunately, when users input only one word, with no surrounding context, it’s sometimes hard for a conversation engine to correctly interpret the intent. Be aware of this when creating your training data.

For example, the garage wrote a system to help users plan holidays. We were expecting to be able to understand destinations we didn’t know about from the surrounding context, but we found users preferred single words. This made understanding a lot harder. For example, it was easy to answer “I want to fly to Chippawa” with “I’m sorry, we don’t fly there”, but what’s the correct answer to “Chippawa”; it could be a food, or a place, or a type of kayak … 
 As well as supporting single-word inputs, consider supporting one-tap responses. For example, if a user needs to respond with “yes” or “no,” don’t force them to type that out; give buttons. The same is true for cases where there are only a handful of possible responses, and even cases where there are an arbitrary number of responses but a subset are very popular. Buttons can also be useful for starting a conversation, to let users know what kinds of things they can ask.

Don’t repeat yourself

One pattern we’ve noticed in chatbot interactions is that the conversation may often circle back to the same node in the conversation tree. This will almost always frustrate users, so there are a few things you should do to handle this:

  • Provide multiple responses for each node. The Watson conversation service allows you to specify a range of responses for a single node in the conversation tree. It can select one at random or work sequentially through them. Take advantage of this.
  • Detect repeat visits to a single node, and do something about it. Almost certainly, if the conversation is going in circles, something has gone wrong. This is a good point to hand off to a person or make the scope and limitations of the chatbot clear, perhaps with a humorous apology.

Detect frustration, and handle it.

More generally, use the technical capabilities available to understand your users’ emotions. If they’re happy, that’s great, but if they’re frustrated, that should be addressed. For example, “I can see this isn’t going so well. Would you like to talk to a real person?”

Hand off to a human

Some interactions are too complex for a computer to handle — or rare enough that it’s not worth teaching a computer to handle them. Detect these, and hand off to a person. Even if a chatbot just handles the first, mechanical, part of a conversation or the most common questions, there can be great efficiency gains.

Context is important

Users will expect you to preserve context. The Watson conversation service has a mechanism to add things to the context and read information from the context, but what to do with the context will be particular to your domain. Too little context-dependent behaviour will frustrate users, but too much will increase the complexity of your conversation tree beyond what’s manageable. Too many closely-related nodes with subtly different behaviour for different conversation histories are a maintenance headache.

Slow down responses

Users are less likely to trust a response that seems to come back implausibly fast. Consider inserting a small pause before printing out a response, or print the response a word at a time for a more natural effect (yep, more skeuomorphism!). 


Open user intents

Open user intents are particularly hard to handle, and can lead to user frustration if intents are misinterpreted.

Here are some things we’ve found can help:

  • Lots and lots of examples to help the tool
  • Turn an open intent into a large, but, closed intent (by programatically creating an entity)
  • Pre-process with Alchemy Language to add extra semantic understanding (for example, Alchemy can detect the input language)

Then, observe your users

User Testing

Observing real users interacting with your product is an essential part of user-centred design. It’s extra-essential (if that’s possible) with chatbots, because supporting natural user interactions is the defining characteristic of the system. The range of possible interactions is almost unlimited, which makes all of design, implementation, and user testing a bigger challenge.

Users doing a user test may behave differently from users in the field

One thing we’ve observed is that because chatbots are a relatively new technology, users in a user study would often try and test the limits of the technology. For example, in a chatbot designed to help users track a lost parcel, test users (who in most test groups, hadn’t actually lost a parcel) might be more interested in the answers to questions like “how are you?” or “how smart are you?” than “where is my [fictious] parcel?“. This can skew the results of user testing. It doesn’t mean user testing shouldn’t be done, but it does mean user testing isn’t a substitute for monitoring and tuning interactions in the live system.

Test, Monitor, Tune

One of the first things you do when developing a chatbot system is instrument it, so that it can be properly monitored in the field. Make a dashboard where you can review what queries did not work, everyday. The Watson Conversation services pre-instruments the conversation and comes with a built-in dashboard. Like with search engine development, looking at the mistakes (and there will be mistakes) is what helps you improve. The requirement to iterate fast and catch problems early is even higher in chatbots than it is in apps.

It’s important to be prepared for the fact that users will use the tool in ways you don’t expect. You will miss obvious user intents in the development phase, because you have an idea how it will be used and users have different ideas.

Solicit user feedback

If things go well, at some point manual review of all of the conversation logs will become unsustainable because you’ve got so many users. This is great, but it doesn’t mean there’s no longer any need to monitor or tune. At this point, you should switch focus to identifying and correcting cases where the bot’s confidence was low, or tone analysis indicated users were frustrated. It’s also a very good idea to let your users tell you when things are going wrong using a non-conversational mechanism. There are lots of ways to do this, such as thumbs up and thumbs down buttons, a star ratings field, or a ‘This answer did not help me’ button. As well as providing you with a useful way of capturing cases where the conversation is going wrong, giving feedback may make your users feel better, and you can use the feedback to take corrective action within the conversation, or hand off to a human.

In summary:

  • User-test as much as possible before launch
  • Closely monitor dashboard post-launch and tune conversation tree daily
  • Add new paths to the integration tests to ensure they keep working as the tree is tuned
  • Make the conversation tree as rich as possible, to cover common paths (but keep the scope realistic)

… and don’t forget to observe the rest of the system (technical considerations)

You need a lot of automated tests - and you should write them first

Good conversation trees can get pretty complex. Just like good code, they need a robust set of tests to make sure that they behave the way you want, and that stuff which works continues to work as you layer in more function. (Just as working code can be regressed by new function, a working conversation tree can change in unexpected and unwanted ways when new nodes or examples are added.)

You will probably be exploring the behaviour of your conversation tree manually as you go along, but this isn’t enough. The Watson conversation service provides a REST API which lends itself well to automated tests, so get these built into your process as soon as possible. We use test-driven development in the Garage, so we write the automated tests for the conversation before we change anything in the tree itself.

For each intent, we’ll check a couple of example user inputs and confirm that the output from the conversation tree includes the expected intent and any expected context. We didn’t want our tests to be brittle, so we generally checked just for the returned intent, and not the precise text of the returned message. We also found writing some automated tests for entities helpful, although the entity detection is more straightforward than the intent recognition.

Think about Devops

One problem we have found in the past is that our automated tests would fail, to let us know that there was a problem with the conversation, but it was too late - the problematic changes had been made to a live conversation workspace. Conversation workspace changes should be managed using the same blue-green-deployment principles as the rest of the application.

In other words, both in order to avoid regressions, and in order to avoid delivering half-baked changes, it’s a good idea to edit a copy of the conversation workspace, and then switch the application over to use that copy only when you’re satisfied everything is working. This can be done programmatically as part of your build script, or as a separate build process. You can either update the live workspace with the json of the staging workspace once the tests pass, or keep the staging workspace as it is and use a user provided service to switch which workspace the application uses. The old ‘live’ workspace then becomes the new ‘staging’ workspace.

Backup your work

As part of the devops flow to test and promote workspaces, take backups! A conversation workspace is a valuable asset, and it should be source-controlled.

Finally …

There’s a reason bots are generating so much interest. After decades of buttons on screens, they offer us a different way of interacting with computers. From a business perspective, they can automate repetitive communication, open up the possibility of voice transactions. For us as developers and designers, they’re an interesting new challenge in user experience design and a new set of patterns to work out. We’ve enjoyed the Garage chatbot projects, and we’re confident there will be many more to come.

25 Oct 2016, 10:28 Containers Secure Gateway Connectivity Docker

Using the Secure Gateway to Connect from Your Local Network to the Cloud

The IBM Bluemix Secure Gateway service allows connectivity from Bluemix applications to local/on-premise destinations. There are many excellent articles which cover this scenario, such as:

However, a lesser-known scenario is that it can also connect back from your on-premise system to cloud-based destinations. Although this may not be the primary motivation for using it in many scenarios, it can be very useful with restrictive on-premise networks which don’t allow outbound connectivity.

This blog post walks through a very simple example of setting up this connectivity to connect to a public JSON-based service (http://loripsum.net/api). Although on your local test system, it is most likely that direct connectivity to that endpoint is possible, in many on-premise enterprise scenarios, that may not be so straightforward. This Secure Gateway setup thus becomes one possible option. We will define a Secure Gateway service, and use the Docker-based client, running locally on your machine, to connect back to the Secure Gateway.

Prerequisites

Instructions for this tutorial are written assuming you are using OS X, although they can probably be adapted to other platforms fairly easily.

Define the Secure Gateway

  • In a browser in which you are logged in to Bluemix, navigate to the URL https://console.ng.bluemix.net/catalog/services/secure-gateway/. Click on “Create” to create the Secure Gateway service. Note: if you already have an instance of the Secure Gateway service in the space in which you are working, you will be unable to create a new one. You can use the existing instance of the service you have and simply create a new Gateway within it (navigate to the Secure Gateway Dashboard in the Bluemix interface). Note also that the Secure Gateway has a cost attached once you exceed a certain threshold of data. Please make sure you are comfortable with this restriction.

  • You will now be in the Secure Gateway Dashboard. At this point, click “Add Gateway”. Specify a name for the Gateway (e.g. “GatewayFromLocal”). For this simple tutorial, you can disable “Require security token to connect clients” and “Token Expiration”. Click “Add Gateway”. It will look like this:

    Add Gateway

  • Once you’ve created your Gateway, you’ll need to add the Client (the local Docker container which connects back from your machine to Bluemix). Locate the Gateway on the administration console (it will have a red X to show it is not yet connected). It will look like this:

    Gateway Disconnected

Click on it.

  • Click on “Add Clients”. This will bring up a dialog with multiple connectivity options. Click on “Docker”. This will show the precise Docker command required to run the local Gateway client on your local machine. The dialog should look like this:

    New Gateway

The identifier at the end of docker run command is the ID of this individual gateway.

  • You will need to modify the docker command to make sure that the port number we pick later on (called the “Client Port”) is mapped from the Docker container (which has a separate network, with a separate set of ports) to your host machine. Add a parameter like -p 6655:6655 to your command (you can pick any number you want, although it’s recommended to be above 1024). It should look like this:

    docker run -p 6655:6655 -it ibmcom/secure-gateway-client vJOMUBbT9Ax_prod_ng
    

Run the docker run command locally on your machine. You should see output that looks a bit like this (your client ID will, of course, be different from the one shown).

    IBM Bluemix Secure Gateway Client Version 1.6.0
    ************************************************************************************************
    You  are running the  IBM Secure  Gateway Client for Bluemix. When you enter the provided docker
    command the IBM Secure Gateway Client  for Bluemix automatically downloads as a Docker image and
    is executed on your system/device. This is released under an IBM license. The  license agreement
    for IBM  Secure Gateway Client for Bluemix is available at the following location:

    http://www.ibm.com/software/sla/sladb.nsf/lilookup/986C7686F22D4D3585257E13004EA6CB?OpenDocument

    Your use of the components of the package and  dependencies constitutes your acceptance  of this
    license agreement. If you do  not want to accept the license, immediately quit  the container by
    closing the  terminal  window or by  entering 'quit' followed by the ENTER key. Then, delete any
    pulled Docker image from your device.

    For client documentation, please view the ReadMe located at:
    .rpm and .deb installers: /opt/ibm/securegateway/docs/
    .dmg installer:           <installation location>/ibm/securegateway/docs/
    .exe installer:           <installation location>\Secure Gateway Client\ibm\securegateway\docs\
    ************************************************************************************************


    <press enter for the command line>
    [2016-10-25 04:57:37.944] [INFO] (Client ID 1) No password provided. The UI will not require a password for access
    [2016-10-25 04:57:37.953] [WARN] (Client ID 1) UI Server started. The UI is not currently password protected
    [2016-10-25 04:57:37.953] [INFO] (Client ID 1) Visit localhost:9003/dashboard to view the UI.
    [2016-10-25 04:57:38.169] [INFO] (Client ID 11) Setting log level to INFO
    [2016-10-25 04:57:39.249] [INFO] (Client ID 11) The Secure Gateway tunnel is connected
    [2016-10-25 04:57:39.546] [INFO] (Client ID Hh2eUmsFUi2_u7U) Your Client ID is Hh2eUmsFUi2_u7U
  • Because you specified the -it flag to docker, the docker client will continue to run. Leave the window open so you can monitor the client easily. Very shortly afterward, the red cross against the Gateway in the Secure Gateway Dashboard should turn to green to show that the Docker client has successfully made a connection. If it does not, proceed no further and debug.

  • Now, we’ll add a “Destination” for a Cloud-based service which we can invoke locally. From within the Gateway control panel, click on “Add destination”:

    Add Destination

  • Click “Advanced Setup” and fill out the screen as shown. The “Client Port” is the port locally within the Docker container - choose the same number you chose earlier for the docker run command. Click on ‘Add Destination’:

    Add Destination Details

(Occasionally I have found that the process of adding a destination can hang at this point. If this happens, refresh the page - you will probably see that the destination has not been added to the gateway. Then, add the destination again, using the same details, but select “TCP” as the type. Once it is created, go in and edit the details, changing the type to “HTTP”. This should work).

  • If the destination is successfully created, you should see a message that looks like this:

    [2016-10-25 05:19:55.346] [INFO] (Client ID Hh2eUmsFUi2_u7U) Creating TCP server for reverse connections. Listening on port 6655
    
  • You can now test this. Open a browser to the URL http://loripsum.net/api. You should see some pseudo-HTML containing some Lorem Ipsum text. Your network may allow this directly, depending on your proxy settings, but of course this scenario is most useful when this direct connectivity is not possible!

  • Now, open a browser to http://localhost:6655/api. You should see exactly the same result. This means that your browser is connecting through your local Docker client, across the Secure Gateway, and out onto the public internet endpoint. Even though in this case your network may allow direct connectivity to the original endpoint, this illustrates the values of the Secure Gateway in “bridging” connectivity.

Addenda

Any questions or comments, contact me at andrew DOT ferrier AT uk DOT ibm DOT com.

04 Oct 2016, 10:40 Containers Splunk Logging Docker

Using Splunk in a Docker Container on Bluemix

One of the things that the IBM Bluemix Platform (based on Cloud Foundry) supports is logging to external logging providers. I was recently reading Takehiko Amano’s excellent article on Splunk integration with IBM Bluemix. Splunk is becoming very popular in the industry generally for log aggregation, indexing, and searching.

Takehiko’s solution is excellent, but still requires somewhere to deploy Splunk. However, Bluemix itself provides the IBM Containers offerings (based on Docker technology) where Splunk can be run. This probably isn’t suitable for robust production environments, but for quick ‘n’ dirty testing, it’s really useful. I’ve documented below some steps that you need to get this up and running with Splunk Light, which is the simpler, lighterweight edition of Splunk.

Prerequisites

Instructions for this tutorial are written assuming you are using OS X, although they can probably be adapted to other platforms fairly easily.

Build the Splunk Container

You need to build the Docker container for Splunk locally before pushing it up to the Bluemix containers repository. There’s already a well-established GitHub project for a Splunk Docker container, but we need to add the RFC5424 add-on as per Takehiko’s article to get Splunk to recognize the logging format.

I’ve already forked the GitHub repository and added most of the changes required to do that, but you will need to download the add-on itself first.

  • Open a terminal and clone my repository, checking out the bluemix branch:

    git clone -b bluemix https://github.com/andrewferrier/docker-splunk/
    
  • Download the RFC 5424 add-on. You will need to sign up for a free splunk.com ID if you don’t already have one. Put the .tgz file in the splunklight directory inside your checked-out git repository.

  • Now build the Docker image (which may take a little while):

    cd <your_checkout_directory>/splunklight
    docker build -t andrewferrier/splunk:latest-light .
    

(If you wish, you can substitute your own namespace prefix in place of andrewferrier - as long as you use it consistently below).

Push the Splunk Container up to Bluemix and start it running

Firstly, log into Bluemix and initialize the container runtime:

bx login
bx ic init

You will need to specify an organisation and space within which to work on Bluemix.

Next, double-check what your IBM Containers “namespace” is. If you’ve worked with Containers before, you’ve probably already got one specified. You can check it with bx ic namespace-get. If you haven’t, you’ll need to set one with bx ic namespace-set (I use andrewferrier, for example - but you can set it as anything that’s meaningful to you - it will have to be unique across all users using shared Bluemix).

Now, tag your built image to prepare it for upload to the remote registry:

docker tag andrewferrier/splunk:latest-light registry.ng.bluemix.net/andrewferrier/splunk:latest-light

(Note that the first andrewferrier above is the prefix we specified previously when we build the image. The second is the namespace on Bluemix itself as just discussed. If you want to work with the UK instance of Bluemix, rather than the US one, change all references to .ng. to .eu-gb.)

Now actually push the image to the remote registry (this may take a little while):

docker push registry.ng.bluemix.net/andrewferrier/splunk:latest-light

Now, we need to create some persistent volumes for both the /opt/splunk/etc and the /opt/splunk/var filesystems within the container:

bx ic volume create splunk-etc
bx ic volume create splunk-var

Start the container running. Notice that we are exposing two TCP ports, 8000 (which will be used over HTTP to access the Splunk console), and 5140 (which will be used to push syslog messages from Bluemix to Splunk).

bx ic create -m 1024 -p 8000 -p 5140 --env SPLUNK_START_ARGS="--accept-license" --volume splunk-etc:/opt/splunk/etc --volume splunk-var:/opt/splunk/var registry.ng.bluemix.net/andrewferrier/splunk:latest-light

Once the container has started running, the Bluemix CLI will print out the container ID. You typically only need the first few characters - enough to uniquely identify it (e.g. abc1234).

Now check which public IP addresses you have free to assign to the container:

bx ic ips

This should print a list of IPs (probably two if you are working with a trial Bluemix account) - pick any IP which is not assigned to a container (if you have no unassigned addresses, you’ll either need to pay for more or unbind one from an existing container first). Now bind that IP address to your newly-created container:

bx ic ip-bind 1.2.3.4 abc1234

Now you’ll need to create a user-provided service to stream the logs from your application(s) to Splunk:

bx cf cups splunk -l syslog://1.2.3.4:5140

Setting up a TCP listener within Splunk

Now we need to set up a data listener within Splunk to listen for data on TCP port 5140 (essentially, this is the same procedure as Takehiko’s original article).

Open the Splunk console in a browser using the URL http://1.2.3.4:8000 (obviously, change the IP address for the one you picked above). Log in using the default username/password pair admin/changeme (Splunk will then encourage you to immediately change the password, which you should).

On the home screen, click “Add Data” to add a data source:

Add Data

Select “Monitor”:

Select Monitor

Select “TCP/UDP” to add a TCP-based data listener:

Select TCP

Enter Port 5140 (the same port we exposed from the Splunk Docker container above):

Enter 5140

Select rfc5424_syslog as the source type (which corresponds to the Splunk add-on we installed previously). You may find it easiest to type rfc into the dropdown box to select this. Also, you may want to create a new index to index data from Bluemix. In this case, I’ve created one called bluemix:

Input Settings

Review the settings you’ve entered and add the data listener.

Clone and push a demo application

In this article, we’ll clone a sample Node.JS application locally and then push it to Bluemix, so we can bind it to the user-provided service we just defined to use it to test the Splunk integration.

cd <some_temporary_directory>
git clone https://github.com/IBM-Bluemix/get-started-node
cd get-started-node
curl https://new-console.ng.bluemix.net/get-started/docs/manifest.yml > manifest.yml

Now edit manifest.yml to change name and host to a unique name (e.g. TestAppForSplunkAF (note that this name must be unique within the whole of Bluemix, which is why I use my initials to make this unique).

You also need to modify lines of the server.js file to look like this:

var port = process.env.VCAP_APP_PORT || 8080;

(This ensures that the application will pick up the correct port number from the Bluemix deployed environment).

Now push the application up to Bluemix:

bx cf push

Bind that service to any application you wish:

bx cf bind-service TestAppForSplunkAF splunk

And restage each application:

bx cf restage TestApp

Testing the logging mechanism

Probably, just in the act of restaging your application, you’ll already have generated some logs. However, to make things a bit more interesting, open the endpoint for your application (e.g. http://testappforsplunk.mybluemix.net/ - or similar, modify for the name of your application!) in a browser, and refresh it a few times.

Now, you should start to see your logging information appearing through Splunk. Assuming you set Splunk up as shown above, and created a new non-default index called bluemix, you should simply be able to search for everything in the bluemix index:

Search Terms

You should see some search results appear like this:

Search Results

Further Steps

The world is now your Oyster! You can use any standard Splunk searching mechanism to find logs.

Any questions or comments, contact me at andrew DOT ferrier AT uk DOT ibm DOT com.

26 Aug 2016, 19:21 MVP Bluemix

Using Bluemix to build Minimum Viable Products

When setting out to build a new product, product development teams are confronted with a number of unknowns.

  • What capabilities should the team incorporate in the product?
  • Will the product be compelling to its intended users – i.e. achieve product/market fit?
  • How much effort should the team invest in building the product?

In the London Garage, as practitioners of the lean startup method, we use a technique called the Minimal Viable Product (MVP) to help us answer these questions.

What is an MVP?

Unfortunately there is no straightforward definition of an MVP. Eric Ries, the author of Lean Startup, offer this definition.

“The minimum viable product is that version of a new product which allows a team to collect the maximum amount of validated learning about customers with the least effort.”

However, the definition is also a source of confusion.

Is an MVP just a trendy name for something that used to be called proof-of-concept? After all, the definition speaks of a product to collect validated learning.

Is an MVP a product with a subset of capabilities with more to be added at a later time, in other words a beta version?

Some practitioners of Lean Startup have even suggested a MVP could be delivered in the form of a PowerPoint presentation, a video, etc. If so, can a PowerPoint version of an MVP be really considered a product?

What about least effort? How little effort would need to expanded to be considered least?

These are questions that are beyond the scope of this blog. For the Garage, we characterise an MVP as having these attributes:

  • A product with just enough capabilities to achieve product/market fit;
  • And a minimally viable or workable product or something that can be used by users to provide feedback;
  • And a product where we can make changes without incurring high costs.

To build an MVP, we in the Garage use IBM’s Bluemix cloud platform.

Why use Bluemix?

Bluemix is a platform that enable organisations to quickly create infrastructure and host apps in the cloud. Developers creating apps that require complex services such as databases, Watson analytics and many others, can do so by connecting to these services hosted in Bluemix via standard interfaces. This means that we could easily swap one services for another simply by reconfiguring the interfaces. With Bluemix’s OpenWisk, Cloud Foundry and Container technologies (based on Docker technology), developers are also offered the option to build apps in modular components known as microservices.

In the not too distant past, the effort needed to build up similar capabilities as Bluemix to host apps would have been considerable and we have not even begun to invest effort in building the apps! Clearly, this option for building MVP would not qualify as requiring least effort.

However, using Bluemix to build MVP offers these benefits:

  • A working apps with a functional backend systems can now be built quickly;
  • Whilst an MVP does not require a completely functioning backend system but building one with these capabilities, even a limited one, will enable us to more accurately predict work for the next iteration of development;
  • Should an MVP be validated to achieve product/market fit, it would not require additional effort to convert it in a production app;
  • Being able to build apps in modular architecture and with minimal effort, discarding capabilities that are found to have no value would incur little cost.

How to use Bluemix to build MVP?

We’ll use two projects that were built in the Garage to illustrate how we use Bluemix to build MVPs.

The first example is an app we refer to as the Marathon App. Whilst this is mainly an iOS app that runs on a phone and outside the Bluemix infrastructure, it requires a backend database. The app was built using Xcode. The backend was facilitated through a Bluemix Cloudant database service that took very little time and manpower to activate and configure. The connection between the iOS and Cloudant was implemented via a RESTful API. Had we decided to switch to a different database all we needed to do was to reconfigure the REST API.

The MVP was developed in an iterative manner; at the end of each iteration, a working app was presented to the user for review. After three iterations, about three weeks, the MVP was considered production ready and was used to support Simon Wheatcroft, a blind marathon runner, in his bid the do a 155-mile ultra-marathon in the Namibian desert.

The second example was a project the Garage delivered to a medical service provider. This was a more complex project than the Marathon App and the MVP had three separate elements:

  • A Web-based application;
  • An app utilising Bluemix Watson IoT services;
  • An iOS app.

For this project, we were able to build a minimally functional MVP with end-to-end connection, using a combination of REST API and MQTT protocol, across the three elements in the first iteration within a week. By having a working system available for the user to use, the engineering team was also able to predict the engineering challenges for the second iteration and adjust accordingly. The MVP only took three iterations to be declared acceptable for the project sponsor, who in turn was able to use it as a basis to secure more funds for follow-on work.

Conclusion

In the two projects described earlier and other Garage projects, Bluemix was a vital technology to rapidly build a working MVP - real code, not just PowerPoint! This enabled us to validate with the client that the solution that had been built suited their business needs, and make swift changes accordingly.

Without Bluemix, we would have had to incur the cost of setting up a hosting environment and backend services. With Bluemix we were able to use the backend services out-of-the-box, so to speak, and focus our effort in building a working product; and taking it from validation to production state quickly.

24 Aug 2016, 12:25 Sizing Estimation Project management Agile Lean XP

Working out how much a project will cost

When the London Garage meet with customers, one of the most frequently asked questions is “how much is my project going to cost?”.

It’s generally accepted that - without a working time machine - at least one corner of the project management triangle needs to be flexible. Trying to nail down scope, cost, and schedule at the beginning of a project is a notorious anti-pattern; delivering a pre-determined list of features on-schedule within a given budget may give the illusion of success, but quality will inevitably suffer. This isn’t malicious or a sign of a bad team; tired people make mistakes.

In the Garage, we’re convinced that time-boxing iterations while keeping scope flexible is the way to go. We work at a sustainable pace, instead of rushing to meet a deadline, injecting lots of exhaustion-induced defects, and then collapsing in a burned-out pile to recover until the next rush. In order to enable this sustainable pace, we can’t commit to a detailed feature list at the beginning of the project. This makes products way better, for two important reasons. No one is cutting corners to meet the letter of a contractual obligation. More importantly, the beginning of a project is when you know least about what the actual requirements are. Requirements will to change as the project progresses; if they don’t, that means you’re not learning as you go. Writing the wrong requirements in stone into a contract at the beginning of a project is pretty self-defeating. The end result is that the development team are too busy implementing requirements that no one really wants, just because they’re in the contract, to be able to pivot and work on the stuff that we’ve learned does have value. Over-specifying feature lists at the beginning of a project is a terrible risk, both technically and in terms of the user value of the finished product.

Should we be estimating?

So far, so good - until we get back to that “how much is this project going to cost?” question. The garage developers have been watching the #NoEstimates movement with interest, because much of it aligns well with our thinking. We want to be as lean and reflective as possible and only do things which add user value (lean startup). We release often and checkpoint even more often (these are extreme programming values). We know the real measure of value is user experience, not metrics on a spreadsheet or feature lists (that’s one of the reasons we do design thinking).

#NoEstimates suggests that trying to estimate project duration is inaccurate and wasteful. In the Garage, we know that we can’t know how much it will cost to build something until we fully understand the technical landscape, and we’ll only get that knowledge once we’re deep into the implementation (and even then, there will always be surprises). More fundamentally, we won’t know what we really need to build for our users until we’re getting regular feedback from them about what we’ve built so far. Does that mean estimation is always a bad idea?

In the Garage, we proud of what we do. I know my team is awesome, and they deliver great value. However, if we’re working with a customer we’ve never worked with before, they don’t necessarily already know we’re awesome. There’s no pre-existing trust relationship, and they’re possibly comparing our value proposition to that of another vendor. It’s difficult for a customer to evaluate value-for-money of the Bluemix Garage unless we give some kind of estimate about how much a project will cost (the money), as well as describing how we can provide unique solutions for business problems (the value). There’s another aspect, too. In general businesses will have capped budgets, determined either by how much seed funding they’ve received, or by internal budgeting decisions. A business will need to decide if they can afford to build the minimum viable product that allows them to start testing business hypotheses, before starting that build. Building half a product, that never reaches viability, and then throwing it out because the money’s run out, is bad business. In other words, a customer needs enough information to be able to decide whether to go ahead with a project, and by definition that information is needed at the beginning of a project. Our job in the garage is to give them that viability assessment, so that we can then help them build a great project.

Sizing methodologies

There’s an growing mountain of academic and industry research about the optimum technique for sizing, including a whole range of models for software cost, from the simple to the super complex. Some are surprisingly old (back to the 1980s), and some are impressively mathematical. We aim for continual improvement in the Garage, so we’re experimenting with a couple of different sizing methodologies that have the potential to be best practice. We need something low cost, because experience has taught us that spending more time on estimates doesn’t actually reduce project risk. On the other hand, the estimate needs to have enough predictive value to allow a customer to make a sound go/no-go decision.

These are the ones we’ve tried so far:

  • Our starting point is the process described in Kent Beck and Martin’s Fowler’sPlanning Extreme Programming. The basic principle is to break the project down into smaller pieces of work, estimate them, and then add back up.
  • A much faster approach is to estimate projects based on our experience of similar projects. This is surprisingly effective - when we’ve tried the two techniques side by side, we’ve found that ‘gut feel’ estimates line up pretty well with ‘break down and add up’ estimates, and of course they’re way faster to produce. The gut feel approach falls down when we do a project which is the first of its kind, and since the Garage specialises in innovation, that’s actually pretty often.
  • One way of adding a bit more rigour to the ‘gut feel’ estimate, is to lay out a straw-man architecture with CRC cards. Since we’re thinking at the architectural level, instead of “Class-Responsibility-Collaborator”, we do something more like “Component-Responsibility-Collaborator”. Actually, it’s “Component-or-Actor-Responsibility-Collaborator”, but that acronym gets pretty messy. We use different colour post-its for “things we write”, things we interact with”, and “actors”. Our aim is to get some of these relationships out on paper and get a sense of the shape of the project, rather than to produce a final architecture. What the rough architecture gives us is a “landscape” that we can then compare to other projects we’ve done in the past, to produce an experience-based effort estimate.
  • Another approach is to make the gut feel less rigorous. In other words, the best way to handle the huge uncertainty in cost estimation is just to acknowledge it. What the ‘real’ project ends up costing will land somewhere on a spectrum, and there are a whole bunch of business and technical factors that can influence that. So rather than trying to guess those factors away, we can simply advise a customer to plan for both a likely-best and likely-worst case: “we think this project will take somewhere between two and four three week phases.” I call this spectrum-sizing.
  • Agile methodologies emphasise the importance of estimating in terms of points, rather than weeks, so we decided to try a points-based project estimation technique. Break a project down into hills and goals, and then for each one assess the complexity of its user interface, data store, business logic, and integration. We don’t want to get hung up on the precise complexity score. Instead we use score to guide categorising the goal into a Fibonacci complexity bucket, and we can sum up how many goals are in each bucket, and derive an estimate in weeks. As we use the technique more, our calibration between ‘bucket count’ and pair-weeks should improve. This technique has the potential to be extremely accurate, but we found it a bit too labour-intensive.
  • Some of our mobile development colleagues have developed a quite extraordinary spreadsheet which encodes years of team experience about how long stuff takes into Excel formulas. Starting with inputs about the number of user screens, supported platforms, and backend interactions, along with a whole bunch of experience-led (but adjustable) assumptions about the other activities that will be required, it produces a detailed cost estimate. It even takes into account things like re-use efficiencies for subsequent stories. The thing I like about it is that it stores our collective experience, and it’s detailed enough to act as an aide-memoire about some things that take time, and that which we should be doing, but which are easy to forget. However, for most of our projects it assumes too much up-front knowledge about the final UX design.

To date, we’ve had the best success with the first three, but we’re still looking out for improvements. We haven’t tried them yet, but we’re intrigued by some of the Monte Carlo modelling and randomised branch sampling. They leverage advanced mathematics to offer the promise of more accurate estimates for less effort, which is pretty awesome. They do assume that a reasonably detailed breakdown of a project into epics and at least some stories has already been done, so we’ll need to decide if they’re appropriate at the very-very beginning of a project.

Conclusion

As I was writing this blog, Kent Beck, the father of extreme programming, posted a blog on the same subject. I won’t try and reproduce it here (you should just go read it, because it’s really good!), but I was pleased to see that some of his arguments line up with what I’d already written. Kent points out that in an ideal world one would do everything, but in a world where resources are finite, and doing one thing means not doing another thing, estimates help us make informed choices about where we should put our resources. He summarises his position as “Estimate Sometimes”. “Estimate Sometimes” isn’t the catchiest strap line, but it’s the right thing to do, for us and our customers. We need to make sure, though, that our estimates are not turned into prescriptions about duration or commitments about detailed feature lists, because we don’t want to be making those sorts of decisions at the point in the project cycle where we know least. So estimate sometimes, and then leave that estimate aside and use all the feedback we can get over the course of the project to make sure we deliver the right thing.

27 Jul 2016, 11:57 General Jobs

What it takes to be a Bluemix Garage developer

The Bluemix Garage is a highly dynamic, multi-disciplinary consulting practice working with clients and partners of any size across all industries.

Being a Bluemix Garage developer means being exposed to forward-thinking methodologies as well as a large variety of technologies and other talented designers, developers and software architects. Bluemix Garage developers are customer focused, highly communicative, flexible, adaptable and know how to empathize and effectively work with their team members and their skills.

Projects run in the Bluemix Garage are guided by the Bluemix Garage Method, a set of methodologies, activities and techniques that help the team to stay focused and enable the client to learn fast.

Design Thinking and Lean Startup inform the user-centric and business-hypothesis-driven nature of the Bluemix Garage projects.

Another integral part of the Bluemix Garage Method is the eXtreme Programming methodology. Bluemix Garage developers embrace this methodology and its values.

  • Following the value of simplicity Bluemix Garage developers implement what is needed and nothing more in small simple steps, mitigating failures as they happen.
  • Communication means working on everything together to create the best possible solution.
  • A set of feedback activities guides the team to demonstrate results early and often. Bluemix Garage developers listen carefully to feedback and adapt where necessary.
  • Bluemix Garage developers have the courage to immediately identify when a process is stuck or an estimate is wrong.
  • Team members respect others and are being respected by others.

In addition, activities such as attending meetups, presenting at conferences, taking part in hackathons and contributing to open source are very welcome and well received.

10 Jun 2016, 16:45 Projects

Helping a blind athlete run four marathons, without a guide

Our mission in the Garage is to do innovative and exciting projects, so we were thrilled to help Simon Wheatcroft in his bid to be the first blind runner to complete the 4 Deserts Namibia race solo. We wrote a native iOS app for Simon which is pre-loaded with routes (such as Simon’s training route near his house, the Boston Marathon, and the 4 deserts stages). If Simon goes more than ten metres off course, the app beeps to let him know that he needs to veer left or right. The app does a low beep for left, and a higher beep for right, and a bit like a car reversing sensor, the more frantic the beeping, the further off course Simon is.

Simon Wheatcroft looking determined

One of the underpinnings of the Bluemix Garage Method is that you don’t know everything at the beginning of a project. The aim of development is to reduce uncertainty about the business problem, while minimising risk. We had a concrete example of “incomplete knowledge” with Simon’s app, because it turns out the pre-plotted race route didn’t end up being the actual race route. For example, one part of the run is along a beach, and the course was about 100 metres to the right of our coordinates. Because of how we’d designed the app, Simon was able to cope with the change - he just had to keep to a path which kept the app beeping at a “medium frantic” level. It can’t have been very relaxing, but it worked. We also made assumptions that the desert would be pretty flat and obstacle-free, so when we prioritised our user stories, things like “obstacle detection” were at the bottom of the backlog. It turns out that parts of the desert are rocky. Because he couldn’t see what kind of surface he was putting his feet on, Simon hurt his feet and legs trying to race over the rock fields - he was badly enough injured that he had to withdraw from the race after running only (!) three and three quarter marathons.

Another aspect of the Garage method is responding quickly to change, both in terms of our planning, and also our implementation. Normally we bake devops into our development so that it’s a matter of minutes to get updates out to users. In the area of the Namib desert where Simon was running, there was no data coverage, so once he landed there, we couldn’t make any more improvements to the app. In other words, there’s no devops in the desert.

The most important part of the Garage method is the user. We were lucky that we were able to work closely with Simon through the whole project, from the ideas stage to inception to after the event. We hope our applications delight users, but with Simon we knew it did; all of us in the Garage were moved when Simon came back and told us how he’d found himself in tears halfway through the first leg of the race, from exhilaration at having achieved his dream of racing solo.

Further reading

A lot has been written about Simon and the app we wrote for him. I can’t even link to a fraction of it, but here are some of my favourites:

07 Jan 2016, 16:45 Continuous Delivery

Automatically Managing an Apple TV Build Radiator

I’ve recently joined IBM’s Bluemix Garage in London. The Garage is all about being lean and innovative. They have some really nice setups, including two-screen two-keyboard pairing stations, and Apple TVs instead of the more traditional data projectors. Whenever any of us want to present, we just mirror our screen to an enormous TV using AirPlay. No more fiddling with projector cables - bliss! When we’re not using the TV for a presentation, we have it showing our build radiator and our telepresence window. We commit tens of times a day, so having the build radiator always-on means the team notice almost instantly if one of our commits causes a build problem:

In my previous roles, I’ve always loved it when we brought fresh blood into my team. New people don’t know all our habits, and they don’t take stuff for granted, so they ask questions. I’ve now found myself on the other side of that relationship; I’ve been the one asking a lot of - mostly dumb - questions. One of those questions was “why isn’t the TV showing our build radiator anymore?” It turns out that every time we used our TV to share a presentation, we had to walk over and manually re-enable the sharing from the build monitor machine.

Well, I’ve never seen a problem I didn’t want to automate. When I heard the word ‘manual’ I had to drop what I was doing (“thud”) and run off and write a script.

Scripting mirroring a display to an Apple TV

There are some excellent resources on how to use AppleScript to drive the AirPlay menu, and it’s pretty easy (the applescript file I ended up with isn’t very long).

tell application "System Events"
    tell process "SystemUIServer"
        click (menu bar item 1 of menu bar 1 whose description contains "Displays")
        set displaymenu to menu 1 of result
        -- Tolerate numbers in brackets after the tv name --
        click ((menu item 1 where its name starts with tvname) of displaymenu)
    end tell
 end tell

One complexity is that the name of our Apple TV box sometimes gets a number in brackets tacked on to the name, so we can’t just hardcode the name. Using a “starts with” selector fixes that problem.

Driving the script automatically

I also set up a launch agent (a .plist file) to drive the display mirroring every five minutes. The exact plist file will depend on where the scripts are extracted, but copying our .plist file to ‘~/Library/LaunchAgents’ and updating the paths is a good start.

Detecting when an Apple TV is already showing something

This solution wasn’t quite good enough, though. If one of us was ~currently~ sharing our screen, we didn’t want the build radiator to grab control back every five minutes. How could we tell if the screen was currently being used? It turns out that this is harder. My initial assumption was that we’d get a ‘Garage TV is being used by someone else’ prompt if an AirPlay share was active, but we didn’t. I suspect it depends on the version of the Apple TV and also the type of share (photos, videos, mirroring … ). Looking at the AirPlay protocols, there doesn’t seem to be any HTTP or RTSP endpoint on the Apple TV which reports whether it’s currently active. That’s ok - we can figure it out ourselves.

AirPlay mirroring traffic is all handled on port 7000 of the Apple TV (other types of share uses different ports). We can sniff the network traffic to see if there are packets flying around, and only start AirPlay mirroring if there aren’t. Bonjour allows us to find an ip address if we know the device name, and tcpdump shows us traffic.

To get the IP address of an Apple TV, replace spaces with hyphens in the name, append a .local domain, and then ping it. The ping output will include the ip address, which can be parsed out. For example,

 # Substitute dashes for spaces to find the Bonjour name
 tv hostname=${tvname/ /-}.local
 ipaddress=$(ping -c 1 $tvhostname | awk -F'[()]' '/PING/{print $2}')

Then tcpdump can be used to monitor traffic to that host. My initial implementation was this:

 sudo tcpdump tcp port 7000 and host $ipaddress > /var/tmp/airplay-tcpdump-output

The tcpdump command will run indefinitely, so we let it run for ten seconds, then stop it:

# Get the PID of the tcpdump command
pid=$!
# Capture 10 seconds of output, then kill the job
sleep 10
sudo kill $pid

You may need to add tcpdump (and kill) to the sudoers file so that it doesn’t prompt for a password:

 buildmachine ALL=(ALL:ALL) NOPASSWD: /usr/sbin/tcpdump
 buildmachine ALL=(ALL:ALL) NOPASSWD: /bin/kill

This worked great when I tested locally, but when I tried it on a bigger network, I discovered that on a wireless network, traffic is point-to-point, and tcpdump can’t gather packets promiscuously. In other words, I could only see traffic to the Apple TV when it was coming from my machine. Useful, but not useful enough.

Switching to monitor mode allows all packets to be viewed, but since packets on a secure network are encrypted, tcpdump can’t filter on tcp-level details, like the port and IP address. Luckily, tcpdump does allow filtering by MAC address, and the arp -n command can be used to work out the MAC address from the IP address.

 arp -n $ipaddress &> /var/tmp/arp-output
 fieldindex='$4'
 macaddress=`awk -F"[ ]" "/($ipaddress)/{print $fieldindex}" /var/tmp/arp-output`

Finally, the output file can be parsed with awk to count how many packets have flown past.

# Process the output file to see how many packets are reported captured
packetcount=`awk -F'[ ]' '/captured/{print $1}' /var/tmp/airplay-tcpdump-output`

A heartbeat packet is sent every second, so I assumed that after listening for 10 seconds an in-use device will always generate some traffic. Listening to the tcp packets on port 7000 (my first attempt) doesn’t yield very many packets, so any packet count greater than 0 indicates the device is in use. Once I started listening to the lower-level packets, there was always some traffic, even when the tv isn’t in use, so I used a threshold of 20:

if [ $packetcount -gt 0 ]
then
    # Handle in-use case
else
    # Handle not-in-use case
fi

The pieces are connected together in the shell script.

Digging in the depths of Apple TV

If you want to do more hacking with AirPlay, the following command lists all devices on a network: dns-sd -B _airplay._tcp

To see the details for an individual device from the list, which includes version and configuration information, you can do (for example)

dns-sd -L "6C94F8CF0F3F@Garage TV" _raop._tcp local

That gives

Lookup 6C94F8CF0F3F@Garage TV._raop._tcp.local
DATE: ---Thu 05 Nov 2015---
16:38:28.679  ...STARTING...
16:38:28.680  6C94F8CF0F3F@Garage\032TV._raop._tcp.local. can be reached at Garage-TV.local.:7000 (interface 4)cn=0,1,2,3 da=true et=0,3,5 ft=0x5A7FFFF7,0x1E md=0,1,2 am=AppleTV3,2 pk=d8b7d729ee2cd56d19e1fe2c9396f63fa41b227727e1b510dc925396d2d86668 sf=0x204 tp=UDP vn=65537 vs=220.68 vv=2

07 Jan 2016, 16:45 Cloud Architecture

Learn to be an IBM Cloud Architect in less than 1500 words

In this article, I’m going condense down the essentials of being an IBM Cloud Architect to less than 1500 words. This article is the big picture of the IBM Cloud (Public) offering. It’s condensed and probably over-simplified! This is a personal, opinionated perspective on the high-level knowledge needed to be an IBM Cloud Architect - it is not a formal IBM document and has not gone through review - so it’s based on my experiences (If it conflicts with formal documentation, announcements, etc. you should assume the error lies here). Others will have their own perspective. YMMV. I’ll include a lot of links for further reading, so please follow them as needed.

Caveats and Assumptions

  • A quick history lesson: until recently, this article would have been titled “Learn to be a Bluemix Architect…“. Bluemix was originally IBM’s PaaS platform. As of October last year, Softlayer (IBM’s IaaS) platform was merged into Bluemix to create one combined offering. To recognise the widening scope of IBM’s cloud offerings, Bluemix has recently been renamed IBM Cloud. In this article we’ll refer to “IBM Cloud”, but you’ll still find a lot of webpages, documentation, and other collateral referring to Bluemix.

  • This article will be focusing mostly on the PaaS components of IBM Cloud only, since that is typically how we would encourage new application construction. We’re also going to be thinking only about the IBM Cloud in its Public cloud form-factor (it’s also available in a Dedicated form-factor where it’s available for a specific customer).

  • There is a distinction between cloud-native applications (also called born-on-the-cloud) and applications that are to be migrated to the cloud. We’re focusing only on the former here.

The Basics

  • Learn about the 12-factor principles. They are important for building cloud applications. In particular though, the ones which cause controversy and confusion compared to the “old” on-premise way applications were typically architected are Processes, Concurrency, and Disposability.

  • At its most fundamental level, there are three pieces to the (PaaS) IBM Cloud Platform:

    1. Compute Models - this is how you develop and run your own code, and there are three you should be interested in for PaaS applications: IBM Cloud Functions (formerly called OpenWhisk), Cloud Foundry, and the IBM Container Service (based on Docker/Kubernetes).

    2. Services - also sometimes (but technically incorrectly) called the Cloud Foundry services, these are services that you can configure and use to add additional functionality to your application. They come in a variety of different categories, including Data and Analytics, DevOps, Integration, IoT, Mobile, Security, Watson (aka Cognitive or AI) etc.

    3. DevOps Toolchains - these are sometimes considered Services, but I logically separate them. This is basically functionality to allow for automated deployments, and other DevOps capabilities, in particular targeted the Compute Models above.

Compute Models

In general, the way you deploy code into each compute model differs. For example, in IBM Cloud Functions you create actions. In Cloud Foundry, you deploy applications. We will call these units of code individually compute model instances (some folks might refer to them as microservices, but that’s an architectural pattern which we will cover later).

IBM Cloud Functions

This is the most abstract of all the compute model types. The basic idea is that you construct actions by providing code that runs in response to triggers (for example, a RESTful request). The Cloud Function runtime takes care of deciding when to provision a server to run your action based on the trigger being fired. In practice, actions are provisioned within 100s of milliseconds. This model charges for code run based on seconds of execution and GBs of memory consumed, so it is particularly suitable for bursty, occasional or unpredictable variable-volume workloads. For more information, start here.

Cloud Foundry

Cloud Foundry is historically the compute model that’s been part of IBM Cloud the longest. It provides a reasonable compromise between “out-of-the-box” developer-friendly behaviour (it’s relatively easy to deploy your code into a Cloud Foundry runtime and get it going), but you do need to think about individual instances of each Cloud Foundry ‘application’, and how many you have - particularly important for reliability (failover) and scalability. For more information, start here.

IBM Containers Service

IBM Containers service is a Docker and Kubernetes-based way of implementing your applications’ code. Docker is a containerization technology which allows you to package your application together with its dependencies, and Kubernetes is a container orchestrator that builds on top of Docker to provide facilities to manage your containers more easily. For more information, start here.

IBM Cloud Services

The IBM Cloud services are a set of pre-built services which exist on the IBM Cloud Platform to enable you to accelerate development of applications. The charging models (plans) available depends on the service you use. It’s not an official tool but you can find out a great overview of all the services available here.

Regions, Organizations and Spaces

IBM Cloud is available in four different geographic regions: US-South (aka Dallas), UK (aka London), Germany (aka Frankfurt), and Australia (aka Sydney). Each of those regions is largely completely separated from each other. However, some caveats you should be aware of:

  • Most simple solutions that don’t have high uptime requirements should probably just run in a single region. This makes things way simpler. But of course this does make you susceptible to issues that are isolated to that region.

  • In general, solutions that require very high availability should run in multiple regions. This means you should be either aiming for passive resilience (here, we are calling that Disaster Recovery - DR), or active resilience (in other words cross-region HA - meaning that the application is always running actively in multiple regions). To achieve either of these requires a cross-region load balancer (e.g. Akamai or Dyn) and the application to be architected in such a way that it can handle this. Both DR and cross-region HA are outside of the scope of this article, although if you pick the right components (a eventually consistent database such as Cloudant, and remove state from application runtimes) cross-region HA can be achieved.

Non-Functional Considerations

Security is generally broken down into “data in transit” and “data at rest” - considering securing/encrypted data over the network, versus stored in a database.

Security - Data in Transit

In general, data in transit discussions are generally fairly straightforward - you want to ensure all connections in the entire system are secured with HTTPS (HTTP over SSL, aka TLS). In general, most of the core Cloud Services expose HTTPS endpoints, and you should make sure you use these. You also want to make sure any connections between your compute instances are HTTPS secured (in general, this is straightforward for Cloud Foundry as all connections are HTTPS by default).

Security - Data at Rest

You also need to think about Data at Rest security (in other words, whether data is encrypted). This is obviously particularly relevant to the database services. For example, with Cloudant, data is encrypted at rest. For other services, the answer is more complex.

Authentication

You need to think about component-to-component authentication (meaning both between compute model instances and IBM Cloud services) is secured on a case-by-case basis. In general, follow the following guidelines:

  • Most of the primary IBM Cloud Services provide an authentication mechanism out-of-the-box (usernames/passwords or at least an API key). Understand how it works and use it. In most cases, you can dynamically inject the credentials into the compute model instance. For example, see here for an example of how that works with Cloud Foundry.

Key Services

There are approximately over 100 services in IBM Cloud, and many are important. Some of the most key are:

Performance/Scalability

In general, cloud-native applications should be scaled horizontally. IBM Cloud Functions should handle this “automatically” be provisioning new instances on-demand. For Cloud Foundry, investigate and understand the Auto-Scaling service. For Kubernetes, add extra instances of a container/pod.

Some Other Things to Think About

You also want to think about architectural patterns for cloud development. One of those is understanding microservice development. For now we’ve been mostly ignoring this, and simply referring to compute model instances. However, there are patterns emerging for treating those specifically as microservices with well-defined RESTful interfaces. You can find out more information here and here.

24 Apr 2015, 11:45 Projects IBM Design Thinking

Connecting low power Things to Bluemix

We have been working with IBM Research Zurich on integrating their Long-Range Signaling and Control (LRSC) platform into Bluemix. Based on the LoRa technology, LRSC allows very low power devices to communicate wirelessly over a range of several miles.

Sensors connect to a physical gateway device, which bridges between the LoRa wireless technology and standard IP networking. The gateway then connects to a centralised LRSC server, which manages the network. The server connects to our bridge, which then passes messages to the IBM Internet of Things Foundation (IoTF) service on Bluemix.

Messages appearing in IoTF from LRSC

IoTF makes it easy to access the sensor data from custom applications running on Bluemix. It provides an MQTT interface that allows a developer to subscribe to updates from any LRSC device, as well as to publish control messages down to the end device.

The Lamplighter demo application

To help us demonstrate the bridge, we have created a mobile web application that allows a street light installer to easily test a new street light in the field.

We used IBM’s Design Thinking process to create the following Minimum Viable Product:

As a Device Installer, I can test the end-to-end bi-directional operation of a newly installed device within a minute of installing it and without leaving its location.

We worked with our designers through a series of storyboards and low fidelity prototypes, which led to the following design for the user interface:

The Lamplighter User Interface

The sidebar is dynamically updated with any new devices that connect to LRSC. The device installer can then select the new device, send a test command to the device, and finally confirm its location on the map.

The application is deployed on Bluemix using the Ionic Framework for the front end, and Express.js running in the nodejs buildpack for the back end.

We use the IMST SK-iM880A kit for our test devices and the Kerlink LoRa IoT Station for our gateway.

Using the Bridge

The bridge code is available as a sample on github.

20 Mar 2015, 16:45 XP Projects

Garden Bridge

In 2014 IBM UKI General Manager David Stokes made the offer to the Garden Bridge Trust for IBM to do the development and initial hosting of the gardenbridge.london website. The London Bluemix Garage has set up the various apps that are behind the website on IBM Bluemix and has developed the integration with the payment gateway run by Elavon for processing donations.

The gardenbridge.london website is made up of three apps:

1. Donate app

It handles all the communication with the payment gateway, triggers the correct e-mails to be sent based on the donation journey that a visitor takes and gracefully handles payment gateway unavailability.

This was developed by the London Bluemix Garage using eXtreme Programming practices including pairing, test-driven development, continuous delivery and blue-green deployment. All Garden Bridge apps are continuously delivered using blue-green deployment.

From the Bluemix service catalog, we’ve chosen SendGrid for all outgoing e-mails. Load Impact proved highly efficient and nice to work with when load testing all Garden Bridge apps. Monitoring & Analytics gives us sufficient insight into historical app availability and response timings.

As for the code stack, we’ve chosen Mocha, Chai & Sinon.js for the test suite, Express for the web server and couple of Node.js modules for integrating with third parties.

2. CMS app

It provides a UI for Garden Bridge Trust employees running the website to make content changes & maintain a newsletter without needing to involve an IT person. This piece, and the beautiful website design, was largely done by Wilson Fletcher, GBT’s chosen design agency.

This app is powered by Keystone.js, an open source framework built on Express (Node.js), and uses the Bluemix MongoDB service for persistence. As this service was still in the experimental phase, automated MongoDB database backups that we’ve previously talked about came in handy.

3. Web app

This app proxies requests to the previous 2 apps, handles URL rewrites and redirects, and provides a static page caching layer for the rare situations when the CMS app is unavailable.

Behind the scenes, this is powered by nginx, a most capable reverse proxy server, via the staticfile-buildpack CloudFoundry buildpack.

XP in action

With no estimates or scoping exercises, the donate app was delivered ahead of schedule. All apps were live, in production, weeks ahead of the public launch of the website. Even though Garden Bridge Trust has not yet launched their public fundraising campaign, all apps are meeting production requirements including high availability, horizontal scalability and fault tolerance.

While the CMS and web app were developed by a third party, Bluemix allowed us to collaborate on the different Garden Bridge components seamlessly and effortlessly. The deployment pipelines ensured that business value was continuously delivered, without any downtime, by separate teams.

Focusing on outcomes

XP is an agile development methodology which is focussed on business outcomes. Every story delivered should add business value, and all code is developed test driven. By adopting this approach, the Garage team is able to satisfy the business requirements with a code base that is bug-free, and deliver working code to the user in super quick time.

16 Mar 2015, 13:20 Services

MongoDB automated database backups

Even though the Bluemix MongoDB service is still in the experimental phase, it was a requirement for one of the client projects that we’ve recently had in the London Garage. In spite of all the warnings associated with an experimental service, MongoDB worked well and performance was satisfactory. Because of the experimental nature of the service, we have chosen to regularly backup the data to an IBM supported Cloudant service. This is the simplest possible automated solution that we’ve come up with.

The free Cloudant plans gives us sufficent data storage for our needs (20GB) and provides a mature, highly-available and clustered environment. Storing MongoDB backups in Cloudant is done through a simple HTTP call. Same for retrieving specific backups. Cloudant’s Dashboard gives us another alternative to browse all available MongoDB backups.

The backup app itself is designed to be run as a standalone app on Bluemix. It consists of a few shell scripts which call unix commands such as tar, curl and statically compiled Linux 64bit binaries such as mongodump & jq. Before deploying to Bluemix, we need to set the correct MongoDB & Cloudant service names in manifest.backup.yml. In the same cf manifest, we can configure the backup interval in seconds via the MONGO_BACKUP_INTERVAL environment variable. Once configured, the app is deployed to Bluemix as a worker app (i.e. no-route option) via cf push -f manifest.backup.yml

A handy behaviour of the backup utility app is that it runs the MongoDB backup on startup. This means that when we want to perform an ad-hoc MongoDB backup to Cloudant, all we have to do is restart the app. Otherwise, the next backup will run when MONGO_BACKUP_INTERVAL is reached.

This utility app would only be half as useful without a restore mechanism for the MongoDB backups. The restore functionality is part of the same code but deployed to Bluemix as a separate app. Service instance configuration is done the same as for the backup app, but in the manifest.restore.yml file. To restore a specific backup, first download the backup.tgz from Cloudant, and then place the file in the same directory as the cf manifest. With this backup file in place, you’re ready to just cf push -f manifest.restore.yml and the bound MongoDB service instance will be restored from this backup.

If you would like to get automated MongoDB database backups working on Bluemix today, feel free to download and deploy our mongo-backup-restore utility app to your Bluemix account.

13 Mar 2015, 11:06 Agile XP IBM Design Thinking

Using Agile Methods in the Bluemix Garage

With the cloud technologies and platforms available today, we finally have the opportunity to build applications in an agile fashion, that in the past has been hindered by the inability of traditional platforms to accommodate change.

In the Bluemix Garage we offer clients the opportunity to experience various agile methods through a number of different engagement types, which attempt to take the most relevant aspects of these practices and employ them to rapidly achieve the outcome the client is looking for. Design Thinking and Extreme Programming have very different heritages and differing focuses, but they are highly complementary provided the terminologies are understood and the appropriate hand offs are managed. In this article we overview the two key methods we use in the Garage and discuss how they focus on different stages of the development process, and why we believe they represent a string combination for organisations to adopt when building out the new innovative applications they want to bring to the cloud.

Key concepts of IBM Design Thinking

Design Thinking is all about getting into the mindset of the end user. It starts in the Explore stage with a detailed examination of the user personas being targeted, even giving them names so that the team develop a strong empathy with them going forward.

The IBM Design Thinking framework encourages the use of design thinking methods to envision the user experience. Design thinking requires diverging on many possible solutions and converging on a focused direction. You will practice design thinking using four mental spaces: Understand users, Explore concepts, Prototype designs, and Evaluate with users and stakeholders. This work may be iterative or non-linear, and can be used whenever you need to push the experience forward, diverge, or check in with users.

In the Explore phase, the team develops ‘Hills’ which are a concise written statement of what the proposed solution will deliver to the target user, stating what they will be able to do in the imagined future, and why this will be on benefit to them. In the Prototype phase, the imagined future experience is developed through a set of increasing fidelity prototypes which are shown to Stakeholders and Expert users through ‘Playbacks’, so that the whole team moves forward together against a common view of what the solution will look and feel like to the user. In the Evaluate phase, measurements are made as to how successful the prototypes are in achieving the objectives stated in the Hills. Feedback is taken on board and used in the next refinement of the system. A design thinking team is multi-disciplinary, with Designers, Product Managers, Engineers, Stakeholders and Expert Users working together as tightly as possible.

In this way the whole team has a collective agreed view of what needs to be created, and everyone has a chance to bring their ideas & concerns to the table. Design Thinking is an excellent way of ensuring pre-conceptions of what needs to be done does not prevent the creative process from coming up with even better solutions, whilst focussing in strongly on delivering the stated benefits to end users.

Key concepts of Extreme Programming

Extreme Programming (XP) is a discipline of software development based on values of simplicity, communication, feedback, courage, and respect. It works by bringing the whole team together in the presence of simple practices, with enough feedback to enable the team to see where they are and to tune the practices to their unique situation.

XP is laser focussed on delivering the minimum amount of code to satisfy the acceptance criteria in a backlog of stories that the team have agreed represent a prioritised list of work that represents a minimum viable product.

XP requires that there is strong Product Management with a clear view of what is good and what is not, and a team of engineers who are able to work together very closely through the practice of Pair programming. All coding is done test first, with the tests being written to support the acceptance criteria and then the code being written to pass the tests. No additional code is written. XP relies on good communications, a willingness for all members of the team to go on a learning journey together with the courage to embrace change at all times.

A team practicing XP will recognise they need to change direction and then do it straight away rather than blindly continuing down the wrong alley. This is achieved through fast feedback loops that are enabled by co-located teams, delivering of working software at the end of every story and continuous integration through automated builds deploying to production. Technical debt is acknowledged openly and represented in the backlog by refactoring chores and bugs. The likely future delivery of functionality is defined by the past ‘velocity’ of the team projected through the estimated size of the stories remaining in the backlog.

Common Ground

IBM Design Thinking and XP share a number of common philosophies and practices. Probably the most important one is fast feedback loops, which is baked into XP through the process of story acceptance, stand up meetings, iteration planning meetings and demos. Design Thinking employs the process of regular playbacks and evaluation of prototypes to achieve the same thing. Both methodologies stress the importance of a committed, multi-disciplinary team, preferably co-located at all times as key to really making this work. Expert users and stakeholders are particularly key in Design Thinking whereas in XP the Product Manager is seen as the one to represent the interests of those parties.

Another key shared philosophy is that of Minimum Viable Product (MVP). Both approaches focus on delivering what the business has defined to be the most valuable functions, and anything that does not contribute to that value is either removed, or, most obviously in XP, highlighted as a ‘chore’ which may have to be done but will have a measurable cost against the delivery of the overall project.

Differences of Focus

XP absolutely requires that the Product Manager has a clear view of what needs to be done to meet the business need. If this is not the case then the project ‘inception’ should not go ahead. IBM Design Thinking is an approach that can be used much earlier in the innovation process. A problem may have been identified but the solution does not have to be clear for a Design Thinking session to go ahead - in fact it is often better if the team are not already over focused on a solution in order that some more outlandish ideas can be shared and potentially lead to a more radical end result than we previously imagined to come to the fore. XP, as the name suggests, has a strong focus on delivering working software. It’s not untypical for an XP project to be delivering working code on the day after the project inception, and because the team is working from a prioritised backlog, every accepted story should be delivering value.

Test Driven Development (TDD) ensures that only the code that is needed to satisfy the story is written, and no code is written without tests. Concepts such as continuous Integration, automated builds and blue/green deployment ensure that every new piece of code delivered makes its way into production through a standardised, repeatable process and any problems introduced are immediately flagged to the engineers who wrote the code so they can fix it or back it out. In this way, new function can be rapidly tried out by selected users and any modifications needed can be quickly identified and prioritised using bug reports or new stories in the backlog.

Design Thinking is much more focussed on the user experience, and therefore various techniques for avoiding writing code that doesn’t meet the requirements are employed instead of the XP approach of quickly producing something and then changing it. Low fidelity prototypes, sometimes created on paper, are used to facilitate early user testing, and then storyboards, wireframes and mockups can be used in subsequent playbacks before the engineers start writing any UI code. Of course this doesn’t mean that the engineers are not involved until late in the process - far from it. They should be part of the Design Thinking process right from the beginning so they can give their views on feasibility of the ideas being suggested, as well as providing ideas of their own. The engineers also need to identify any ‘technical foundation’ required to support the UI designs being created, and they may well decide to start building some of this underpinning before the final UI designs have been agreed.

Working Together

IBM Design Thinking and XP will be used in different doses in different projects, but there is a clear synergy between them which will be exploited in the IBM Bluemix Garage. By merging Design Thinking with XP, it should be possible to come up with highly innovative solutions to client problems, and to then very rapidly turn them into working systems to use as Proof of Concepts or even live production applications. The start point is to have a team assembled that has the right combination of disciplines either committed to the project, or easily accessible to the project team with the importance of the work to the business firmly agreed and prioritised for the key contributors. The engineers and product management must have day job time dedicated to the project, even if it is not 100% of that time. Design expertise also needs to be readily available, especially early on in the Design Thinking sessions, and then for performing the evaluation of user feedback in order to assess likely success of the solution and suggest refinements. Empowered stakeholders in the business, who ultimately will own the solution and reap its benefits need to be closely involved in the project, or prepared to delegate the responsibility to the product manager.

Even in the latter case, they should be regularly updated on progress and have opportunities to feedback through demos or access to early system deliveries. Expert users are also key. Ideally they are part of the project team, but if not they need to be identified and methods for engaging them regularly to get feedback worked out. In terms of the process itself, once the Design Thinking sessions have successfully completed a Hills Playback, it should be possible for the team to attempt an inception in order to identify the Activities, Stories and Risks that the XP process needs in order to produce a Pointed Prioritised Backlog for the engineers to start work from. The Hills themselves will have directly identified the Actors, but if not enough is understood yet about what needs to be built in order to do an inception then the team could continue with Design Thinking to get to the Playback Zero, where a low fidelity prototype should be available and a lot more detail have been thrashed out. Beyond this point it should be possible for working code based on agreed designs to start to be built, and for the project as a whole to be able to reduce the time between subsequent playbacks or maybe even the number of playbacks needed because XP is driving the production of working code at a much faster rate. Another side effect could be that more granular playbacks can be achieved, possibly even with the concept of ‘continuous playbacks’, because the XP process is able to deliver small increments of new capability against each Hill in turn without a large overhead in producing code releases.

Summary

It has always been a stated intention of the Bluemix Garage that we would use the disciplines of IBM Design Thinking and Extreme Programming to deliver rapid value to clients exploiting cloud technology centred on Bluemix. Having gained some experience of this through our early projects, and after engaging acknowledged experts in both philosophies to discuss the similarities and differences, it is only now we are starting to appreciate how the practicalities of this will work going forward.

The good news is that the assertions that IBM Design Thinking and Extreme Programming are complementary appears to be sound, and we can now look forward to taking these approaches forward into varied projects in the future and discover more about what the ‘Bluemix Garage Way’ should look like.

Acknowledgements

Thanks to Steve Haskey from the IBM Design team in Hursley and Colin Humphreys, CEO of CloudCredo for their engagement with this discussion and valuable views based on their vast experience of IBM Design Thinking and Extreme Programming, respectively.

10 Mar 2015, 15:52 Bluemix Continuous Delivery XP

Blue-Green Deployment

One of our core principles is to deliver stories with the highest business value from day one. To deliver something means to put it in front of our customer. Regardless of whether our customer decides to make it public or not, it is available in production from the very first cf push. We strongly believe that having a partially complete solution in production from day 1 is more valuable than having the full solution delivered later.

All Bluemix Garage projects are continuously and automatically deployed into production, regardless whether they are internal tools or software components that we develop for customers.

One of the goals with any production environment is maintaining high availability for your application. The CloudFoundry platform helps with this by giving you the ability to scale your application horizontally to prevent downtime. There is another availability challenge when continuously deploying applications - if you cf push an existing application in production, your instances must be restarted to update your code. Not only does this inevitably introduce downtime, but you are left without a fallback if something goes wrong.

Blue Green deployment is not a new concept by any means. Nor is it tied to any particular platform. In short the idea is:

  1. Deploy a new instance of your application (leaving your current “green” environment untouched).
  2. Run smoke tests to verify the new environment, before taking it live.
  3. Switch over production traffic, keeping the old environment as a hot backup.

As we are deploying straight to production, using blue-green is imperative.

The current status-quo

We discovered these two tools:

  • Buildpack Utilities from the IBM WAS team — An interactive Bash script that will take you through the process
  • Autopilot from Concourse — A plugin for cf (We really like that it’s built into the tooling)

Both tools in fact do roughly the same thing:

  1. Rename the existing app while keeping all routes in place
  2. Push the new instance, which will automatically map the correct routes to it
  3. Delete the old instance

There are three problems with this approach:

  • They do not support running tests against the new app before making it available on the production route.
  • They immediately delete the app, making it impossible to revert to the old environment if something goes wrong
  • There is a chance that the old app could be serving requests at the time it is deleted.

While Blue-Green is quite a simple concept, we discovered several nuances in how we wanted to approach the problem so we decided to build our own.

Our Take

Our continuous delivery environment already consisted of a couple of shell scripts that provide app lifecycle management on top of the cf cli. We decided to build an extension to these scripts to manage Blue-Green with this process:

  1. Push new app instance to Bluemix with a different name
  2. Run end-to-end integration tests against the new app instance using the auto-generated route
  3. Map production routes (including domain aliases) to the new app instance
  4. Unmap routes from previous app instance
  5. Clean up all older app instances, leaving only the previous instance running as a hot backup

If any step fails then the CI server immediately gives up. We don’t want to continue the deployment if anything goes wrong!

The crucial difference is that we do not touch the production app until we know we have a good instance to replace it with. This means using a new app name for every deploy — as easy as passing a specific application-name when calling cf push. We chose to append the date and time for every deploy, but you could also use the git hash or CI build number.

After the new instance has been tested — with simple smoke tests in our case, the tool will then cf map-route the production route to the application. Now the new instance is handling production traffic, cf unmap-route can be used to disconnect the old instance from the world, leaving it untouched until the next deploy.

Deploying with a new application name each time, means we always run in a clean environment. However, without cleaning up, we’ll be left with a lot of redundant, idle applications consuming resources. This is easy to deal with, we simply delete all previous instances except the new app and the instance we just unmapped.

Blue, Green, Cloud.

Arguably what we’re doing here isn’t strictly Blue-Green deployment. The name “Blue-Green” stems from there being two separate environments on physical hardware, each of which could run an instance of the application. It wasn’t possible to throw away the old hardware and start with a completely fresh system for each install so it was necessary to flip between them.

Developing for a Cloud environment such as IBM Bluemix, means that creating whole new application environments is a simple cf push away, and deleting them is just as easy. With such a cheap way to create whole new environments it just doesn’t make sense to keep flipping between two apps when you can always start afresh.

We’ve been running several applications in this configuration for months now and we’re really pleased with what we’ve built. Everyone has different needs, this works for us but it’s just a different way to approach the problem.

Update 07/09/2015

We have recently open sourced our Blue-Green deploy script. It is now available as a Cloud Foundry CLI plugin. The source code and documentation can be found here:

https://github.com/bluemixgaragelondon/cf-blue-green-deploy

06 Mar 2015, 11:15 Hugo XP Continuous Delivery

Continuously delivered to Bluemix, served by Hugo

This website runs on Bluemix and is built with Hugo, a static site generator written in Go. Pages are served directly by hugo server which thinly wraps the http package from Go’s stdlib.

The entire website setup is versioned by git including the Hugo binaries, continuous delivery scripts, content & assets.

Every local commit gets pushed to a Jazz Hub git repository. Our continuous delivery environment, which runs on Jenkins, triggers a new build every time there is a new commit to this repository. This translates to pulling the latest version from Jazz Hub, running cf push, checking that the new version of the website is up and running and finally making it available to the public. Widely known as zero downtime deployment, we call it Blue-Green deployment since it describes the process more accurately.

Why Hugo?

After having considered WordPress, KeystoneJS & Ghost, we have settled on Hugo for the following reasons:

  • a platform-specific binary file is the only dependency
  • no database required
  • fast feedback when creating new content
  • fits perfectly with our way of working (version control, instant feedback, Continuous Delivery)

Why Bluemix?

We could try other popular alternatives which needed a PHP & node.js runtime with no effort. Rather than spending time with programming language or database setup, we could focus on finding the solution which best met our needs. Bluemix Containers came in handy while evaluating Ghost & WordPress. As for KeystoneJS, a new instance was a simple a cf push away.

Once we have settled on Hugo, we had continuous delivery and blue green deployment set within minutes. We can scale to as many app instances as we need with a single command. We went with the Go buildpack initially, then settled on a generic CloudFoundry binary buildpack as it made our deployment 50% quicker.

Why Continuous Delivery?

All Bluemix Garage projects are continuously delivered, without exception. Whether it’s a utility such as our Bluemix Blue Green Deployment, cf cli plugins, our website or any projects which we deliver for our clients, they are all developed using Extreme Programming practices.

No code is written before there is a failing test. We keep the cost of change low by continuously refactoring. We always pair program. We keep our commits small and constantly push to a central repository. Our Continuous Delivery environment pushes directly to production. We keep all feedback loops as small as possible so that we can quickly learn what works and what doesn’t. There is no better way of consistently delivering working software of the highest quality.