The IBM Bluemix Secure Gateway service allows connectivity from Bluemix applications to local/on-premise destinations. There are many excellent articles which cover this scenario, such as:

• Securing Destinations with TLS in the Bluemix Secure Gateway.

• Create a Secure Gateway.

However, a lesser-known scenario is that it can also connect back from your on-premise system to cloud-based destinations. Although this may not be the primary motivation for using it in many scenarios, it can be very useful with restrictive on-premise networks which don’t allow outbound connectivity.

This blog post walks through a very simple example of setting up this connectivity to connect to a public JSON-based service (http://loripsum.net/api). Although on your local test system, it is most likely that direct connectivity to that endpoint is possible, in many on-premise enterprise scenarios, that may not be so straightforward. This Secure Gateway setup thus becomes one possible option. We will define a Secure Gateway service, and use the Docker-based client, running locally on your machine, to connect back to the Secure Gateway.

Prerequisites

• You have signed up for IBM Bluemix (Public) and have an ID to log in with.

• You have installed Docker locally on your machine.

Instructions for this tutorial are written assuming you are using OS X, although they can probably be adapted to other platforms fairly easily.

Define the Secure Gateway

• In a browser in which you are logged in to Bluemix, navigate to the URL https://console.ng.bluemix.net/catalog/services/secure-gateway/. Click on “Create” to create the Secure Gateway service. Note: if you already have an instance of the Secure Gateway service in the space in which you are working, you will be unable to create a new one. You can use the existing instance of the service you have and simply create a new Gateway within it (navigate to the Secure Gateway Dashboard in the Bluemix interface). Note also that the Secure Gateway has a cost attached once you exceed a certain threshold of data. Please make sure you are comfortable with this restriction.

• You will now be in the Secure Gateway Dashboard. At this point, click “Add Gateway”. Specify a name for the Gateway (e.g. “GatewayFromLocal”). For this simple tutorial, you can disable “Require security token to connect clients” and “Token Expiration”. Click “Add Gateway”. It will look like this:

  

• Once you’ve created your Gateway, you’ll need to add the Client (the local Docker container which connects back from your machine to Bluemix). Locate the Gateway on the administration console (it will have a red X to show it is not yet connected). It will look like this:

  

Click on it.

• Click on “Add Clients”. This will bring up a dialog with multiple connectivity options. Click on “Docker”. This will show the precise Docker command required to run the local Gateway client on your local machine. The dialog should look like this:

  

The identifier at the end of docker run command is the ID of this individual gateway.

• You will need to modify the docker command to make sure that the port number we pick later on (called the “Client Port”) is mapped from the Docker container (which has a separate network, with a separate set of ports) to your host machine. Add a parameter like -p 6655:6655 to your command (you can pick any number you want, although it’s recommended to be above 1024). It should look like this:

  docker run -p 6655:6655 -it ibmcom/secure-gateway-client vJOMUBbT9Ax_prod_ng
  

Run the docker run command locally on your machine. You should see output that looks a bit like this (your client ID will, of course, be different from the one shown).

    IBM Bluemix Secure Gateway Client Version 1.6.0
    ************************************************************************************************
    You  are running the  IBM Secure  Gateway Client for Bluemix. When you enter the provided docker
    command the IBM Secure Gateway Client  for Bluemix automatically downloads as a Docker image and
    is executed on your system/device. This is released under an IBM license. The  license agreement
    for IBM  Secure Gateway Client for Bluemix is available at the following location:

    http://www.ibm.com/software/sla/sladb.nsf/lilookup/986C7686F22D4D3585257E13004EA6CB?OpenDocument

    Your use of the components of the package and  dependencies constitutes your acceptance  of this
    license agreement. If you do  not want to accept the license, immediately quit  the container by
    closing the  terminal  window or by  entering 'quit' followed by the ENTER key. Then, delete any
    pulled Docker image from your device.

    For client documentation, please view the ReadMe located at:
    .rpm and .deb installers: /opt/ibm/securegateway/docs/
    .dmg installer:           <installation location>/ibm/securegateway/docs/
    .exe installer:           <installation location>\Secure Gateway Client\ibm\securegateway\docs\
    ************************************************************************************************


    <press enter for the command line>
    [2016-10-25 04:57:37.944] [INFO] (Client ID 1) No password provided. The UI will not require a password for access
    [2016-10-25 04:57:37.953] [WARN] (Client ID 1) UI Server started. The UI is not currently password protected
    [2016-10-25 04:57:37.953] [INFO] (Client ID 1) Visit localhost:9003/dashboard to view the UI.
    [2016-10-25 04:57:38.169] [INFO] (Client ID 11) Setting log level to INFO
    [2016-10-25 04:57:39.249] [INFO] (Client ID 11) The Secure Gateway tunnel is connected
    [2016-10-25 04:57:39.546] [INFO] (Client ID Hh2eUmsFUi2_u7U) Your Client ID is Hh2eUmsFUi2_u7U

• Because you specified the -it flag to docker, the docker client will continue to run. Leave the window open so you can monitor the client easily. Very shortly afterward, the red cross against the Gateway in the Secure Gateway Dashboard should turn to green to show that the Docker client has successfully made a connection. If it does not, proceed no further and debug.

• Now, we’ll add a “Destination” for a Cloud-based service which we can invoke locally. From within the Gateway control panel, click on “Add destination”:

  

• Click “Advanced Setup” and fill out the screen as shown. The “Client Port” is the port locally within the Docker container - choose the same number you chose earlier for the docker run command. Click on ‘Add Destination’:

  

(Occasionally I have found that the process of adding a destination can hang at this point. If this happens, refresh the page - you will probably see that the destination has not been added to the gateway. Then, add the destination again, using the same details, but select “TCP” as the type. Once it is created, go in and edit the details, changing the type to “HTTP”. This should work).

• If the destination is successfully created, you should see a message that looks like this:

  [2016-10-25 05:19:55.346] [INFO] (Client ID Hh2eUmsFUi2_u7U) Creating TCP server for reverse connections. Listening on port 6655
  

• You can now test this. Open a browser to the URL http://loripsum.net/api. You should see some pseudo-HTML containing some Lorem Ipsum text. Your network may allow this directly, depending on your proxy settings, but of course this scenario is most useful when this direct connectivity is not possible!

• Now, open a browser to http://localhost:6655/api. You should see exactly the same result. This means that your browser is connecting through your local Docker client, across the Secure Gateway, and out onto the public internet endpoint. Even though in this case your network may allow direct connectivity to the original endpoint, this illustrates the values of the Secure Gateway in “bridging” connectivity.

Addenda

Any questions or comments, contact me at andrew DOT ferrier AT uk DOT ibm DOT com.

One of the things that the IBM Bluemix Platform (based on Cloud Foundry) supports is logging to external logging providers. I was recently reading Takehiko Amano’s excellent article on Splunk integration with IBM Bluemix. Splunk is becoming very popular in the industry generally for log aggregation, indexing, and searching.

Takehiko’s solution is excellent, but still requires somewhere to deploy Splunk. However, Bluemix itself provides the IBM Containers offerings (based on Docker technology) where Splunk can be run. This probably isn’t suitable for robust production environments, but for quick ‘n’ dirty testing, it’s really useful. I’ve documented below some steps that you need to get this up and running with Splunk Light, which is the simpler, lighterweight edition of Splunk.

Prerequisites

• You have a git client installed locally.

• You have signed up for IBM Bluemix (Public) and have an ID to log in with.

• Install Docker locally on your machine. This will give you access to the Docker command-line tools.

• Install the IBM Bluemix command-line tools, which will require installing the prerequisite Cloud Foundry command line tools.

Instructions for this tutorial are written assuming you are using OS X, although they can probably be adapted to other platforms fairly easily.

Build the Splunk Container

You need to build the Docker container for Splunk locally before pushing it up to the Bluemix containers repository. There’s already a well-established GitHub project for a Splunk Docker container, but we need to add the RFC5424 add-on as per Takehiko’s article to get Splunk to recognize the logging format.

I’ve already forked the GitHub repository and added most of the changes required to do that, but you will need to download the add-on itself first.

• Open a terminal and clone my repository, checking out the bluemix branch:

  git clone -b bluemix https://github.com/andrewferrier/docker-splunk/
  

• Download the RFC 5424 add-on. You will need to sign up for a free splunk.com ID if you don’t already have one. Put the .tgz file in the splunklight directory inside your checked-out git repository.

• Now build the Docker image (which may take a little while):

  cd <your_checkout_directory>/splunklight
  docker build -t andrewferrier/splunk:latest-light .
  

(If you wish, you can substitute your own namespace prefix in place of andrewferrier - as long as you use it consistently below).

Push the Splunk Container up to Bluemix and start it running

Firstly, log into Bluemix and initialize the container runtime:

bx login
bx ic init

You will need to specify an organisation and space within which to work on Bluemix.

Next, double-check what your IBM Containers “namespace” is. If you’ve worked with Containers before, you’ve probably already got one specified. You can check it with bx ic namespace-get. If you haven’t, you’ll need to set one with bx ic namespace-set (I use andrewferrier, for example - but you can set it as anything that’s meaningful to you - it will have to be unique across all users using shared Bluemix).

Now, tag your built image to prepare it for upload to the remote registry:

docker tag andrewferrier/splunk:latest-light registry.ng.bluemix.net/andrewferrier/splunk:latest-light

(Note that the first andrewferrier above is the prefix we specified previously when we build the image. The second is the namespace on Bluemix itself as just discussed. If you want to work with the UK instance of Bluemix, rather than the US one, change all references to .ng. to .eu-gb.)

Now actually push the image to the remote registry (this may take a little while):

docker push registry.ng.bluemix.net/andrewferrier/splunk:latest-light

Now, we need to create some persistent volumes for both the /opt/splunk/etc and the /opt/splunk/var filesystems within the container:

bx ic volume create splunk-etc
bx ic volume create splunk-var

Start the container running. Notice that we are exposing two TCP ports, 8000 (which will be used over HTTP to access the Splunk console), and 5140 (which will be used to push syslog messages from Bluemix to Splunk).

bx ic create -m 1024 -p 8000 -p 5140 --env SPLUNK_START_ARGS="--accept-license" --volume splunk-etc:/opt/splunk/etc --volume splunk-var:/opt/splunk/var registry.ng.bluemix.net/andrewferrier/splunk:latest-light

Once the container has started running, the Bluemix CLI will print out the container ID. You typically only need the first few characters - enough to uniquely identify it (e.g. abc1234).

Now check which public IP addresses you have free to assign to the container:

bx ic ips

This should print a list of IPs (probably two if you are working with a trial Bluemix account) - pick any IP which is not assigned to a container (if you have no unassigned addresses, you’ll either need to pay for more or unbind one from an existing container first). Now bind that IP address to your newly-created container:

bx ic ip-bind 1.2.3.4 abc1234

Now you’ll need to create a user-provided service to stream the logs from your application(s) to Splunk:

bx cf cups splunk -l syslog://1.2.3.4:5140

Setting up a TCP listener within Splunk

Now we need to set up a data listener within Splunk to listen for data on TCP port 5140 (essentially, this is the same procedure as Takehiko’s original article).

Open the Splunk console in a browser using the URL http://1.2.3.4:8000 (obviously, change the IP address for the one you picked above). Log in using the default username/password pair admin/changeme (Splunk will then encourage you to immediately change the password, which you should).

On the home screen, click “Add Data” to add a data source:

Select “Monitor”:

Select “TCP/UDP” to add a TCP-based data listener:

Enter Port 5140 (the same port we exposed from the Splunk Docker container above):

Select rfc5424_syslog as the source type (which corresponds to the Splunk add-on we installed previously). You may find it easiest to type rfc into the dropdown box to select this. Also, you may want to create a new index to index data from Bluemix. In this case, I’ve created one called bluemix:

Review the settings you’ve entered and add the data listener.

Clone and push a demo application

In this article, we’ll clone a sample Node.JS application locally and then push it to Bluemix, so we can bind it to the user-provided service we just defined to use it to test the Splunk integration.

cd <some_temporary_directory>
git clone https://github.com/IBM-Bluemix/get-started-node
cd get-started-node
curl https://new-console.ng.bluemix.net/get-started/docs/manifest.yml > manifest.yml

Now edit manifest.yml to change name and host to a unique name (e.g. TestAppForSplunkAF (note that this name must be unique within the whole of Bluemix, which is why I use my initials to make this unique).

You also need to modify lines of the server.js file to look like this:

var port = process.env.VCAP_APP_PORT || 8080;

(This ensures that the application will pick up the correct port number from the Bluemix deployed environment).

Now push the application up to Bluemix:

bx cf push

Bind that service to any application you wish:

bx cf bind-service TestAppForSplunkAF splunk

And restage each application:

bx cf restage TestApp

Testing the logging mechanism

Probably, just in the act of restaging your application, you’ll already have generated some logs. However, to make things a bit more interesting, open the endpoint for your application (e.g. http://testappforsplunk.mybluemix.net/ - or similar, modify for the name of your application!) in a browser, and refresh it a few times.

Now, you should start to see your logging information appearing through Splunk. Assuming you set Splunk up as shown above, and created a new non-default index called bluemix, you should simply be able to search for everything in the bluemix index:

You should see some search results appear like this:

Further Steps

The world is now your Oyster! You can use any standard Splunk searching mechanism to find logs.

Any questions or comments, contact me at andrew DOT ferrier AT uk DOT ibm DOT com.