We had a fantastic weeks holiday up in the Peak District staying in Hathersage. We fitted in lots of exploring of the surrounding area, but still a lot more to see and do.

On the way up to the Peak District, we visited my parents for a couple of days before heading on further. From there we stopped at Hardwick Hall on the way where we started with lunch before looking around the hall. We learnt about the formidable Bess of Hardwick who had built both this and Chatsworth. We then explored the grounds - particular highlights being the swifts swooping across the lawn in front of the house and the impressive stumpery.

Then as we neared Hathersage, we made one more stop at Owler Tor for our first taste of some tor scrambling for the holiday.

Sunday was our only really wet day so we waited until the evening to set out exploring and headed across through Winnats Pass to the National Trust Mam Tor car park and walked from there up to the trig point at the top. After the blustery walk at the top of Mam Tor, we headed back down for a walk through the pass.

We then finished the day off returning to the car park by Owler Tor, but this time headed off the other direction to Surprise View to watch the sunset.

The next day we set off early to visit Ladybower and Derwent reservoirs where we rented bikes and cycled around the side of the Derwent reservoir. The Derwent Dam is where the dambusters practised their low flying techniques using converging spotlights from the nose and tail of the plane to gauge their height. The dam was chosen because of it’s resemblence to the German dams.

On Tuesday we walked from the cottage to the River Derwent and then followed the path along the side of the river to the Hathersage Stepping Stones where we had our picnic lunch. The girls had a great time going back and forth across the river on the stepping stones, before we headed back across the fields and railway into Hathersage to finish off with a drink. This was a lovely riverside walk with fantastic views across to the peaks along the way.

We set off early on Wednesday morning up to Stanage Edge to get there before it got busy and had the edge to ourselves. After that we headed into Bakewell to look round and (of course) get a Bakewell tart! On the way back we visited the David Mellor design museum in Hathersage - where is was interesting to see the wide range of design that he had been involved in from cutlery to traffic lights and bus shelters.

On Thursday, we spent the day at Chatsworth House. Starting off with the first slot to visit the house and then the rest of the day in the grounds.

After the day at Chatsworth, I still managed to make it up to Stanage Edge again to the High Neb Trigpoint for Sunset.

Friday morning we booked an early bird ticket for the Treak Cliff Cavern - they had a brilliant mobile app based system for a self-guided tour which would play the commentry as you made your way through the cave system. We were the first group in so it was nice having the caves mostly to ourselves and not have anyone in front of us. After this and a look around Castleton we headed up to Bamford Edge where we had fantastic views of the reservoirs we’d visited earlier in the week and the vivid heather.

Saturday marked the end of our time in the peak district, so after we’d packed up we made one more stop on our way back at the Longshaw Estate where we had a short walk with great views back over the edges we’d been exploring followed by a snack before heading on our way home.

As jobs are often short running and finish before you can check anything within the pod, often it’s helpful to make the pod run for longer to be able to inspect the environment and re-run the job manually. One way to do this is to extract the yaml definition for a previous run of the job and create a pod definition replacing the command with an endless sleep.

kubectl get pod {previous-job-pod} -o yaml > pod.yaml

Then edit the pod.yaml, removing the status block and the additional metadata referring to the previous job so the metadata only contains name and namespace

The comand then needs to be updated/specified with something like this:

  command:
    - /bin/sh
    - '-c'
    - while true; do echo hello; sleep 10000; done

Then use kubectl apply -f pod.yaml to deploy the new pod definition to your cluster and you will have a long running pod with the environment and code for the job.

Today I harvested our first row of potatoes from our raised beds:

Quite pleased with them, and looking forward to tasting them!

API Connect Reserved instance provides the ability to add remote API gateways so that you can co-locate the gateway service with your back-end systems for improved performance. With the new announcement of IBM Cloud Satellite, you can make use of this to securely expand your API Connect footprint across cloud providers and into the on-premise datacenter close to where your applications are running.

Create your Satellite location

To create a Satellite location you will need 3 hosts for the control plane, and at least one host to deploy DataPower on. For each of the hosts you will need to do the following:

• Set up the host prior based on the host requirements for Satellite.

• Register the host for RedHat updates using subscription-manager register - then apply a subscription to it in the RedHat Customer Portal

• Refresh the packages and enable the repositories

subscription-manager refresh
subscription-manager repos --enable rhel-server-rhscl-7-rpms
subscription-manager repos --enable rhel-7-server-optional-rpms
subscription-manager repos --enable rhel-7-server-rh-common-rpms
subscription-manager repos --enable rhel-7-server-supplementary-rpms
subscription-manager repos --enable rhel-7-server-extras-rpms

• Run the attach script obtained from the IBM Cloud Satellite UI to attach the host.

For the three hosts to form the control plane, you will need to assign them to the Satellite control plane through the UI or CLI.

Install DataPower in the Satellite location

• Download the DataPower rpms from your reserved instance.
• Install DataPower on your RHEL VM. In order to do this along with the pre-reqs I used the following commands:

yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum install schroot ipvsadm telnet
yum install idg_cloud1.10011.common.x86_64.rpm idg_cloud1.10011.image.x86_64.rpm

Set up link endpoint

• Create a link endpoint pointing to the API Connect Gateway management endpoint (usually port 3000)

Configure DataPower for API Connect

• Follow the steps to configure your DataPower for use with API Connect.
• For the certificates, in order to avoid a hostname mismatch create a self-signed keypair for the management interface for the hostname generated for your link endpoint.

Configure Certificate Manager service

• If you don’t already have one, create a Certificate Manager instance
• Ensure API Connect Reserved Instance is authorised through IAM to access this certificate manager instance.

Add Remote gateway to your reserved instance

• Register your gateway in your API Connect Reserved Instance, filling in the details as follows:
  • Management endpoint - enter the full URL and port generated by IBM Cloud Satellite for the Link Endpoint prefixed with https://
  • Certificate - the certificate to present for the mutual TLS communication with the Gateway management
  • CA bundle - the certificate in certificate manager to use to verify the gateway management endpoint (either your CA or the certificate itself)
  • Base URL - the endpoint you want APIs to be called through - should map to the API gateway address configured in the API Connect Gateway service either directly or through a load balancer.

Currently the v10 Reserved Instance of API Connect doesn’t yet have a simple approach for headless use of the CLI toolkit. The following details how to use an IBM Cloud IAM bearer token with the API Connect CLI and REST API in a headless environment such as a CI/CD pipeline.

For interactive use of the API Connect CLI, you can login using the --sso option, retreive an api key with your browser and provide that to the CLI for example:

apic-slim login --server {apic-api-endpoint} --sso

However if you want to use the CLI in a non-interactive context such as a CI/CD pipeline you need to retrieve an IBM Cloud IAM Bearer token for the toolkit to use. This can be obtained using ibmcloud iam oauth-tokens and then placed in ~/.apiconnect/token for the apic CLI to use.

The token file needs to contain:

{apic-api-endpoint}/api: |
  refresh_token: ""
  access_token: {access_token}   

This can be done programmatically using something like this:

ibmcloud iam login --apikey {api-key}
ic iam oauth-tokens | sed 's/IAM token:  Bearer /api.9a6e-bd639816.us-south.apiconnect.cloud.ibm.com\/api: |\n  refresh_token: ""\n  access_token: /' > ~/.apiconnect/token

apic orgs --my --server {apic-api-endpoint}
production    [state: enabled]   https://api...apiconnect.cloud.ibm.com/api/orgs/9123ae60-427c-4997-8a6b-ddd75b169bfb
test-porg     [state: enabled]   https://api...apiconnect.cloud.ibm.com/api/orgs/b73708ea-a7b5-4d27-b562-80767e0b238e

Invoking the API Connect REST APIs

To invoke the API Connect REST APIs in Reserved Instance v10, you can use an IBM Cloud IAM token which can be obtained using the ibmcloud iam oauth-tokens CLI command or with an API call as detailed in the IAM API documentation. This token can then be used as a bearer token to invoke the API Connect REST APIs.

The full process looks like this:

Firstly use your IBM Cloud API key to retrieve an IAM token

curl -X POST \
  "https://iam.cloud.ibm.com/identity/token" \
  --header 'Content-Type: application/x-www-form-urlencoded' \
  --header 'Accept: application/json' \
  --data-urlencode "grant_type=urn:ibm:params:oauth:grant-type:apikey" \
  --data-urlencode "apikey=[your api key]"

{"access_token":"[access token would be here]","refresh_token":"not_supported","token_type":"Bearer","expires_in":3600,"expiration":1615370557,"scope":"ibm openid"}

Then take the value of the access_token and use it to call the API Connect API e.g.

curl https://{{ api_host }}/api/orgs \
  -H "Accept: application/json" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer [access_token]

{
  "total_results": 2,
  "results": [
    {
      "type": "org",
      "org_type": "provider",
    ...
    }
  ]
}

Even though a lot has changed within the API Connect product and the types and numbers of stacks we’re running since I first posted an overview of monitoring API Connect , the main areas we monitor haven’t.

We are still using Grafana as a central location for dashboarding and analysing data across different data sources but some of the tools we’re using to collect the data have changed. Having access to all the data in a single UI is really powerful, especially when troubleshooting or investigating events across the systems, being able to identify correlations between data from external load balancing, response times parsed from logs and pod utilisation metrics can really help narrrow in on specific components and how they impact the wider solution.

Metrics

For metrics we’re making use of the IBM Cloud Monitoring with Sysdig to gather metrics from across the kubernetes deployment, including metrics from kubernetes itself and recognisable container applications such as nginx. We also supplement this with our own custom metrics exporter, Trawler, which we built for API Connect to extract key application specific data and expose them to a prometheus compatible monitoring tool or send them to graphite. Examples of data gathererd are counts of objects within API Manager and DataPower and analytics call counts. For endpoint and availability monitoring we are continuing to use Hem which is a simple python application to call HTTP(s) endpoints and send the metrics to our graphtie stack. All of these then come together to view within our grafana dashboards - and to be used within new exploratory dashboards whilst problem solving as needed.

Logging

For our logging infrastructure, we continue to use Elastic, making use of the filebeat agent within our clusters to gather and tag the container logs, then some custom parsing in logstash to parse out the significant elements from the different logs so that we can easily correlate these with events going on in the system. A lot of the time this data is then viewed in timeseries graphs within grafana, but also linked to Kibana views to dig deeper in the logs themselves.