Even though a lot has changed within the API Connect product and the types and numbers of stacks we're running since I first posted an overview of monitoring API Connect , the main areas we monitor haven't.

We are still using Grafana as a central location for dashboarding and analysing data across different data sources but some of the tools we're using to collect the data have changed. Having access to all the data in a single UI is really powerful, especially when troubleshooting or investigating events across the systems, being able to identify correlations between data from external load balancing, response times parsed from logs and pod utilisation metrics can really help narrrow in on specific components and how they impact the wider solution.

Metrics

For metrics we're making use of the IBM Cloud Monitoring with Sysdig to gather metrics from across the kubernetes deployment, including metrics from kubernetes itself and recognisable container applications such as nginx. We also supplement this with our own custom metrics exporter, Trawler, which we built for API Connect to extract key application specific data and expose them to a prometheus compatible monitoring tool or send them to graphite. Examples of data gathererd are counts of objects within API Manager and DataPower and analytics call counts. For endpoint and availability monitoring we are continuing to use Hem which is a simple python application to call HTTP(s) endpoints and send the metrics to our graphtie stack. All of these then come together to view within our grafana dashboards - and to be used within new exploratory dashboards whilst problem solving as needed.

Logging

For our logging infrastructure, we continue to use Elastic, making use of the filebeat agent within our clusters to gather and tag the container logs, then some custom parsing in logstash to parse out the significant elements from the different logs so that we can easily correlate these with events going on in the system. A lot of the time this data is then viewed in timeseries graphs within grafana, but also linked to Kibana views to dig deeper in the logs themselves.

As part of our work in running and monitoring our API Connect cloud deployments we've built some of our own tooling to assist with monitoring what is going on within the deployments. Trawler is one of these items which is used to gather metrics from a Kubernetes based deployment of API Connect.

Trawler runs within kubernetes alongside API Connect and identifies the API Connect components and exposes metrics to prometheus (or other compatible monitoring tooling)

This data can then be used to feed into dashboards such as this one in Grafana:

Trawler is open-source and available on github and docker hub - See the installation guide for more information on using trawler for yourself.

The kind of metrics that trawler collects are currently as follows:

Management subsystem:

• API Connect version information (apiconnect_build_info)
• Total users (apiconnect_users_total)
• Number of provider_orgs (apiconnect_provider_orgs_total)
• Number of consumer orgs (apiconnect_consumer_orgs_total)
• Number of catalogs (apiconnect_catalogs_total)
• Number of draft products / apis (apiconnect_draft_products_total / apiconnect_draft_apis_total)
• Number of products / apis (apiconnect_products_total / apiconnect_apis_total)
• Number of subscriptions (apiconnect_subscriptions_total)

DataPower subsystem:

• TCP connection stats (datapower_tcp...)
• Log target stats: events processed, dropped, pending (datapower_logtarget...)
• Object counts e.g. SSLClientProfile, APICollection, APIOperation etc. (datapower_{object}_total)
• HTTP Stats (datapower_http_tenSeconds/oneMinute/tenMinutes/oneDay)

Analytics subsystem

• Cluster health status (analytics_cluster_status)
• Number of nodes in the cluster (analytics_data_nodes_total/analytics_nodes_total)
• Number of shards in states - active, relocating, initialising, unassigned (analytics_{state}_shards_total)
• Number of pending tasks (analytics_pending_tasks_total)

• 3.8 miles

Today's walk was only a short section of the route as it was my birthday which meant a later start and fitting in a lunch out. The next logical section (Ovington to Cheesefoot Head) would have taken us too long so instead we opted for a cicular route, planned as a figure of eight, but private roads made it more circular and a lot more road than hoped.

The walk took us along through Ovington past the quaint thatched cottages to a short stop at St Peter's Church - which helpfully had a portaloo just outside. We continued along the road overlooking the fields where we saw a pair of Roe Deer skipping joyfully across the crops. After the fields we passed through a wooded section at the top of the hill, seeing an unidentified bird of prey circling and hovering through a clearing and then headed out of the woods to cross the A31 to rejoin the footpath the other side. Fortunately the road didn't prove to be too busy and crossing was easy.

Along beside two more fields led us to the town of Tichborne, another town with pretty thatched cottages with perfectly trimmed hedges and Wisteria growing up their walls. This was where todays walk departed from the Allan King Way and we attempted to circle back. When planning the route initially I'd failed to distinguish between a path (which turned out to be private) and a public footpath. So after walking past where I'd expected the path to be, passing Raimes Sparkling Wines which was also disappointingly closed today, we ended up back tracking and walking along the road back round Tichborne.

We conveniently found a bench to gather round for a coffee stop and Laura surprised me with a birthday cake and candle that she'd secretly been carrying - so we had a lovely birthday cake and coffee stop then followed the road along until we reached a path through the fields back up to the A31.

The A31 at this point was much busier but we crossed safely and were pleased to see the wildflower verges there on both sides and on the roundabout. We then rejoined the St Swithun's Way and Watercress Path along the road into Ovington - a road we'd driven along a couple of times in the two days, but hadn't appreciated the beauty of the stunning foliage above and the water either side with wild watercress growing in it.

• 6.8 miles

Inspired by the Wildmans' Wildmen on the borders, we were encouraged to start a walking project of our own. The Allan King Way is a 45 mile path from Portchester to Winchester. As we live in Portchester and I work near Winchester it seemed an appropriate walk to set out on, although we decided we wanted to walk towards home, so made Winchester our start point.

I dropped Laura and Abi off in Winchester by St Alfred's statue to find the start of the walk and headed off to park my car and meet my parents at the end of the days route. We then headed back and met them by the tunnel under the A34 by Kings Worthy - hence I can only claim that as my start point!

From there we headed along past St Mary's church in Kings Worthy and along by the pretty thatched cottages towards the A33 and over into the countryside. The path went along the edge of a copse with trees in a fantastic variety of colours from lime green to a rusty red before heading down to our first stretch along the side of the Itchen. Even on this short first stretch we see so many routes coinciding for a while and then going their own way.

We found a nice tree to sit under for our lunch which was a bit chilly as the wind caught us but still a lovely lunch break and we were ready for it! As predicted though, a short way on afterwards we came across a fantastic area with full sun and a bench which would have been a much better lunch stop, but instead made a spot for tree climbing and exploring along the waters edge.

We then made it along to St John the Baptist church at Avington where we discovered they had a stamp to collect as one of the churches on the Pilgrims Way so started a collection. Then after a stretch along by Avington hall and golf course, we found ourselves entering a field where the sky was full of Swifts and Swallows. We were mesmerised watching them soaring around above us and sometimes so close to our heads that we could hear the air being pushed away as they flew past.

We then headed across to Itchen Stoke and back down to the Itchen for a beautiful last stretch for the day along the river, ending at the Bush Inn, Ovington just in time for a drink even though they were just closing.