Netherlands 2026

• 31 Jul - Folkestone
• 1 Aug - Doorn
• 3 Aug - Utrechtse Heuvelrug National Park
• 4 Aug - Utrecht
• 5 Aug - 📷 Zaanse Schans
• 6-7 Aug - Amsterdam
• 8 Aug - 📷 Kinderdijk
• 9 Aug - Henschotermeer
• 10 Aug - Wijk bij Duurstede
• 11 Aug - Calais

Folkestone

To start the holiday we headed across to Folkestone where we stayed a night at the Holiday Inn Express before our early morning booking for Le Shuttle the next morning. After checking into the hotel we noticed that Nova was licking her paw a lot, so decided we should get it checked out. We managed to find a nice helpful local vet who could see her that evening and it turned out she had a grass seed stuck in between her paw pads. He managed to remove it and gave her some medicine to stop it from bothering her as much. We then had Fish and Chips on the sea front and enjoyed exploring the conversion of the station area.

Doorn

Border controls were busy in Folkestone so we ended up on the next train, but the crossing itself went very smoothly. We stopped to get a drink before heading on to the motorway. We stopped by Fort Merksem for a lunch stop and a walk around the Fort.

After this we headed on to Doorn and after a GPS input failure taking us to the same road in a different town we made it to the house we'd be staying at. In the evening Kai and I went to Jumbo to do some food shopping and Nova enjoyed exploring the garden.

On Sunday we had a quiet day to recover from the journey and enjoy the surroundings of Doorn.

Utrechtse Heuvelrug National Park

On Monday we headed out to explore the trails in the National Park on our doorstep. We parked by Chalet Helenaheuvel and headed out on the trails from there. The area we walked in was a lovely forest area with clear trails and we walked through the forest to the Uitkijktoren de Kap and went up to the top, thankfully it was still early so not too hot! At the top there were great views for a long way around 3 sides of the tower - we could see Utrecht and the Pyramide van Austerlitz

Utrecht

Utrecht was the plan for Tuesday and originally we were planning to take Nova in the morning, but in the end just Kai, Jess and me headed in. We took the Park and Ride from Westraven and got the tram to Utrecht Centraal

Utrecht is a beautiful city with pretty canals built around a central tower. Originally it was part of the Northern border of the Roman empire and as such a fortress on the edge of the Rhine. We took the Dom Under tour where we learnt more about the history of the city and saw some of the foundations from the past under the Dom square.

Zaanse Schans

Photos from Zaanse Schans

Amsterdam

Originally the plan had been for me to go to Amsterdam with Jessica on the Thursday and Laura to go with Kai on the Friday, but as Laura wasn't feeling well I ended up going both days. We parked at Driebergen-Zeist station and got the train to Amsterdam.

Each day we took a different route from the station through the city exploring the canals and catching a number of different sights throughout the route. With Jessica we headed south straight away - quickly finding we'd got there too early for a lot of the shops and cafes but were able to see the Royal Palace without any crowds on the square. We then headed to Negen Straatjes and on to the Vondelpark - a beautiful park with lots going on and space to sit, relax and enjoy watching things going on around. After this we went to take a look at the WorldPride Village but again we were too early for there to be much happening yet.

With Kai we started off heading across to Nemo and up onto its roof so we could see the views across the city, the steps were so wide that you had to take two paces on each one which made it quite a climb up but the views were well worth it. We then stopped for a drink at the cafe at the top before heading on to explore the city.

We then worked our way through the canals to find ourselves somewhere for lunch. Then after lunch we went to see the Royal Palace and the National Monument (much busier than they had been the day before). We then headed on round to Seven Bridges and round to Museumplein and the Van Gogh museum.

Kinderdijk

Photos from Kinderdijk

In the evening after our trip to Kinderdijk, we went out to have pancakes for dinner at Het Wapen van Sandenburg, followed by a walk in the forest enjoying the evening light.

Sunday 9th

Henschotermeer is a lake surrounded by beaches that you can swim in - as dogs aren't allowed on site I took Kai and Jessica there to explore. As I didn't have my swimming things with me I just paddled but Kai swam. I wished I'd had my things as the water was a lovely temperature and it was a really nice location!

In the afternoon Laura, Kai and Jessica went to Huis Doorn - I dropped them off and then headed back with Nova - we took a scenic route and ended up on some seemingly quite narrow roads with water either side of them!

In the evening Laura and I went out to a local restaurant, Spott where we had a lovely evening. This was walking distance from where we were staying, had a fantastic atmosphere and we were well looked after by the friendly staff.

We had the High Wine menu which was amazing, and consisted of 3 different wines each served with a platter of food alongside. We started with a glass of Cava with fresh bread with olives and salad. Then there was a Portuguese Verdelho served alongside a seafood platter consisting of Oysters, Salmon, Tuna, Dutch Shrimp and Mackerel. We then finished with the Château Cavalier Rosé from Provence along with a hot platter including Nachos, Prawns and Bitterballen. The wines were exquisite and the food was delicious - really good value for money!

Monday 10th

First thing we visited the nearby town of Wijk bij Duurstede where there is a mill and a castle on the banks of the Nederrijn (Lower Rhine). We went for a nice walk along the river to the castle and back along through the streets. We then headed back to the house to sort our things and to take Nova to the local vet for her required Tapeworm treatment so we would be allowed to return to the UK.

Tuesday 11th

Our last day so we finished packing up, took Nova for a last wander around the local woods and set off on our way to Calais - taking a different route this time so we could see some of the sea defence dams on the way. We were fortunate to see one of the locks in operation with the road lifted up to let the boats past. We also stopped for a walk around Sluis to break the journey further which was a busy little town to visit, and lovely to explore around the fortifications.

As architect for the API Connect Cloud offerings one of the key areas I am involved in is deployment and in particular deployment automation. With my SRE background the main goal here is that anything we need to do more than once should be automated - deployment being a big one of these. Across our cloud environments - single-tenant and multi-tenant, IBM Cloud, AWS and Azure from dev through to production we probably have over 80 deployments of API Connect. These all use automation to deploy from declarative deployment models - in particular GitOps so our git repo is the source of truth for everything that gets deployed.

This is split into two primary areas - deployment of cloud services dependencies (Kubernetes clusters, object storage set up etc. ) and the application deployment itself.

Infrastructure - Terraform

For the cloud service deployments we use Terraform so we can declaratively describe the infrastructure requirements and have clean separation between the base template and the per environment configuration (region, size, ha-requirements) through variables. This all lives in git.

Application - ArgoCD

For everything that goes into kubernetes we use ArgoCD - which means all of the manifests for deployment live in our git repos and are watched by ArgoCD for changes. These are structured making use of ArgoCD's SyncWaves so that they are deployed in the right order and we know that the dependencies are in place before moving on. For example:

• -200 - Pre-requisites (cert-manager & issuers)
• -100 - Monitoring tooling
• 300 - API Connect CRDs and operators
• 400 - Management subsystem
• 500 - Analytics subsystem
• 600 - Developer portal
• 700 - Gateways

Just like in the old days of basic we've used multiples of 100 to allow us to add things in between later without reshuffling everything! In ArgoCD the default syncwave for anything without one specified is zero - so using negative waves for pre-reqs means we don't have to annotate all the CRDs and if we miss anything we know the pre-reqs are already there. In the manifests this would look like:

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "100"

This is a simplified overview, in our SaaS environments we add additional cloud specific components that handle tenant provisioning and ui based configuration addons which have their own waves in the deployments. We follow a similar pattern with the manifests that we do with the terraform - a base template and environment specific configuration. In this case we have used kustomize & helm based charts in different places as they both have distinct benefits for different use cases.

Use Kustomize when environments have significant differences. It uses a base template and overlays, making it much easier to follow what is actually happening compared to complex Helm syntax.

Use Helm when environments are identical except for minor elements (like hostnames). It uses values files to pass variables into charts, making it perfect for simple toggles (e.g., dev vs. prod).

In a repo the difference looks like this:

bases/
    management/
        management_cr.yaml
        kustomization.yaml
    gateway/
        gateway_cr.yaml
        kustomization.yaml
    overlays/
        dev/
            kustomization.yaml
            management-patch.yaml
            gateway-patch.yaml
        prod/
            kustomization.yaml
            management-patch.yaml
            gateway-patch.yaml

Chart.yaml
values.yaml
values-dev.yaml
values-prod.yaml
templates/
    management_cr.yaml
    gateway_cr.yaml

With Kustomize the bases/management/management_cr.yaml holds the common shape of the ManagementCluster CR and overlays/dev/management-patch.yaml only contains the fields that differ for that environment (endpoint hostnames, profiles, feature enablement). The patch only carries the fields that differ for this environment:

# overlays/prod/management-patch.yaml
apiVersion: management.apiconnect.ibm.com/v1beta1
kind: ManagementCluster
metadata:
  name: management
  namespace: apic-prod
spec:
  profile: n3xc4.m16
  endpoints:
    management: mgmt.prod.example.com

The overlays/dev/kustomization.yaml then pulls it all together:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../bases/management
  - ../../bases/gateway
patchesStrategicMerge:
  - management-patch.yaml
  - gateway-patch.yaml

With Helm the same effect comes from templating the differing fields in templates/management_cr.yaml and overriding them per environment in the values file:

# values-prod.yaml
management:
  profile: n3xc4.m16
  endpoints:
    management: mgmt.prod.example.com

We use Helm for our multi-tenant SaaS environments where the regional deployments need to be the same and Kustomize for single tenant deployments where we need to support more per-tenant differences.

Secrets

Everything that is not sensitive lives in the git repo - the secrets are then injected into the charts at deploy time from the relevant secrets manager for the cloud provider. These are abstract primarily through the use of ArgoCD Vault Plugin which supports a number of different secret backends.

This means our secret manifests have references rather than the actual secrets. For example the Secret we mount for the Management subsystem's DB credentials in prod looks like this:

apiVersion: v1
kind: Secret
metadata:
  name: mgmt-admin-pass
  namespace: apic
type: Opaque
stringData:
  password: <path:apiconnect/prod/management#db.password>

At sync time ArgoCD Vault Plugin resolves each <path:...#...> reference against the configured secret backend (IBM Cloud Secrets Manager, HashiCorp Vault, AWS Secrets Manager etc.) and rewrites the manifest with the literal values before it is applied to the cluster - the rendered manifest never lands in git, and the cluster ends up with a standard Secret that the operator can consume.

For customer owned secrets we've integrated External Secrets Operator to provide a similar abstraction at runtime.

A valuable technique I've found is treating everything I can as code - checked in, reviewed, and versioned. This post covers what that looks like across infrastructure, deployments, APIs, and even documentation, and why the discipline matters more than any specific tool.

What does "as code" mean?

By 'as code' I mean storing as a version controlled source of truth in a declarative form. This is distinct from building custom code as an automation in that what is being stored represents the desired state, not the steps to get there.

What can be treated as code?

• Infrastructure: defining the infrastructure components and configuration through a declarative language to be used with a tool such as Terraform to deploy and maintain state.
• Application deployments: for deployments to Kubernetes, you can use YAML manifests in a repository and then ArgoCD will keep the cluster in sync with what you have defined.
• API definitions: Store your OpenAPI specifications and gateway policy definitions in version control and use the API Connect toolkit to publish them to your gateway and developer portal - example with GitHub Actions. API Studio also makes it easy to work directly with your git repo.
• Governance rulesets: See also Git is your governance source of truth
• Process documentation & runbooks: Storing your documentation and runbooks as markdown in a repo gives you that built in change history and management rather than having to remember to update the changelog manually in the front of a word document.
• Diagrams: If you make use of a text based diagramming tool such as Mermaid, you can get these same benefits for your architecture diagrams making it easier to see over time how things have changed. Whilst you could also store draw.io diagrams in git and follow the changes, it is slightly less clear with its XML-based format.

Advantages

• Traceability — you know who made which changes.
• Currency — you know when it was last updated.
• Time travel — git history lets you revisit any prior state.
• Repeatable and Consistent — you can always instantiate from the declarative source
• Peer review via PRs — there is a built in system for proposing and reviewing changes
• Drift detection — you can see when your system diverges from the source

How this works in practice

This model is fundamental to how my team at IBM works. All our internal documentation and runbooks live as markdown files in git repos. For our cloud deployments the infrastructure is defined in Terraform, all the manifests are stored in git and deployed with ArgoCD. Having this consistency and repeatability is essential when you're maintaining over 80 stacks. We deploy a set of our own APIs to every stack using the toolkit. My next post will cover more detail on our deployments.

Pitfalls to watch

Whilst treating everything as code has its advantages there are a number of things to watch out for. You need to make sure you have a way to handle sensitive data separately and ideally scanning to avoid these being unintentionally leaked through your version control system. Not all data types work well in a system designed around text files, for example large or binary data that you can't easily compare changes in. This will balloon the repo size and start to make things painful for everyone using it. The review process can become tedious if not handled carefully - it needs to be thought through so that you get the benefit of peer review without it becoming a checkbox or a blocker.

Conclusion

The value in treating everything as code really comes from the mindset and the discipline around it, which is much more powerful than any specific tool. If you adopt this approach you get a common set of consistent practices across many different artifacts: traceability, review and a way to revert changes. In my next post I'll dig into how this is done in our API Connect cloud deployments.

Have you ever reached the end of the week and felt like you haven't had any time to actually complete anything you set out to do? We're building tools to save time but the resulting pressure to produce more and faster often leaves us with less time to think about why we are building anything. At the end of the day, computers are there to help us as humans. With APIs we build them for computer interactions, but with a clear aim of shared value to produce something useful for us; they're not an end in itself. The same should be the case with AI.

When the goal becomes AI integration itself however we risk shifting our focus from the human outcome (time-saving productivity) to the technological output. When this happens it feels like it doesn't matter how it's being used or where, but rather the usage itself is the goal. Wide claims of productivity gains from AI are leading to new expectations of how much can be achieved in a given time period resulting in more and more things being built without the time for thinking.

Thinking is the key part of the development process - refining and synthesizing ideas - this is way we grow through working through the different options and elements of the idea getting through this messy stage into a valuable refined idea. If we skip this and jump straight to the implementation we end up with so many things being built - some with no purpose, some without enough time to ensure they are sufficient quality. Others started and never completed because of a pivot to the latest technology trend.

AI doesn't look to be going away any time soon and it can bring some value and benefits if we are careful around how we make use of it. We need to approach it as any other tool with a clear purpose and strategy to reach our goals - using the AI to handle the "busy work" to reclaim that time for the valuable deep work. Focusing on using AI to solve problems that are tedious or complicated for humans to work on rather than to replace creative human thinking. AI shouldn't replace inter-human communication - I'd much prefer to read raw thoughts in a couple of paragraphs than a 4 page polished AI generated document that was built from that.

The goal isn't to build more, it's to build better. AI can help give us some of the speed but only if we clearly articulate the direction.

I've always been interested in ensuring my blog was readable to humans and machines - so over the years I've added and experimented with different ways to do this as what it means to be machine readable has changed. I currently have a range of these supported from embedded metadata to alternate versions that can be used with an LLM. Currently on this blog I am using the following:

• RSS Feed - my RSS feed
• JSON Feed (validator)- my json feed
• hentry (validator)
• TechArticle (validator)
• OpenGraph (validator)
• Alternate markdown version - this page in markdown
• standard.site (validator)

As I recently added the alternate markdown version and the standard.site integration here's some details on how:

Adding standard.site to my hugo static blog

Initially I started using Sequoia which is a great tool that can handle processing and posting documents from a static site and whilst it did a great job of processing and posting my content, I wanted a bit more control. In particular the way I handle featured images didn't fit with its model and I didn't want to inject links into the generated html - I wanted this as part of the main templates. Adding the meta tags to the template was straight forward - always including the publication reference and the document one where specified in the frontmatter.

Now the next thing to do was to find a new way to post the documents and obtain the did for the standard.site document. I came across this useful guide which gave me some good pointers on the basics through nodejs examples, but as I prefer to write scripts in python, I ported the ideas into my own script. Whilst building this I used pds.ls to check on the results.

You can see my current script in github - this will either take in an md file as a parameter or identify one from the staged commit. The script then builds the standard.site.document lexicon content:

document_record = {
    '$type': 'site.standard.document',
    'site': blog_did,
    'title': post_data['frontmatter'].get('title', 'Untitled'),
    'publishedAt': post_data['frontmatter'].get('date'),
    'textContent': text_content,
    'canonicalUrl': post_url,
    'contributor': {   
        'did': agent.me.did,
        'role': 'author',
        'displayName': "Ricky Moorhouse",
    }
}
# If there is a featured image, upload as blob and get the reference
if 'featured' in post_data['frontmatter']:
    featured_image = post_data['frontmatter']['featured']

    # get relative path to the post file, removing the file name
    post_path = post.rsplit('/', 1)[0]

    # Upload to atmosphere and get the blob reference
    blob_ref = upload_featured_image(agent, post_path, featured_image)
    document_record['cover'] = blob_ref

If I have an atUri already in the front-matter, the existing document is updated, otherwise a new one is published and adds the atUri to the front-matter. At the moment my workflow is to write the post, stage the commit and then run the script before updating the commit and pushing to GitHub where my actions will then publish the blog post, but eventually I plan to move the script to be run from the actions as part of the publishing. I'm also experimenting with using an LLM to generate a post for social media to link to the article.

Markdown versions of posts

As well as extending to push documents to atproto using the standard.site lexicon I've added markdown files for easier consumption by AI Agents - inspired by isitagentready. These files are generated through hugo as part of the publishing flow.

To add markdown versions of posts, I just added markdown as an output format, created .md versions of my page and home template and included metadata pointing to the alternate version in the header - you can see how in this commit.

Once the files were being generated, I added a Bunny CDN Edge rule to handle content negotiation so that if content type of text/markdown is requested, then I redirect to the md file. This way the same URL can be used for reference and humans and agents will each get the most appropriate version.

Agents are only as good as the access they have to accurate data and business functions. LLMs have a cut off point as to what they know - they're trained on a set of data and anything beyond that is unknown to them - therefore to build useful agents you need to provide a way for them to access data beyond that corpus. This could be more your own business data or just more recent information. To enable this Anthropic introduced Model Context Protocol (MCP) as an open standard to build these connections to data-sources and tooling. Since then, MCP has exploded into the most common way of providing tools to AI Agents.

There are 2 main types of MCP Server - local which you download and run on your machine (just like a CLI tool) and remote that you connect to through a URL (just like an API). They can both be configured and they both have their risks - so we need to consider how we safely manage these in our environment, building on how we control and manage APIs and software and APIs.

Local MCP Servers (stdio)

There are thousands of pre-built MCP servers out there to download and use - some official but mostly unofficial community built servers created by interested developers or their AI Agents. According to Equixly 43% of MCP servers contain command injection flaws and 30% permit unrestricted URL fetching, so it is important to understand what you are running. This highlights the fact that traditional security practices still apply to the agentic space - often more than ever. Supply chain security and least privilege are more critical than ever when you have non-deterministic agent flows handling data through community tooling.

Often these are documented with a single command to download and run that you add to your agent config - using npx or uvx, however you can usually still download, build and run in the traditional way you would opensource projects to ensure you stick to the version you have verified. This way you have a level of control over the tools you are using but would still need to be sure to keep dependencies up to date.

Remote MCP Servers (streamable-http / sse)

In a lot of ways use of a remote MCP server is very similar to using a third-party API as part of your application, in fact some places I've seen MCP tools provided where there wasn't previously even an API or access to the MCP be made easier than it was previously to get access to the API. There are however some key differences:

As Agents connect to MCP servers they will initialise a session and list the tools available with their specifications dynamically as a kind of discovery each session which is loaded into the agent context. The ensures they always have the latest tool definitions, can adapt accordingly and don't need to review a separate spec or changelog if the tooling is updated.

As they are being used in a non-deterministic process, you don't know up front what your agent will be sending them. For example agents build the tool call based on their language model, your input and the tool specification without formal object declarations, so it is all open to interpretation which may lead to unexpected sequences of tools or needing to retry to get parameter formats correct. Whilst APIs would typically define input types using json schema, MCP defines tools with textual descriptions. One example I've seen was with timestamp formats where the backend the tool interacted with was expecting ISO8601 format but the agent was trying to use relative times.

Security checklist for MCP

Ultimately, the responsibility for the security and integrity of these systems remains entirely with the user. Because agentic flows are non-deterministic the risk of an unintended action, triggered by a flawed community tool or a prompt injection, is higher than in traditional software.

• Audit the tooling - whether this is the source code downloaded or the provider of the hosted server - before you connect you should ensure you can trust it with your data.
• Isolate downloaded tools to limit the risk of command injection, for example running the MCP server within a Docker container controlling what they can access. For remote MCP servers, routing calls through a gateway gives similar control to restrict, observe and adjust interactions as needed.
• Credentials - Use dedicated credentials for MCP tooling that can be revoked, and apply the principle of least-privilege so they only have permissions they need.
• Approvals - Ensure that sensitive actions require a human approval.
• Review that the tools continue to work as you intended them to.

Also see API Evangelist: MCP discovery and governance

Unintentionally logging sensitive customer data (PII, financial info) is a major compliance risk. When deploying APIs, ensuring that this data is kept out of your logs requires robust, layered controls. Whilst developing APIs that will deal with this sensitive content you will often need to see what is being passed to ensure that everything is being handled as expected. Very often in a development or test environment you will be using a dummy payload to ensure the API is working as expected so it is key to ensure it is reflected at the different stages of processing as part of your test suites.

When you progress to partner integration testing with an API you are no longer as in control of the payload passed in through the client, so at this point become dependent on payload logging to see what is being passed and processed. At this point as we would still be dealing with dummy data this would be acceptable in most cases.

Once the API use progresses towards production and real sensitive data is being transferred you need safeguards in place to ensure the data is no longer being logged. Of course data security requires a layered approach and this article only covers one area of this - payload controls. This needs to be paired with additional strong security controls such as ensuring there is encryption in transit and at rest across your system and sensitive data isn't being transmitted in the query string which could be logged by other network components (owasp ref).

In API Connect there are several features that can be used to assist with this and the combination to use will be different depending on your specific requirements. Depending on your goal will determine which one is most appropriate in your scenario:

• To see the data in lower environments to assist in development, make use of full payload activity logging
• Protect specific fields in logs using redaction policies
• Ensure compliance standards are met across the board make use of Governance Rulesets
• To prevent any logging of certain data globally, make use of analytics Filters

Let's examine each of these features in turn:

Activity Log settings

The default behaviour in API Connect is to log activity for all APIs and only log payload (request and response body) on error - but this can be customised on a per-API basis. For the DataPower API Gateway, this is configured in the activity-log extension and the options are payload, activity, headers or none and in the yaml it looks something like this:

activity-log:
  success-content: headers
  error-content: headers
  enabled: true

This could also be configured to reference Catalog properties so that you make these settings at a catalog level and all the APIs reference this with syntax like $(catalog-error-content) in your api definition.

Redaction policy for field level controls

If you don't want to restrict the entire body of the API and it is specific known fields that contain sensitive data, you can use the redaction policy to remove or replace these fields with asterisks. The policy uses JSONata to identify the fields to modify. For this you need to first use the log policy in gather-only mode in order to modify the logging. The gather-only mode in the log policy is crucial because it instructs the policy to collect the full, sensitive payload and place it into the internal log context object. This allows the subsequent redact policy to operate on the logging data itself, prior to it being written to analytics, ensuring the client receives an unmodified response while the sensitive data is masked in the logs.

- log:
  title: Gather log to redact
  version: 2.0.0
  mode: gather-only
- redact:
    version: 2.0.0
    title: redact birthday in the logged data
    redactions:
      - action: redact
        path: $.**.birthday"
    root: log.response_body

Whilst this can also be used to remove or redact the full body it is more typically used on individual fields as adding a policy to each API when you can just set the activity log level is mostly overkill.

Governance rulesets to check

Whichever approach you take, you can make use of governance rulesets to ensure that all APIs meet your standards - these can be used in a CI CD pipeline for checking the status before publish or even configured on a catalog to block publish if the rules aren't met. Governance within API Connect supports the use of spectral rulesets to scan API specifications at development time, through CI/CD pipelines and in periodic catalog scans.

extends: spectral:oas

rules:
  # Ensure activity-log.success-content is header
  activity-log-success-content-header:
    description: "activity-log.success-content must be set to 'header'"
    message: "activity-log.success-content must be 'header'"
    severity: error
    given: $.x-ibm-configuration.activity-log
    then:
      field: success-content
      function: pattern
      functionOptions:
        match: "^header$"

  # Ensure activity-log.error-content is header
  activity-log-error-content-header:
    description: "activity-log.error-content must be set to 'header'"
    message: "activity-log.error-content must be 'header'"
    severity: error
    given: $.x-ibm-configuration.activity-log
    then:
      field: error-content
      function: pattern
      functionOptions:
        match: "^header$"

Analytics filters

If you need to ensure these are never logged in the whole API Connect deployment you can set up filters in the Analytics Cluster configuration. There are two settings here one for storage and one for offload.

spec:
  ...  
  ingestion:
    ...
    filter: |
      mutate {
        remove_field => ["response_body"]
      }

Navigating payload security in API Connect is not about choosing one feature; it's about implementing a defense-in-depth strategy. Start with foundational security (encryption in transit and at rest). Then, apply appropriate Activity Log configuration to get appropriate visibility at each stage. Where you need more fine-grained control, use Redaction Policies to target specific fields.

To continuously validate your configurations meet your compliance requirements make use of governance rulesets in CI/CD and finally, use Analytics Filters as a system-wide preventative guardrail. By stacking these tools appropriately you move from merely tracking data to actively securing it throughout its entire lifecycle.

Lovely way to spend the bank holiday on a circular walk around Cheriton. The route picks up a section of the Itchen Way around Cheriton, joining it into a circular walk with paths around the surrounding fields with gorgeous views across the landscape.

We parked near the village store in the centre of Cheriton and started off along School Lane until we saw the footpath signs to head off along the path between two fields - this was initially quite enclosed and narrow, but soon we turned right onto a wider path along the edge of the field. We continued along this path until we reached Broad Lane where we headed up to where it and Hinton lane joined. We followed Hinton lane up further to Prite lane where there were lovely views across the fields and down into the town.

We headed down Prite lane across the bridge to the Old Mill where we were greeted with the sounds and sights of the River Itchen again. This is where we rejoined the Itchen way along the river past the mill. Further along the river there was a lovely spot that was ideal for Nova to have a nice paddle and play in the water.

Teams spend a lot of energy arguing about whether to put the version in the URL, a header, or a query parameter. That debate usually misses the point. The real question is: what is your change management discipline, and how do you protect consumers from breaking changes?

Taking a step up from the versioning question - how are you managing changes to your API contract? Ideally this would be transparent to consumers unless there is something they are looking to take advantage of in the new changes. If your API is designed well it can abstract any internal requirements and technical limitations from your clients such that you can update the implementation internally and their applications will continue to operate without even needing to know about it.

So a versioning strategy really comes back to what sort of changes you are expecting to your API. As with most things having an understanding of what you will do in this space up front is going to make things smoother down the line.

• Externally versioned standards (e.g. FHIR, OpenAPI specs tied to a standard) — your hands are largely tied. Both you and your consumers need to track versions explicitly, and you'll likely need to support multiple versions in parallel during transitions.
• Additive changes (new fields, new endpoints) — these can often be transparent to consumers, as long as clients are told upfront to ignore unknown fields. No formal versioning strategy may be needed.
• Breaking changes (removing fields, restructuring data types) — you need an explicit consumer-facing versioning strategy.

You'll still need to pick an approach so here's how the main methods compare:

• URL Path (/v1/) - very common and highly visible approach, but maintaining a separate set of endpoints for every version.

• Header-based - keeps URLs clean, but no standard and complicates caching.

• Query Parameters - sitting somewhere between the two above - easy to see what is going on without multiplying endpoints for each version.

• Gateway Managed - instead of client specified versioning you can handle this provider side through an API Gateway, routing consumers to the version of the API they are subscribed to.

No matter which way you choose to handle changes to an API you need to start from your strategy around how you intend to handle changes to your APIs. This needs to be an upfront plan and integrated into your API Contract commitments to your clients rather than something you leave until you need it. Once you have a clear strategy and have committed to it, you can handle technical requirements within that framework much more easily.

When none of these options feel right or you see that breaking changes are frequent and the versioning overhead is getting in the way it may be time to look at GraphQL where you let the client determine the data they are requesting rather than dictating the content from the provider side.

What can you actually see in your API platform?

To go from an API working for the developer building it to knowing it is working for your consumers is where API observability is key and can give you the confidence in the decisions you made building it.

I once made the mistake of turning off logging of metrics for one of our APIs and only discovered this when we needed to debug an issue, at which point we were blind to what was happening with it. As well as having a strategy for observability, it's important to check it is working as expected!

What do your error responses tell the consumer?

The API Consumer needs visibility too, not the same view as those running the platform but enough to figure out if their calls are successful and when they're not why. Status codes and error responses are the primary way to do this and providing something clear and actionable makes it a lot easier for consumers to self-resolve issues.

Beyond the API response itself it is also key to give them a way to see how their applications are calling the API once they are deployed and probably can't see the response directly. This is often through the developer portal so they can easily find out how their applications are using the APIs alongside the documentation and credential management.

Clear actionable responses and observability is even more crucial when the consumer is an AI agent. An agent won't infer what's going on from an obscure error code and outdated documentation. It will retry and seek to workaround the problem likely in ways you didn't intend. For agent consumers, having clear and actionable errors are the difference between recovering gracefully and getting stuck in a runaway loop.

Correlation and trace propagation as contract commitments

If you're committed to providing this traceability of requests to your API, why not include the appropriate headers in your OpenAPI specification as part of the contract. This makes it clear for the consumer this is something the API provides you rather than there being mysterious extra headers on requests that you don't know if you can rely on to use. If it makes sense to and is possible then supporting propagation of trace headers from inbound requests makes sense - in multi-region distributed systems these can be really valuable in understanding what is happening.

Designing for debuggability, not just availability

As you build out your API architecture, ask yourself: "how will I know what is happening here?" If you can answer that at design time, you'll have it covered when you need it in production.

The time to think about what you'd need to debug a problem is when you're designing the system — not at 3am when you're already in the middle of a failure.

A consistent, unique request ID propagated through all your logs and traces is one of the simplest things you can do. When something goes wrong, being able to follow a single transaction end-to-end across your system is invaluable.

Observability isn't something that you bolt on after your API is live - it is worth thinking through as you design your system. A well thought out strategy here will save you and your consumers time in production.

The requirement to be multi-region usually comes suddenly, an immediate business demand, a compliance deadline, or the aftermath of an incident. When designing API Connect SaaS, the decision was to build each region completely independently. Within a region it's highly available across availability zones, but the regions themselves don't replicate. The reasoning is straightforward: as a shared API management platform serving customers with fundamentally different requirements - data residency, cross-region resilience, or both - independent regions let each customer build the topology that fits their constraints, rather than the platform prescribing one.

What is your requirement anyway?

The first thing to consider when designing a multi-region API deployment is to make sure you are clear on what the fundamental business requirement is that is driving you towards a multi-region strategy. There are a number of routes that lead towards multi-region planning and they will all drive different variations to the design.

• Latency - ensuring your APIs respond quickly from the regions you have consumers calling them
• Resilience - providing a failover for if there are issues in one region.
• Compliance - ensuring data and/or traffic to remain in a particular region

For improved latency across regions - is this required primarily around read APIs or do you need to handle write operations as well? If it is primarily reads, then you may be able to reduce your latency with the use of a CDN with edge caching without the complexity and cost of a full multi-region deployment.

If you're looking for resilience, will your requirements be satisfied with a primary region that normally will handle everything and a secondary location that can pick things up if the primary is down or do you need them both to be able to handle traffic in parallel? How long can you cope with a failover between regions taking - seconds or hours? These are the factors that will drive how you go about implementing a multi-region solution.

An API is only as good as its data

Whilst it is relatively easy to replicate API Gateways and application servers between regions - this only gets you so far, as at the end of the day the API depends on a data source. Where does the data for your API reside - what requirements for data storage need to be factored into the overall architecture?

If you have an API that is read-heavy and the data is not time critical you may be able to use replication with eventual consistency saving on the performance hit, cost and complexity of ensuring strong consistency between replicas.

If you have a write heavy API - how will you handle conflicts between writes across regions - will you need to add application or data level locking or would you enforce some form of queuing?

To improve performance across regions the other approach could be to have regional caching to reduce the need to retrieve data from the master source, but this will require careful management of time to live values for cached data so that it remains within an acceptable freshness threshold which will differ between data types.

Auth dependencies are the hidden failure mode

If your identity provider or authorization service is centralised then a regional API failure may be the least of your worries if auth itself is unavailable. API Connect SaaS uses IBMID for it's management plane identity provider, highly available in it's own right, but not part of our regional deployment. Importantly though it's not involved in runtime authentication for customer APIs or their developer portals so any failure doesn't cascade to impact live traffic.

If you can validate JWTs locally without needing to call out to a central service then in most cases you are good, but what happens if you need to revoke a token - do you have a robust system for sharing a revocation list?

How are you handling rate-limiting - is this maintained within the region or do you have a way to track usage across the regions? Within DataPower API Gateway the rate-limiting is maintained through the peering group, requiring a low-latency connection to keep them in-sync. For the DataPower Nano Gateway this can be externalised to your own Valkey or Redis deployment which you could deploy using cross-region replication. Whatever solution your rate limiting is based around, ensure that the period over which the rate is set aligns with the latency in your replication - for example if the replication may sometimes take more than a second you can't use per second rate limiting and would be better to calculate the equivalent over a larger period.

Routing and traffic management

How do you route traffic through your system - starting from the consumer - do you want to expose the regions to them or should they just continue to use a known endpoint and it just works? If you are maintaining a global endpoint you will need some form of global traffic routing - either DNS-based or some form of load-balancing. If you stick with regional endpoints - how are they discovered by the consumer - are they all just servers listed in the OpenAPI specification? Do you provide guidance to the developers building against your API on how to handle retries and failover across regions?

If there are cases where there is state involved, maybe you need some form of session-stickiness to keep the traffic in the region it starts. The techniques you would use here would completely depend on the type of clients you are building for - if this is pure API clients you would need something client IP based, if it's browser based clients, cookies are an option.

If you have global endpoints what controls the routing or failover to a different region - is this purely based on health-check response, latency based or some intelligence based on the inbound IP address and which region is closest to them? For the stateless common components in our API Connect Reserved Instance console we have them available in each region and use broad GeoIP based routing to try and serve inbound traffic from their nearest endpoint with a failover - GeoIP based routing helps to a degree but IP geo-location isn't always accurate.

Failure modes and failover

If there is an issue with one of the regions, how do you detect it and what happens next? Of course if you have to serve traffic within region it may just be a case of waiting it out.

How do you test the failover? Any failover plan is only any use if it works so it is crucial that you have a way to test this. Ideally you would identify a way simulate a failure and trigger the failover - then you can ensure your observability covers this to show what is happening during a failover. Also worth considering when you look at observability for a multi-region solution is whether you define Service Level Objectives by region or across the solution and how you track these, along with the resilience of your monitoring tooling as well.

Actual failures aren't always as cut and dried as the test however - you may find that a region is only partially impacted or continues to run just without any network connectivity. These are key factors to consider when determining the trigger for failover but also for clarifying the state of the system during the failover and what actions will be needed to move back to normal operation after the failover. It may be that you need to treat a region as completely down even when it isn't so as not to end up with split-brain across the regions data - then restore a backup there in order to return to normal afterwards. This was a fundamental part of how we used to deal with replication in Informix - where if there was a network partition we would find multiple nodes continuing to run as primary and we would need to select which node to reset. This was usually the one with a secondary server still attached, but occasionally there would be workload continuing on the standalone primary and nothing happening in the clustered one. The same principle applies here at a higher level: sometimes the safest recovery path is to treat a degraded region as completely lost and restore cleanly rather than trying to reconcile state.

Are there parts of your system that need to be disabled during a regional failure and run in a degraded mode as they add more risk or aren't as crucial to maintain availability on?

Most multi-region requirements aren't as simple as just adding another geographic location - there is a lot more to think through across the different layers of your system - data, auth, routing and how it actually will handle failures. Going back to where we started, the right answer will depend on the requirement you are building for. Latency, resilience and compliance will each take you in different directions as you design so it is key to have clarity up-front rather than trying to do everything and over-engineering a solution. For more on designing with reliability in mind from the start see SRE Mindset in API Architecture or for an example of how this could be done see my post on building a global deployment with API Connect.

I spent a good chunk of my career working in SRE and then when the opportunity came up I took the decision to move over into an Architecture role - in some ways a change and in many ways a wider remit to continue with the types of things I'd been doing and working on in the SRE team.

How SRE reliability principles shape architecture

Core principles from my time in SRE have definitely shaped the way I look at bringing our product to the cloud - and whilst there's still a lot of things that I (and our SRE team) wish we had done better with, I think it's a valuable insight to have when building out these systems. I think the biggest area this has influenced my way of working is a desire to be driven by real data where possible and validate my thinking with what we are seeing in reality.

Sizing - planning for reality, not hopes

When we're building something new we often get the numbers wrong - either we're too optimistic and expect viral exponential growth or we're too pessimistic and assume no-one is interested. If however you're building something that you have experience with either directly or in a similar area there is real data available to reference.

This means you can inform your sizing estimations with a grounding in historic real world usage data. It won't always be precise and these things usually don't repeat in the same way but at least it means you are learning from something and building on the lived experience.

Even though we can capture some data to base our sizing on we still want to validate it - key to any architecture is testing the decisions you've made - both in small parts and end to end. For load testing I would want to use realistic numbers that can be proportioned against the expectations - so validating each point in the scale up - over a short term to understand the characteristics - then do some additional longer running load tests to ensure things continue to operate over time as you would expect. This also needs to be combined with your observability plan - if you get these in place early you can use them as part of your load testing validation and capture valuable insights into your system.

This then leads into the next set of sizing decisions - how and when do you scale? For each part of your system you need to look at how scaling would apply and the impact this has on the neighbouring components - e.g. if we just scale up the API Gateway to handle more load but haven't ensured we have the right rate limiting or considered the capacity of the backend microservices serving the API we could just be opening ourselves up for too much traffic and different types of failure. At each layer we need to consider how much capacity we need, how long the different scaling strategies would take and the cost implications of allowing this to scale automatically. Some parts of the system may be so critical or growth is directly proportional to your revenue so that an increase in cost with autoscaling may be the desired outcome.

Observability - you can't fix what you can't see

Can you tell what is going on in the system? As systems become more complex and more microservices are added it is fundamental to be able to answer these types of questions about specific areas. Building systems for observability from the start makes it a lot easier than bolting this on afterwards. Defining standard approaches and formats for observability means that you're not deciding for each component independently but you have something that everything in the system can adopt. These need to cover the three main pillars of observability - Metrics, Logs and Traces.

• Metrics give you a lightweight numeric representation of parts of your system that you can graph over time - they are useful for indicators as to how loaded things are or how slow certain flows are performing. For example, response times, payload sizes and error count
• Traces let you follow the end to end journey of a request through your system detailing the steps it has taken along the way and connections between microservices and processes. This would show the inbound request from when it hits your API Gateway on to each microservice that is involved in responding to that request - giving you full details of what and when at each stage. In fact your traces are really the living illustration of your architecture.
• Logs give you insight into the path through your code that is being taken or the error cases you are hitting. Having consistently formatted structured logs with request ids to correlate what request the log relates to or tie it to the trace without guessing.

I read a comment somewhere that diagnosing problems in a microservices architecture is like a murder mystery - however much you like solving these, you don't want to have to at 3am at the end of a pager, so in logging we need to think about this up front.

Metrics will give you an easy snapshot of if there is a problem, Traces will show you which service is causing the problem and the logs will tell you why. These will also have a significant impact on the costing of your solution - metrics being the cheapest and logs being the most expensive as they will scale up rapidly as your volumes increase - 1 transaction could result in 10-100 log lines depending on the complexity and how verbose your logging is. This is also why it is key to be able to dynamically adjust log levels - you can keep the general volume down and increase it as necessary.

As with sizing we mustn't get carried away with the trillions of transactions we're hoping to see on day 1 with 100% uptime - we need to set some expectations (Service Level Objectives) as to what real use of the system needs to look like and what our customers would accept. This is not what the SLA is or what the ideal requirement is (that would be super fast and always available) but the acceptable level of availability and responsiveness day to day. In some scenarios waiting for 5 seconds for data is fine, and others it is needed within milliseconds and each delay costs money. Just like some scenarios can cope with a retry and others can't.

Sizing the system, ensuring we can answer the crucial questions about what is happening in it and appropriately meet our users' expectations are some of the ways my SRE experience has directly fed into my architecture work. Equally important is designing for resilience - ensuring the system can fracefully handle failure rather than just hoping it won't happen. That's one of the next topics I plan to explore here, looking at failure modes and resilience patterns building from what I've covered here.