Teams spend a lot of energy arguing about whether to put the version in the URL, a header, or a query parameter. That debate usually misses the point. The real question is: what is your change management discipline, and how do you protect consumers from breaking changes?

Taking a step up from the versioning question - how are you managing changes to your API contract? Ideally this would be transparent to consumers unless there is something they are looking to take advantage of in the new changes. If your API is designed well it can abstract any internal requirements and technical limitations from your clients such that you can update the implementation internally and their applications will continue to operate without even needing to know about it.

So a versioning strategy really comes back to what sort of changes you are expecting to your API. As with most things having an understanding of what you will do in this space up front is going to make things smoother down the line.

• Externally versioned standards (e.g. FHIR, OpenAPI specs tied to a standard) — your hands are largely tied. Both you and your consumers need to track versions explicitly, and you'll likely need to support multiple versions in parallel during transitions.
• Additive changes (new fields, new endpoints) — these can often be transparent to consumers, as long as clients are told upfront to ignore unknown fields. No formal versioning strategy may be needed.
• Breaking changes (removing fields, restructuring data types) — you need an explicit consumer-facing versioning strategy.

You'll still need to pick an approach so here's how the main methods compare:

• URL Path (/v1/) - very common and highly visible approach, but maintaining a separate set of endpoints for every version.

• Header-based - keeps URLs clean, but no standard and complicates caching.

• Query Parameters - sitting somewhere between the two above - easy to see what is going on without multiplying endpoints for each version.

• Gateway Managed - instead of client specified versioning you can handle this provider side through an API Gateway, routing consumers to the version of the API they are subscribed to.

No matter which way you choose to handle changes to an API you need to start from your strategy around how you intend to handle changes to your APIs. This needs to be an upfront plan and integrated into your API Contract commitments to your clients rather than something you leave until you need it. Once you have a clear strategy and have committed to it, you can handle technical requirements within that framework much more easily.

When none of these options feel right or you see that breaking changes are frequent and the versioning overhead is getting in the way it may be time to look at GraphQL where you let the client determine the data they are requesting rather than dictating the content from the provider side.

What can you actually see in your API platform?

To go from an API working for the developer building it to knowing it is working for your consumers is where API observability is key and can give you the confidence in the decisions you made building it.

I once made the mistake of turning off logging of metrics for one of our APIs and only discovered this when we needed to debug an issue, at which point we were blind to what was happening with it. As well as having a strategy for observability, it's important to check it is working as expected!

What do your error responses tell the consumer?

The API Consumer needs visibility too, not the same view as those running the platform but enough to figure out if their calls are successful and when they're not why. Status codes and error responses are the primary way to do this and providing something clear and actionable makes it a lot easier for consumers to self-resolve issues.

Beyond the API response itself it is also key to give them a way to see how their applications are calling the API once they are deployed and probably can't see the response directly. This is often through the developer portal so they can easily find out how their applications are using the APIs alongside the documentation and credential management.

Clear actionable responses and observability is even more crucial when the consumer is an AI agent. An agent won't infer what's going on from an obscure error code and outdated documentation. It will retry and seek to workaround the problem likely in ways you didn't intend. For agent consumers, having clear and actionable errors are the difference between recovering gracefully and getting stuck in a runaway loop.

Correlation and trace propagation as contract commitments

If you're committed to providing this traceability of requests to your API, why not include the appropriate headers in your OpenAPI specification as part of the contract. This makes it clear for the consumer this is something the API provides you rather than there being mysterious extra headers on requests that you don't know if you can rely on to use. If it makes sense to and is possible then supporting propagation of trace headers from inbound requests makes sense - in multi-region distributed systems these can be really valuable in understanding what is happening.

Designing for debuggability, not just availability

As you build out your API architecture, ask yourself: "how will I know what is happening here?" If you can answer that at design time, you'll have it covered when you need it in production.

The time to think about what you'd need to debug a problem is when you're designing the system — not at 3am when you're already in the middle of a failure.

A consistent, unique request ID propagated through all your logs and traces is one of the simplest things you can do. When something goes wrong, being able to follow a single transaction end-to-end across your system is invaluable.

Observability isn't something that you bolt on after your API is live - it is worth thinking through as you design your system. A well thought out strategy here will save you and your consumers time in production.

The requirement to be multi-region usually comes suddenly, an immediate business demand, a compliance deadline, or the aftermath of an incident. When designing API Connect SaaS, the decision was to build each region completely independently. Within a region it's highly available across availability zones, but the regions themselves don't replicate. The reasoning is straightforward: as a shared API management platform serving customers with fundamentally different requirements - data residency, cross-region resilience, or both - independent regions let each customer build the topology that fits their constraints, rather than the platform prescribing one.

What is your requirement anyway?

The first thing to consider when designing a multi-region API deployment is to make sure you are clear on what the fundamental business requirement is that is driving you towards a multi-region strategy. There are a number of routes that lead towards multi-region planning and they will all drive different variations to the design.

• Latency - ensuring your APIs respond quickly from the regions you have consumers calling them
• Resilience - providing a failover for if there are issues in one region.
• Compliance - ensuring data and/or traffic to remain in a particular region

For improved latency across regions - is this required primarily around read APIs or do you need to handle write operations as well? If it is primarily reads, then you may be able to reduce your latency with the use of a CDN with edge caching without the complexity and cost of a full multi-region deployment.

If you're looking for resilience, will your requirements be satisfied with a primary region that normally will handle everything and a secondary location that can pick things up if the primary is down or do you need them both to be able to handle traffic in parallel? How long can you cope with a failover between regions taking - seconds or hours? These are the factors that will drive how you go about implementing a multi-region solution.

An API is only as good as its data

Whilst it is relatively easy to replicate API Gateways and application servers between regions - this only gets you so far, as at the end of the day the API depends on a data source. Where does the data for your API reside - what requirements for data storage need to be factored into the overall architecture?

If you have an API that is read-heavy and the data is not time critical you may be able to use replication with eventual consistency saving on the performance hit, cost and complexity of ensuring strong consistency between replicas.

If you have a write heavy API - how will you handle conflicts between writes across regions - will you need to add application or data level locking or would you enforce some form of queuing?

To improve performance across regions the other approach could be to have regional caching to reduce the need to retrieve data from the master source, but this will require careful management of time to live values for cached data so that it remains within an acceptable freshness threshold which will differ between data types.

Auth dependencies are the hidden failure mode

If your identity provider or authorization service is centralised then a regional API failure may be the least of your worries if auth itself is unavailable. API Connect SaaS uses IBMID for it's management plane identity provider, highly available in it's own right, but not part of our regional deployment. Importantly though it's not involved in runtime authentication for customer APIs or their developer portals so any failure doesn't cascade to impact live traffic.

If you can validate JWTs locally without needing to call out to a central service then in most cases you are good, but what happens if you need to revoke a token - do you have a robust system for sharing a revocation list?

How are you handling rate-limiting - is this maintained within the region or do you have a way to track usage across the regions? Within DataPower API Gateway the rate-limiting is maintained through the peering group, requiring a low-latency connection to keep them in-sync. For the DataPower Nano Gateway this can be externalised to your own Valkey or Redis deployment which you could deploy using cross-region replication. Whatever solution your rate limiting is based around, ensure that the period over which the rate is set aligns with the latency in your replication - for example if the replication may sometimes take more than a second you can't use per second rate limiting and would be better to calculate the equivalent over a larger period.

Routing and traffic management

How do you route traffic through your system - starting from the consumer - do you want to expose the regions to them or should they just continue to use a known endpoint and it just works? If you are maintaining a global endpoint you will need some form of global traffic routing - either DNS-based or some form of load-balancing. If you stick with regional endpoints - how are they discovered by the consumer - are they all just servers listed in the OpenAPI specification? Do you provide guidance to the developers building against your API on how to handle retries and failover across regions?

If there are cases where there is state involved, maybe you need some form of session-stickiness to keep the traffic in the region it starts. The techniques you would use here would completely depend on the type of clients you are building for - if this is pure API clients you would need something client IP based, if it's browser based clients, cookies are an option.

If you have global endpoints what controls the routing or failover to a different region - is this purely based on health-check response, latency based or some intelligence based on the inbound IP address and which region is closest to them? For the stateless common components in our API Connect Reserved Instance console we have them available in each region and use broad GeoIP based routing to try and serve inbound traffic from their nearest endpoint with a failover - GeoIP based routing helps to a degree but IP geo-location isn't always accurate.

Failure modes and failover

If there is an issue with one of the regions, how do you detect it and what happens next? Of course if you have to serve traffic within region it may just be a case of waiting it out.

How do you test the failover? Any failover plan is only any use if it works so it is crucial that you have a way to test this. Ideally you would identify a way simulate a failure and trigger the failover - then you can ensure your observability covers this to show what is happening during a failover. Also worth considering when you look at observability for a multi-region solution is whether you define Service Level Objectives by region or across the solution and how you track these, along with the resilience of your monitoring tooling as well.

Actual failures aren't always as cut and dried as the test however - you may find that a region is only partially impacted or continues to run just without any network connectivity. These are key factors to consider when determining the trigger for failover but also for clarifying the state of the system during the failover and what actions will be needed to move back to normal operation after the failover. It may be that you need to treat a region as completely down even when it isn't so as not to end up with split-brain across the regions data - then restore a backup there in order to return to normal afterwards. This was a fundamental part of how we used to deal with replication in Informix - where if there was a network partition we would find multiple nodes continuing to run as primary and we would need to select which node to reset. This was usually the one with a secondary server still attached, but occasionally there would be workload continuing on the standalone primary and nothing happening in the clustered one. The same principle applies here at a higher level: sometimes the safest recovery path is to treat a degraded region as completely lost and restore cleanly rather than trying to reconcile state.

Are there parts of your system that need to be disabled during a regional failure and run in a degraded mode as they add more risk or aren't as crucial to maintain availability on?

Most multi-region requirements aren't as simple as just adding another geographic location - there is a lot more to think through across the different layers of your system - data, auth, routing and how it actually will handle failures. Going back to where we started, the right answer will depend on the requirement you are building for. Latency, resilience and compliance will each take you in different directions as you design so it is key to have clarity up-front rather than trying to do everything and over-engineering a solution. For more on designing with reliability in mind from the start see SRE Mindset in API Architecture or for an example of how this could be done see my post on building a global deployment with API Connect.

I spent a good chunk of my career working in SRE and then when the opportunity came up I took the decision to move over into an Architecture role - in some ways a change and in many ways a wider remit to continue with the types of things I'd been doing and working on in the SRE team.

How SRE reliability principles shape architecture

Core principles from my time in SRE have definitely shaped the way I look at bringing our product to the cloud - and whilst there's still a lot of things that I (and our SRE team) wish we had done better with, I think it's a valuable insight to have when building out these systems. I think the biggest area this has influenced my way of working is a desire to be driven by real data where possible and validate my thinking with what we are seeing in reality.

Sizing - planning for reality, not hopes

When we're building something new we often get the numbers wrong - either we're too optimistic and expect viral exponential growth or we're too pessimistic and assume no-one is interested. If however you're building something that you have experience with either directly or in a similar area there is real data available to reference.

This means you can inform your sizing estimations with a grounding in historic real world usage data. It won't always be precise and these things usually don't repeat in the same way but at least it means you are learning from something and building on the lived experience.

Even though we can capture some data to base our sizing on we still want to validate it - key to any architecture is testing the decisions you've made - both in small parts and end to end. For load testing I would want to use realistic numbers that can be proportioned against the expectations - so validating each point in the scale up - over a short term to understand the characteristics - then do some additional longer running load tests to ensure things continue to operate over time as you would expect. This also needs to be combined with your observability plan - if you get these in place early you can use them as part of your load testing validation and capture valuable insights into your system.

This then leads into the next set of sizing decisions - how and when do you scale? For each part of your system you need to look at how scaling would apply and the impact this has on the neighbouring components - e.g. if we just scale up the API Gateway to handle more load but haven't ensured we have the right rate limiting or considered the capacity of the backend microservices serving the API we could just be opening ourselves up for too much traffic and different types of failure. At each layer we need to consider how much capacity we need, how long the different scaling strategies would take and the cost implications of allowing this to scale automatically. Some parts of the system may be so critical or growth is directly proportional to your revenue so that an increase in cost with autoscaling may be the desired outcome.

Observability - you can't fix what you can't see

Can you tell what is going on in the system? As systems become more complex and more microservices are added it is fundamental to be able to answer these types of questions about specific areas. Building systems for observability from the start makes it a lot easier than bolting this on afterwards. Defining standard approaches and formats for observability means that you're not deciding for each component independently but you have something that everything in the system can adopt. These need to cover the three main pillars of observability - Metrics, Logs and Traces.

• Metrics give you a lightweight numeric representation of parts of your system that you can graph over time - they are useful for indicators as to how loaded things are or how slow certain flows are performing. For example, response times, payload sizes and error count
• Traces let you follow the end to end journey of a request through your system detailing the steps it has taken along the way and connections between microservices and processes. This would show the inbound request from when it hits your API Gateway on to each microservice that is involved in responding to that request - giving you full details of what and when at each stage. In fact your traces are really the living illustration of your architecture.
• Logs give you insight into the path through your code that is being taken or the error cases you are hitting. Having consistently formatted structured logs with request ids to correlate what request the log relates to or tie it to the trace without guessing.

I read a comment somewhere that diagnosing problems in a microservices architecture is like a murder mystery - however much you like solving these, you don't want to have to at 3am at the end of a pager, so in logging we need to think about this up front.

Metrics will give you an easy snapshot of if there is a problem, Traces will show you which service is causing the problem and the logs will tell you why. These will also have a significant impact on the costing of your solution - metrics being the cheapest and logs being the most expensive as they will scale up rapidly as your volumes increase - 1 transaction could result in 10-100 log lines depending on the complexity and how verbose your logging is. This is also why it is key to be able to dynamically adjust log levels - you can keep the general volume down and increase it as necessary.

As with sizing we mustn't get carried away with the trillions of transactions we're hoping to see on day 1 with 100% uptime - we need to set some expectations (Service Level Objectives) as to what real use of the system needs to look like and what our customers would accept. This is not what the SLA is or what the ideal requirement is (that would be super fast and always available) but the acceptable level of availability and responsiveness day to day. In some scenarios waiting for 5 seconds for data is fine, and others it is needed within milliseconds and each delay costs money. Just like some scenarios can cope with a retry and others can't.

Sizing the system, ensuring we can answer the crucial questions about what is happening in it and appropriately meet our users' expectations are some of the ways my SRE experience has directly fed into my architecture work. Equally important is designing for resilience - ensuring the system can fracefully handle failure rather than just hoping it won't happen. That's one of the next topics I plan to explore here, looking at failure modes and resilience patterns building from what I've covered here.

I've been fascinated by the idea of AI agents for a while now - programs that can not just chat, but actually do things. A few weeks back I came across Clawdbot - now called OpenClaw and it clicked something in my brain. This had a great potential for proactive workflows with a regular trigger to identify things that needed doing and the ability to extend itself with additional code and updates to it's own configuration. Very powerful, but also very risky if not controlled. I set this up intially as it's own user account and then fairly soon moved it into an isolated VM.

I really liked the potential here but wasn't comfortable with the complexity of the code and the power I was giving it or would need to give it to do more interesting tasks - I've still not let it access calendar or e-mail for example. However if I could build something I could read all the code for and understand - with my own selection of controls and tools that are limited according to my own sense of risk level.

So rather than just using someone elses platform I embarked on building something smaller and more personal that I could tune to the things I wanted it to do. This became a great learning experience in understanding the basics at the core of what is needed in an Agent but also how this can get much more complex as you start to deal with things like memory and control.

Why Go?

Python would have been the obvious choice. It's the default language of AI, with libraries for everything, but I wanted to start with API calls rather than libraries and Go offered something different:

• Single binary deployment - no virtual environments, no dependency hell
• Straightforward concurrency - agents need to do multiple things at once
• Static typing - catching mistakes at compile time matters when your agent is taking real actions

The Architecture

The agent follows a pattern you'll recognise if you've looked at OpenClaw or similar frameworks:

1. Model - The brain. Currently I'm switching between Ollama, MiniMax and Claude all using their Anthropic compatible API
2. Planner - Takes the user's request and figures out what steps are needed
3. Executor - Runs the planned steps, handling tool calls and results
4. Tools - The verbs the agent can use (files, commands, web search, etc.)

User Request → Planner → [Tool Calls] → Executor → Results
                   ↑                         │
                   └────── Feedback ←────────┘

The key insight from OpenClaw that shaped this was deliberate action. The agent doesn't just guess what to do - it reasons about the next step, executes it, observes the result, and decides what to do next. That loop is the heart of it.

The Tool Selection

So far I've created a set of basic tools:

The interesting part wasn't adding tools - it was deciding what not to add. Every tool is a potential attack surface or a source of confusion. The discipline of a small, thoughtful toolbox is more valuable than a big one.

Lessons from Building It

Context is everything

The agent is only as good as what it knows about the world it's operating in. Early versions missed crucial details because I didn't pass the right context to the model. Now I spend more time thinking about what context to provide than about the tool infrastructure itself.

Tool design matters enormously

Badly designed tools create bad agent behaviour. The model will find the problems with the tools and either hit them multiple times causing more to clean up or work around them in interesting new ways. A tool that returns too much information overwhelms the context. Too little, and the agent is flying blind. The best tools do one thing well and return structured, predictable output. One of the things I found that helped here was when the agent was struggling with the tools - either through bugs or a lack of clarity I had it look at improving them in context so the specific error fed into the solution. An example of this was it kept trying to use the exec tool to write new files and hitting issues as my protections were blocking multi-line pipes, where as I'd intended this to be done through the edit tool - so asked the agent to improve the description and metadata for the edit tool to make it clear it could be used for this.

Trust but verify

The agent can take actions - that's the point. But it needs guardrails. I've built in permissions and observability so I can see what it's trying to do before it does something irreversible. The goal is helpful, not dangerous.

What's Next

Right now the agent is genuinely useful for my workflow. It writes drafts, fetches information, organises files, and saves me clicks. But there's more to do:

• Memory - I'm storing memory in markdown but want to explore doing something better here for easier retrieval
• Multi-step tool composition - chaining tools intelligently - often today it feels like there are more steps used than really necessary so seeing potential for optimisation
• Better planning - fewer back-and-forth iterations

The thing I'm most proud of isn't any single feature. It's that it exists, that I built something that actually works, and that it reflects how I think about AI: not as magic, but as a tool. I'll probably share more findings as I experiment more - for now this is working for me as a collaborator on code development (including it's own code base) and a research assistant.

The code lives in a private repo for now as this is intended as a personal agent not a framework, but the concepts here are universal. If you're thinking about building an agent, start small, choose your tools carefully, and remember: the goal is to build something that helps you.

References I found helpful

• How to build an agent from Amp Code - I used this one as a starter to see the basic principles.
• Anthropic API guide and Ollama anthropic compatibility helped me go beyond the SDK use from the first example
• Telegram Bot Go Framework let me go beyond CLI based chat to talk to my agent from anywhere