When Failure Is Not an Option: How Form3 Built a Three-Cloud Payment Platform in Go

Multi-cloud Isn’t a Myth. Form3 Proves it Runs in Go — Every Day, at Scale.

It was Thursday evening, June 12th, 2024. Kevin Holditch was on the train home, phone out, ready to watch a football tactics breakdown on YouTube.

The YouTubers he followed were usually polished — professional graphics, live chat flying, the works. But that evening’s stream was just two people staring at a black screen, microphones cutting out. Kevin glanced at the chat and immediately understood what was happening.

AWS was down. At a massive scale.

He closed the football video, opened his work laptop, and pulled up Form3’s monitoring dashboard.

Not a single payment was affected.

That moment is the entire point of Kevin’s keynote at GopherCon UK 2025. Not “here’s how you should architect multi-cloud” — but “here’s proof it actually works, validated by a real AWS outage.”

What Does Form3 Do?

Before the architecture, the business context matters, because the architecture was forced by the business requirements.

Form3 provides payment scheme integration. Every country has its own payment infrastructure: the UK has BACS and FPS (Faster Payment Scheme), Europe has SEPA, and so on around the world. Each scheme is completely different in how you connect (VPN? Private line? Internet?), what message format it uses (space-delimited formats from the 1970s? XML?), and how it handles errors and message flows.

Form3’s product is a single REST API that abstracts all of this away. Banks and fintechs integrate once and gain access to every payment scheme globally, without having to care about the underlying complexity.

That positioning means their platform cannot go down. Ever.

V1: Go All-In on AWS

Form3 was founded in late 2016 with four engineers. Kevin was one of them.

The strategy was straightforward: offload operational complexity to AWS. The stack reflected that:

Microservices in Java + Spring Boot
Containers on AWS ECS (AWS’s proprietary container runtime — think Kubernetes with an AWS lock-in)
Messaging via AWS SQS
Data in PostgreSQL on AWS RDS

This was the right call for a 4-person team trying to prove a business. AWS handles your database, your message broker, and your SLAs. When something breaks, AWS support is on the hook, not you. It worked — Form3 landed their first customers, mostly smaller fintechs.

Then success brought new problems.

The Regulatory Trigger

As Form3 started attracting top-tier UK high street banks, those banks came with entirely different requirements.

The UK’s Prudential Regulation Authority (PRA) — the body that oversees banks — is deeply nervous about cloud concentration risk. One of their core mandates: banks cannot be fully dependent on a single cloud provider and must be able to exit that cloud relatively quickly. The regulator doesn’t tell you how to do it. They just say you must.

Banks passed that requirement down to Form3: run across multiple clouds, or we can’t use you.

Two architectural paths emerged:

Option A: Re-deploy on a second cloud (say, GCP), maintain both in parallel, and fail over in a disaster.

The problems are obvious — two codebases to maintain, and a big-bang failover event that itself becomes a risk. Form3 rejected this.

Option B: Treat each cloud provider like an availability zone. Run active-active-active across all three simultaneously. If any one fails, it’s no different from a single AZ going down.

They chose Option B.

The trade-off was significant: abandon all AWS-proprietary services and adopt cloud-agnostic, open-source alternatives — accepting the operational burden of running those technologies. The gain: engineers write code without ever needing to know or care which cloud it runs on.

V2: The Three-Cloud Architecture

Here’s what the new architecture looks like:

Each cloud provider (AWS, GCP, Azure) runs an independent Kubernetes cluster. Customers can ingress through any cloud. All three clusters are privately networked together so pods can communicate directly via IP address across clouds.

The technology stack swaps were deliberate:

Two of these deserve more explanation.

Why CockroachDB?

The database requirements for a payment system can be summarized in four properties: always-on, strongly consistent, horizontally scalable, and zero-downtime upgrades. That’s a demanding combination. Very few databases satisfy all four simultaneously.

CockroachDB checks every box — and two of its properties made migration practical:

It implements the PostgreSQL wire protocol, so services already migrated to Go could switch with minimal code changes
It uses Raft-based consensus natively, which means it handles cross-region and cross-cloud consistency out of the box

Form3’s configuration: a write is acknowledged as successful only after at least two cloud providers have accepted it. A small latency trade-off in exchange for an absolute guarantee that payments are never duplicated or lost. For a payment platform, that trade-off is non-negotiable.

Why NATS JetStream?

NATS itself is written in Go and was built by former engineers from Tibco, a company that developed high-performance message buses for real-time trading platforms. JetStream is NATS’s durable messaging layer, also Raft-based, with the same cross-cloud fault tolerance properties.

The key property it delivers is that the message bus is no longer single-cloud — it’s a single logical cluster spanning all three providers. Losing an entire cloud is handled transparently by the NATS consensus layer.

Why Go Instead of Java?

Kevin mentioned three concrete reasons for the Java → Go migration.

1. Cold Start Time

Form3’s integration tests work by spinning up real Docker containers to simulate all dependencies, then running tests end-to-end. Java + Spring Boot applications routinely take 30 seconds to start. With Go, startup is near-instant. Multiply that across dozens of services and hundreds of daily test runs, and the productivity gain is significant.

2. No Magic

Kevin described a colleague who wrote a beautiful Java integration test — three lines, given/when/then, which automatically launched Docker Compose, rewired all database clients, ran the tests, and cleaned up. It looked elegant. When Kevin asked how it worked: “There’s an annotation at the top.”

That annotation pulled in dozens of JAR files and a web of auto-configuration logic. Beautiful to read, nearly impossible to debug when something breaks.

Go has no magic. What you see is what happens. In a microservices architecture where engineers regularly work across dozens of codebases, explicit beats elegant.

3. Natural Fit for Microservices

When you maintain many services simultaneously, readability and reasonability matter more than cleverness. Go’s explicit style means a new engineer can pick up any service and understand it without first mastering a framework’s implicit conventions. That consistency compounds at scale.

Three Engineering Challenges — and How Go Helped Solve Them

The architecture works in production. Getting there required solving three specific problems.

Challenge 1: Pod Disruption Budgets Don’t Cross Cluster Boundaries

Kubernetes Pod Disruption Budgets (PDB) are a clean mechanism: tell Kubernetes how many pods can be unavailable during maintenance operations such as node pool rolling updates.

The problem is that PDBs are cluster-scoped.

Form3 runs 3 CockroachDB pods per cluster across 3 clusters — 9 pods total. Quorum requires at least 5 pods online at all times.

Setting maxUnavailable=1 on each cluster individually is technically valid. But it means all three clusters could simultaneously take one pod down each — leaving 6 online, dangerously close to quorum failure. Worse, there's no global constraint preventing this.

Their solution: they built and open-sourced XPDB (Cross-cluster Pod Disruption Budget), a Kubernetes operator that spans multiple clusters. With XPDB, you can declare a single global constraint: at most 1 pod unavailable across all clusters combined, regardless of which cluster triggers the disruption.

This is Kubernetes extensibility at its best — define a domain-specific language through CRDs and implement cross-cluster coordination logic through operators.

Challenge 2: Node Pool Upgrades at Scale

Kubernetes releases frequently. Managed control planes (EKS, GKE, Ask) upgrade with a button click. But the underlying node pools require rolling every single node to adopt new Kubernetes versions.

Form3’s scale: multiple node pool types (apps, CockroachDB, NATS, etc.) × 3 clouds × 3 environments (dev/staging/prod) × multiple geographic jurisdictions. All managed via Terraform. A single Kubernetes version bump means a large number of PRs, strictly sequenced dev → staging → prod. Essentially unmanageable at speed.

Their solution: the Cluster Lifecycle Operator, a custom Kubernetes operator that takes ownership of node pool lifecycle. Engineers simply declare the desired Kubernetes version via a CRD, and the operator handles all the coordination.

A Kubernetes version upgrade went from dozens of Terraform PRs to 3 CRD changes — one per environment.

Challenge 3: DR Testing Isn’t a Once-a-Year Event

Traditional disaster recovery testing at most organizations looks like this: retrieve a Word document from a drawer once a year, check boxes, sign off, and put it back. Maybe update it. Repeat next year.

This is useless when your codebase changes daily. Yesterday’s DR test is invalidated the moment you deploy new code.

Form3’s approach: build a fake customer system in a pre-production environment — a fully independent deployment that simulates real customer behavior, sending real payments every second against fake payment scheme implementations. Continuously alert to fires as soon as any payment fails.

Every night, Chaos Mesh Combined with custom Go scenarios, it injects failures into the platform: taking down an entire cloud provider, severing network connections, and killing the database.

Every morning, engineers get a detailed report on each failure scenario — which payments were affected, what was recovered, and how long it took.

DR testing became a continuous engineering feedback loop. Any code change gets tested against real failure scenarios by the next morning.

The Moment It All Came Together

Back to the evening of June 12th.

The AWS outage hit. Form3’s three-cloud architecture had been running in production for years, stress-tested by hundreds of internal chaos engineering sessions. But there’s always uncertainty until a real failure arrives.

Kevin logged in from the train. System metrics: normal. Customer impact: zero. For the entire duration of a massive AWS outage that took down many high-profile services globally, Form3’s customers — including several top UK banks — experienced nothing.

That’s not luck. That’s years of engineering investment, validated in a single moment.

Summary

Kevin’s closing lessons are worth writing down directly:

Multi-cloud is hard, but achievable — stand on the shoulders of open-source giants; don’t reinvent distributed systems
Kubernetes extensibility is your leverage — CRD + Operator patterns let you express any cross-cluster operational requirement declaratively
Active-active beats active-passive — no failover event means no failover failure
Make DR testing a first-class engineering discipline — daily, automated, with reports, not annual checkbox theater
Go was a significant enabler — fast startup, explicit code, easy reasoning across many services

Multi-cloud isn’t a PowerPoint slide. It’s running in Go, in production, across three clouds — every single day.

Resources

Original talk (GopherCon UK 2025): youtube.com/watch?v=vrnIrHsG7HE[1]