Distributed Transaction Consistency in Go
Saga, TCC, and the Local Message Table
Golang
Distributed Transaction Consistency in Go
Chapter 1: The Landscape
As microservice architectures became the default, Go grew into a mainstream language for distributed systems. Its concurrency model, lightweight goroutines, and cloud-native fit make it a natural choice. But once a business is split across services, the single-node ACID transaction no longer applies, and data consistency turns into one of the hardest problems standing between you and a reliable system.
The Go ecosystem doesn’t have a global-transaction framework as mature as Java’s Spring XA. What it has instead is a set of lighter approaches that fit the cloud-native path better: TCC, Saga, message-based eventual consistency, and newer open-source implementations like Seata-Go and Dtm-Go. Picking one means weighing performance, observability, rollback reliability, and the cognitive load on the developers who have to maintain it.
Comparing the common consistency patterns
Quickly validating the Saga pattern
Here’s a snippet that starts a simple Saga transaction with dtm-client-go:
// Initialize the Dtm client (the dtm service must already be deployed)
dtmClient := dtmcli.NewHTTPDtmClient("http://localhost:36789")
// Define two sub-transactions: deduct inventory, then create the order
saga := dtmcli.NewSaga(dtmClient, dtmcli.MustGenGid(dtmClient)).
Add("http://stock-service/Reduce", "http://stock-service/ReduceRevert",
map[string]interface{}{"product_id": 1001, "amount": 1}).
Add("http://order-service/Create", "http://order-service/CreateRevert",
map[string]interface{}{"user_id": 123, "product_id": 1001})
// Submit and wait for the result
if err := saga.Submit(); err != nil {
log.Fatal("saga failed: ", err) // if any forward step fails, dtm triggers the matching compensation
}The call talks to the DTM coordinator over HTTP, and the coordinator manages the global transaction lifecycle. You only write the business logic and the compensation endpoints. In production, pair this with OpenTelemetry spans and a correlation ID in your logs so the cross-service path remains traceable and the state remains auditable.
Chapter 2: A Deep Dive into Saga
2.1 The theoretical model and compensation semantics
Saga breaks a long transaction into a series of local transactions. Each local transaction has a matching compensation, and together they deliver eventual consistency.
The core constraints:
- Every forward operation (
T_i) must have an idempotent, reversible compensation (C_i). C_imust undo the business effect ofT_iwithout depending on whetherT_{i+1}ran.- A Saga can be coordinated in two ways: choreography (event-driven) or orchestration (a centralized coordinator).
Here’s a forward path expressed as orchestration. In Go, instead of an exception handler, you compensate for the steps that have already been committed, in reverse order:
func executeSaga(ctx context.Context) error {
// T1: reserve inventory
if err := reserveInventory(ctx); err != nil {
return err // nothing committed yet, no compensation needed
}
// T2: charge payment
if err := chargePayment(ctx); err != nil {
compensateReserveInventory(ctx) // C1
return err
}
// T3: schedule delivery
if err := scheduleDelivery(ctx); err != nil {
compensateChargePayment(ctx) // C2
compensateReserveInventory(ctx) // C1
return err
}
return nil
}When a step fails, you only roll back what has already succeeded. Each compensation function has to be independently idempotent. Its arguments carry the transaction context ID and a version number, which are used for the idempotency check and for comparison against the state snapshot.
2.2 Building a Saga orchestrator with channels and context
A Saga orchestrator coordinates several local transactions and must support timeouts, cancellations, and rollbacks on error. Go’s channels give you a natural way to decouple the event flow, and context.Context give you one place to manage the lifecycle and propagate cancellation.
type SagaOrchestrator struct {
steps []SagaStep
done chan error
ctx context.Context
cancel context.CancelFunc
}steps: an ordered slice of local transactions; each one has anExecute()and aCompensate()method.done: a write-once channel carrying the final result (nilmeans success).ctx/cancel: external interruption and timeout control, e.g.ctx, cancel = context.WithTimeout(parent, 30*time.Second).
Two guarantees worth calling out. Every Execute call responds to interruption through select { case <-ctx.Done(): ... }. And the compensation chain runs in reverse, synchronously. That last choice is deliberate: a slow rollback is acceptable, an inconsistent one is not.
2.3 Persisting the Saga log and resumable execution
Whether a Saga can recover depends on persisting its log with strong consistency and precisely locating the resume point.
A Saga log entry needs: saga_id (globally unique), step_id (execution order), status (PENDING / COMPENSATED / SUCCESS), payload (the serialized context), and compensate_cmd (the compensation instruction).
For persistence, use a write-ahead log: the state change is written to disk before the transaction proceeds. The backend can vary: MySQL for strong consistency, PostgreSQL with a JSONB column for the payload, or a dedicated event store.
Inserting a log entry in Go with database/sql:
// Insert a Saga execution log entry, including the compensation command
_, err := db.ExecContext(ctx,
`INSERT INTO saga_log (saga_id, step_id, status, payload, compensate_cmd, created_at)
VALUES (?, ?, ?, ?, ?, NOW())`,
sagaID, stepID, "PENDING", jsonPayload, compensateCommand,
)
if err != nil {
return fmt.Errorf("write saga log: %w", err)
}jsonPayload serializes the business context (order ID, amount) so a later step can deserialize it. compensateCommand is a precompiled description of the compensation — a SQL statement or an RPC call — so nothing has to be parsed at runtime. NOW() lets the database generate the timestamp, which avoids ordering anomalies from clock drift between nodes.
2.4 State-machine-driven coordination in Go microservices
Modeling the Saga lifecycle as an explicit state machine improves observability and makes error recovery easier to reason about.
type SagaState string
const (
Pending SagaState = "pending"
Executing SagaState = "executing"
Compensating SagaState = "compensating"
Succeeded SagaState = "succeeded"
Failed SagaState = "failed"
)Pending means not yet triggered. Executing means forward steps are running. Compensating means something failed, and rollback has started. Succeeded and Failed are terminal. Transitions are one-directional, and each one validates its preconditions.
2.5 Load testing and bottleneck analysis in a high-concurrency order flow
The load test used 10K concurrent users. The order path was: create order → deduct inventory → reserve payment → reserve shipment. The orchestrator ran in choreography mode, with services communicating asynchronously over RabbitMQ.
The first bottleneck was the Saga log write:
// Before: flush the saga log synchronously on every step
if err := sagaLogRepo.Save(ctx, step); err != nil { // blocking I/O, ~87ms on average
return err
}A synchronous write per step forced a database fsync every time, and that capped throughput. The step value also carried a JSON-serialized field, which added GC pressure. Switching to batched asynchronous writes — a buffered channel feeding a writer that flushes every 100 records — brought the log-write latency down to 3.2ms.
The second bottleneck was in the compensation path:
The real cause was an undersized consumer-worker pool (default: 5), which allowed compensation messages to pile up. Raising it to 32 and enabling prefetch=10 dropped the p99 compensation latency from 12.4s to 860ms.
Chapter 3: TCC in the Go Ecosystem
3.1 How the three-phase protocol maps onto Go’s concurrency model
TCC (Try-Confirm-Cancel) maps cleanly onto Go’s lightweight scheduling: each phase can be wrapped in its own goroutine, and the phases coordinate through channels.
The Try phase kicks off async resource reservation. It blocks briefly, so it fits a non-blocking go tryOp() launch. Confirm and Cancel are the terminal handling — the main goroutine uses select{} to watch for a timeout or a result on a channel.
type TCCTransaction struct {
ID string
TryCh chan error // unbuffered, so Try runs serialized
DoneCh chan struct{} // signals Confirm is done
}
func (t *TCCTransaction) Confirm() {
select {
case <-t.DoneCh:
return // already done
default:
// run the confirm logic...
close(t.DoneCh)
}
}TryCh is unbuffered, which forces Try operations to execute in order. A closed DoneCh means Confirm completed idempotently, so a duplicate submit is a no-op.
3.2 A generic TCC resource manager with Go generics and interfaces
A TCC resource manager needs type-agnostic lifecycle control. Go generics plus an interface let you model the Try / Confirm / Cancel phases uniformly.
type TCCTransaction[T any] interface {
Try(ctx context.Context, data T) error
Confirm(ctx context.Context, data T) error
Cancel(ctx context.Context, data T) error
}The generic interface constrains the business data type T as a parameter, which keeps the three phases semantically consistent over the same data structure. context.Context handles timeout and cancellation propagation.
Generics remove the uncertainty that comes with runtime type erasure, so the resource manager is both extensible and deterministic.
3.3 Defending against TCC’s three classic problems
TCC in Go runs into three well-known traps: the empty rollback (Cancel is called even though Try never ran), the hung transaction (Cancel arrives late, after Try has timed out), and idempotency (a duplicate instruction corrupts the state). The defenses belong in the framework layer.
A state-machine-driven transaction context handles the first two. Use a TransactionStatus enum (TRYING, CONFIRMING, CANCELLING, COMMITTED, CANCELLED, UNKNOWN) and validate the prior state with an atomic Redis operation.
For idempotency, use a token with a double check:
func (s *TCCService) Cancel(ctx context.Context, req *CancelRequest) error {
// Build the idempotency key from the globally unique txID plus the action type
idempotentKey := fmt.Sprintf("tcc:ido:%s:%s", req.TxID, "cancel")
// The Lua script guarantees the write only happens if the key is absent
script := `
if redis.call('GET', KEYS[1]) == false then
redis.call('SET', KEYS[1], ARGV[1], 'EX', ARGV[2])
return 1
else
return 0
end
`
result, err := s.redis.Eval(ctx, script, []string{idempotentKey}, "processed", "3600").Result()
if err != nil {
return err
}
if result == int64(0) {
return ErrIdempotentRejected // already processed, reject the duplicate Cancel
}
// ... run the business Cancel logic
return nil
}The idempotentKey binds the transaction ID to the action type, which avoids collisions across phases. EX 3600 lets the token expire on its own so it doesn't sit around forever. Running it as a Lua script makes the check-and-set atomic, which closes the race.
Chapter 4: The Local Message Table in Go
4.1 The model and MySQL Binlog coordination
The local message table acts as the on-disk receipt for a transaction: before the main business transaction commits, you insert the pending message. The Binlog then captures that change for downstream consumers. The two stay aligned through transaction atomicity and log ordering.
-- Insert into the message table, in the same transaction as the business operation
INSERT INTO local_message (topic, payload, status)
VALUES ('order.created', '{"id":123,"amount":99.9}', 0);
-- status = 0 means only the Binlog consumer updates it idempotently, never local codeThat statement has to sit in the same BEGIN...COMMIT block as the business UPDATE/INSERT. It leans on InnoDB's ACID properties so the message and the business state are persisted together. Validate the payload against a JSON schema, and use create_time to order later retries.
4.2 Reliable delivery with a worker pool and Redis Stream
To prevent messages from being lost, duplicated, or becoming untraceable under high concurrency, use Redis Streams as the persistent channel and a Go worker pool for controlled-concurrency consumption.
Redis Stream gives you consumer groups, an ACK mechanism, and a pending list for free. The worker pool throttles consumption, which protects you from a thundering herd that would otherwise exhaust Redis connections or trigger an OOM.
// Initialize the worker pool (5 fixed workers)
pool := make(chan func(), 5)
for i := 0; i < cap(pool); i++ {
go func() {
for job := range pool {
job() // process a single message
}
}()
}
// Submit to the pool (non-blocking)
pool <- func() {
if err := processMessage(msgID, payload); err != nil {
redisClient.XClaim(ctx, "mystream", "mygroup", "myworker", 5000*time.Millisecond, msgID).Err()
}
}processMessage wraps the business logic. XClaim reassigns a message that hasn't been ACKed; the 5000ms is the minimum idle time, which gives you at-least-once delivery. cap(pool) caps the number of concurrent consumers — tune it together with the Redis connection pool size.
4.3 Sharding the message table with GORM and ShardingSphere-Go
The message table is under three kinds of pressure — heavy writes, time-ordered queries, and long-term retention — so the sharding design has to balance routing efficiency against transaction consistency.
For the shard key, prefer user_id (the high-frequency query dimension) or msg_id (globally unique, snowflake-compatible). Avoid created_at, which skews the data.
db, err := gorm.Open(mysql.Open("shardingsphere://"), &gorm.Config{
Dialector: sharding.NewShardingDialector(
sharding.WithProps(map[string]string{
"mode.type": "Standalone", // in-memory mode for quick validation
"rules[0].tables.t_message.actualDataNodes": "ds_${0..1}.t_message_${0..3}",
"rules[0].tables.t_message.tableStrategy.standard.shardingColumn": "user_id",
"rules[0].tables.t_message.tableStrategy.standard.shardingAlgorithmName": "t_message_table_inline",
}),
),
})This config says: shard t_message by user_id modulo 4 into four physical tables (t_message_0 through t_message_3), spread evenly across the two logical databases ds_0 and ds_1. shardingAlgorithmName points at the inline algorithm defined in YAML, which runs user_id % 4 for routing.
4.4 End-to-end latency and TPS in a flash-sale scenario
The local message table keeps the initial state consistent through a same-database “write business + write message” transaction. The consumer side polls, pulls, and delivers idempotently. It avoids distributed-transaction overhead, at the cost of some added latency.
// Consumer: idempotency check, then the real deduction
func handleSeckill(ctx context.Context, msg SeckillMessage) error {
// Use the message ID as the idempotency key, 10-minute TTL
ok, err := rdb.SetNX(ctx, "msg:"+msg.ID, "1", 10*time.Minute).Result()
if err != nil {
return err
}
if !ok {
return nil // already processed, skip
}
return seckillSvc.Execute(ctx, msg.OrderID) // the real deduction
}SetNX deduplicates using the message ID as the key; the 10-minute TTL keeps a stale lock from lingering. Execute still runs its own eventual-consistency check internally — comparing against an inventory snapshot — so it never oversells.
Chapter 5: A Unified Evaluation and a Decision Framework
After validating prototypes for three core modules (a microservice gateway, real-time log aggregation, and high-concurrency order fulfillment), the team reached the decision stage. We built an evaluation matrix covering four dimensions: performance, maintainability, ecosystem fit, and team capability. Three Go-based candidate architectures went head-to-head: Plan A (standard Gin + GORM + Redis), Plan B (the ZeroRPC microservice framework + Ent ORM), and Plan C (eBPF-assisted observability + fx dependency injection + the pgx native driver).
We calibrated the weights with the Analytic Hierarchy Process: performance 35%, maintainability 25%, ecosystem maturity 20%, and team ramp-up cost 20%. Each dimension had measurable sub-metrics — the performance dimension, for example, included p99 latency (ms), single-node throughput (QPS), and cold-start time (ms); maintainability included unit-test coverage, average CI build time, and the typical bug-fix turnaround (hours).
A few technical debts surfaced. In the order-fulfillment scenario, Plan A hit the GORM preload N+1 problem, and the measured p99 latency jumped to 89ms. Plan B’s ZeroRPC default serializer panicked on overlong log fields, and we had to swap it for gogoproto. Plan C's fx initialization chain ran too deep; reloading config at runtime leaked goroutines, which we fixed by taking explicit lifecycle control with fx.Decorate.
To validate the team-capability mapping, we ran a three-day “Go architecture workshop” in which every backend engineer implemented the same order state machine endpoint across all three plans. The averages: Plan A took 2.1 hours, Plan B took 3.4 hours, and Plan C took 6.7 hours. But Plan C’s ability to automatically pinpoint a goroutine blocking point after a stress test won everyone over.
The constraints the chosen plan had to satisfy:
- Compatible with the existing Kubernetes 1.24+ environment and the Istio 1.17 service mesh.
- Every component must ship an official ARM64 image (verified for pgx, Ent, and zerolog).
- The log format must follow OpenTelemetry Protocol v1.3.0.
- No CGO (which rules out cgo-enabled sqlite, OpenSSL bindings, and the like).
The weighted scores were Plan A: 76.4, Plan B: 84.2, and Plan C: 82.9. Plan C had one advantage the others couldn’t match: on SLO assurance, its eBPF trace tooling could locate a slow-SQL source within 3 seconds, while Plan B’s application-layer instrumentation averaged 17 seconds. So production runs Plan B as the trunk architecture, with Plan C’s eBPF probe module embedded as a sidecar in every critical service.
That decision has been running in the Canary cluster for 14 days. It has intercepted an average of 3.2 potential P0 incidents per day. Two of those were automatic scale-outs triggered just before the database connection pool would have been exhausted.
