Amazon MemoryDB: Enhancing Data Consistency and Durability with a Memory-First Approach

Amazon MemoryDB: A Fast and Durable Memory-First Cloud MemoryDB

developed based on Redis, primarily aims to address the shortcomings of Redis in terms of data consistency and persistence.

From a product positioning perspective, MemoryDB is positioned as a database rather than a cache. This is evident from its vocabulary and its focus on high-throughput write scenarios such as IoT, finance, and advertising, which are sensitive to latency and demand millions of writes per second.

The paper's core lies in how it addresses the persistence and consistency issues of Redis within the community through a distributed log system. The design and implementation insights provided can serve as valuable references for those seeking to achieve similar functionalities.

Overall Design

MemoryDB was developed by Amazon in 2021. As the Redis code was not yet adjusted to a non-open-source license at the time, the design primarily focused on minimizing changes to the Redis code to avoid the difficulty of merging upstream code. From the following design diagram, it can be observed that the main modifications to MemoryDB are in the master-slave replication part: data synchronization no longer occurs by directly synchronizing changes to the slave after writing to the master, but rather by first writing to a distributed transaction log system, which is then pulled by the slave to apply these changes.

The write and sync process is as follows:

1. Changes to keys are written to a tracking buffer (tracker), and write to clients is blocked. Note that these write changes are not visible until they receive an ACK from the distributed log system.
2. Submit change logs to the distributed transaction log system across multiple AZs.
3. After successful logging, the master executes changes from the tracking buffer and modifies them to be visible to clients, unblocking writing to clients.
4. Slaves actively pull and replay changes to maintain eventual consistency between master and slave data.

Returning to MemoryDB's design goal: data strong consistency and reliable persistence. How is this achieved through this design?

The reliable persistence mechanism of data primarily relies on a strongly consistent multi-AZ distributed transaction log service to ensure that write requests must wait for the distributed transaction log service to commit before returning, thereby ensuring that once committed, data will not be lost. Moreover, during master-slave switching, only the slave that has caught up with the latest data can be selected as the new master. Consistency depends on whether reading from the slave is allowed:

• Reading only from the master ensures sequential strong consistency.
• Allowing reads from the slave can only guarantee eventual consistency since slave updates are asynchronous.

The reason why reading only from the master can achieve sequential strong consistency is primarily because Redis command processing is single-threaded, naturally ensuring the sequentiality of writing to the distributed transaction log. Additionally, MemoryDB also controls and optimizes command read and write, where changes need to wait for successful submission before becoming visible. Similarly, read requests for keys that have not been committed also wait until after submission to return, ensuring linear strong consistency between reads and writes.

This is something Redis cannot achieve primarily because Redis lacks an MVCC mechanism, and changes take effect immediately, allowing other clients to see changes before they are officially ACK'd.

An example is as follows:

1. Client A sends a SET A 2 request and waits for all slaves to return via the WAIT command.
2. Client B sends a GET A request while the WAIT has not returned. However, Client B sees the latest value as 2.

This means Client B can see updates from Client A that have not been confirmed, as MemoryDB places changes into a buffer (tracker) before writing to the distributed transaction log system, so other clients must wait for completion before returning when reading keys that are being written.

Allowing multiple slaves can only achieve eventual consistency, which is easier to understand because master-slave replication is through asynchronous log synchronization, so slave data is a snapshot at a certain time window rather than the latest data.

Additionally, the paper mentions a fine detail: query requests, are executed immediately but must check the write buffer (tracker) before returning. If a key being read is currently being written, the response is delayed until after the write is successful. For example, if Client A executes SET A 1 but is not completed, and Client B requests Get B, it will return immediately. However, if requesting Get A, it needs to wait until Client A's write is successful before returning.

On-mutating operations can be executed immediately but must consult the tracker to determine if their results must also be delayed until a particular log write completes. Hazards are detected at the key level.

Data Recovery Mechanism

From the design, it can be seen that master-slave synchronization is achieved through a distributed transaction log system. Therefore, without other auxiliary means, instances need to replay historical change logs in full when restarting, which is evidently unacceptable in terms of time and availability. MemoryDB implements periodic creation of Snapshots to reduce the volume of logs that need to be replayed. This can also be understood as RDB + incremental AOF:

This process is handled by an external control plane service (off-box), which decides whether to create a new Snapshot (RDB) and upload it to S3 based on timing and status. The Snapshot records the offset of the distributed transaction log system. The data recovery process involves loading the Snapshot and then replaying incremental data from this offset.

Performance Comparison

In terms of throughput, in read-only pressure testing scenarios, MemoryDB performs better than community Redis on machine types above 2xlarge. MemoryDB's peak OPS can reach 500K Op/s, while community Redis is approximately 330K Op/s. In write-only pressure testing scenarios, community Redis's performance is approximately twice that of MemoryDB. The main reason is that MemoryDB requires each write to be committed to the distributed transaction log system, resulting in slightly higher latency.

In terms of latency, MemoryDB and community Redis are close in P50/P99 in read-only latency (Figure a). For write-only scenarios, MemoryDB's P99 is approximately 6ms, while community Redis is around 3ms. In mixed read-write scenarios, community Redis also has significantly lower latency than MemoryDB, suggesting that MemoryDB's performance may be slightly worse in mixed read-write scenarios.

Additionally, the paper also mentions the impact of BGSAVE (generating RDB files) on latency. It can be observed that BGSAVE has almost no impact on average latency, but there are noticeable spikes in P100 latency. The main reason is that BGSAVE requires forking a new process, during which the kernel needs to copy memory page mappings, taking approximately 12ms per GiB. Moreover, as the red line rises (increased SWAP usage due to COW), both throughput and latency deteriorate. However, in actual production environments, it is recommended to disable SWAP, so these observations may not have significant practical value.

Conclusion

In addition, the paper also mentions adjustments in the design of Slot migration and leader selection. Leader selection no longer relies on the Cluster Bus for detection but rather uses a Raft Lease-like approach, initiating leader selection if no heartbeat logs from the leader are received within a certain period. The Slot migration process is similar to master-slave replication, where all keys corresponding to the Slot are serialized and sent to the target node, followed by incremental synchronization of newly written parts.

Overall, MemoryDB's approach enhances Redis's reliability in data consistency and persistence through a distributed transaction log system while sacrificing some performance. Both in design and implementation, it provides valuable insights and is worthy of reference.