AlloyDB For PostgreSQL- Google Cloud

What is new?

Fully managed
PostgreSQL-compatible database
Enterprise-grade transactional and analytical workloads
Elastic storage and compute
Intelligent caching, and AI/ML-powered management
4x faster on transactional workloads, and up to 100x faster analytical queries than standard PostgreSQL.

Disaggregation of compute and storage

AlloyDB for PostgreSQL was built on the fundamental principle of disaggregation of compute and storage.
Designed to leverage disaggregation at every layer of the stack.
AlloyDB to offload many database operations to the storage layer through the use of a log-processing system.

AlloyDB design overview

The AlloyDB storage layer is a distributed system comprised of three major parts:

A low-latency, regional log storage service for very fast write-ahead log (WAL) writing
A log processing service (LPS) that processes these WAL records and produces “materialized” database blocks
Failure-tolerant, sharded, regional block storage that ensures durability even in case of zonal storage failures.

Log Processing Service (LPS)

The primary database instance (of which there is only one) persists WAL log entries, reflecting database modification operations (such as INSERT/DELETE/UPDATE) to the low-latency regional log store.
the log processing service (LPS) consumes these WAL records for further processing. Because the log processing service is fully aware of the semantics of the PostgreSQL WAL records and the PostgreSQL storage format,
it can continuously replay the modification operations described by these WAL records and materialize up-to-data database blocks to a sharded, regional storage system.
these blocks can then either be served back to the primary database instance (in the case of a restart or simply when a block falls out of cache) or to any number of replica instances that might be in any of the zones within the region where the storage service operates.
To keep the local caches of the replica instances up-to-date, AlloyDB also streams WAL records from the primary to the replica instances to notify them about recent changes.

Benefit OF LPS

Full compute/storage disaggregation even within the storage layer. LPS can scale out based on workload patterns, and transparently add more compute resources to process logs when needed to avoid hotspots. Since the log processors are purely compute-attached to a shared regional storage, they can flexibly scale out/in without needing to copy any data.
Storage-layer replication: By synchronously replicating all blocks across multiple zones, the storage layer automatically protects the system from zonal failures without any impact on or modifications to the database layer.
Efficient IO paths / no full-page writes: For update operations, the compute layer only communicates the WAL records to the storage layer, which is continuously replaying them. In this design, there is no need to checkpoint the database layer, or any reason to send complete database blocks to the storage layer (e.g., to safeguard against the torn pages problem). This allows the database layer to focus on query processing tasks, and allows the network between the database and storage layer to be used efficiently.
Low-latency WAL writing: The use of low-latency, regional log storage allows the system to quickly flush WAL log records in case of a transaction commit operation. As a result, transaction commit is a very fast operation and the system achieves high transaction throughput even in times of peak load.
Fast creation of read replica instances: Since the storage service can serve any block in any zone, any number of read replica instances from the database layer can attach to the storage service and process queries without needing a “private” copy of the database. The creation of a read replica instance is very fast as data can be incrementally loaded from the storage layer on demand — there’s no need to stream a complete copy of the database to a replica instance before starting query processing.
Fast restart recovery: Since the log processing service continuously replays WAL log records during online operations, the amount of write-ahead log that needs to be processed during restart recovery is minimal. As a consequence, system restarts are accelerated significantly (because WAL-related recovery work is kept to a minimum).
Storage-layer backups: Backup operations are completely handled by the storage service, and do not impact the performance and resources of the database layer.

Life of a write operation

A SQL INSERT statement over the client’s TCP connection to the primary instance on the database layer.
The primary instance processes the statement (updating its data and index structures in-memory) and prepares a WAL log record that captures the semantics of the update operation.
Upon transaction commit, this log record is first synchronously saved to the low-latency regional log storage, and then asynchronously picked up by the log processing service in the next step.
Storage layer is intentionally decomposed into separate components, optimizing for the separate tasks performed by the storage layer — log storage, log processing, and block storage
To reduce transaction commit latency, it is important to durably store the log records as fast as possible and achieve transaction durability.
To ensure regional durability for the materialized blocks, multiple instances of the log processing service (LPS) run in each of the zones of the region

Life of a read operation

Reads begin with a SQL query that’s sent to a database server; this can either be the primary instance or one of the (potentially many) replica instances used for read-only query processing.
The database server performs the same query-parsing, planning and processing as a conventional PostgreSQL system. If all the required blocks are present in its memory-resident buffer cache, there’s no need for the database to interact with the storage layer at all.
To allow for very fast query processing, even in cases where the working set does not fit into the buffer cache, AlloyDB integrates an ultra-fast block cache directly into the database layer. This cache extends the capacity of the buffer cache significantly, thereby further accelerating the system in those cases.
If a block is missing in both of the caches, a corresponding block fetch request is sent to the storage layer.
The use of a specific LSN here ensures that the database server always sees a consistent state during query processing. This is particularly important when evicting blocks from the PostgreSQL buffer cache and subsequently reloading them, or when traversing complex, multi-block index structures like B-trees that might be (structurally) modified concurrently.
the log processing service is also responsible for serving the block fetch requests.
Every LPS has its own instance of the PostgreSQL buffer cache — if the requested block is already in the LPS buffer cache, it can be returned to the database layer immediately without any I/O operation. If the requested block is not present in the cache, the LPS retrieves the block from the sharded, regional storage and sends it back to the database layer.

Article By:

“Rajendra Pachouri Hands-on Software Engineering, API, Cloud, IOT & DevOps experience, and a proven track record of architecting and delivering reliable and scalable systems in a variety of areas such as e-commerce, payments, security, home automation, financial systems, messaging services, EDI, EAI Microservices”.