We recently switched to version 7 (v7) uuid primary keys and saw significantly faster inserts for some tables.

The databases were running Postgres 18.4 and mostly used v1 with some v4 uuid values for primary keys.

Changing the column default involved running a single alter table command, but did require an exclusive lock on the table, blocking everything including selects.

To solve that, we used a short lock timeout and lots of retries.

The biggest speedup was 23x faster average execution time for a multi-row insert query called 12,000 times per minute on a table with billions of rows.

History and trade-offs with UUIDs

The system uses UUID primary keys throughout. I typically recommend starting with bigint and sequences over UUID v4 primary keys, although here uuid v1 was used. Insert performance is not as bad for v1 compared with v4.

Still though, v7 brings better performance than both for inserts and can also result in smaller indexes with fewer page splits meaning less CPU and IO.

What drives bad performance for v4 and to a lesser extent v1? Let’s do a quick refresher. As new table rows are inserted and a primary key is defined, primary key values are maintained in sorted order in a b-tree index. Just like table rows, index entries in Postgres are stored in fixed size 8kb pages.

Postgres needs to know in which page to place the new index entry. For sorted order, the first bytes of new uuid values are compared.

For v4 given new values are very random and not monotonically increasing (they lack “monotonicity”), values can be earlier or later, meaning they’re unlikely to be placed into the same recently accessed page. This is bad for caching!

When new values are monotonically increasing, the recently accessed page is “hot” in the Postgres buffer cache (in memory copy of the on-disk page).

When Postgres is not able to use the hot index page for the newly inserted value, that page could be outside the buffer cache, not in the OS cache, and ultimately result in a much slower disk read which increases latency.

Besides the worse insert performance, since v4 values are scattered to more pages, this means there are more “page splits” when new inserts are attempted in full pages. Page splits cause more latency from increase WAL and IO.

We experimented and benchmarked with v1, v4, and v7 uuid formats and we leveraged the research and write-ups from external sources like the ones below.

1. How Sequential UUIDv7 Boosts Ingestion Performance
2. Simplicity and power of UUID v7
3. PostgreSQL UUID Performance: Benchmarking Random (v4) and Time-based (v7) UUIDs

Benchmarks are great, but what kind of real world results did we see?

What kinds of improvements did we see?

We decided to make this the new default unless v4 was needed for more randomness. After all qualified tables were changed, I began going through insert queries for each changed table. For many of the tables, there wasn’t an obvious change.

However, for a handful we saw an immediate and significant improvement. I picked 5 with speedups of 6x, 8x, 9x, 20x, and 23x.

The PgAnalyze graphs for the 23x, 9x, and 6x queries are shown below.

Showing PgAnalyze insert query graphs for tables A, B, C:
Table A - 23x reduction. 0.7ms to 0.03ms, 12000 calls/min

Table B - 9x reduction. 0.6ms to 0.07ms, 2000 calls/min

Table C - 6x reduction. 0.50ms to 0.08ms, 9500 calls/min

Now that we’ve seen the results, let’s talk about how this was done and the challenges.

Auditing where the UUIDs came from

The UUID values came from various sources:

• The uuid_generate_v1() function from the uuid-ossp module
• The function gen_random_uuid() added in Postgres 13 that generates v4 UUIDs natively
• UUID v4 values sent by a client application, which meant the column default function was not used

We replaced most of these with the uuidv7() function in Postgres 18. To do that, we needed to run a single alter table ... alter column statement per table.

The statement ran fast, so no problem, right?

How did the switch go?

One wrinkle we found was that modifying the column default while fast, required an access exclusive lock.

This lock type conflicts with every read and write operation including regular select statements.

For our highest queried tables, they’re queried constantly, so this was a problem. There was almost never a “window” to perform this operation, and we didn’t want to take downtime for this switch.

While heavily queried tables were a challenge, infrequently queried tables did not pose a problem for this alter table statement at all.

For those, we could use our migrations framework (Active Record in Ruby on Rails) and perform the alter table using a regular old migration.

For those, we did add some safeguards, by creating an explicit transaction and using set local to control timeout values. We’d set short timeouts for the alter table to give up quickly if it didn’t work or ran too long.

From psql:

BEGIN;

SET LOCAL lock_timeout = '50ms';
SET LOCAL statement_timeout = '100ms';

ALTER TABLE my_table ALTER COLUMN id SET DEFAULT uuidv7();

COMMIT;

For the higher activity tables, we’d need some retries. We’d use a manual psql session:

SET lock_timeout = '50ms';
SET statement_timeout = '100ms';
ALTER TABLE my_table ALTER COLUMN id SET DEFAULT uuidv7();

The alter table would commit if it grabbed the lock within 50ms, or we’d get an error that the lock_timeout was reached.

The benefit of the manual approach was we could retry until successful and backfill a Rails migration to keep everything in sync. A more sophisticated solution might have automated retries within Ruby.

However, for our most heavily queried tables, we wanted even more control over the retries.

How did we do that?

Bringing in the big retry machinery

Sometimes one or two retries would do the job. Great, we’d move on.

However, for our most heavily queried table that didn’t work.

What ended up working was using the same strategy of retries, but just adding more sophistication with looping and backoffs.

Claude helped me cook up the PL/pgSQL looping retry function below, I did some testing and was ready to try it. It has these features:

• Try up to 50 times (max attempts is configurable)
• Add a pause in between retries, with a jittered backoff of 50-250ms

From psql:

SET statement_timeout = 0;
SET lock_timeout = '100ms';

DO $$
DECLARE
  attempt      INT := 0;
  max_attempts INT := 50;
BEGIN
  LOOP
    attempt := attempt + 1;
      BEGIN
        EXECUTE 'ALTER TABLE my_table ALTER COLUMN id SET DEFAULT uuidv7()';
        RAISE NOTICE 'Succeeded on attempt %', attempt;
        EXIT;
      EXCEPTION WHEN lock_not_available THEN
        IF attempt >= max_attempts THEN
          RAISE EXCEPTION 'Failed to acquire lock after % attempts', attempt;
        END IF;
      PERFORM pg_sleep(0.05 + random() * 0.2);  -- jittered backoff, 50-250ms
    END;
  END LOOP;
END $$;

By using the function above, we were able to find a small window to perform the alter table after several dozen quick retries!

Actively cancelling lock-holding queries

In cases where even many retries won’t work, and we don’t want downtime, we may be left with needing to actively monitor lock holder queries and to cancel them (assuming that’s ok).

Thanks to Ants Aasma from the community PostgreSQL Slack for this idea.

We didn’t end up needing to do this, but here was my prep for this. It’s still useful to review lock holder queries.

First we’d inspect live queries:

SELECT
  pid, state, left(query,100), xact_start, state_change,
  age(clock_timestamp(), xact_start) AS tx_duration
FROM
  pg_stat_activity
WHERE
  state != 'idle'
ORDER BY xact_start;

And identify queries holding locks:

SELECT
  blocked_locks.pid AS blocked_pid,
  blocking_locks.pid AS blocking_pid,
  blocked_activity.query AS blocked_statement,
  blocking_activity.query AS current_statement_in_blocking_process
FROM
  pg_catalog.pg_locks blocked_locks
JOIN
  pg_catalog.pg_stat_activity blocked_activity
  ON blocked_activity.pid = blocked_locks.pid
JOIN pg_catalog.pg_locks blocking_locks
  ON blocking_locks.locktype = blocked_locks.locktype
  AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation
  AND blocking_locks.pid != blocked_locks.pid
JOIN pg_catalog.pg_stat_activity blocking_activity
  ON blocking_activity.pid = blocking_locks.pid
WHERE NOT blocked_locks.granted;

If we find them, we could cancel them to create a window to run our alter table. That could mean bad user experience so you’d need to figure that out for your database.

SELECT pg_cancel_backend(blocking_pid);

We’d likely want to stack up our alter table operation to occur immediately after. Fortunately we didn’t end up needing to do this, but I’d be interested to hear the stories from others with heavily queried databases.

Downsides of uuid v7

Since uuid v7 values use a timestamp in their first bits, this timestamp can be easily decoded. This can be viewed as “leaking” or exposing the creation time of the record via that timestamp, which could be a downside for your database. You’ll have to decide that. v4 UUIDs do not expose the creation time.

Wrap Up

We found some significant speedups for insert queries after switching to uuidv7() primary keys, for a relatively low effort change. A nice ROI.

The only wrinkle was the exclusive lock alter table ... alter column required, but we solved that with short lock related timeouts and many retries.

Although this didn’t benefit 100% of our tables, the gains for some were significant and uuid v7 has become our new default choice for uuid primary keys.

Thanks to the Postgres core team for creating this new capability within Postgres. The availability in core made it possible to adopt on AWS RDS which supports a limited amount of extensions.

Thanks for reading, and until next time.

On Christmas Day 2024, Postgres infrastructure powering the Aura Frames API had problems under peak load, being unavailable for three hours and disrupting the experience for new customers. The team knew it would need improvements to handle the surge for Christmas 2025 and beyond.

One year later, much of the resource intensive data access was reworked, the Postgres infrastructure was upsized, and this approach not only survived, but thrived, providing reliable service through the holiday season.

The sum of Transactions Per Second (TPS) across the DBs peaked at 226,000, with more than 100K TPS sustained for 10 hours and repeating on multiple days after Christmas, with an average query time of 25 microseconds.

The improved reliability meant customers could smoothly set up new frames and add photos, and they did it more than ever, with the Aura Frames app reaching #1 in U.S. and Canadian Apple and Android App Stores on Christmas Day.

In this post we’ll look back at the months of planning and execution that went into achieving that outcome!

A second post in this series will dig into the Ruby on Rails side, while this one will focus on Postgres.

What’s Aura Frames?

Aura Frames (Aura Home, Inc.) is the company behind modern, high-quality, Wi-Fi connected digital photo frames that customers love.

The frames are easy to use via free iOS and Android apps, don’t require a subscription, and offer unlimited cloud storage for photos and videos. Once set up, family members can be invited to contribute photos and videos via the app from anywhere. Typically Aura frames have an average of 4 contributors adding content.

In 2025, more than 1 billion photos were shared to Aura frames globally.

While public engineering blog posts are limited, Aura was featured on the AWS Storage Blog in the past. Link: How Aura improves database performance using Amazon S3 Express One Zone for caching.

Disclosures

I began working with Aura in 2025. Aura does not have a public engineering blog, so we discussed me writing a post here where I regularly write about Postgres, Ruby on Rails, and scaling databases.

This post was written by me and I do not speak for the company. The company had the opportunity to review and make minor edits before publication.

The Christmas Day outage was a painful reality of scaling fast, and I appreciate Aura’s willingness to discuss it here.

I’m biased, but from my view the company is dedicated to continually improving the customer experience, in part with strategic investments in technical infrastructure.

With that covered, let’s take a look at how the frames are used and what drives the traffic.

What causes the sharp increase in traffic?

On Christmas Day, millions of customers set up hundreds of thousands of new Aura frames. The backend platform needs to work well for both existing customers and handle the load from new customer activity. For new customers it’s especially critical they have a good experience from their first moments with the product.

While the holiday timing is predictable, the rate of new frames and new photos added each year increases, adding a new amount of pressure to infrastructure components. Postgres is not easily horizontally scalable, and is costly to operate.

The average amount of increased peak TPS for all DBs on Christmas Day was ~4.5x, with the biggest being ~18x the normal value. To meet this demand, advanced financial planning and vertical scaling were needed. Resources were all shrunk back down after to save on costs.

Scaling over the years

The team has executed a variety of scaling tactics over the last half decade by employees and in conjunction with Postgres consultants. Scaling efforts often focused on reducing pressure on Postgres within the constraints of a single primary instance, while preserving its operational simplicity. (See: Squeeze the hell out of the system you have for a similar philosophy).

Scaling is more straightforward on the stateless, HTTP side. Aura uses AWS and has leveraged Auto Scaling Groups (ASGs), which can scale up to thousands of EC2 instances running the web application stack, image processing, PgBouncer, and other services.

For Postgres, vertical scaling of a single primary instance was leveraged as long as possible.

Here’s a look at the primary database instance at peak for Christmas Day 2024. Note that the db.r6g.48xlarge instance was the largest instance available for RDS.

Without a larger instance to move to, the team could not rely on vertical scaling for Christmas 2025 and beyond.

⚠️ For Christmas 2024, the team had a Dedicated Log Volume (DLV) in place for replication from the main instance. Replication management was one of the main challenges. A DLV reduces latency and improves reliability for replication.

Quoting from AWS Docs on Using a Dedicated Log Volume (DLV):

A DLV moves PostgreSQL database transaction logs … to a storage volume that’s separate from the volume containing the database tables.

While replication lag is “normal” (asynchronous streaming replication) and varies due to write pressure, vacuum activity, and more, performance on the primary had not previously been affected by replication lag before.

Unfortunately that changed during peak load on Christmas 2024. More details are below in the Christmas 2024 Retrospective section.

Before diving into that, let’s briefly cover some of the generic challenges of reaching the scalability limits of a single instance.

Postgres Scaling Challenges and Solutions

The use of Postgres at Aura faced all kinds of common Postgres scaling challenges.

• Insert latency. To help reduce latency, foreign key constraints are not used. Indexes on high write tables are minimized. Indexes are periodically rebuilt (reindex concurrently).
• Replication. The product needs read-after-write behavior and can have high replication lag, so read replicas have historically not been used for read queries. Read queries run on the primary.
• Buffer cache and high cache hit rates. It’s critical to have page and index content for key queries in the buffer cache to achieve sub-millisecond query durations. Buffer cache memory access also reduces storage device IOPS.
• IOPS spikes that exceed the max Provisioned IOPS are problematic, resulting in queuing and high latency.
• The team faced CPU spikes in the single primary configuration, during vacuum, reproduced in load testing. High query latency across the DB would follow.
• During peak load periods, the system needs to handle tens of thousands of client connections from the application.
• The application tracks per-user counts that constantly change. For example, social media-style likes, comments, and activity feeds. These counts are stored in Memcached when possible, with connections managed by HAProxy.
• Autovacuum triggered vacuums during busy periods are disruptive. To minimize disruption, Autovacuum is throttled to run slower (autovacuum_vacuum_cost_limit, autovacuum_vacuum_cost_delay). Tables with heavy dead tuple growth are vacuumed manually in a low activity period overnight, not by Autovacuum.
• Index bloat. The database uses a primary key data type that isn’t 100% ideal for minimizing bloat. Indexes are periodically rebuilt, but that process adds a lot of IOPS pressure so the timing needs coordination and PIOPS need to be upsized.
• Configuration complexity. Postgres parameters (GUCs) are modified beyond what RDS provides sparingly. One exception is Autovacuum parameters which are monitored and adjusted often to help control spikes in IOPS.
• Background work state and queue style data could be managed separately. The team had previously created a separate Postgres database to manage the state of background work.

Christmas 2024 Retrospective

Unfortunately on Christmas Day 2024, the team, platform, and customers faced a significant outage. A root cause analysis revealed that the main contributor was running out of space on the DLV, due to the growth of write-ahead logs (WAL) filling it up.

The team had provisioned a Dedicated Log Volume (DLV) offering lower latency as a dedicated volume for WAL log storage.

The team traced the root cause back to a change introduced in RDS Postgres 14.1 (the prior year ran Postgres 13.x which used S3 for WAL archival), which began using replication slots for replication.

“We weren’t aware RDS had changed in-region replication to use replication slots by default in Postgres 14.1. This caused the amount of WAL stored on the primary to be unbounded when the replica lagged.”

(AWS source: RDS “Monitoring replication slots for your RDS for PostgreSQL DB instance”) “RDS for PostgreSQL 14.1 and higher versions use replication slots for in-Region read replicas.”

The Postgres docs describe the benefits of replication slots: “Replication slots provide an automated way to ensure that the primary server does not remove WAL segments until they have been received by all standbys” in 26.2.6. Replication Slots.

The trade-off can be severe, as there’s the possibility of unbounded slot growth for inactive (or heavily lagging) slots, resulting in pg_wal filling the storage volume and causing Postgres to shut down. There’s a big “Caution” about this in the documentation, which looks new since Postgres version 17.

This is documented in community Postgres under Disk Full Failure (Docs for Postgres 16.x) or under “No space left on device” ENOSPC on the wiki.

The server will crash and run crash recovery.

One way to limit the growth is to set max_slot_wal_keep_size (Postgres replication docs) (new in 13, default value is -1 which means uncapped). wal_keep_size, max_slot_wal_keep_size, and max_wal_size were all updated going forward. Why set these? In a worst case, the replica would become unusable but not cause the primary to shut down.

With the replication issue solved going forward, the team faced a second issue, problematic occasional CPU spikes during vacuum. The issue was reproducible under load testing. A variety of theories were explored, evidence collected within the constraints of RDS, but no very strong causes and sources of evidence were discovered.

RDS Postgres limits access to the underlying host OS making it impossible to directly use Linux profiling tools like perf (as compared with something like running community Postgres on EC2).

Given this experience and the unsolved issue, the team began exploring moving to multiple Postgres instances, beyond a single instance. This would bring new complexity, but expanded capacity.

Proofs of concept for sharding from the application were explored. The team preferred mature solutions and a high degree of owner-operator control, thus a custom solution with Ruby on Rails framework capabilities had the lowest friction.

Ruby on Rails Horizontal Sharding was partially built out and could have been a viable solution, but ultimately was not chosen.

With more than half of 2025 gone, Christmas Day 2025 was looming and daunting. An additional constraint on possible solutions was what could be built and supported in a matter of a few months by a small team.

The clock was ticking!

Postgres Christmas 2025

The solution that seemed to fit the best would be a custom solution, rewriting a lot of the application query layer, taking direct control of key queries, high call volume, on big tables, and distributing the work to more instances.

To prepare, the top 10 tables by write operations and size were analyzed. All queries for those tables would need to be analyzed for incompatible elements that don’t work across a database boundary, like joins and some subqueries.

Ultimately the 10 tables were distributed to 7 new primary instances (making 8 in total), some DBs with as few as 1 table. All reads and writes continued to flow from the same Ruby on Rails codebase, not new microservices. To achieve that we’d use Active Record Multiple Databases support. That meant that each primary database would get the full accoutrement, including its own named config, the option of a read replica, and the ability to manage schema definition DDL changes (Rails “Migrations”). The production configuration would be mirrored in all lower environments so that the extensive unit test suite would run across all 8 databases. The only difference in the development environment was the 8 databases ran on one Docker Postgres container.

With the plan in place, it was time to start coding! We got started in earnest around August 2025 with 3 months to execute and validate the plan ahead of Christmas.

With each instance dedicated to one or a couple of tables, there was much more CPU, Memory, and IOPS available in total. This allowed each of the instances to be over provisioned temporarily before Christmas, adding headroom, availability, and reliability.

We determined the query workload for the biggest table by size, row count, call frequency and % of IO would still fit ok on a single big instance, without needing to shard the table rows.

Here’s what the instances were scaled up to for Christmas 2025. (Older generation: Graviton2 ARM and DDR4 memory)

With ~4.7x more CPU, Memory, and more IOPS, the expanded capacity provided plenty of power through Christmas! We’ll look at how we scaled down to reduce costs after Christmas in an upcoming section.

Let’s look at the sharding strategy in more detail.

Workload-driven “Whole table sharding”

We ended up using the term “whole table sharding” and the tables picked tended to have the most writes, the most rows, and be the most challenging to vacuum quickly or rebuild indexes for.

We were able to gradually modify all the application queries and get everything rolled out where it was backwards compatible, then we could cut over.

To transition the row data, wanting to initially replicate it, we tried using AWS pglogical and logical replication directly but the initial replication was too slow.

We may revisit that in the future, however we ultimately decided on physical replication which copied the whole instance, before cutting over to the new one. While more wasteful initially, we knew we could operate that approach reliably and with a minimal amount of downtime.

The major downside of this approach was that we had to repeat it 7 times, duplicating the entire database, consuming a ton of extra space temporarily.

We decided the trade-off was worth it; we could re-provision new instances and reduce space after Christmas by using the AWS Blue/Green deployments. More on that later.

Let’s look at some metrics from Christmas 2025.

Postgres and Infra Metrics Christmas 2025

All Postgres instances were upgraded to 17.6 in the Fall of 2025. TPS measured by Odarix. PgBouncer, Memcached, HAProxy metrics from CloudWatch. Query and schema details from PgAnalyze.

Main DB 133K TPS Peak Christmas Day Odarix Screenshot

Traffic grows in the week before Christmas, but on Christmas day (December 25) it really takes a sharp upward trajectory. The peak load period lasts for more than 10 hours on Christmas Day, with the main DB receiving more than 100K TPS from 10:00 to 20:00 US Central Time.

Employees noticed the free Aura Frames iOS and Android apps moving up in the App Store ranks. Excitement built into the evening as the Aura Frames app moved into the top 10, top 5, and eventually reached the #1 rank. 🎉 I grabbed the screenshot below at around 11:30 PM CT December 25.

Screenshot showing the Aura Frames app at the #1 rank in the U.S. Apple App Store

Switchover to New DBs

Let’s look at how the actual switchover happened. To switch over to new server instances, effectively “relocating” the tables, it was critical to not lose any write operations and to minimize user-facing downtime.

The steps were roughly as follows:

Switchover preparation steps:

1. For the primary database, create a new replica. It will need as much allocated space as the primary instance. Use physical replication.
2. Set up AWS SSM parameters for the new database to be used by Ruby on Rails and PgBouncer.
3. Create a new PgBouncer Auto Scaling Group (ASG) for the new DB. Set the SSM parameter to the network load balancer endpoint.
4. Route application traffic through the new PgBouncer but have it continue to point at the original DB via an environment variable. Changing this param would be the sole change in the brief downtime period.

Switchover steps:

1. Bring all PgBouncer instances down (set ASG desired capacity to 0). No connectivity to DB now, no writes, fully down.
2. Wait for replication lag to reach zero. Promote the read replica to be a primary instance and wait for restart. Now it’s ready for traffic.
3. Change the environment variable for PgBouncer described above to now point at the newly promoted primary instance.
4. Bring PgBouncer instances back, setting the ASG desired capacity back to the original value.

This process involved 5-10 minutes of user-facing downtime. We performed it in an off-peak time to minimize user-facing problems. Much of the frame activity occurs in the background, so the main impact is app use. Thanks to the customer support team for helping through this period internally and externally.

Clean up of unneeded tables (due to physical replication): As mentioned, due to the choice of physical replication, the majority of the tables on each instance were not needed and should be removed.

1. Drop the relocated tables from the main DB. Carefully review this with team members. Initially rename table, double check again, then drop the renamed table.
2. Drop all unneeded tables replicated to the new whole-table shard DBs, which was most of them. This was partly scripted, and partly done manually for review and close monitoring.

With all the unneeded tables cleaned up, we now had way more allocated space than needed. Provisioned space costs money. AWS launched Blue/Green Deployments which has made it easier to replace instances with newly configured ones.

We set up a Blue/Green Deployment with the Blue as the newly promoted primary, and the Green would be a replacement instance with less space provisioned. Once replication was caught up, we cut over to Green. This process was smooth, and the result was right-sized space and cost.

Reflecting back on the plan

Some of the key contributors to successfully delivering this plan:

1. Having an extensive test suite running tests continuously (CI) catching regressions as code was refactored, along with PR reviews from long-tenured team members (invaluable)!
2. As refactorings happened in large batches, slicing out smaller chunks as smaller PRs for easier review, and less risk as releases.
3. Using a canary release process for widespread changes, released to a single instance vs. the whole fleet, which helped validate correctness with a small blast radius for issues that were difficult to verify in unit tests or outside of the production environment.
4. Having an extensive pre-production load testing capability to validate the accumulated changes under high load, across most of the API surface area of the platform, drilling into identified performance regressions.
5. Having a large AWS infrastructure budget 😅 to work with and strategic spending, in order to over-provision instance sizes and IOPS temporarily to gain more reliability, thanks in part to being a profitable company!
6. Having comprehensive CloudWatch metrics, dashboards, web, and Postgres logs for analysis (AWS Athena), time-series metrics galore (formerly StatHat), and best-in-class Postgres observability (PgAnalyze), to empower backend engineers with data access layer visibility.
7. Running recent versions of Postgres and Ruby on Rails, unlocking useful features for bigger scale platforms.
8. Experienced, long-tenured colleagues helping guide changes, focusing on high leverage opportunities, while generously sharing their knowledge and experience.

Thank You and Looking Forward

While the biggest payoff was seeing Postgres operate reliably through peak holiday traffic, it was equally rewarding to work with great engineers, be well supported by leadership, and benefit from years of accumulated scalability engineering all over the codebase. A special thank you to Josh, Ronnie, and EJ.

For 2026 we’re forming plans to further improve Postgres reliability, scalability, and cost efficiency.

If these types of posts are interesting to you, please consider subscribing to my blog or buying my book (details below).

If you’re an engineer reading this, thinking that these types of challenges would be fun to work on, please get in touch!

What’s next? Check out Part 2 which covers the Ruby on Rails side of the house.

And as always, please contact me with any questions or suggestions. Thanks for reading.

Ruby on Rails has helped make it possible to scale out the database layer, meeting the demands of millions of Aura Frames customers enjoying their digital photo frames.

In late 2025, the team added additional primary databases to expand capacity for peak write and read load ahead of Christmas Day, the busiest day of the year for the company. Rails manages queries and schema changes for each primary database within the same codebase, and now with the additional capacity of many primary databases.

With 8 primary databases in total, each server instance can be vertically scaled ahead of peak load. When load returns to normal levels, instances are scaled down for cost savings.

The team leveraged native support for Multiple Databases and the disable_joins: true feature in Active Record, the ORM for Ruby on Rails. The disable_joins feature replaces SQL joins, issuing multiple SELECT statements to combine data in the application from different databases.

This post looks back at the technical details of that plan, as well as a variety of additional data layer scaling tactics, that culminated in a successful Christmas 2025 season, with peak U.S. and Canadian Apple App Store and Google Play Store rankings of #1.

Building With Ruby on Rails

The Aura Frames platform has been built with Ruby on Rails since the beginning (more than 10 years ago!). Christmas 2025 was the busiest day of the year for the company and technical platform, serving a peak of 41 million API requests per hour (~11.4K requests per second), and processing a peak of 11.8 million background jobs per hour (~3300 jobs/second). On the database side, the sum of DB peak transactions per second (TPS) was 226K.

For an introduction to the Aura Frames company and products, and a deeper dive on the Postgres side of things, please check out Part 1 of this series.

Brief Recap from Part 1: Besides Ruby on Rails, Aura Frames uses PostgreSQL and AWS as key technologies.

Due to not being easily scalable horizontally for write operations, the database layer of PostgreSQL and Active Record often became a bottleneck. The team relied on vertically scaling the single primary server instance through Christmas of 2024.

The largest instance available for RDS at the time was the 48x family (192 vCPU, 1.5 TB RAM). Even with that jumbo-sized instance, the platform had reliability issues at peak load on Christmas 2024, driving a need to re-design for reliability improvements before Christmas 2025.

To handle greater levels of peak traffic reliably, the team decided to introduce application-level sharding using multiple primary databases. Several alternative approaches were considered. One goal was to leverage the existing code as much as possible, with minimal changes, and control the sharding distribution from the application level.

Another choice was whether to do traditional sharding at the row level, which distributes rows across multiple instances with databases having the same schema.

Fortunately Ruby on Rails was enhanced through more than 15 years of development, to support the needs of mature, scaled-up platforms with billions of rows and terabytes of data.

Before getting into the solution details, let’s look at some technical metrics from Christmas Day 2025 to help set context.

Technical Metrics

On Christmas Day, the Aura Frames platform sees a 4-5x increase in load. Below are some HTTP and background jobs oriented metrics that Rails developers might find interesting.

An exciting development for the team was seeing the free iOS and Android Aura Frames app rise in ranking throughout the day.

Late on Christmas Day, the app reached a peak rank of #1 among all free apps in the U.S. and Canadian App Stores, beating apps from big companies like OpenAI (ChatGPT) and Meta (Meta AI)!

Screenshot showing the Aura Frames app at the #1 rank in the U.S. Apple App Store

Although there is a lot of interesting history from how the Aura Frames Ruby on Rails codebase evolved over a decade, in this post we’ll focus on changes made from mid-2025 to prepare for the surge of traffic on Christmas Day, as well as some general data layer scaling tactics.

Getting Started With Multiple Databases

From earlier, you learned that the Aura Frames platform was expanded from a single primary application DB to a total of 8.

To do that, heavy refactoring was performed to Active Record query layer code.

Queries for tables must not span a database boundary, and given some of the big tables were being moved to a new database, queries would break.

The development environment uses a Docker Postgres container. To keep things simple locally, all 8 databases run within the single container, but are spread out as separate Postgres databases. This meant that queries still “broke” (helpfully) when they spanned a database boundary, making them easy to find through unit tests and manual testing.

The gist of the changes were pretty straightforward: find breaking queries, unravel joins or other incompatible SQL, and change connections for those queries to access the correct database. Their query results were then passed around in Ruby as input to queries in other databases.

With hundreds of failing tests to sift through, the refactoring work took a long time as test failures were addressed one by one, but progress was easy to measure.

Eventually, weeks later, all tests were passing! The nice property about this design was the same query changes without SQL joins could be performed on the existing main DB, meaning the query changes were backwards compatible and could be rolled out on the single primary DB.

Some of the main changes were evaluating all Active Record relationships (has_many, belongs_to, etc.) and any subquery expressions or other incompatible code, and using disable_joins: true, removing subquery expressions and table references that spanned the boundary.

From SQL Joins to Multiple SELECTs

The key parts of Ruby on Rails and Active Record that made this possible were Multiple Databases launched in 6.0, and the disable_joins feature for has_many :through and has_one :through relationships to query across databases launched in Rails 7.0 (2021).

Both of these were possible with custom code or third party library code (Ruby gems) prior to those releases, but having native support in Rails was a differentiator. Native support meant more real world use, bug fixes, improved documentation, and a longer term commitment to support.

Having disable_joins as a consistent pattern also helped with comprehension by the team, enabling “learn how it works once, then re-use it all over the codebase.”

Due to the increase in SELECT queries (and loss of join efficiency), the team had concerns about additional read query volume. Fortunately the team had a load testing tool in place and was able to verify through load testing that the additional read queries performed would not be a problem. With that said, over time we have replaced certain usages of disable_joins associations code with more targeted queries based on slow query logs or query cancellations. These queries are index supported, select minimal fields, and narrow ranges of rows by using batching.

Here’s a simple example using Author and Post models illustrating how disable_joins: true works:

• Author (table_name: authors)
• Post (table_name: posts)
• AuthorPost (table_name: author_posts)

Author model has an existing association defined as: has_many :posts, through: :author_posts.

To change this association, the disable_joins: true option is added like this:

class Author < ApplicationRecord
  has_many :posts, through: :author_posts, disable_joins: true
end

What’s happening in SQL? Previously to get an Author’s posts, we’d query the author_posts table and join to the posts table on the author’s id.

Instead of that, there will now be two SELECT queries. One queries author_posts by author_id (an important foreign key column to index) to get post id values. Then a second query to the posts table by id (which uses the primary key index) gets the rows from the second table.

select * from author_posts where author_id = '<some id>';
select * from posts where id IN (?);

Active Record handles the query change and presents the objects and collections in the same way to the developer.

Besides the query changes, what else needed to change?

New Database Configuration

Although we rolled out the query changes on the single primary DB architecture initially, that was intended to be temporary to further validate the changes without needing the new DBs in place.

The main plan was to use separate DB server instances, relocating the largest, busiest tables to their own instances in order to add more capacity and distribute the load.

For that we’d need to provision all the new DBs and connect to them from Rails. The first thing we needed was new YML config entries (config/database.yml) for each of them. The Multiple Databases Documentation uses “animals” and my_animals_db as the second primary database, so we’ll use that too for examples here.

This configuration is where we’ll store the Postgres connection string details and other application config like whether migrations are used, the schema dump path, etc.

Second, Active Record classes that previously inherited (OOP style) from ApplicationRecord < ActiveRecord::Base would get a new parent class.

The new parent class would introduce the new DB config for my_animals_db and have the concept of “writing” and “reading” roles, shown below.

The new class is AnimalsRecord, which is a child class that inherits from ApplicationRecord. Extending this new parent class becomes the “interface” for any additional Active Record classes that wish to read or write to this new DB.

Examples from Rails’ Documentation:

class AnimalsRecord < ApplicationRecord
  self.abstract_class = true

  connects_to database: {
    writing: :animals,
    reading: :animals_replica
  }
end

Models/tables inherit from AnimalsRecord to work with that database. For example a Dog class:

class Dog < AnimalsRecord
  # Talks automatically to the animals database.
end

Eventually all new DBs and infra (PgBouncer, config vars) were set up, and could be switched over to. In the Rails app, the new DBs were named generically as they didn’t really correspond to a particular grouping of activity like a service, we just wanted a bunch of DBs with unique names.

The names of the DBs were part of the parent model, now the parent of the original model. Since the original model doesn’t change apart from having a new parent class, the new DB details are nicely encapsulated.

disable_joins for has_many :through relationships ended up covering a lot of what was needed for query database separation, however there were other issues encountered on the way.

What were they?

Post-split: Subquery Expressions

Multiple tables exist in subquery expressions (aka “subqueries”), and these tables need to be in the same database for the statement to be valid.

When we found those, they needed to be restructured so that the DBs containing the table could be queried.

Post-split: EXISTS Clauses

Post-split, the SQL below doesn’t work when users and posts (example models) aren’t in the same DB.

SELECT users.*
FROM users
WHERE EXISTS (
  SELECT 1 FROM posts WHERE posts.user_id = users.id
)

Post-split: Aggregating And Grouping Multiple Tables

Post-split, there can be SQL fragments like this lurking, and this code needs to be changed when these tables are moved to separate databases.

SQL fragments are wrapped in Arel.sql('').

Users.select("users.*, COUNT(posts.id) AS posts_count")

Post-split: Merging Scopes

If the tables for User and Post aren’t in the same database, we can’t merge a scope like this:

User.merge(Post.recent)

Post-split: References Method

references API Documentation which adds a SQL join, is used in conjunction with includes() to specify a table. However, this won’t work if the table is no longer in the same database.

Other Associations: has_and_belongs_to_many

We did have a handful of has_and_belongs_to_many (HABTM) relationships (API Documentation). With these there is still a join table, but there is no Active Record model for it. The tables are also slim, no primary key or timestamp columns.

For HABTM relationships that would span a DB boundary, we decided to keep the table definitions as is, but introduce a model class and convert the code-level relationship from HABTM to has_many :through (HMT) so that we could use disable_joins: true.

Not all HABTM relationships were changed. When the HABTM relationship tables stayed in the same DB, we left those untouched.

Let’s shift gears into some general additional data layer scaling tactics.

Scaling Inserts and Updates

With Multiple Databases and disable_joins: true covered, what other database scalability tactics are used?

Rails supports bulk inserts and upserts (either an insert or an update), however the helper method for mass-inserting data didn’t support what we needed. A limitation was that the ON CONFLICT clause couldn’t be customized for insert_all() (API Documentation), which we needed.

For example attempting an INSERT and specifying the DO NOTHING option for handling unique constraint violations.

However, upsert_all() did get support for an :on_duplicate option.

Aura Frames has custom code for mass insert with direct control over the ON CONFLICT clause. Being able to batch inserts like this is a critical part of write scalability, consolidating the overhead of a batch of row insertions (e.g. 1000) into a single commit.

Scaling Reads With Batching

Rails supports batched read queries with a few Active Record methods: find_each(), in_batches(), and find_in_batches().

Aura Frames has custom code for batched finding, specifying an arbitrary column on the table and a sorting direction.

Rails 6.1 did add support for find_in_batches() (API Documentation to control ordering, but only the primary key column was supported for ordering in ascending or descending order. We needed to order on arbitrary columns.

Chedli Bourguiba pointed out that Rails 8 added a :cursor option to support cursor pagination on a non-primary key column. See more at: JetThoughts - Ruby on Rails 8: How to Batch with Custom Columns. I will be evaluating this in the future.

Reading a batch of rows using keyset pagination is a critical tactic for stable query execution times with varying result set sizes.

Scaling Reads With Paginated Queries

Aura Frames has custom code to perform keyset pagination, and generally does not use LIMIT and OFFSET style pagination built-in to Active Record. LIMIT and OFFSET pagination works for smaller amounts of data, but doesn’t scale well for deep pagination levels or when working with tables with billions of rows.

Keyset pagination with a high cardinality indexed column works well for fetching batches at a time, even when querying multi-billion row tables given they have good supporting indexes. The trick is to index a high cardinality column like a timestamp column, then filter on that with a WHERE clause and use LIMIT for a batch of rows. Note that timestamps can be duplicated, so you may need an additional column in that case.

An example fetch might be from a value with a >= or < operator and a LIMIT of 1000 as a batch size. The last accessed value then becomes the cursor position to start from.

This is an incredibly useful pattern and commonly used for API requests and other spots. To my knowledge Active Record doesn’t have a generic keyset style pagination helper.

Counter Cache Maintenance for Frequently Updated Counters

Rails supports counter_cache columns (Blog post) as a running counter, which is kept updated at write time.

Caveats are row churn and possible lock contention. Even updates of a single column create a new immutable row version behind the scenes. This adds dead row versions and more work for Vacuum, but this trade-off may be worth it.

Aura Frames does something similar but keeps counter cache columns in a separate but related table (plus counters in Memcached, see below). This reduces contention and places the churn more on a separate utility table.

Random Values and Sampling

Ordering by RANDOM() is slow. To avoid that, the Aura codebase uses TABLESAMPLE in Postgres (a contrib module which means no extensions are needed to install). A tablesample can be specified in a FROM clause (Postgres Documentation).

As a quick example from Rideshare using the Trip model (trips table) as SQL:

Trip.connection.select_all("SELECT * FROM trips TABLESAMPLE SYSTEM (10)").to_a

Or as more conventional Active Record, requesting 5% of rows (1000 rows in this table):

Trip.from("trips TABLESAMPLE SYSTEM (5)").count
Trip.from("trips TABLESAMPLE BERNOULLI (5)").count

The sampling_method options (built-in) above were SYSTEM and BERNOULLI. With 1000 rows in my local table, I should get about 50 rows by specifying 5%, but if you run this you’ll notice it’s only approximate. System is faster, but neither are fully accurate.

To get a precise amount of rows, enable the tsm_system_rows extension (Postgres Documentation) and specify a row count like this:

Trip.from("trips TABLESAMPLE SYSTEM_ROWS(50)").count
  Trip Count (11.9ms)  SELECT COUNT(*) FROM trips TABLESAMPLE SYSTEM_ROWS(50)
  => 50

You’ll notice above we got exactly 50 rows.

This tactic is used in various scripts and utilities where samples are needed from tables with millions of rows.

Using Memory Key Value Cache Stores

Aura Frames makes use of the Active Support Cache Store via Memcached with HAProxy performing connection management.

Keeping certain values in Memcached is a key part of the scaling strategy, values like per-user counters, per-feature rate limiting, or cached environment variable values with TTLs.

Managing Schema Changes with Multiple Databases

With the 7 new DBs and configs in config/database.yml, we wanted to continue managing DDL changes via Rails Migrations like normal.

Fortunately this is supported. Each DB has its own schema.rb, a Ruby representation of the schema definition, a directory for migration files, and a place for config options.

Since each “new” database was not actually new, but based on an existing table definition, we started a new “first” migration for it using the existing table definition dumped via pg_dump.

This migration version was written to be idempotent meaning the table was added only when it didn’t exist. Initially the table would not exist in dev, test, and staging, but based on how we planned to migrate the tables in production, the table would exist there.

Brief recap from Part 1: The plan was to use physical replication from the original primary instance to create a read only replica, then promote it to become a writer database. The replication was used as the means of moving all of the row data. This approach proved very reliable, but it did mean we had the former table copy on the original DB to clean up later, plus a ton of unneeded tables on all the new DBs to clean up (more on that in the other post).

Imagine the new migration version was 1234567890. Once switched over, we’d TRUNCATE its schema_migrations table, then manually insert the new migration version into schema_migrations to keep the state consistent.

insert into schema_migrations (version) values ('1234567890');

That was repeated for each new DB. Once that was done, schema management via migrations worked like normal, each seeded with their initial create table DDL as of that moment in time.

Migrations could be generated with their own directory for files, applied with rails db:migrate and schema.rb kept updated.

Some example commands:

rails g migration --database new_db

rails db:migrate --database my_animals_db
# or rails db:migrate for all databases

rails db:schema:cache:dump

Wrap Up

Ruby on Rails has been a critical technology for Aura Frames to build with for more than a decade, enabling a small team to continually ship improvements to customers from the same codebase, with the expanded capacity of many primary databases.

Enhancements in the last handful of versions like Multiple Databases support, disable_joins: true for associations have helped the team expand DB capacity, and still ship quickly to continue to deliver higher performance, and more reliable solutions to customers.

If these types of posts are interesting to you, please consider subscribing to my blog or buying my book.

If you’re an engineer interested in working on these types of challenges, please get in touch.

Thanks for reading!

Updates

2026-06-18 Chedli Bourguiba

• :cursor option for find_in_batches()
• Replaced two instances of documentation with Edge Rails documentation

With the recent announcement of PostgreSQL 19 Beta 1, it’s a great time to do that. Let’s get an instance up and running and test some of the new capabilities in Postgres 19.

Pre-Release Versions of Postgres with Docker

First, you’ll need to install Docker for your OS! Grab the version needed for your OS and processor architecture, for example ARM or AMD/Intel/x86.

On MacOS run uname -m or sw_vers in your Terminal to learn more about your hardware details.

For Windows check Install Docker Desktop on Windows

Building and Running

Official Postgres images for Docker Postgres are limited to fully released versions.

Fortunately @yosifkit created a PR to add 19 Beta 1 (merged by @tianon) with instructions for how to use docker buildx to build pre-release versions.

This command downloads and builds postgres:19beta1-trixie:

docker buildx build -t postgres:19beta1-trixie \
    'https://github.com/infosiftr/postgres.git#19-rc:19/trixie'

With that built, I could invoke docker run with postgres:19beta1-trixie. I named mine pg19.

I also passed the env vars below based on how I run other Docker Postgres containers (these options may not be necessary).

The final command:

docker run \
--name pg19 \
--env POSTGRES_USER=postgres \
--env POSTGRES_PASSWORD=postgres \
--detach postgres:19beta1-trixie

To check if it’s running, I run docker ps -a. For logs I’d run: docker logs -f postgres:19beta1-trixie.

Released Versions

Shortly after writing the initial post on June 5th, the postgres:beta1 image became available for download without any special steps:

docker run --detach postgres:19beta1

Connect via psql

The container is running and the logs have what we want: “database system is ready to accept connections”.

Let’s connect to the postgres database using psql on the container:

docker container exec -it pg19 psql -U postgres

We should see output like showing version 19:

psql (19beta1 (Debian 19~beta1-1.pgdg13+1))
Type "help" for help.

New Feature Testing in 19

Great. Let’s try out some things in 19.

19 Added a new system view for checking out locks. Let’s try it out:

postgres=# select * from pg_stat_lock;

We get a lot of new data like waits counts, wait_time and more.

What about the new pg_plan_advice extension? First let’s load it and then create a table t to experiment with:

postgres=# LOAD 'pg_plan_advice';
postgres=# create table t (id int);

With that in place we can show the output via EXPLAIN with a new PLAN_ADVICE parameter:

postgres=# EXPLAIN (PLAN_ADVICE) SELECT * FROM t;
                     QUERY PLAN
-----------------------------------------------------
 Seq Scan on t  (cost=0.00..35.50 rows=2550 width=4)
 Generated Plan Advice:
   SEQ_SCAN(t)
   NO_GATHER(t)
(4 rows)

I wonder why the rows estimate is 2550 by default? Let’s run analyze t;.

After doing that, it looks more sensible with a rows estimate of 1:

postgres=# EXPLAIN (PLAN_ADVICE) SELECT * FROM t;
                   QUERY PLAN
-------------------------------------------------
 Seq Scan on t  (cost=0.00..0.00 rows=1 width=4)
 Generated Plan Advice:
   SEQ_SCAN(t)
   NO_GATHER(t)
(4 rows)

Additions to pg_stat_statements

The extension pg_stat_statements gained new capabilities in 19.

Let’s try it out:

postgres=# select * from pg_stat_statements;
ERROR:  relation "pg_stat_statements" does not exist

Oops, we need to add it to shared_preload_libraries first. We can see that’s currently not the case:

postgres=# show shared_preload_libraries;
 shared_preload_libraries
--------------------------

(1 row)

One way to do that with docker run is the -c parameter as follows:

docker run \
--name pg19 \
--env POSTGRES_USER=postgres \
--env POSTGRES_PASSWORD=postgres \
--detach postgres:19beta1-trixie \
-c shared_preload_libraries=pg_stat_statements

Now we see what we want in shared_preload_libraries:

postgres=# show shared_preload_libraries;
 shared_preload_libraries
--------------------------
 pg_stat_statements
(1 row)

We have not yet enabled the extension though given \dx doesn’t list it. Let’s do that:

psql> create extension if not exists pg_stat_statements;

Now \dx shows it, and we’re ready to query it.

One of the additions is tracking the use of prepared statements. Let’s create a basic table and prepared statement.

Create table again if needed:

create table if not exists t (id int);

Create a simple prepared statement get_t and execute it. The goal here is for pg_stat_statements to increment the generic_plan_calls field.

PREPARE get_t AS
SELECT *
FROM t;

Now let’s execute it:

EXECUTE get_t;

Did it work?

postgres=# select left(query,100),generic_plan_calls from pg_stat_statements limit 1;
       left       | generic_plan_calls
------------------+--------------------
 PREPARE get_t AS+|                  1
 SELECT *        +|
 FROM t           |

It worked! We see generic_plan_calls was incremented.

This looks very useful to monitor the use of prepared statements.

Repack Concurrently for Tables

We’ve had the ability to use reindex concurrently to rebuild indexes since Postgres 12, but have lacked the ability to rebuild tables.

That changes in 19, with the introduction of repack concurrently for tables.

Let’s try it out quick and bloat a table by updating every row. We’ll compare the size before and after repacking it.

Run these examples using psql:

create table bloat (id int primary key);
CREATE TABLE

insert into bloat (id) select i from generate_series(1, 1_000_000) as t(i);
INSERT 0 1000000

SELECT pg_size_pretty(pg_total_relation_size('bloat'));
 pg_size_pretty
----------------
 56 MB

analyze bloat;
ANALYZE

-- Update every row
update bloat set id = id;
UPDATE 1000000

analyze bloat;
ANALYZE

-- The table size has doubled
SELECT pg_size_pretty(pg_total_relation_size('bloat'));
 pg_size_pretty
----------------
 112 MB

repack (concurrently, verbose) bloat;
INFO:  repacking "public.bloat" in physical order
INFO:  "public.bloat": found 0 removable, 1000000 nonremovable row versions in 8850 pages
DETAIL:  0 dead row versions cannot be removed yet.
CPU: user: 0.21 s, system: 0.04 s, elapsed: 0.28 s.
REPACK

-- Returns to original table size
SELECT pg_size_pretty(pg_total_relation_size('bloat'));
 pg_size_pretty
----------------
 56 MB

Very useful! Check out the repack documentation for more info.

Wrapping Up

Please give this a shot and experiment with new features in Postgres 19!

• Add 19.x builds (currently beta 1)

The first internal item is an “SLRU.” The acronym stands for “simple least recently used.” The LRU portion refers to caches and how they work, and SLRUs in Postgres are a collection of these caches.

SLRUs are small in-memory item stores. Since they need to persist across restarts, they’re also saved into files on disk. Álvaro Herrera¹ calls SLRUs “poorly named for a user-facing feature.” If they’re internal, why are they worth knowing about as Postgres users?

They’re worth knowing about because there can be a couple of possible failure points with them, due their fixed size. We’ll look at those later in this post.

Before getting into that, let’s cover some basics about what they are and look at a specific type.

Main purpose of SLRUs

The main purpose of SLRUs is to track metadata about Postgres transactions.

SLRUs are a general mechanism used by multiple types. Like a lot of things in Postgres, the SLRU system is extensible which means extensions can create new types.

The “least recently used” aspect might be recognizable from cache systems. LRU refers to how the oldest items are evicted from the cache when it’s full, and newer items take their place. This is because the cache has a fixed amount of space (measured in 8KB pages) and thus can only store a fixed amount of items.

Old SLRU cache items are periodically cleaned up by the Vacuum process.

What about the buffer cache?

The buffer cache (sized by configuring shared_buffers) is another form of cache in Postgres. Thomas Munro proposed unifying the SLRUs and buffer cache mechanisms.

However, as of Postgres 17 and the upcoming 18 release (Update: PostgreSQL 18 released September 25, 2025), SLRUs are still their own distinct type of cache.

What types of data is stored in SLRUs?

What type of data is tracked in SLRUs?

Transactions are a core concept for relational databases like Postgres. Transactions are abbreviated “Xact,” and Xacts are one of the types of data stored in SLRUs.

Besides regular transactions, there are variations of transactions. Transactions can be created inside other transactions, which are called “nested transactions.”

Whether parent or nested transactions, they each get their own 32-bit integer identifier once they begin modifying something, and these are all tracked while they’re in use. The SAVEPOINT keyword (blog post: You make a good point! — PostgreSQL Savepoints saves the incremental status for a transaction.

Another variation of a transaction is a “multi-transaction,” (multiple transactions in a group) or “MultiXact” in Postgres speak.

What are MultiXacts?

A MultiXact gets a separate number from the transaction identifier. I think of it like a “group” number. The group might be related to a table row, but each transaction in the group is doing something different. Think of multiple transactions all doing a foreign key referential integrity check on the same referenced primary key.

Here’s a definition of MultiXact IDs:

A MultiXact ID is a secondary data structure that tracks multiple transactions holding locks on the same row.

When MultiXacts are created, their identifier is stored in tuple header info, replacing the transaction id that would normally be stored in the tuple header.

As this buttondown blog post (“Notes on some PostgreSQL implementation details”)² describes, the tuple (row version) header has a small fixed size. The MultiXact id replaces the transaction id using the same size identifier (but a different one), to keep the tuple header size small (as opposed to adding another identifier).

Transaction IDs and MultiXact IDs are both represented as a circular 32-bit integer space, meaning it’s possible to store a max of around ~4 billion values (See: Transactions and Identifiers, half in the past, half in the future.

We can get the current transaction id value by running select pg_current_xact_id();.

What do we mean by transaction metadata? One example is with nested transactions, the parent transaction, the “creator”.

If you’d like to read how AWS introduces MultiXacts, check out this post. This post describes them: What are MultiXacts? https://aws.amazon.com/blogs/database/multixacts-in-postgresql-usage-side-effects-and-monitoring/

When do MultiXacts get created?

When do MultiXacts get created?

MultiXacts get created only for certain types of DML operations and for certain schema definitions. In other words, it’s possible that your particular Postgres database workload does not create MultiXacts at all, or it’s possible they’re heavily used. Let’s look at what creates MultiXacts:

• Foreign key constraint enforcement
• SELECT FOR SHARE

If you use no foreign key constraints or your application (or ORM) never creates SELECT FOR SHARE, then your Postgres database may have no MultiXacts.

Let’s go back to SLRUs.

More about SLRUs

SLRUs have a fixed size (prior to Postgres 17) measured in pages. When items are evicted from the SLRU cache, a page replacement occurs.

The page being replaced is called the “victim” page and Postgres must do a little work to find a victim page. Since SLRUs survive Postgres restarts, they’re saved in files in the PGDATA directory.

The directory name will depend on the SLRU type. For example for MultiXacts, the directory name is pg_multixact. SLRU buffer pages are written to the WAL and to disk, meaning that if the primary instance fails, the state can be recovered.

See the slru.c SlruPhysicalWritePage function comments which describes writing WAL and writing out data:

Honor the write-WAL-before-data rule, if appropriate, so that we do not
write out data before associated WAL records.

Each SLRU instance implements a circular buffer of pages in shared memory, evicting the least recently used pages. A circular buffer is another interesting Postgres internal concept but is beyond the scope of this post. How can we observe what’s happening with SLRUs?

Using pg_stat_slru

Since Postgres 13, we have the system view “pg_stat_slru” to query to inspect cumulative statistics about the SLRUs. https://www.postgresql.org/docs/current/monitoring-stats.html#PG-STAT-SLRU-VIEW To list only the names of the built-in SLRU types:

select name from pg_stat_slru;
      name
-----------------
 CommitTs
 MultiXactMember
 MultiXactOffset
 Notify
 Serial
 Subtrans
 Xact
 Other

To determine if our system is creating MultiXact SLRUs, we can query the pg_stat_slru view. We’d see non-zero numbers in rows below when the system is creating SLRU data.

select name from pg_stat_slru;
                     View "pg_catalog.pg_stat_slru"
    Column    |           Type           | Collation | Nullable | Default 
--------------+--------------------------+-----------+----------+---------
 name         | text                     |           |          | 
 blks_zeroed  | bigint                   |           |          | 
 blks_hit     | bigint                   |           |          | 
 blks_read    | bigint                   |           |          | 
 blks_written | bigint                   |           |          | 
 blks_exists  | bigint                   |           |          | 
 flushes      | bigint                   |           |          | 
 truncates    | bigint                   |           |          | 
 stats_reset  | timestamp with time zone |           |          |

To look at the pg_xact SLRU:

select * from pg_stat_slru where name = 'Xact';
 name | blks_zeroed | blks_hit | blks_read | blks_written | blks_exists | flushes | truncates |          stats_reset
------+-------------+----------+-----------+--------------+-------------+---------+-----------+-------------------------------
 Xact |         460 | 30686596 |        44 |         2030 |           0 |    1684 |         0 | 2024-11-19 09:52:33.506794-06

“Hit” and “read” refer to reads from the SLRU that where the desired pages were already in the SLRU or they were not.

When new pages are allocated, we see this reflected in “blks_zeroed” as they’re written out with zeroes.

When new pages are written (blks_written) into the SLRU this creates “dirtied” pages that eventually will be written out (flushes).

SLRUs can also be truncated (“Truncates” count).

Some of the source code for SLRUs in Postgres is in the file backend/access/transam/slru.c. https://github.com/postgres/postgres/blob/master/src/backend/access/transam/slru.c

Now that we know some basics about SLRUs and a specific type, the MultiXact SLRU, what are some operational concerns or things that can go wrong?

What can go wrong with SLRUs and Xacts?

Operational problems can stem from the fact that SLRUs use a 32-bit number and for high scale Postgres, it’s possible to consume these fast enough that the number can “wrap around.”

Two examples with public write-ups related to SLRU operational problems are:

• Subtransactions overflow: Using subtransactions, each use of a subtransaction creates an id to track. At a high enough creation rate it’s possible to run out of values. This was written up in the GitLab post: Why we spent the last month eliminating PostgreSQL subtransactions.

MultiXact member space exhaustion: MultiXact or multiple transactions can occur in a few scenarios.

• An explicit row lock: SELECT … FOR SHARE
• SELECT … FOR UPDATE

Written up in the Metronome blog post: Root Cause Analysis: PostgreSQL MultiXact member exhaustion incidents (May 2025).

A scenario for that could be a foreign key constraint lookup on a high insert table referencing a low cardinality table.

Another type of problem in the buttondown post² is the quadratic growth of MultiXacts.

Dilip Kumar talked about: “Long running transaction, system can go fully to cache replacement, TPS drops, with subtransactions ids (need to get parent ids).” See Dilip’s presentation for more info.³

What do we do with info as Postgres operators?

This is a huge topic and this post just scratches the surface.

However, let’s wrap this up here a bit with some takeaways.

If operating a high scale Postgres instance when it comes to SLRUs, what’s worth knowing about?

• Know about the SLRU system in general, how to monitor it, and don’t forget about extensions
• Learn about SLRUs limitations and possible failure points, for the various types
• Determine whether your workload is using SLRUs, monitor their growth, and learn about the possible failure points based on your use

What’s changing with SLRUs in new Postgres versions?

In Postgres 17, the MultiXact member space and offset is now configurable beyond the initial default size. The unit is the number of 8KB pages.

• multixact_member_buffers, default is 32 8kb pages
• multixact_offset_buffers, default is 16 8kb pages

In the recent episode of postgres.fm MultiXact member space exhaustion,⁴ the Metronome engineers discussed working on a patch related to MultiXact member exhaustion.

Lukas covers changes in Postgres 17 to adjust SLRU cache sizes. Each of the SLRU types can now be configured to be larger in size. https://pganalyze.com/blog/5mins-postgres-17-configurable-slru-cache

Conclusion

I’m still learning about MultiXacts, SLRUs, and failure modes as a result of these. If you have feedback on this post or additional useful resources, I’d love to hear about them. Please contact me here or on social media.

Thanks for reading!

Resources

Dilip Kumar presentation 2024 - PostgreSQL Development Conference https://www.youtube.com/watch?v=74xAqgS2thY

MultiXacts Dan Slimmon https://blog.danslimmon.com/2023/12/11/concurrent-locks-and-multixacts-in-postgres/

5 minutes of Postgres LWLock Lock Manager https://pganalyze.com/blog/5mins-postgres-LWLock-lock-manager-contention

SLRU Improvements Proposals Wiki https://wiki.postgresql.org/wiki/SLRU_improvements

Corrections

September 27, 2025: An earlier version of this post inaccurately described SLRU buffers as not being WAL logged. Thank you to Laurenz Albe for writing in to correct this and providing a pointer into the source code to learn more.

Over the last decade, when working on databases with UUID Version 4¹ as the primary key data type, these databases have usually had bad performance and excessive IO.

UUID is a native data type in Postgres stored as binary data. Various UUID versions are in the RFC. Version 4 has 128 bits, with 122 being random, obfuscating information like when the value was created or where it was generated.

Version 4 UUIDs are easy to generate in Postgres using the gen_random_uuid()² function since version 13 (released in 2020).

I’ve learned there are misconceptions about UUID Version 4, and sometimes these are the reasons users pick this data type.

Because of the poor performance, misconceptions, and available alternatives, I’ve come around to a simple position: Avoid UUID Version 4 for primary keys.

My more controversial take is to avoid UUIDs in general, but I understand there are some legitimate reasons for them without practical alternatives.

As a database enthusiast, I wanted to have an articulated position on this classic “Integer v. UUID” debate.

Among databases folks, debating this may be tired and clichéd. However, from my consulting work, I can say I work with databases using UUID v4 in 2024 and 2025, and still see the issues discussed in this post.

Let’s dig in.

UUID context for this post

• UUIDs (or GUID in Microsoft speak)³) are long strings of 36 characters, 32 digits, 4 hyphens, stored as 128 bits (16 byte) values, stored using the binary uuid data type in Postgres
• The RFC documents how the 128 bits are set
• The bits for UUID Version 4 are mostly random values
• UUID Version 7 includes a timestamp in the first 48 bits, which works much better with database indexes compared with random values

Although unreleased as of this writing, and pulled from Postgres 17 previously, UUID V7 is part of Postgres 18⁴ scheduled for release in the Fall of 2025.

What kind of app databases are in scope for this post?

Scope of web app usage and their scale

The kinds of web applications I’m thinking of with this post are monolithic web apps, with Postgres as their primary OLTP database. The apps could be in categories like social media, e-commerce, click tracking, or business process automation apps.

The types of performance issues discussed here are related to inefficient storage and retrieval, meaning they happen for all of these types of apps.

What’s the core issue with UUID v4?

Randomness is the issue

The core issue with UUID Version 4, given that the 122 bits they’re made up of are “random or pseudo-randomly generated values”¹, is how the values are maintained in indexes. Since primary keys are backed by indexes by default, each insert is less efficient compared with inserts for sequentially ordered values.

For lookups, each update and delete for individual items or for ranges of items are less efficient, due to increased traversal of non-sequential index pages in Postgres.

Since the randomly generated values aren’t inserted sequentially (or in sequential/adjacent pages), it’s less efficient to find them later for updates or deletes. Each of these workload types use the primary key index.

UUID v4s don’t have a useful natural ordering that aligns with how they’re stored, and thus both storage and retrieval is less efficient.

Later in the post we’ll look at just how many more Postgres pages need to be accessed for equivalent data, and what that means in terms of performance.

Despite the inefficiencies, UUID v4s and UUIDs in general remain (or at least were) popular in the last decade based on my experience consulting in Postgres.

Given the popularity, what use cases for UUID are there?

Why choose UUIDs at all? Generating values from one or more client applications

One use case for UUIDs is when there’s a need to generate an identifier on a client or from multiple services, then passed to Postgres for persistence.

For web apps, generally they instantiate objects in memory and don’t expect an identifier to be used for lookups until after an instance is persisted as a row (where the database generates the identifier).

In a microservices architecture where the apps have their own databases, the ability to generate identifiers from each database without collisions is a use case for UUIDs. The UUID could also identify the database a value came from later, vs. an integer.

For collision avoidance (see HN discussion⁵), we can’t practically make the same guarantee with sequence-backed integers. There are hacks, like generating even and odd integers between two instances, or using different ranges in the int8 range.

There are also alternative identifiers like using composite primary keys (CPKs), however the same set of 2 values wouldn’t uniquely identify a particular table.

The avoidance of collisions is described this way on Wikipedia:⁶

The number of random version-4 UUIDs which need to be generated in order to have a 50% probability of one collision: 2.71 quintillion

This number would be equivalent to:

Generating 1 billion UUIDs per second for about 86 years.

Are UUIDs secure?

Misconceptions: UUIDs are secure

One misconception about UUIDs is that they’re secure. However, the RFC describes that they shouldn’t be considered secure “capabilities.”

From RFC 4122¹ Section 6 Security Considerations:

Do not assume that UUIDs are hard to guess; they should not be used as security capabilities

How can we create obfuscated codes from integers?

Creating obfuscated values using integers

While UUID V4s obfuscate their creation time, the values can’t be ordered to see when they were created relative to each other. We can achieve those properties with integers with a little more work.

One option is to generate a pseudo-random code from an integer, then use that value externally, while still using integers internally.

To see the full details of this solution, please check out: Short alphanumeric pseudo random identifiers in Postgres⁷

We’ll summarize it here.

• Convert a decimal integer like “2” into binary bits. E.g. a 4 byte, 32 bit integer: 00000000 00000000 00000000 00000010
• Perform an exclusive OR (XOR) operation on all the bits using a key
• Encode each bit using a base62 alphabet

The obfuscated id is stored in a generated column. By reviewing the generated values, they are similar, but aren’t ordered by their creation order.

The values in insertion order were 01Y9I, 01Y9L, then 01Y9K.

With alphabetical order, the last two would be flipped: 01Y9I first, then 01Y9K second, then 01Y9L third, sorting on the fifth character.

If I wanted to use this approach for all tables, I’d try a centralized table that was polymorphic, storing a record for each table that’s using a code (and a foreign key constraint).

That way I’d know where the code was used.

Why else might we want to skip UUIDs?

Reasons against UUIDs in general: they consume a lot of space

UUIDs are 16 bytes (128 bits) per value, which is double the space of bigint (8 bytes), or quadruple the space of 4-byte integers. This extra space adds up once many tables have millions of rows, and copies of a database are being moved around as backups and restores.

A more considerable impact to performance though is the poor characteristics of writing and reading random data into indexes.

Reasons against: UUID v4s add insert latency due to index page splits, fragmentation

For random UUID v4s, Postgres incurs more latency for every insert operation.

For integer primary key rows, their values are maintained in index pages with “append-mostly” operations on “leaf nodes,” since their values are orderable, and since B-Tree indexes store entries in sorted order.

For UUID v4s, primary key values in B-Tree indexes are problematic.

Inserts are not appended to the right most leaf page. They are placed into a random page, and that could be mid-page or an already-full page, causing a page split that would have been unnecessary with an integer.

Planet Scale has a nice visualization of index page splits and rebalancing.⁸

Unnecessary splits and rebalancing add space consumption and processing latency to write operations. This extra IO shows up in Write Ahead Log (WAL) generation as well.

Given fixed size pages, we want high density within the pages. Later on we’ll use pageinspect to check the average leaf density between integer and UUID to help compare the two.

Excessive IO for lookups even with orderable UUIDs

B-Tree page layout means you can fit fewer UUIDs per 8KB page. Since we have the limitation of fixed page sizes, we at least want them to be as densely packed as possible.

Since UUID indexes are ~40% larger in leaf pages than bigint (int8) for the same logical number of rows, they can’t be as densely packed with values. As Lukas says, “All in all, the physical data structure matters as much as your server configuration to achieve the best I/O performance in Postgres.”⁹

This means that for individual lookups, range scans, or UPDATES, we will incur ~40% more I/O on UUID indexes, as more pages are scanned. Remember that even to access one row, in Postgres the whole page is accessed where the row is, and copied into a shared memory buffer.

Let’s insert and query data and take a look at numbers between these data types.

Working with integers, UUID v4, and UUID v7

Let’s create integer, UUID v4, and UUID v7 fields, index them, load them into the buffer cache with pg_prewarm.

I will use the schema examples from the Cybertec post Unexpected downsides of UUID keys in PostgreSQL by Ants Aasma.

View andyatkinson/pg_scripts PR #20.

On my Mac, I compiled the pg_uuidv7 extension. Once compiled and enabled for Postgres, I could use the extension functions to generate UUID V7 values.

Another extension pg_prewarm is used. It’s a module included with Postgres, so it just needs to be enabled per database where it’s used.

The difference in latency and the enormous difference in buffers from the post was reproducible in my testing.

“Holy behemoth buffer count batman” - Ants Aasma

Cybertec post results:

• 27,332 buffer hits, index only scan on the bigint column
• 8,562,960 buffer hits, index only scan on the UUID V4 index scan

Since these are buffer hits we’re accessing them from memory, which is faster than disk. We can focus then on only the difference in latency based on the data types.

How many more pages are accessed for the UUID index? 8,535,628 (8.5 million!) more 8KB pages were accessed, a 31229.4% increase.

• 68,285,024 KB or ~68.3 GB! more data that’s accessed

Calculating a low and high estimate of access speeds for memory:

• Low estimate: 20 GB/s
• High estimate: 80 GB/s

Accessing 68.3 GB of data from memory (shared_buffers in PostgreSQL) would add:

• ~3.4 seconds of latency (low speed)
• ~0.86 seconds of latency (high speed)

That’s between ~1 and ~3.4 seconds of additional latency solely based on the data type using this simplistic method of calculation.

Please note this is not a rigorous calculation method, just something to do some comparisons.

Here we used 10 million rows and performed 1 million updates, but the latencies will get worse as data and query volumes increase.

Inspecting density with the pageinspect extension

We can inspect the average fill percentage (density) of leaf pages using the pageinspect extension.

The uuid_experiments/page_density.sql (andyatkinson/pg_scripts PR #20) query in the repo gets the indexes for the integer and v4 and v7 uuid columns, their total page counts, their page stats, and the number of leaf pages.

Using the leaf pages, the query calculates an average fill percentage.

After performing the 1 million updates on the 10 million rows mentioned in the example, I got these results from that query:

 idxname             | avg_leaf_fill_percent
---------------------+-----------------------
 records_id_idx      |                 97.64
 records_uuid_v4_idx |                 79.06
 records_uuid_v7_idx |                 90.09
(3 rows)

This shows the integer index had an average fill percentage of nearly 98%, while the UUID v4 index was around 79%.

UUID Downsides: Worse cache hit ratio

The Postgres buffer cache is a critical part of good performance.

For good performance, we want our queries to produce cache “hits” as much as possible.

The buffer cache has limited space. Usually 25-40% of system memory is allocated to it, and the total database size including table and index data is usually much larger than that amount of memory. That means we’ll have trade-offs, as all data will not fit into system memory. This is where the challenges come in!

When pages are accessed they’re copied into the buffer cache as buffers. When write operations happen, buffers are dirtied before being flushed.¹⁰

Since the UUIDs are randomly located, additional buffers will need to be copied to the cache compared to ordered integers. Buffers might be evicted to make space that are needed, decreasing hit rates.

Mitigations: Rebuilding indexes with UUID values

Since the tables and indexes are more likely to be fragmented, it makes sense to rebuild the tables and indexes periodically.

Rebuilding tables can be done using pg_repack, pg_squeeze, or VACUUM FULL if you can afford to perform the operation offline.

Indexes can be rebuilt online using REINDEX CONCURRENTLY.

While the newly laid out data in pages, they will still not have correlation, and thus not be smaller. The space formerly occupied by deletes will be reclaimed for reuse though.

Mitigation: Shared buffers and work_mem memory sizing

If possible, size your primary instance memory to be 4x than the size of database. If your database size is 25GB, run an instance with at least 100GB of memory if its in the budget.

A 128GB memory with 25% of memory allocated to buffer cache provides 32GB (controllable via shared_buffers) which could be enough to keep all table and index pages in memory cache.

Use pg_buffercache¹¹ to inspect the contents, and pg_prewarm¹² to populate tables into it.

One tactic I’ve used when working with UUID v4 random values where sorting is happening, is to provide more memory to sort operations.

To do that in Postgres, we can change the work_mem setting. This setting can be changed for the whole database, a session, or even for individual queries.

Check out Configuring work_mem in Postgres on PgMustard for an example of setting this in a session.

Mitigation in Rails: UUID and implicit order column Active Record

Since Rails 6, we can control implicit_order_column.¹³ The database_consistency gem even has a checker for folks using UUID primary keys.

When ORDER BY is generated in queries implicitly, it may be worth ordering on a different high cardinality field that’s indexed, like a created_at timestamp field.

Mitigating poor performance by clustering on orderable field

Cluster on a column that’s high cardinality and indexed could be a mitigation option.

For example, imagine your UUID primary table has a created_at timestamp column that’s indexed with idx_on_tbl_created_at, and clustering on that.

CLUSTER table_with_uuid_ok USING idx_on_tbl_created_at;

I don’t see CLUSTER used ever really though as it takes an access exclusive lock. The CLUSTER is a one-time operation that would also need to be repeated regularly to maintain its benefits.

Recommendation: Stick with sequences, integers, and big integers

For new databases that may be small, with unknown growth, I recommend plain old integers and an identity column (backed by a sequence)¹⁴ for primary keys. These are signed 32 bit (4-byte) values. This provides about 2 billion positive unique values per table.

For many business apps, they will never reach 2 billion unique values per table, so this will be adequate for their entire life. I’ve also recommended always using bigint/int8 in other contexts.

I guess it comes down to what you know about your data size, how you can project growth. There are plenty of low growth business apps out there, in constrained industries, and constrained sets of business users.

For Internet-facing consumer apps with expected high growth, like social media, click tracking, sensor data, telemetry collection types of apps, or when migrating an existing medium or large database with 100s of millions or billions of rows, then it makes sense to start with bigint (int8), 64-bit, 8-byte integer primary keys.

UUID v4 alternatives: Use time-ordered UUIDs like Version 7

Since Postgres 18 is not yet released, generating UUID V7s now in Postgres is possible using the pg_uuidv7 extension.

If you have an existing UUID v4 filled database and can’t afford a costly migration to another primary key data type, then starting to populate new values using UUID v7 will help somewhat.

Fortunately the binary uuid data type in Postgres can be used whether you’re storing V4 or V7 UUID values.

Another alternative that relies on an extension is sequential_uuids.¹⁵

Summary

• UUID v4s increase latency for lookups, as they can’t take advantage of fast ordered lookups in B-Tree indexes
• For new databases, don’t use gen_random_uuid() for primary key types, which generates random UUID v4 values
• UUIDs consume twice the space of bigint
• UUID v4 values are not meant to be secure per the UUID RFC
• UUID v4s are random. For good performance, the whole index must be in buffer cache for index scans, which is increasingly unlikely for bigger data.
• UUID v4s cause more page splits, which increase IO for writes with increased fragmentation, and increased size of WAL logs
• For non-guessable, obfuscated pseudo-random codes, we can generate those from integers, which could be an alternative to using UUIDs
• If you must use UUIDs, use time-orderable UUIDs like UUID v7

Do you see any errors or have any suggested improvements? Please contact me. Thanks for reading!

Learn More

• Franck Pachot for AWS Heroes has an interesting take on UUID in PostgreSQL
• Brandur has a great post: Identity Crisis: Sequence v. UUID as Primary Key
• 5mins of Postgres: UUIDs vs Serial for Primary Keys - what’s the right choice?
• andyatkinson/pg_scripts PR #20

Updates

• 2025-12-15: Appeared on front page of Hacker News. Updating the “Randomness is the issue” section for improved clarity.

In this post, we’ll cover some database design principles and package them up into a catchy mnemonic acronym.

Software engineering is loaded with acronyms like this. For example, SOLID principles describe 5 principles, Single responsibility, Open-closed, Liskov substitution, Interface segregation and Dependency inversion, that promote good object-oriented design.

Databases are loaded with acronyms, for example “ACID” for the properties of a transaction, but I wasn’t familiar with one the schema designer could keep in mind while they’re working.

Thus, the motivation for this acronym was to help the schema designer, by packaging up some principles of good design practices for database schema design. It’s not based in research or academia though, so don’t take this too seriously. That said, I’d love your feedback!

Let’s get into it.

Picking a mnemonic acronym

In picking an acronym, I wanted it to be short and have each letter describe a word that’s useful, practical, and grounded in experience. I preferred a real word for memorability!

The result was “CORE.” Let’s explore each letter and the word behind it.

Constraint-Driven

The first word (technically two) is “constraint-driven.” Relational databases offer rigid structures, but the ability to be changed while online, a form of flexibility in their evolution. We evolve their structure through DDL. They use data types and constraints to make changes, as entities and relationships evolve.

Constraint-driven refers to leveraging all the constraint objects available, designing for our needs today, but also in a more general sense applying constraints (restrictions) to designs in the pursuit of data consistency and quality.

Let’s look at some examples. Choose the appropriate data types, like a numeric data type and not a character data type when storing a number. Use NOT NULL for columns by default. Create foreign key constraints for table relationships by default.

Validate expected data inputs using check constraints. For small databases, use integer primary keys. If tables get huge later, no problem, we can migrate the data into a bigger more suitable structure.

The mindset is to prefer rigidity initially, design for today, then leverage the flexibility available to evolve later, as opposed to designing for a hypothetical future state.

Optimized

Databases present loads of optimization opportunities. Relational data is initially stored in a normalized form to eliminate duplication, but later denormalizations can be performed when read access is more important.

When our use cases are not known at the outset, plan to iterate on the design, changing the structure to better support the use cases that emerge. This will mean evolving the schema design.

This applies to tables, columns, constraints, indexes, parameters, queries, and anything that can be optimized to better support real use cases.

Queries are restructured and indexes are added to reduce data access. Strive for highly selective data access (a small proportion of rows) on high cardinality (uniqueness) data to reduce latency.

Critical background processes like VACUUM get optimized too. Resources (workers, memory, parallelization) are increased proportionally.

Responsive

When problems emerge like column or row level unexpected data, missing referential integrity, or query performance problems, engineers inspect logs, catalog statistics, and parameters, from the core engine and third party extensions to diagnose issues.

When DDL changes are ready, the engineer applies them in a non-blocking way, in multiple steps as needed. Operations are performed “online” by default when practical.

DDL changes are in a source code file, reviewed, tracked, and a copy of the schema design is kept in sync across environments.

Parameter (GUC) tuning (Postgres: work_mem, etc.) happens in a trackable way. Parameters are tuned online when possible, and scoped narrowly, to optimize their values for real queries and use cases.

Efficient

It’s relatively costly to store data in the database, compared with file storage! The data consumes limited space and accessing data unnecessarily adds latency.

Data that’s stored is queried later or it’s archived.

To minimize space consumption and latency, tables, columns, constraints, and indexes are removed continually by default, when they no longer are required, to reduce system complexity.

Server software is upgraded at least annually so that performance and security benefits can be leveraged.

Huge tables are split into smaller tables using table partitioning for more predictable administration.

CORE Database Design

There’s lots more to evolving a database schema design, but these principles are a few I keep in mind.

Did you notice anything missing? Do you have other feedback? Please contact me with your thoughts.

Thank You

Over the years, I’ve learned a lot from Postgres.fm hosts Nikolay and Michael, and other community leaders like Lukas and Franck, as they’ve shaped my database design choices.

I’m grateful to them for sharing their knowledge and experience with the community.

Thanks for reading!

Much of the time taken processing HTTP requests in web apps is SQL queries. To minimize that, we want to avoid unnecessary and duplicate queries, and generally perform as few queries as possible.

Think of the work that needs to happen for every query. The database engine parses it, creates a query execution plan, executes it, and then sends the response to the client.

When the response reaches the client, there’s even more work to do. The response is transformed into application objects in memory.

How do we see how many queries are being created for our app actions?

Count the queries

When doing backend work in a web app like Rails, monitor the number of queries being created directly, by the ORM, or by libraries. ORMs like Active Record can generate more than one query from a given line of code. Libraries can generate queries that are problematic and may be unnecessary.

Over time, developers may duplicate queries unknowingly. These are all real causes of unnecessary queries from my work experience.

Why are excessive queries a problem?

Why reduce the number of queries?

Besides parsing, planning, executing, and serializing the response, the client is subject to a hard upper limit on the number of TCP connections it can send to the database server.

In Postgres that’s configured as max_connections. The application will have a variable number of open connections based on use, and its configuration of processes, threads and its connection pool. Keeping the query count low helps avoid exceeding the upper limit.

What about memory use?

What about app server memory?

With Ruby on Rails, the cost of repeated queries is shifted because the SQL Cache is enabled by default, which stores and serves results for matching repeated queries, at the cost of some memory use.

As an side, from Rails 7.1 the SQL Cache uses a least recently used (LRU) algorithm. We can also configure the max number of queries to cache, 100 by default, to control how much memory is used.

Counting queries prior to Rails 7.2

Prior to Rails 7.2, I recommend adding the query_count gem which does a simple thing, it shows the count of SQL queries processed for an action.

The count is in the Rails log file like this: SQL Queries: 100 (50 cached). In this case, 100 queries were performed and 50 used the SQL Cache.

Built-in from Rails 7.2 onward

From Rails 7.2 onward, the count of queries is now built in, so query_count is no longer needed.

Rails 7.2 onward looks like this: ActiveRecord: 105.5ms (10 queries, 1 cached). Here 10 queries ran, and 1 used the SQL Cache.

Repeated queries

While the SQL Cache saves the roundtrip for a repeated query, ideally we want to eliminate the repeated query. It’s worth hunting for it and considering refactoring or restructuring data access.

Another tactic is using memoization to store results for the duration of processing one controller action. Read more about that: Speeding up Rails with Memoization.

How do I get started?

Finding the source code location of the queries

To get started, identify some slow API endpoints in production, run them locally in development, and begin monitoring their quantity of SQL queries. Find the Source code locations for database queries in Rails with Marginalia and Query Logs.

Determine how to factor out data access that can be shared.

How many queries are “a lot?”

It’s hard to give a generic number. However, duplicate queries are a category to remove.

Let’s say you’ve got a Book model for your bookstore app. Scan your Rails log file for a pattern like this:

Book Load (4.3ms) …
Book Load (5.0ms) …
Book Load (0.5ms) …
Book Load (2.3ms) …

If you see that sort of pattern, track down the source locations, and eliminate any repeated loads. Let’s assume this is not a N + 1 queries problem, but repeated access to the same data from different source code locations.

You may be able to factor out and consolidate a data load. You may be able to use an existing loaded collection for an existence check, or use memoization to use previously calculated results.

Using these tactics, I’ve reduced controller actions with 250+ SQL queries (a ton!) to 50 or fewer (still a lot), by going through these steps. Monitor the log, find source locations for first party code, ORM generated queries, query code from libraries (gems), Rails controller action “before filters,” and other sources, then eliminate and consolidate.

When faced with a lot of queries, I find it helpful to study the bare minimum of what’s needed by the client, working outside in, then look to see if it’s possible to reduce the tables, rows, and columns to only what’s needed.

Wrap Up

• Track the count of SQL queries performed in different versions of Rails
• Remove unnecessary queries so they don’t use limited system resources
• Eliminate repeated queries to keep the count as low as possible
• Only access data that’s needed for client application use cases

If you’ve created web apps with relational databases and ORMs like Active Record (part of Ruby on Rails), you’ve probably experienced database performance problems after a certain size of data and query volume.

In this post, we’re going to look at a specific type of problematic query pattern that’s somewhat common.

We’ll refer to this pattern as “Big INs,” which are queries with an IN clause that has a big list of values. As data grows, the length of the list of values will grow. These queries tend to perform poorly for big lists, causing user experience problems or even partial outages.

We’ll dig into the origins of this pattern, why the performance of it is poor, and explore some alternatives that you can use in your projects.

IN clauses with a big list of values

The technical term for values are a parenthesized list of scalar expressions.

For example in the SQL query below, the IN clause portion is WHERE author_id IN (1,2,3) and the list of scalar expressions is (1,2,3).

SELECT * FROM books
WHERE author_id IN (1, 2, 3);

The purpose of this clause is to perform filtering. Looking at a query execution plan in Postgres, we’ll see something like this fragment below:

Filter: (author_id = ANY ('{1,2,3}'::integer[]))

This of course filters the full set of books down to ones that match on author_id.

Filtering is a typical database operation. Why are these slow?

Parsing, planning, and executing

Remember that our queries are parsed, planned, and executed. A big list of values are treated like constants, and don’t have associated statistics.

Queries with big lists of values take more time to parse and use more memory.

Without pre-collected table statistics for planning decisions, PostgreSQL is more likely to mis-estimate cardinality and row selectivity.

This can mean the planner chooses a sequential scan over an index scan, causing a big slowdown.

How do we create this pattern?

Creating this pattern directly

In Active Record, a developer might create this query pattern by using pluck(:id) to collect some ids in a list, then pass that list as an argument to another query.

Here’s an example of that:

author_ids = Author.
  where("created_at >= ?", 1.year.ago).
  pluck(:id)

The author_ids are supplied as the argument querying books by author_id foreign key:

Book.where(author_id: author_ids)

Another scenario is when this query is created from ORM methods. What does that look like?

Active Record ORM methods that create this pattern

This query pattern can happen when using eager loading methods like includes() or preload().

This Crunchy Data post mentions how eager loading methods produce IN clause SQL queries.

The post links to the Eager Loading Associations documentation which has examples in Active Record and the resulting SQL that we’ll use here.

Let’s first discuss N+1 with these examples.

Fixing N+1s

Let’s study the examples here. Here’s some Active Record for books and authors:

# N+1
books = Book.limit(10)

books.each do |book|
   puts book.author.last_name
end

The issue above is the undesirable N+1 query pattern, where a table is repeatedly queried in a loop, instead of bulk loading all of the desired authors.

To fix the N+1, we’ll add the includes(:author) eager loading method to the code above.

That looks like this:

books = Book.includes(:author).limit(10) 👈

books.each do |book|
   puts book.author.last_name
end

We’ve now eliminated the N+1 queries, but we’ve opened ourselves up to a new possible problem.

Eager loading with includes or preload

While the includes(:author) fixed the N+1 queries, Active Record is now creating two queries, with the second one having an IN clause.

Here’s the example from above as SQL:

SELECT books.* FROM books LIMIT 10;

SELECT authors.* FROM authors
  WHERE authors.id IN (1,2,3,4,5,6,7,8,9,10);

Here we only have 10 values for the IN clause, so performance will be fine. However, once we’ve got hundreds or thousands of values, we will run into the problems described above.

Performance will tank if the authors.id primary key index isn’t used for this filtering operation.

Are there alternatives for eager loading?

Eager loading using eager_load

Besides includes() and preload() which create two queries with the second having an IN clause, there’s another way to do eager loading in Active Record.

An alternative method eager_load works a little bit differently. It produces a single SQL query that uses a LEFT OUTER JOIN.

Here’s an example of eager_load from the Active Record documentation:

books = Book.eager_load(:author).limit(10)

books.each do |book|
  puts book.author.last_name
end

The following single SQL query is produced. Note that it has no IN clause.

SELECT
    "books"."id" AS t0_r0,
    "books"."title" AS t0_r1
FROM
    "books" LEFT OUTER JOIN "authors"
    ON "authors"."id" = "books"."author_id"
LIMIT 10;

Since we’re now using a join operation, we’ve got statistics available from both tables. This makes it much more likely PostgreSQL can correctly estimate selectivity and cardinality.

The planner also isn’t needing to parse and store a large list of constant values.

While IN clauses might perform fine with smaller inputs of 100 values or fewer, for large lists we should try and restructure the query to use a join operation instead.

Besides restructuring the queries into joins, are there other alternatives?

Alternative approaches using ANY

Crunchy Data’s post Postgres Query Boost: Using ANY Instead of IN describes how IN is more restrictive on the input.

A more usable alternative to IN can be ANY (or synonym SOME), which has more flexibility in handling the list of values.

Here’s A CTE example using ANY:

WITH author_ids AS (
  SELECT id FROM authors
)
SELECT title
FROM books
WHERE author_id = ANY (
      SELECT id
      FROM author_ids);

However, ANY is not generated by Active Record. What if we want to generate these queries using Active Record?

One option is to use the any method provided by the ActiveRecordExtended gem.

Let’s talk at another alternative approach using a VALUES clause.

A VALUES clause

In the comments in the PR above, Vlad and Sean discussed an alternative for IN using a VALUES clause.

Let’s look at an example with a CTE and VALUES clause:

WITH ids(author_id) AS (
  VALUES(1),(2),(3)
)
SELECT title
FROM books
JOIN ids USING (author_id);

Or we can write this as a subquery:

SELECT title
FROM books
WHERE author_id IN (
  SELECT id
  FROM (VALUES(1),(2),(3)) AS v(id)
);

This is better because the IN list is a big list of scalar expressions, where the VALUES clause is treated like a relation (or table). This can help with join strategy selection.

A temporary table of ids

Yet another option for big lists of values is to put these into a temporary table for the session. The temporary table can even index the ids.

CREATE TEMP TABLE temp_ids (author_id int);
INSERT INTO temp_ids(author_id) VALUES (1),(2),(3);
CREATE INDEX ON temp_ids(author_id);

SELECT title
FROM books b
JOIN temp_ids t ON t.author_id = b.author_id;

Using ANY and an ARRAY of values

Another form is using ANY with an ARRAY:

SELECT title
FROM books
WHERE author_id = ANY (ARRAY[1, 2, 3]);

The ANY form can perform better. With an IN list, the values are parsed like a chain of OR operations, with the planner handling one branch at a time.

ANY is treated like a single functional expression.

This form also supports prepared statements. With prepared statements, the statement is parsed and planned once and then can be reused.

Here’s an example of fetching books by author:

PREPARE get_books_by_author(int[]) AS
SELECT title
FROM books
WHERE author_id = ANY ($1);

EXECUTE get_books_by_author(ARRAY[1,2,3,4,5]);

Testing the alternative query structures

Unfortunately generic guidelines here won’t guarantee success in your specific database. Row counts, data distributions, cardinality, or correlation are just some of the factors that affect query execution.

My recommended process is to test on production-like data, work in the SQL layer, then try out restructured queries using these tactics, and study their query execution plans collected using EXPLAIN (ANALYZE, BUFFERS).

Query plan collection and analysis is outside the scope of this post, but in brief, you’ll want to compare the plans and look to access fewer buffers, at lower costs, with fewer rows evaluated, fewer loops, for more efficient execution.

If you’re working in Active Record, you’d then translate your SQL back into the Active Record source code location where the queries were generated.

How do we find problematic IN queries that ran earlier in Postgres?

Finding IN clause queries in pg_stat_statements

To find out if your query stats include the problematic IN queries, let’s search the results of pg_stat_statements by querying the query field.

Unfortunately these don’t always group up well, so there can be duplicates or near-duplicates. You may have lots of PGSS results to sift through.

Here’s a basic query to filter on query for '%IN \(%':

SELECT
    query
FROM
    pg_stat_statements
WHERE
    query LIKE '%IN \(%';

See the linked PR for a reproduction set of commands to create these tables, queries, and then inspect the query statistics using PGSS.

While you can find and restructure your queries towards more efficient patterns, are there any changes coming to Postgres itself to better handle these?

Improvements in Postgres 17

As part of the PostgreSQL 17 release in 2024, the developers made improvements to more efficiently work with scalar expressions and indexes, resulting in fewer repeated scans, and thus faster execution.

This reduces latency by reducing IO, and the benefits are available to all Postgres users without the need to change their SQL queries or ORM code!

Grouping similar query groups in pg_stat_statements

There are more usability improvements coming for Postgres users, pg_stat_statements, and IN clause queries.

One problem with these has been that similar entries aren’t collapsed together when they have a different numbers of scalar array expressions.

For example IN ('1') was not grouped with IN ('1','2'). Having the statistics for nearly identical entries split across multiple results makes them less useful.

Fortunately, fixes are coming. On the Ruby on Rails side, Sean Linsley is working on a fix by replacing the use of IN with ANY which solves the grouping problem.

Here’s the PR: https://github.com/rails/rails/pull/49388#issuecomment-2680362607

On the PostgreSQL side, there are fixes coming for PostgreSQL 18.

Improvements in PostgreSQL 18

Related improvements are coming to PostgreSQL 18 for 2025.

This commit¹ implements the automatic conversion of x IN (VALUES ...) into ScalarArrayOpExpr.

Another noteworthy commit is: “Squash query list jumbling” from Álvaro Herrera.²

pg_stat_statements produces multiple entries for queries like SELECT something FROM table WHERE col IN (1, 2, 3, ...) depending on the number of parameters, because every element of ArrayExpr is individually jumbled. Most of the time that’s undesirable, especially if the list becomes too large.

This commit³ mentions the original design was for a GUC query_id_squash_values, but that was removed in favor of making this the default behavior.

Conclusion

In this post, we looked at a problematic query pattern, big IN lists. You may have instances of this pattern in your codebase from direct means or from using some ORM methods.

This type of query performs poorly for big lists of values, as they take more resources to parse, plan, and execute. There are fewer indexing options compared with an alternative structured as a join operation. Join queries provide two sets of table statistics from both tables being joined, that help with query planning.

We learned how to find instances of these using pg_stat_statements for PostgreSQL. The post then considers several alternatives.

Our main tactics are to convert these queries to joins when possible. Outside of that, we could consider using the ANY operator with an array of values, a VALUES clause, and consider using a prepared statement.

The next time you see big IN lists causing database performance problems, hopefully you feel more prepared to restructure and optimize them!

Thanks for reading this post. I’d love to hear about any tips or tricks you have for these types of queries!

In this post, we’ll cover a way to generate short, alphanumeric, pseudo random identifiers using native Postgres tactics.

These identifiers can be used for things like transactions or reservations, where users need to read and share them easily. This approach is an alternative to using long, random generated values like UUID values, which have downsides for usability and performance.

We’ll call the identifier a public_id and store it in a column with that name. Here are some example values:

SELECT public_id
FROM transactions
ORDER BY random()
LIMIT 3;

 public_id
-----------
 0359Y
 08nAS
 096WV

Natural and Surrogate Keys

In database design, we can use natural or surrogate keys to identify rows. We won’t cover the differences here as that’s out of scope.

For our public_id identifier, we’re going to generate it from a conventional surrogate integer primary key called id. We aren’t using natural keys here.

The public_id is intended for use outside the database, while the id integer primary key is used inside the database to be referenced by foreign key columns on other tables.

Whle public_id is short which minimizes space and speeds up access, the main reason for it is for usability.

With that said, the target for total space consumption was to be fewer bytes than a 16-byte UUID. This was achieved with an integer primary key and this additional 5 character generated value, targeting a smaller database where this provides plenty of unique values now and into the future.

Let’s get into the design details.

Design Properties

Here were the desired design properties:

• A fixed size, 5 characters in length, regardless of the size of the input integer (and within the range of the integer data type)
• Fewer bytes of space than a uuid data type
• An obfuscated value, pseudo random, not easily guessable. While not easily guessable, this is not meant to be “secure”
• Reversibility back into the original integer
• Only native Postgres capabilities, no extensions, client web app language can be anything as it’s within Postgres
• Non math-heavy implementation

Additional details:

• public_id is stored using text not not char(5), following recommendations for best practices
• PL/PgSQL functions, native Postgres data types and constraints are used, like UNIQUE, NOT NULL, and CHECK, and a stored generated column.
• Converts integers to bits, uses exclusive-or (XOR) bitwise operation and modulo operations.

Limitations

• Did not set out to support case insensitivity now, possible future enhancement
• Did not try to exclude similar-looking characters (see: Base32 Crockford below), possible future enhancement

PL/PgSQL Functions

Here are the functions used:

This function obfuscates the integer value using exclusive or (XOR) obfuscation.

• Uses a Hexadecimal key 0x5A3C1 (make this any key you want)
• Sets a max value for the data type range 62^5, which is just under 1 billion possible values. This was enough for this system and into the future, but a bigger system would want to use bigint
• Converts integer bytes into bits

Main entrypoint function:

• obfuscate_id(id INTEGER)

This converts the obfuscated value into the public_id alphanumeric value, used within obfuscate_id().

This is “base 62” with the 26 upper and lower case characters, and 10 numbers (0-9).

• to_base62_fixed(val BIGINT, width INT DEFAULT 5)

Reverses the public_id back into the original integer.

• deobfuscate_id(public_id TEXT)

Used within deobfuscate_id():

• from_base62_fixed(str TEXT)

For a length of 5 with this system, we can create up to around ~1 billion unique values. This was sufficiently large for the original use case.

For use cases requiring more values, by storing 6 characters for public_id then up to ~56 billion values could be generated, based on a bigint primary key.

Table Design

Let’s create a sample transactions table with an integer primary key with a generated identity column.

Besides the use in the identity column, we’ll again use the GENERATED keyword to create a STORED column for the public_id.

The public_id column uses the id column as input, obfuscates it, encodes it to base 62, producing a 5 character value.

DROP TABLE IF EXISTS transactions;
CREATE TABLE transactions (
  id INTEGER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,  -- 4-byte integer ID
  public_id text GENERATED ALWAYS AS (obfuscate_id(id)) STORED UNIQUE NOT NULL, -- 5-character obfuscated Base62 value
  amount NUMERIC,
  description TEXT
);

How do we guarantee public_id conforms to our expected data properties? Constraints!

• public_id gets a UNIQUE constraint and NOT NULL, so we know we have a unique value
• A CHECK constraint is added to validate the length

For an existing system, we could add a unique index CONCURRENTLY first as follows:

CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS idx_uniq_pub_id ON transactions (public_id);

Then we can add the unique constraint using the unique index, along with the CHECK constraint:

ALTER TABLE transactions
    ADD CONSTRAINT uniq_pub_id UNIQUE USING INDEX idx_uniq_pub_id, -- depends on index above
    ADD CONSTRAINT public_id_length CHECK (LENGTH(public_id) <= 5);

Insert Data

Let’s insert data into the transactions table:

INSERT INTO transactions (amount, description) VALUES
  (100.00, 'First transaction'),
  (50.00, 'Second transaction'),
  (0.25, 'Third transaction');

Let’s query the data, and also make sure it’s reversed (using the deobfuscate_id(public_id TEXT) function) properly:

Access the rows

SELECT
    id,
    public_id,
    deobfuscate_id(public_id) AS reversed_id,
    description
FROM
    transactions;


 id | public_id | reversed_id |    description
----+-----------+-------------+--------------------
  1 | 01Y9I     |           1 | First transaction
  2 | 01Y9L     |           2 | Second transaction
  3 | 01Y9K     |           3 | Third transaction

Additional time spent on inserts

Let’s compare the time spent inserting 1 million rows into an equivalent transactions table without the public_id column or value generation.

That took an average of 2037.906 milliseconds, or around 2 seconds on my machine.

Inserting 1 million rows with the public_id took an average of 6954.070 or around 7 seconds, or about 3.41x slower. Note that these times were with the indexes and constraints in place on the transactions table in the second example, but not the first, meaning their presence contributed to the total time.

Summary: Creating this identifier made the write operations 3.4x slower for me locally, which was an acceptable amount of overhead for the intended use case.

Performance

Compared with random values, the pseudo random public_id remains orderable, which means that lookups for individual rows or ranges of rows can use indexes, running fast and reliably even as row counts grow.

We can add a unique index on the public_id column like this:

CREATE UNIQUE INDEX CONCURRENTLY
IF NOT EXISTS
idx_uniq_pub_id ON transactions (public_id);

We can verify that individual lookups or range scans use this index, by inspecting query execution plans for this table.

PL/pgSQL Source Code

https://github.com/andyatkinson/pg_scripts/pull/15

Feedback

Feedback on this approach is welcomed! Please use my contact form to provide feedback or leave comments on the PR.

Future enhancements to this could include unit tests using pgTAP for the functions, packaging them into an extension, or supporting more features like case insensitivity or a modified input alphabet.

Thanks for reading!

Alternatives

• Base32 Crockford - An emphasis ease of use for humans: removing similar looking characters, case insensitivity.
• ULID - Also 128 bits/16 bytes like UUIDs, so I had ruled these out for space consumption, and they’re slightly less “usable”
• NanoIDs at PlanetScale - I like aspects of NanoID. This is random generation though like UUID vs. encoding a unique integer.