The first internal item is an “SLRU.” The acronym stands for “simple least recently used.” The LRU portion refers to caches and how they work, and SLRUs in Postgres are a collection of these caches.

SLRUs are small in-memory item stores. Since they need to persist across restarts, they’re also saved into files on disk. Alvaro¹ calls SLRUs “poorly named” for a user-facing feature. If they’re internal, why are they worth knowing about as Postgres users?

They’re worth knowing about because there can be a couple of possible failure points with them, due their fixed size. We’ll look at those later in this post.

Before getting into that, let’s cover some basics about what they are and look at a specific type.

Main purpose of SLRUs

The main purpose of SLRUs is to track metadata about Postgres transactions.

SLRUs are a general mechanism used by multiple types. Like a lot of things in Postgres, the SLRU system is extensible which means extensions can create new types.

The “least recently used” aspect might be recognizable from cache systems. LRU refers to how the oldest items are evicted from the cache when it’s full, and newer items take their place. This is because the cache has a fixed amount of space (measured in 8KB pages) and thus can only store a fixed amount of items.

Old SLRU cache items are periodically cleaned up by the Vacuum process.

What about the buffer cache?

The buffer cache (sized by configuring shared_buffers) is another form of cache in Postgres. Thomas Munro proposed unifying the SLRUs and buffer cache mechanisms.

However, as of Postgres 17 and the upcoming 18 release (released September 9, 2025), SLRUs are still their own distinct type of cache.

What types of data is stored in SLRUs?

What type of data is tracked in SLRUs?

Transactions are a core concept for relational databases like Postgres. Transactions are abbreviated “Xact,” and Xacts are one of the types of data stored in SLRUs.

Besides regular transactions, there are variations of transactions. Transactions can be created inside other transactions, which are called “nested transactions.”

Whether parent or nested transactions, they each get their own 32-bit integer identifier once they begin modifying something, and these are all tracked while they’re in use. The SAVEPOINT keyword (blog post: You make a good point! — PostgreSQL Savepoints saves the incremental status for a transaction.

Another variation of a transaction is a “multi-transaction,” (multiple transactions in a group) or “MultiXact” in Postgres speak.

What are MultiXacts?

A MultiXact gets a separate number from the transaction identifier. I think of it like a “group” number. The group might be related to a table row, but each transaction in the group is doing something different. Think of multiple transactions all doing a foreign key referential integrity check on the same referenced primary key.

Here’s a definition of MultiXact IDs:

A MultiXact ID is a secondary data structure that tracks multiple transactions holding locks on the same row.

When MultiXacts are created, their identifier is stored in tuple header info, replacing the transaction id that would normally be stored in the tuple header.

As this buttondown blog post (“Notes on some PostgreSQL implementation details”)² describes, the tuple (row version) header has a small fixed size. The MultiXact id replaces the transaction id using the same size identifier (but a different one), to keep the tuple header size small (as opposed to adding another identifier).

Transaction IDs and MultiXact IDs are both represented as a unsigned 32-bit integer, meaning it’s possible to store a max of around ~4 billion values (See: Transactions and Identifiers. We can get the current transaction id value by running select pg_current_xact_id();.

What do we mean by transaction metadata? One example is with nested transactions, the parent transaction, the “creator”.

If you’d like to read how AWS introduces MultiXacts, check out this post. This post describes them: What are MultiXacts? https://aws.amazon.com/blogs/database/multixacts-in-postgresql-usage-side-effects-and-monitoring/

When do MultiXacts get created?

When do MultiXacts get created?

MultiXacts get created only for certain types of DML operations and for certain schema definitions. In other words, it’s possible that your particular Postgres database workload does not create MultiXacts at all, or it’s possible they’re heavily used. Let’s look at what creates MultiXacts:

• Foreign key constraint enforcement
• SELECT FOR SHARE

If you use no foreign key constraints or your application (or ORM) never creates SELECT FOR SHARE, then your Postgres database may have no MultiXacts.

Let’s go back to SLRUs.

More about SLRUs

SLRUs have a fixed size (prior to Postgres 17) measured in pages. When items are evicted from the SLRU cache, a page replacement occurs.

The page being replaced is called the “victim” page and Postgres must do a little work to find a victim page. Since SLRUs survive Postgres restarts, they’re saved in files in the PGDATA directory.

The directory name will depend on the SLRU type. For example for MultiXacts, the directory name is pg_multixact. SLRU buffer pages are written to the WAL and to disk, meaning that if the primary instance fails, the state can be recovered.

See the slru.c SlruPhysicalWritePage function comments which describes writing WAL and writing out data:

Honor the write-WAL-before-data rule, if appropriate, so that we do not
write out data before associated WAL records.

Each SLRU instance implements a circular buffer of pages in shared memory, evicting the least recently used pages. A circular buffer is another interesting Postgres internal concept but is beyond the scope of this post. How can we observe what’s happening with SLRUs?

Using pg_stat_slru

Since Postgres 13, we have the system view “pg_stat_slru” to query to inspect cumulative statistics about the SLRUs. https://www.postgresql.org/docs/current/monitoring-stats.html#PG-STAT-SLRU-VIEW To list only the names of the built-in SLRU types:

select name from pg_stat_slru;
      name
-----------------
 CommitTs
 MultiXactMember
 MultiXactOffset
 Notify
 Serial
 Subtrans
 Xact
 Other

To determine if our system is creating MultiXact SLRUs, we can query the pg_stat_slru view. We’d see non-zero numbers in rows below when the system is creating SLRU data.

select name from pg_stat_slru;
                     View "pg_catalog.pg_stat_slru"
    Column    |           Type           | Collation | Nullable | Default 
--------------+--------------------------+-----------+----------+---------
 name         | text                     |           |          | 
 blks_zeroed  | bigint                   |           |          | 
 blks_hit     | bigint                   |           |          | 
 blks_read    | bigint                   |           |          | 
 blks_written | bigint                   |           |          | 
 blks_exists  | bigint                   |           |          | 
 flushes      | bigint                   |           |          | 
 truncates    | bigint                   |           |          | 
 stats_reset  | timestamp with time zone |           |          |

To look at the pg_xact SLRU:

select * from pg_stat_slru where name = 'Xact';
 name | blks_zeroed | blks_hit | blks_read | blks_written | blks_exists | flushes | truncates |          stats_reset
------+-------------+----------+-----------+--------------+-------------+---------+-----------+-------------------------------
 Xact |         460 | 30686596 |        44 |         2030 |           0 |    1684 |         0 | 2024-11-19 09:52:33.506794-06

“Hit” and “read” refer to reads from the SLRU that where the desired pages were already in the SLRU or they were not.

When new pages are allocated, we see this reflected in “blks_zeroed” as they’re written out with zeroes.

When new pages are written (blks_written) into the SLRU this creates “dirtied” pages that eventually will be written out (flushes).

SLRUs can also be truncated (“Truncates” count).

Some of the source code for SLRUs in Postgres is in the file backend/access/transam/slru.c. https://github.com/postgres/postgres/blob/master/src/backend/access/transam/slru.c

Now that we know some basics about SLRUs and a specific type, the MultiXact SLRU, what are some operational concerns or things that can go wrong?

What can go wrong with SLRUs and Xacts?

Operational problems can stem from the fact that SLRUs use a 32-bit number and for high scale Postgres, it’s possible to consume these fast enough that the number can “wrap around.”

Two examples with public write-ups related to SLRU operational problems are:

• Subtransactions overflow: Using subtransactions, each use of a subtransaction creates an id to track. At a high enough creation rate it’s possible to run out of values. This was written up in the GitLab post: Why we spent the last month eliminating PostgreSQL subtransactions.

MultiXact member space exhaustion: MultiXact or multiple transactions can occur in a few scenarios.

• An explicit row lock: SELECT … FOR SHARE
• SELECT … FOR UPDATE

Written up in the Metronome blog post: Root Cause Analysis: PostgreSQL MultiXact member exhaustion incidents (May 2025).

A scenario for that could be a foreign key constraint lookup on a high insert table referencing a low cardinality table.

Another type of problem in the buttondown post² is the quadratic growth of MultiXacts.

Dilip Kumar talked about: “Long running transaction, system can go fully to cache replacement, TPS drops, with subtransactions ids (need to get parent ids).” See Dilip’s presentation for more info.³

What do we do with info as Postgres operators?

This is a huge topic and this post just scratches the surface.

However, let’s wrap this up here a bit with some takeaways.

If operating a high scale Postgres instance when it comes to SLRUs, what’s worth knowing about?

• Know about the SLRU system in general, how to monitor it, and don’t forget about extensions
• Learn about SLRUs limitations and possible failure points, for the various types
• Determine whether your workload is using SLRUs, monitor their growth, and learn about the possible failure points based on your use

What’s changing with SLRUs in new Postgres versions?

In Postgres 17, the MultiXact member space and offset is now configurable beyond the initial default size. The unit is the number of 8KB pages. The default size is X and Y and this is configurable.

• multixact_member_buffers, default is 32 8kb pages
• multixact_offset_buffers, default is 16 8kb pages

In the recent episode of postgres.fm MultiXact member space exhaustion,⁴ the Metronome engineers discussed working on a patch related to MultiXact member exhaustion.

Lukas covers changes in Postgres 17 to adjust SLRU cache sizes. Each of the SLRU types can now be configured to be larger in size. https://pganalyze.com/blog/5mins-postgres-17-configurable-slru-cache

Conclusion

I’m still learning about MultiXacts, SLRUs, and failure modes as a result of these. If you have feedback on this post or additional useful resources, I’d love to hear about them. Please contact me here or on social media.

Thanks for reading!

Resources

Dilip Kumar presentation 2024 - PostgreSQL Development Conference https://www.youtube.com/watch?v=74xAqgS2thY

MultiXacts Dan Slimmon https://blog.danslimmon.com/2023/12/11/concurrent-locks-and-multixacts-in-postgres/

5 minutes of Postgres LWLock Lock Manager https://pganalyze.com/blog/5mins-postgres-LWLock-lock-manager-contention

SLRU Improvements Proposals Wiki https://wiki.postgresql.org/wiki/SLRU_improvements

Corrections

September 27, 2025: An earlier version of this post inaccurately described SLRU buffers as not being WAL logged. Thank you to Laurenz Albe for writing in to correct this and providing a pointer into the source code to learn more.

Over the last decade, when working on databases with UUID Version 4¹ as the primary key data type, these databases have usually had bad performance and excessive IO.

UUID is a native data type in Postgres stored as binary data. Various UUID versions are in the RFC. Version 4 has mostly random bits, obfuscating information like when the value was created or where it was generated.

Version 4 UUIDs are easy to generate in Postgres using the gen_random_uuid()² function since version 13 (released in 2020).

I’ve learned there are misconceptions about UUID Version 4, and sometimes these are the reasons users pick this data type.

Because of the poor performance, misconceptions, and available alternatives, I’ve come around to a simple position: Avoid UUID Version 4 for primary keys.

My more controversial take is to avoid UUIDs in general, but I understand there are some legitimate reasons for them without practical alternatives.

As a database enthusiast, I wanted to have an articulated position on this classic “Integer v. UUID” debate.

Among databases folks, debating this may be tired and clichéd. However, from my consulting work, I can say I work with databases using UUID v4 in 2024 and 2025, and still see the issues discussed in this post.

Let’s dig in.

UUID context for this post

• UUIDs (or GUID in Microsoft speak)³) are long strings of 36 characters, 32 digits, 4 hyphens, stored as 128 bits (16 byte) values, stored using the binary uuid data type in Postgres
• The RFC documents how the 128 bits are set
• The bits for UUID Version 4 are mostly random values
• UUID Version 7 includes a timestamp in the first 48 bits, which works much better with database indexes compared with random values

Although unreleased as of this writing, and pulled from Postgres 17 previously, UUID V7 is part of Postgres 18⁴ scheduled for release in the Fall of 2025.

What kind of app databases are in scope for this post?

Scope of web app usage and their scale

The kinds of web applications I’m thinking of with this post are monolithic web apps, with Postgres as their primary OLTP database. The apps could be in categories like social media, e-commerce, click tracking, or business process automation apps.

The types of performance issues discussed here are related to inefficient storage and retrieval, meaning they happen for all of these types of apps.

What’s the core issue with UUID v4?

Randomness is the issue

The core issue with UUID Version 4, given that the 122 bits they’re made up of are “random or pseudo-randomly generated values”¹, is how the values are maintained in indexes. Since primary keys are backed by indexes by default, each insert is less efficient compared with inserts for sequentially ordered values.

For lookups, each update and delete for individual items or for ranges of items are less efficient, due to increased traversal of non-sequential index pages in Postgres.

Since the randomly generated values aren’t inserted sequentially (or in sequential/adjacent pages), it’s less efficient to find them later for updates or deletes. Each of these workload types use the primary key index.

UUID v4s don’t have a useful natural ordering that aligns with how they’re stored, and thus both storage and retrieval is less efficient.

Later in the post we’ll look at just how many more Postgres pages need to be accessed for equivalent data, and what that means in terms of performance.

Despite the inefficiencies, UUID v4s and UUIDs in general remain (or at least were) popular in the last decade based on my experience consulting in Postgres.

Given the popularity, what use cases for UUID are there?

Why choose UUIDs at all? Generating values from one or more client applications

One use case for UUIDs is when there’s a need to generate an identifier on a client or from multiple services, then passed to Postgres for persistence.

For web apps, generally they instantiate objects in memory and don’t expect an identifier to be used for lookups until after an instance is persisted as a row (where the database generates the identifier).

In a microservices architecture where the apps have their own databases, the ability to generate identifiers from each database without collisions is a use case for UUIDs. The UUID could also identify the database a value came from later, vs. an integer.

For collision avoidance (see HN discussion⁵), we can’t practically make the same guarantee with sequence-backed integers. There are hacks, like generating even and odd integers between two instances, or using different ranges in the int8 range.

There are also alternative identifiers like using composite primary keys (CPKs), however the same set of 2 values wouldn’t uniquely identify a particular table.

The avoidance of collisions is described this way on Wikipedia:⁶

The number of random version-4 UUIDs which need to be generated in order to have a 50% probability of one collision: 2.71 quintillion

This number would be equivalent to:

Generating 1 billion UUIDs per second for about 86 years.

Are UUIDs secure?

Misconceptions: UUIDs are secure

One misconception about UUIDs is that they’re secure. However, the RFC describes that they shouldn’t be considered secure “capabilities.”

From RFC 4122¹ Section 6 Security Considerations:

Do not assume that UUIDs are hard to guess; they should not be used as security capabilities

How can we create obfuscated codes from integers?

Creating obfuscated values using integers

While UUID V4s obfuscate their creation time, the values can’t be ordered to see when they were created relative to each other. We can achieve those properties with integers with a little more work.

One option is to generate a pseudo-random code from an integer, then use that value externally, while still using integers internally.

To see the full details of this solution, please check out: Short alphanumeric pseudo random identifiers in Postgres⁷

We’ll summarize it here.

• Convert a decimal integer like “2” into binary bits. E.g. a 4 byte, 32 bit integer: 00000000 00000000 00000000 00000010
• Perform an exclusive OR (XOR) operation on all the bits using a key
• Encode each bit using a base62 alphabet

The obfuscated id is stored in a generated column. By reviewing the generated values, they are similar, but aren’t ordered by their creation order.

The values in insertion order were 01Y9I, 01Y9L, then 01Y9K.

With alphabetical order, the last two would be flipped: 01Y9I first, then 01Y9K second, then 01Y9L third, sorting on the fifth character.

If I wanted to use this approach for all tables, I’d try a centralized table that was polymorphic, storing a record for each table that’s using a code (and a foreign key constraint).

That way I’d know where the code was used.

Why else might we want to skip UUIDs?

Reasons against UUIDs in general: they consume a lot of space

UUIDs are 16 bytes (128 bits) per value, which is double the space of bigint (8 bytes), or quadruple the space of 4-byte integers. This extra space adds up once many tables have millions of rows, and copies of a database are being moved around as backups and restores.

A more considerable impact to performance though is the poor characteristics of writing and reading random data into indexes.

Reasons against: UUID v4s add insert latency due to index page splits, fragmentation

For random UUID v4s, Postgres incurs more latency for every insert operation.

For integer primary key rows, their values are maintained in index pages with “append-mostly” operations on “leaf nodes,” since their values are orderable, and since B-Tree indexes store entries in sorted order.

For UUID v4s, primary key values in B-Tree indexes are problematic.

Inserts are not appended to the right most leaf page. They are placed into a random page, and that could be mid-page or an already-full page, causing a page split that would have been unnecessary with an integer.

Planet Scale has a nice visualization of index page splits and rebalancing.⁸

Unnecessary splits and rebalancing add space consumption and processing latency to write operations. This extra IO shows up in Write Ahead Log (WAL) generation as well.

Buildkite reported a 50% reduction in write IO for the WAL by moving to time-ordered UUIDs.

Given fixed size pages, we want high density within the pages. Later on we’ll use pageinspect to check the average leaf density between integer and UUID to help compare the two.

Excessive IO for lookups even with orderable UUIDs

B-Tree page layout means you can fit fewer UUIDs per 8KB page. Since we have the limitation of fixed page sizes, we at least want them to be as densely packed as possible.

Since UUID indexes are ~40% larger in leaf pages than bigint (int8) for the same logical number of rows, they can’t be as densely packed with values. As Lukas says, “All in all, the physical data structure matters as much as your server configuration to achieve the best I/O performance in Postgres.”⁹

This means that for individual lookups, range scans, or UPDATES, we will incur ~40% more I/O on UUID indexes, as more pages are scanned. Remember that even to access one row, in Postgres the whole page is accessed where the row is, and copied into a shared memory buffer.

Let’s insert and query data and take a look at numbers between these data types.

Working with integers, UUID v4, and UUID v7

Let’s create integer, UUID v4, and UUID v7 fields, index them, load them into the buffer cache with pg_prewarm.

I will use the schema examples from the Cybertec post Unexpected downsides of UUID keys in PostgreSQL by Ants Aasma.

View andyatkinson/pg_scripts PR #20.

On my Mac, I compiled the pg_uuidv7 extension. Once compiled and enabled for Postgres, I could use the extension functions to generate UUID V7 values.

Another extension pg_prewarm is used. It’s a module included with Postgres, so it just needs to be enabled per database where it’s used.

The difference in latency and the enormous difference in buffers from the post was reproducible in my testing.

“Holy behemoth buffer count batman” - Ants Aasma

Cybertec post results:

• 27,332 buffer hits, index only scan on the bigint column
• 8,562,960 buffer hits, index only scan on the UUID V4 index scan

Since these are buffer hits we’re accessing them from memory, which is faster than disk. We can focus then on only the difference in latency based on the data types.

How many more pages are accessed for the UUID index? 8,535,628 (8.5 million!) more 8KB pages were accessed, a 31229.4% increase. In terms of MB and MB/s that is:

• 68,285,024 MB or ~68.3 GB! more data that’s accessed

Calculating a low and high estimate of access speeds for memory:

• Low estimate: 20 GB/s
• High estimate: 80 GB/s

Accessing 68.3 GB of data from memory (shared_buffers in PostgreSQL) would add:

• ~3.4 seconds of latency (low speed)
• ~0.86 seconds of latency (high speed)

That’s between ~1 and ~3.4 seconds of additional latency solely based on the data type. Here we used 10 million rows and performed 1 million updates, but the latencies will get worse as data and query volumes increase.

Inspecting density with the pageinspect extension

We can inspect the average fill percentage (density) of leaf pages using the pageinspect extension.

The uuid_experiments/page_density.sql (andyatkinson/pg_scripts PR #20) query in the repo gets the indexes for the integer and v4 and v7 uuid columns, their total page counts, their page stats, and the number of leaf pages.

Using the leaf pages, the query calculates an average fill percentage.

After performing the 1 million updates on the 10 million rows mentioned in the example, I got these results from that query:

 idxname             | avg_leaf_fill_percent
---------------------+-----------------------
 records_id_idx      |                 97.64
 records_uuid_v4_idx |                 79.06
 records_uuid_v7_idx |                 90.09
(3 rows)

This shows the integer index had an average fill percentage of nearly 98%, while the UUID v4 index was around 79%.

UUID Downsides: Worse cache hit ratio

The Postgres buffer cache is a critical part of good performance.

For good performance, we want our queries to produce cache “hits” as much as possible.

The buffer cache has limited space. Usually 25-40% of system memory is allocated to it, and the total database size including table and index data is usually much larger than that amount of memory. That means we’ll have trade-offs, as all data will not fit into system memory. This is where the challenges come in!

When pages are accessed they’re copied into the buffer cache as buffers. When write operations happen, buffers are dirtied before being flushed.¹⁰

Since the UUIDs are randomly located, additional buffers will need to be copied to the cache compared to ordered integers. Buffers might be evicted to make space that are needed, decreasing hit rates.

Mitigations: Rebuilding indexes with UUID values

Since the tables and indexes are more likely to be fragmented, it makes sense to rebuild the tables and indexes periodically.

Rebuilding tables can be done using pg_repack, pg_squeeze, or VACUUM FULL if you can afford to perform the operation offline.

Indexes can be rebuilt online using REINDEX CONCURRENTLY.

While the newly laid out data in pages, they will still not have correlation, and thus not be smaller. The space formerly occupied by deletes will be reclaimed for reuse though.

Mitigation: Shared buffers and work_mem memory sizing

If possible, size your primary instance to have 4x the amount of memory of your size of database. In order words if your database is 25GB, try and run a 128GB memory instance.

This gives around 32GB to 50GB of memory for buffer cache (shared_buffers) which is hopefully enough to store all accessed pages and index entries.

Use pg_buffercache¹¹ to inspect the contents, and pg_prewarm¹² to populate tables into it.

One tactic I’ve used when working with UUID v4 random values where sorting is happening, is to provide more memory to sort operations.

To do that in Postgres, we can change the work_mem setting. This setting can be changed for the whole database, a session, or even for individual queries.

Check out Configuring work_mem in Postgres on PgMustard for an example of setting this in a session.

Mitigation in Rails: UUID and implicit order column Active Record

Since Rails 6, we can control implicit_order_column.¹³ The database_consistency gem even has a checker for folks using UUID primary keys.

When ORDER BY is generated in queries implicitly, it may be worth ordering on a different high cardinality field that’s indexed, like a created_at timestamp field.

Mitigating poor performance by clustering on orderable field

Cluster on a column that’s high cardinality and indexed could be a mitigation option.

For example, imagine your UUID primary table has a created_at timestamp column that’s indexed with idx_on_tbl_created_at, and clustering on that.

CLUSTER table_with_uuid_ok USING idx_on_tbl_created_at;

I don’t see CLUSTER used ever really though as it takes an access exclusive lock. The CLUSTER is a one-time operation that would also need to be repeated regularly to maintain its benefits.

Recommendation: Stick with sequences, integers, and big integers

For new databases that may be small, with unknown growth, I recommend plain old integers and an identity column (backed by a sequence)¹⁴ for primary keys. These are signed 32 bit (4-byte) values. This provides about 2 billion positive unique values per table.

For many business apps, they will never reach 2 billion unique values per table, so this will be adequate for their entire life. I’ve also recommended always using bigint/int8 in other contexts.

I guess it comes down to what you know about your data size, how you can project growth. There are plenty of low growth business apps out there, in constrained industries, and constrained sets of business users.

For Internet-facing consumer apps with expected high growth, like social media, click tracking, sensor data, telemetry collection types of apps, or when migrating an existing medium or large database with 100s of millions or billions of rows, then it makes sense to start with bigint (int8), 64-bit, 8-byte integer primary keys.

UUID v4 alternatives: Use time-ordered UUIDs like Version 7

Since Postgres 18 is not yet released, generating UUID V7s now in Postgres is possible using the pg_uuidv7 extension.

If you have an existing UUID v4 filled database and can’t afford a costly migration to another primary key data type, then starting to populate new values using UUID v7 will help somewhat.

Fortunately the binary uuid data type in Postgres can be used whether you’re storing V4 or V7 UUID values.

Another alternative that relies on an extension is sequential_uuids.¹⁵

Summary

• UUID v4s increase latency for lookups, as they can’t take advantage of fast ordered lookups in B-Tree indexes
• For new databases, don’t use gen_random_uuid() for primary key types, which generates random UUID v4 values
• UUIDs consume twice the space of bigint
• UUID v4 values are not meant to be secure per the UUID RFC
• UUID v4s are random. For good performance, the whole index must be in buffer cache for index scans, which is increasingly unlikely for bigger data.
• UUID v4s cause more page splits, which increase IO for writes with increased fragmentation, and increased size of WAL logs
• For non-guessable, obfuscated pseudo-random codes, we can generate those from integers, which could be an alternative to using UUIDs
• If you must use UUIDs, use time-orderable UUIDs like UUID v7

Do you see any errors or have any suggested improvements? Please contact me. Thanks for reading!

Learn More

• Franck Pachot for AWS Heroes has an interesting take on UUID in PostgreSQL
• Brandur has a great post: Identity Crisis: Sequence v. UUID as Primary Key
• 5mins of Postgres: UUIDs vs Serial for Primary Keys - what’s the right choice?
• andyatkinson/pg_scripts PR #20

Updates

• 2025-12-15: Appeared on front page of Hacker News. Updating the “Randomness is the issue” section for improved clarity.

In this post, we’ll cover some database design principles and package them up into a catchy mnemonic acronym.

Software engineering is loaded with acronyms like this. For example, SOLID principles describe 5 principles, Single responsibility, Open-closed, Liskov substitution, Interface segregation and Dependency inversion, that promote good object-oriented design.

Databases are loaded with acronyms, for example “ACID” for the properties of a transaction, but I wasn’t familiar with one the schema designer could keep in mind while they’re working.

Thus, the motivation for this acronym was to help the schema designer, by packaging up some principles of good design practices for database schema design. It’s not based in research or academia though, so don’t take this too seriously. That said, I’d love your feedback!

Let’s get into it.

Picking a mnemonic acronym

In picking an acronym, I wanted it to be short and have each letter describe a word that’s useful, practical, and grounded in experience. I preferred a real word for memorability!

The result was “CORE.” Let’s explore each letter and the word behind it.

Constraint-Driven

The first word (technically two) is “constraint-driven.” Relational databases offer rigid structures, but the ability to be changed while online, a form of flexibility in their evolution. We evolve their structure through DDL. They use data types and constraints to make changes, as entities and relationships evolve.

Constraint-driven refers to leveraging all the constraint objects available, designing for our needs today, but also in a more general sense applying constraints (restrictions) to designs in the pursuit of data consistency and quality.

Let’s look at some examples. Choose the appropriate data types, like a numeric data type and not a character data type when storing a number. Use NOT NULL for columns by default. Create foreign key constraints for table relationships by default.

Validate expected data inputs using check constraints. For small databases, use integer primary keys. If tables get huge later, no problem, we can migrate the data into a bigger more suitable structure.

The mindset is to prefer rigidity initially, design for today, then leverage the flexibility available to evolve later, as opposed to designing for a hypothetical future state.

Optimized

Databases present loads of optimization opportunities. Relational data is initially stored in a normalized form to eliminate duplication, but later denormalizations can be performed when read access is more important.

When our use cases are not known at the outset, plan to iterate on the design, changing the structure to better support the use cases that emerge. This will mean evolving the schema design.

This applies to tables, columns, constraints, indexes, parameters, queries, and anything that can be optimized to better support real use cases.

Queries are restructured and indexes are added to reduce data access. Strive for highly selective data access (a small proportion of rows) on high cardinality (uniqueness) data to reduce latency.

Critical background processes like VACUUM get optimized too. Resources (workers, memory, parallelization) are increased proportionally.

Responsive

When problems emerge like column or row level unexpected data, missing referential integrity, or query performance problems, engineers inspect logs, catalog statistics, and parameters, from the core engine and third party extensions to diagnose issues.

When DDL changes are ready, the engineer applies them in a non-blocking way, in multiple steps as needed. Operations are performed “online” by default when practical.

DDL changes are in a source code file, reviewed, tracked, and a copy of the schema design is kept in sync across environments.

Parameter (GUC) tuning (Postgres: work_mem, etc.) happens in a trackable way. Parameters are tuned online when possible, and scoped narrowly, to optimize their values for real queries and use cases.

Efficient

It’s relatively costly to store data in the database, compared with file storage! The data consumes limited space and accessing data unnecessarily adds latency.

Data that’s stored is queried later or it’s archived.

To minimize space consumption and latency, tables, columns, constraints, and indexes are removed continually by default, when they no longer are required, to reduce system complexity.

Server software is upgraded at least annually so that performance and security benefits can be leveraged.

Huge tables are split into smaller tables using table partitioning for more predictable administration.

CORE Database Design

There’s lots more to evolving a database schema design, but these principles are a few I keep in mind.

Did you notice anything missing? Do you have other feedback? Please contact me with your thoughts.

Thank You

Over the years, I’ve learned a lot from Postgres.fm hosts Nikolay and Michael, and other community leaders like Lukas and Franck, as they’ve shaped my database design choices.

I’m grateful to them for sharing their knowledge and experience with the community.

Thanks for reading!

Much of the time taken processing HTTP requests in web apps is SQL queries. To minimize that, we want to avoid unnecessary and duplicate queries, and generally perform as few queries as possible.

Think of the work that needs to happen for every query. The database engine parses it, creates a query execution plan, executes it, and then sends the response to the client.

When the response reaches the client, there’s even more work to do. The response is transformed into application objects in memory.

How do we see how many queries are being created for our app actions?

Count the queries

When doing backend work in a web app like Rails, monitor the number of queries being created directly, by the ORM, or by libraries. ORMs like Active Record can generate more than one query from a given line of code. Libraries can generate queries that are problematic and may be unnecessary.

Over time, developers may duplicate queries unknowingly. These are all real causes of unnecessary queries from my work experience.

Why are excessive queries a problem?

Why reduce the number of queries?

Besides parsing, planning, executing, and serializing the response, the client is subject to a hard upper limit on the number of TCP connections it can send to the database server.

In Postgres that’s configured as max_connections. The application will have a variable number of open connections based on use, and its configuration of processes, threads and its connection pool. Keeping the query count low helps avoid exceeding the upper limit.

What about memory use?

What about app server memory?

With Ruby on Rails, the cost of repeated queries is shifted because the SQL Cache is enabled by default, which stores and serves results for matching repeated queries, at the cost of some memory use.

As an side, from Rails 7.1 the SQL Cache uses a least recently used (LRU) algorithm. We can also configure the max number of queries to cache, 100 by default, to control how much memory is used.

Counting queries prior to Rails 7.2

Prior to Rails 7.2, I recommend adding the query_count gem which does a simple thing, it shows the count of SQL queries processed for an action.

The count is in the Rails log file like this: SQL Queries: 100 (50 cached). In this case, 100 queries were performed and 50 used the SQL Cache.

Built-in from Rails 7.2 onward

From Rails 7.2 onward, the count of queries is now built in, so query_count is no longer needed.

Rails 7.2 onward looks like this: ActiveRecord: 105.5ms (10 queries, 1 cached). Here 10 queries ran, and 1 used the SQL Cache.

Repeated queries

While the SQL Cache saves the roundtrip for a repeated query, ideally we want to eliminate the repeated query. It’s worth hunting for it and considering refactoring or restructuring data access.

Another tactic is using memoization to store results for the duration of processing one controller action. Read more about that: Speeding up Rails with Memoization.

How do I get started?

Finding the source code location of the queries

To get started, identify some slow API endpoints in production, run them locally in development, and begin monitoring their quantity of SQL queries. Find the Source code locations for database queries in Rails with Marginalia and Query Logs.

Determine how to factor out data access that can be shared.

How many queries are “a lot?”

It’s hard to give a generic number. However, duplicate queries are a category to remove.

Let’s say you’ve got a Book model for your bookstore app. Scan your Rails log file for a pattern like this:

Book Load (4.3ms) …
Book Load (5.0ms) …
Book Load (0.5ms) …
Book Load (2.3ms) …

If you see that sort of pattern, track down the source locations, and eliminate any repeated loads. Let’s assume this is not a N + 1 queries problem, but repeated access to the same data from different source code locations.

You may be able to factor out and consolidate a data load. You may be able to use an existing loaded collection for an existence check, or use memoization to use previously calculated results.

Using these tactics, I’ve reduced controller actions with 250+ SQL queries (a ton!) to 50 or fewer (still a lot), by going through these steps. Monitor the log, find source locations for first party code, ORM generated queries, query code from libraries (gems), Rails controller action “before filters,” and other sources, then eliminate and consolidate.

When faced with a lot of queries, I find it helpful to study the bare minimum of what’s needed by the client, working outside in, then look to see if it’s possible to reduce the tables, rows, and columns to only what’s needed.

Wrap Up

• Track the count of SQL queries performed in different versions of Rails
• Remove unnecessary queries so they don’t use limited system resources
• Eliminate repeated queries to keep the count as low as possible
• Only access data that’s needed for client application use cases

If you’ve created web apps with relational databases and ORMs like Active Record (part of Ruby on Rails), you’ve probably experienced database performance problems after a certain size of data and query volume.

In this post, we’re going to look at a specific type of problematic query pattern that’s somewhat common.

We’ll refer to this pattern as “Big INs,” which are queries with an IN clause that has a big list of values. As data grows, the length of the list of values will grow. These queries tend to perform poorly for big lists, causing user experience problems or even partial outages.

We’ll dig into the origins of this pattern, why the performance of it is poor, and explore some alternatives that you can use in your projects.

IN clauses with a big list of values

The technical term for values are a parenthesized list of scalar expressions.

For example in the SQL query below, the IN clause portion is WHERE author_id IN (1,2,3) and the list of scalar expressions is (1,2,3).

SELECT * FROM books
WHERE author_id IN (1, 2, 3);

The purpose of this clause is to perform filtering. Looking at a query execution plan in Postgres, we’ll see something like this fragment below:

Filter: (author_id = ANY ('{1,2,3}'::integer[]))

This of course filters the full set of books down to ones that match on author_id.

Filtering is a typical database operation. Why are these slow?

Parsing, planning, and executing

Remember that our queries are parsed, planned, and executed. A big list of values are treated like constants, and don’t have associated statistics.

Queries with big lists of values take more time to parse and use more memory.

Without pre-collected table statistics for planning decisions, PostgreSQL is more likely to mis-estimate cardinality and row selectivity.

This can mean the planner chooses a sequential scan over an index scan, causing a big slowdown.

How do we create this pattern?

Creating this pattern directly

In Active Record, a developer might create this query pattern by using pluck(:id) to collect some ids in a list, then pass that list as an argument to another query.

Here’s an example of that:

author_ids = Author.
  where("created_at >= ?", 1.year.ago).
  pluck(:id)

The author_ids are supplied as the argument querying books by author_id foreign key:

Book.where(author_id: author_ids)

Another scenario is when this query is created from ORM methods. What does that look like?

Active Record ORM methods that create this pattern

This query pattern can happen when using eager loading methods like includes() or preload().

This Crunchy Data post mentions how eager loading methods produce IN clause SQL queries.

The post links to the Eager Loading Associations documentation which has examples in Active Record and the resulting SQL that we’ll use here.

Let’s first discuss N+1 with these examples.

Fixing N+1s

Let’s study the examples here. Here’s some Active Record for books and authors:

# N+1
books = Book.limit(10)

books.each do |book|
   puts book.author.last_name
end

The issue above is the undesirable N+1 query pattern, where a table is repeatedly queried in a loop, instead of bulk loading all of the desired authors.

To fix the N+1, we’ll add the includes(:author) eager loading method to the code above.

That looks like this:

books = Book.includes(:author).limit(10) 👈

books.each do |book|
   puts book.author.last_name
end

We’ve now eliminated the N+1 queries, but we’ve opened ourselves up to a new possible problem.

Eager loading with includes or preload

While the includes(:author) fixed the N+1 queries, Active Record is now creating two queries, with the second one having an IN clause.

Here’s the example from above as SQL:

SELECT books.* FROM books LIMIT 10;

SELECT authors.* FROM authors
  WHERE authors.id IN (1,2,3,4,5,6,7,8,9,10);

Here we only have 10 values for the IN clause, so performance will be fine. However, once we’ve got hundreds or thousands of values, we will run into the problems described above.

Performance will tank if the authors.id primary key index isn’t used for this filtering operation.

Are there alternatives for eager loading?

Eager loading using eager_load

Besides includes() and preload() which create two queries with the second having an IN clause, there’s another way to do eager loading in Active Record.

An alternative method eager_load works a little bit differently. It produces a single SQL query that uses a LEFT OUTER JOIN.

Here’s an example of eager_load from the Active Record documentation:

books = Book.eager_load(:author).limit(10)

books.each do |book|
  puts book.author.last_name
end

The following single SQL query is produced. Note that it has no IN clause.

SELECT
    "books"."id" AS t0_r0,
    "books"."title" AS t0_r1
FROM
    "books" LEFT OUTER JOIN "authors"
    ON "authors"."id" = "books"."author_id"
LIMIT 10;

Since we’re now using a join operation, we’ve got statistics available from both tables. This makes it much more likely PostgreSQL can correctly estimate selectivity and cardinality.

The planner also isn’t needing to parse and store a large list of constant values.

While IN clauses might perform fine with smaller inputs, e.g. 100 values or fewer,¹ for large lists we should try and restructure the query to use a join operation instead.

Besides restructuring the queries into joins, are there other alternatives?

Alternative approaches using ANY or SOME

Crunchy Data’s post Postgres Query Boost: Using ANY Instead of IN describes how IN is more restrictive on the input.

A more usable alternative to IN can be using ANY or SOME, which has more flexibility in handling the list of values.

Here’s A CTE example using ANY:

WITH author_ids AS (
  SELECT id FROM authors
)
SELECT title
FROM books
WHERE author_id = ANY (
      SELECT id
      FROM author_ids);

However, ANY is not generated by Active Record. What if we want to generate these queries using Active Record?

One option is to use the any method provided by the ActiveRecordExtended gem.

Let’s talk at another alternative approach using a VALUES clause.

A VALUES clause

In the comments in the PR above, Vlad and Sean discussed an alternative for IN using a VALUES clause.

Let’s look at an example with a CTE and VALUES clause:

WITH ids(author_id) AS (
  VALUES(1),(2),(3)
)
SELECT title
FROM books
JOIN ids USING (author_id);

Or we can write this as a subquery:

SELECT title
FROM books
WHERE author_id IN (
  SELECT id
  FROM (VALUES(1),(2),(3)) AS v(id)
);

This is better because the IN list is a big list of scalar expressions, where the VALUES clause is treated like a relation (or table). This can help with join strategy selection.

A temporary table of ids

Yet another option for big lists of values is to put these into a temporary table for the session. The temporary table can even index the ids.

CREATE TEMP TABLE temp_ids (author_id int);
INSERT INTO temp_ids(author_id) VALUES (1),(2),(3);
CREATE INDEX ON temp_ids(author_id);

SELECT title
FROM books b
JOIN temp_ids t ON t.author_id = b.author_id;

Using ANY and an ARRAY of values

Another form is using ANY with an ARRAY:

SELECT title
FROM books
WHERE author_id = ANY (ARRAY[1, 2, 3]);

The ANY form can perform better. With an IN list, the values are parsed like a chain of OR operations, with the planner handling one branch at a time.

ANY is treated like a single functional expression.

This form also supports prepared statements. With prepared statements, the statement is parsed and planned once and then can be reused.

Here’s an example of fetching books by author:

PREPARE get_books_by_author(int[]) AS
SELECT title
FROM books
WHERE author_id = ANY ($1);

EXECUTE get_books_by_author(ARRAY[1,2,3,4,5]);

Testing the alternative query structures

Unfortunately generic guidelines here won’t guarantee success in your specific database. Row counts, data distributions, cardinality, or correlation are just some of the factors that affect query execution.

My recommended process is to test on production-like data, work in the SQL layer, then try out restructured queries using these tactics, and study their query execution plans collected using EXPLAIN (ANALYZE, BUFFERS).

Query plan collection and analysis is outside the scope of this post, but in brief, you’ll want to compare the plans and look to access fewer buffers, at lower costs, with fewer rows evaluated, fewer loops, for more efficient execution.

If you’re working in Active Record, you’d then translate your SQL back into the Active Record source code location where the queries were generated.

How do we find problematic IN queries that ran earlier in Postgres?

Finding IN clause queries in pg_stat_statements

To find out if your query stats include the problematic IN queries, let’s search the results of pg_stat_statements by querying the query field.

Unfortunately these don’t always group up well, so there can be duplicates or near-duplicates. You may have lots of PGSS results to sift through.

Here’s a basic query to filter on query for '%IN \(%':

SELECT
    query
FROM
    pg_stat_statements
WHERE
    query LIKE '%IN \(%';

See the linked PR for a reproduction set of commands to create these tables, queries, and then inspect the query statistics using PGSS.

While you can find and restructure your queries towards more efficient patterns, are there any changes coming to Postgres itself to better handle these?

Improvements in Postgres 17

As part of the PostgreSQL 17 release in 2024, the developers made improvements to more efficiently work with scalar expressions and indexes, resulting in fewer repeated scans, and thus faster execution.

This reduces latency by reducing IO, and the benefits are available to all Postgres users without the need to change their SQL queries or ORM code!

Grouping similar query groups in pg_stat_statements

There are more usability improvements coming for Postgres users, pg_stat_statements, and IN clause queries.

One problem with these has been that similar entries aren’t collapsed together when they have a different numbers of scalar array expressions.

For example IN ('1') was not grouped with IN ('1','2'). Having the statistics for nearly identical entries split across multiple results makes them less useful.

Fortunately, fixes are coming. On the Ruby on Rails side, Sean Linsley is working on a fix by replacing the use of IN with ANY which solves the grouping problem.

Here’s the PR: https://github.com/rails/rails/pull/49388#issuecomment-2680362607

On the PostgreSQL side, there are fixes coming for PostgreSQL 18.

Improvements in PostgreSQL 18

Related improvements are coming to PostgreSQL 18 for 2025.

This commit¹ implements the automatic conversion of x IN (VALUES ...) into ScalarArrayOpExpr.

Another noteworthy commit is: “Squash query list jumbling” from Álvaro Herrera.²

pg_stat_statements produces multiple entries for queries like SELECT something FROM table WHERE col IN (1, 2, 3, ...) depending on the number of parameters, because every element of ArrayExpr is individually jumbled. Most of the time that’s undesirable, especially if the list becomes too large.

This commit³ mentions the original design was for a GUC query_id_squash_values, but that was removed in favor of making this the default behavior.

Conclusion

In this post, we looked at a problematic query pattern, big IN lists. You may have instances of this pattern in your codebase from direct means or from using some ORM methods.

This type of query performs poorly for big lists of values, as they take more resources to parse, plan, and execute. There are fewer indexing options compared with an alternative structured as a join operation. Join queries provide two sets of table statistics from both tables being joined, that help with query planning.

We learned how to find instances of these using pg_stat_statements for PostgreSQL. The post then considers several alternatives.

Our main tactics are to convert these queries to joins when possible. Outside of that, we could consider using the ANY operator with an array of values, a VALUES clause, and consider using a prepared statement.

The next time you see big IN lists causing database performance problems, hopefully you feel more prepared to restructure and optimize them!

Thanks for reading this post. I’d love to hear about any tips or tricks you have for these types of queries!

In this post, we’ll cover a way to generate short, alphanumeric, pseudo random identifiers using native Postgres tactics.

These identifiers can be used for things like transactions or reservations, where users need to read and share them easily. This approach is an alternative to using long, random generated values like UUID values, which have downsides for usability and performance.

We’ll call the identifier a public_id and store it in a column with that name. Here are some example values:

SELECT public_id
FROM transactions
ORDER BY random()
LIMIT 3;

 public_id
-----------
 0359Y
 08nAS
 096WV

Natural and Surrogate Keys

In database design, we can use natural or surrogate keys to identify rows. We won’t cover the differences here as that’s out of scope.

For our public_id identifier, we’re going to generate it from a conventional surrogate integer primary key called id. We aren’t using natural keys here.

The public_id is intended for use outside the database, while the id integer primary key is used inside the database to be referenced by foreign key columns on other tables.

Whle public_id is short which minimizes space and speeds up access, the main reason for it is for usability.

With that said, the target for total space consumption was to be fewer bytes than a 16-byte UUID. This was achieved with an integer primary key and this additional 5 character generated value, targeting a smaller database where this provides plenty of unique values now and into the future.

Let’s get into the design details.

Design Properties

Here were the desired design properties:

• A fixed size, 5 characters in length, regardless of the size of the input integer (and within the range of the integer data type)
• Fewer bytes of space than a uuid data type
• An obfuscated value, pseudo random, not easily guessable. While not easily guessable, this is not meant to be “secure”
• Reversibility back into the original integer
• Only native Postgres capabilities, no extensions, client web app language can be anything as it’s within Postgres
• Non math-heavy implementation

Additional details:

• public_id is stored using text not not char(5), following recommendations for best practices
• PL/PgSQL functions, native Postgres data types and constraints are used, like UNIQUE, NOT NULL, and CHECK, and a stored generated column.
• Converts integers to bits, uses exclusive-or (XOR) bitwise operation and modulo operations.

Limitations

• Did not set out to support case insensitivity now, possible future enhancement
• Did not try to exclude similar-looking characters (see: Base32 Crockford below), possible future enhancement

PL/PgSQL Functions

Here are the functions used:

This function obfuscates the integer value using exclusive or (XOR) obfuscation.

• Uses a Hexadecimal key 0x5A3C1 (make this any key you want)
• Sets a max value for the data type range 62^5, which is just under 1 billion possible values. This was enough for this system and into the future, but a bigger system would want to use bigint
• Converts integer bytes into bits

Main entrypoint function:

• obfuscate_id(id INTEGER)

This converts the obfuscated value into the public_id alphanumeric value, used within obfuscate_id().

This is “base 62” with the 26 upper and lower case characters, and 10 numbers (0-9).

• to_base62_fixed(val BIGINT, width INT DEFAULT 5)

Reverses the public_id back into the original integer.

• deobfuscate_id(public_id TEXT)

Used within deobfuscate_id():

• from_base62_fixed(str TEXT)

For a length of 5 with this system, we can create up to around ~1 billion unique values. This was sufficiently large for the original use case.

For use cases requiring more values, by storing 6 characters for public_id then up to ~56 billion values could be generated, based on a bigint primary key.

Table Design

Let’s create a sample transactions table with an integer primary key with a generated identity column.

Besides the use in the identity column, we’ll again use the GENERATED keyword to create a STORED column for the public_id.

The public_id column uses the id column as input, obfuscates it, encodes it to base 62, producing a 5 character value.

DROP TABLE IF EXISTS transactions;
CREATE TABLE transactions (
  id INTEGER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,  -- 4-byte integer ID
  public_id text GENERATED ALWAYS AS (obfuscate_id(id)) STORED UNIQUE NOT NULL, -- 5-character obfuscated Base62 value
  amount NUMERIC,
  description TEXT
);

How do we guarantee public_id conforms to our expected data properties? Constraints!

• public_id gets a UNIQUE constraint and NOT NULL, so we know we have a unique value
• A CHECK constraint is added to validate the length

For an existing system, we could add a unique index CONCURRENTLY first as follows:

CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS idx_uniq_pub_id ON transactions (public_id);

Then we can add the unique constraint using the unique index, along with the CHECK constraint:

ALTER TABLE transactions
    ADD CONSTRAINT uniq_pub_id UNIQUE USING INDEX idx_uniq_pub_id, -- depends on index above
    ADD CONSTRAINT public_id_length CHECK (LENGTH(public_id) <= 5);

Insert Data

Let’s insert data into the transactions table:

INSERT INTO transactions (amount, description) VALUES
  (100.00, 'First transaction'),
  (50.00, 'Second transaction'),
  (0.25, 'Third transaction');

Let’s query the data, and also make sure it’s reversed (using the deobfuscate_id(public_id TEXT) function) properly:

Access the rows

SELECT
    id,
    public_id,
    deobfuscate_id(public_id) AS reversed_id,
    description
FROM
    transactions;


 id | public_id | reversed_id |    description
----+-----------+-------------+--------------------
  1 | 01Y9I     |           1 | First transaction
  2 | 01Y9L     |           2 | Second transaction
  3 | 01Y9K     |           3 | Third transaction

Additional time spent on inserts

Let’s compare the time spent inserting 1 million rows into an equivalent transactions table without the public_id column or value generation.

That took an average of 2037.906 milliseconds, or around 2 seconds on my machine.

Inserting 1 million rows with the public_id took an average of 6954.070 or around 7 seconds, or about 3.41x slower. Note that these times were with the indexes and constraints in place on the transactions table in the second example, but not the first, meaning their presence contributed to the total time.

Summary: Creating this identifier made the write operations 3.4x slower for me locally, which was an acceptable amount of overhead for the intended use case.

Performance

Compared with random values, the pseudo random public_id remains orderable, which means that lookups for individual rows or ranges of rows can use indexes, running fast and reliably even as row counts grow.

We can add a unique index on the public_id column like this:

CREATE UNIQUE INDEX CONCURRENTLY
IF NOT EXISTS
idx_uniq_pub_id ON transactions (public_id);

We can very that individual lookups or range scans use this index, by inspecting query execution plans for this table.

PL/pgSQL Source Code

https://github.com/andyatkinson/pg_scripts/pull/15

Feedback

Feedback on this approach is welcomed! Please use my contact form to provide feedback or leave comments on the PR.

Future enhancements to this could include unit tests using pgTAP for the functions, packaging them into an extension, or supporting more features like case insensitivity or a modified input alphabet.

Thanks for reading!

Alternatives

• Base32 Crockford - An emphasis ease of use for humans: removing similar looking characters, case insensitivity.
• ULID - Also 128 bits/8 bytes like UUIDs, so I had ruled these out for space consumption, and they’re slightly less “usable”
• NanoIDs at PlanetScale - I like aspects of NanoID. This is random generation though like UUID vs. encoding a unique integer.

Back in 2022, we covered how to log database query generation information from a web app using pg_stat_statements for Postgres. https://andyatkinson.com/blog/2022/10/07/pgsqlphriday-2-truths-lie

The application context annotations can look like this. They’ve been re-formatted for printing:

application=Rideshare
controller=trip_requests
action=create

I use pg_stat_statements to identify costly queries generated in the web application, often ORM queries (the ORM is Active Record in Ruby on Rails), with the goal of working on efficiency and performance improvements.

The annotations above are included in the query field and formatted as SQL-compatible comments.

Application context usually includes the app name and app concepts like MVC controller names, action names, or even more precise info which we’ll cover next.

How can we make these even more useful?

What’s the mechanism to generate these annotations?

For Ruby on Rails, we’ve used the Marginalia Ruby gem to create these annotations.

Besides the context above, a super useful option is the :line option which captures the source code file and line number.

Given how dynamic Ruby code can be, including changes that can happen at runtime, the :line level logging takes these annotations from “nice to have” to “critical” to find opportunities for improvements.

What’s more, is that besides Marginalia, we now have a second option that’s built-in to Ruby on Rails.

What’s been added since then?

In Rails 7.1, Ruby on Rails gained similar functionality to Marginalia directly in the framework.

While nice to have directly in the framework, the initial version didn’t have the source code line-level capability.

That changed in the last year! Starting from PR 50969 to Rails linked below, for Rails 7.2.0 and 8.0.2, the source_location option was added to Active Record Query Logs, equivalent to the :line option in Marginalia.

PR: Support :source_location tag option for query log tags by fatkodima https://github.com/rails/rails/pull/50969#issuecomment-2797357558

An example of :source_location in action looks like this:

application=Rideshare
controller=trip_requests
➡️ source_location=app/services/trip_creator.rb:26:
in `best_available_driver'
action=create

Nice, now we’ve got the class name, line number, and Ruby method. In this example, we can get to work optimizing the best_available_driver method.

Dima described how the Marginalia :line option was costly in production and even managed to improve that with the Query Logs change.

Safe Logging Locally or in Production

If you’re unsure about source code line logging in production, but want to get started using it, a great place to start is using it in your local development enrivonment.

To avoid enabling the option for all environments, we’ll use an environment variable that’s enabled only for local development.

Here’s a real example I use for Marginalia: MARGINALIA_LINE_NUMBER_ENABLED=true

In config/initializers/marginalia.rb:

Marginalia::Comment.components = [
  :application,
  :controller,
  :action
]

if ENV['MARGINALIA_LINE_NUMBER_ENABLED']
  Marginalia::Comment.components.append(:line)
end

For Query Logs, in config/application.rb (adjust to be for the environments you prefer), the equivalent could look like this:

config.active_record.query_log_tags_enabled = true
config.active_record.query_log_tags = [
  :application,
  :controller,
  :action,
  :source_location
]

The configuration above was tested with Rails 7.2.2.

If your team uses Query Logs :source_location in development or production, I’d love to know!

Wrap Up

Having source code line-level logging for query statistics is critical information that allows backend engineers to quickly zero in on where to fix database performance issues.

With this combo of information, engineers can identify the most heavy queries, then go backwards into the source code to know where to redesign, refactor, or restructure, or even remove costly queries.

What’s next?

The pg_stat_statements extension is critical for this workflow, but it’s not without opportunities for improvement.

One issue that pg_stat_statements has is that many of the entries can be duplicates or near-duplicates, making it tougher to sift through.

Fortunately, fixes are coming for that too! Stay tuned for a future post where we’ll cover upcoming improvements in future versions of Postgres that will help de-duplicate pg_stat_statements entries, as well as options to achieve that with Ruby on Rails even for older versions of Postgres.

Consider subscribing to my newsletter, where I send out occasional issues linking to blog posts, conferences, industry news, and more, so you don’t miss that post!

Thanks for reading.

Some questions

Claire had a lot of thoughtful questions. Here’s a few:

• Why write a technical book? Was there some moment or spark?
• Why write a book about Postgres?
• Why Ruby on Rails?

Fun topics

Claire also brought up a lot of fun points and reactions. Here’s a sample:

• The importance planting seeds and encouraging others with ambitious projects
• Would I consider writing a book for Django and Python for Postgres?
• Where does the book fit in the landscape?
• How long did it take to write this book?
• Did I ever want to quit writing, even for a moment?
• Did I have a party when the book was fully complete?
• I talked about “little parties” with Rails developer communities at events like Rails World and Sin City Ruby
• What was my experience like in working with other publishers
• I shared my deep appreciation for the efforts of the technical reviewers of the book!
• We talked about cheese! 🧀 (stories and connections with Postgres icons David Rowley and Melanie Plageman)
• What was my favorite chapter?
• Is there a frequently asked question I get about databases from Rails developers?
• For my consulting services, do clients hire me for my Rails expertise or my Postgres expertise?

Quote mentioned by Claire:

Writing is thinking. To write well is to think clearly. That’s why it’s so hard.
—David McCullough

It was a real honor to be a guest on this prestigious podcast. I’m lucky to call Claire a friend as well! Thank you for the opportunity Claire, Aaron, and team!

Check out more episodes of Talking Postgres!

Recently I had the chance to work with Python, Django, and Postgres as a long-time and busy Rails developer.

There were some things I thought were really nice. So am I switching?

The team I worked with was experienced with Django so I was curious to learn from them which libraries and tools are popular, and how how to write idiomatic code.

In this post I’ll briefly cover the database parts of Django using Postgres (of course!), highlight libraries and tools, and compare aspects to the Ruby on Rails framework. You’ll find a small Django repo towards the end as well.

Ruby versus Python

Ruby and Python are both general purpose programming languages. On the similarity side, they can both be used to write script style code, or organize code into classes using object oriented paradigms.

In local development, it felt like the execution of Python was perhaps faster than Ruby, however I’ve noticed that new apps are always fast to work with, given how little code is being loaded and executed.

Language runtime management

As a developer we typically need to run multiple versions of Ruby, Python, Node, and other runtimes, to support different codebases, and to avoid modifying our system installation.

In Ruby I use rbenv to manage multiple versions of Ruby, and to avoid using the version of Ruby that was installed by macOS, which is usually outdated compared with the version I want for a new app.

In Python, I used pyenv to accomplish the same thing, which seemed quite similar in use.

Both have concepts of a local and global version, and roughly similar commands to install and change versions.

Library management

In Ruby on Rails, Bundler has been the de facto standard forever, as a way to pull in Ruby library code and make sure it’s loaded and accessible in the Rails application.

In Python, the team selected the poetry dependency management tool.

Commands are similar to Bundler commands, for example poetry install is about the same as bundle install.

Dependencies can be expressed in a pyproject.toml file and poetry creates a lock file with specific library versions. TOML and YAML are similar.

Linting and formatting

In Ruby on Rails, although I personally resisted rule detection etc. for years, Rubocop has become the standard, even being built in to the most recent Rails version 8.

Rubocop has configurable rules that can automatically reformat code or lint code for issues.

Formatters like standardrb are commonly used as well.

For the Django app the team selected ruff, which performed formatting of code and linting for issues like missing imports.

I found ruff fast and easy to use and genuinely helpful.

For example, sometimes I’d fire up a Django shell, having skipped running ruff, only to realize there are issues it would have caught.

On this small codebase, ruff ran instantly, so it was a no-brainer to run regularly, or even include in my code editor.

Postgres adapter

In Rails and Django, SQLite is the default database, however I wanted to use Postgres.

In Ruby, we have the pg gem which connects the application to Postgres as a driver. This does work at a lower level than the application like sending TCP requests, mapping Postgres query result responses into Ruby data types, and much more.

In Python, we used the psycopg2 library and I found it pretty easy to use.

Besides being used by the framework ORM, I created a wrapper class using psycopg2 to use for sending SQL queries outside of models.

For example, we inspected Postgres system catalog views to capture certain data as part of the product features.

Migrations in Rails

Both Ruby on Rails and Django have the concept of Migrations, which are Ruby or Python code files that describe a database structure change, and have a version.

From the Ruby or Python code files, SQL DDL (or DML) statements are generated which are run against the configured database.

For example, to add a table in Rails typically a developer uses the create_table Ruby helper as opposed to writing a CREATE TABLE SQL statement.

Adding or dropping an index or modifying a column type are other types of DDL statements that typically are performed via migrations.

Migrations in Django

The Django approach has noteworthy differences and a slightly different workflow that I enjoyed more in some ways.

For example, changes are started in a models.py file, which contains all the application models (multiple models in a single file), and the database layer details about each model attribute.

This means that we specify database data types for columns, whether fields are unique, indexed, and more in the models file.

The interesting difference compared with Rails is that the next step in Django is to run makemigrations, which generates Python migration files.

This is different from Rails, where Rails developers would first generate a migration file to place changes into.

In Django the generated migration file can be inspected or simply applied using the migrate command. This command is nearly identical to the Rails equivalent command db:migrate.

For a new project where we were rapidly iterating on the models and their attributes, I preferred the way Django worked to how Rails works, or found it at least as productive.

Command line vibes

Here are some commands like running poetry install, or running manage.py commands like shell or makemigrations, to give you a flavor.

poetry install
python manage.py dbshell # psql in postgres
python manage.py shell   # Django shell
python manage.py makemigrations  # Generates Python migration files
python manage.py migrate # Runs migration files

Interactive console (REPL)

Both Django and Rails use interpreted languages, Python and Ruby respectively, that each support an interactive execution environment.

This environment is called a read, eval, print loop or REPL for short.

In Rails, the Ruby REPL “irb” is launched and Rails application code is loaded automatically when running the rails console command.

In Django the equivalent command is running shell, however application code needs to be imported before it can be used, using import statements.

Both frameworks also support opening a database client, by running dbconsole in Rails or dbshell in Django.

When Postgres is configured, these both open a psql session.

Projects and Apps

In Django projects and applications are separate concepts.

In my experimental project I made a “booksproject” project and a “books” app.

Check out the booksproject repo.

Postgres details

The books app models are Author, Publisher, and Books.

The tables for those models are contained in a custom schema booksapp, and Django is configured to access it.

The application connects to Postgres as the booksapp user and the dev database is called books_dev.

No migration safety concept

There’s no concept of what I’d call “safety” for migrations for either framework out of the box.

Operations like adding indexes in Postgres don’t use the concurrently keyword by default for example.

We can add safety using additional libraries like Strong Migrations in Ruby.

At a smaller scale of data and query volume, even unsafe operations will be fine. With that said, I think some visibility into blocking database operations, and how to perform them using safe alternatives is valuable.

Adding a constraint

In models add unique=True to a field definition to add a unique constraint (via a unique index). After running makemigrations a migration for a unique index will be created.

In Active Record we’d generate the migration file first, then fill in the create statement adding a unique index.

Django models

When querying a model like Book we’d use objects which returns a QuerySet object with one or more books.

The filter() method will generate a SQL query with a WHERE clause to filter down the rows or all rows can be accessed using all().

For example:

Model.objects.filter()
Model.objects.first()
Model.objects.all()

Statements in Python are whitespace sensitive so we’d indent the attributes in create() below by 4 spaces:

Thing.models.create(
    attr1=val,
    attr2=val
)

Previewing DDL

The generated SQL DDL isn’t displayed when running migrate by default.

Unlike Rails, Django provides a mechanism to preview it.

To do that run the sqlmigrate command instead of migrate.

For example, to print the 0001 migration DDL:

python manage.py sqlmigrate books 0001

BEGIN;
--
-- Create model Author
--
CREATE TABLE "books_author" ("id" bigint NOT NULL PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, "first_name" varchar(200) NOT NULL, "last_name" varchar(200) NOT NULL);
COMMIT;

Note that Django uses an identity column for the primary key, and as of Rails 8 Active Record does not.

Resources

For the basics of an Author, Publisher, and Books models, or Postgres configuration including a custom schema and user, check out booksproject repo.

To collect random Django tips I’ve created a django-tips page. This page can be used in a similar way as my rails-tips and postgresql-tips pages, mostly as a reference for myself and possibly as a useful resource for others.

Wrap Up

Do you have any similarities and differences between Django and Rails to share? I’d love to hear from you.

😅 And no, I’m not “switching” from Rails and Ruby, but I did enjoy working with Python, Django, and Postgres!

Thanks for reading.

In this post, I’ll describe some sessions, but mostly they’re saved for part 2, once I watch all the sessions I missed now that the full Rails World 2024 Playlist is on YouTube.

As a book author and consultant, the focus for Rails World for me was on meeting people, raising awareness about my book, and generally chatting about how people are using Postgres and Rails. As a long-time Ruby community member, it was great to catch up with a lot of industry friends.

As a first time Rails World visitor, I was really impressed with the energy and vibes.

Let’s get into it.

Arrival: Wednesday

🇨🇦 I Landed in Toronto after a short two hour flight from Minneapolis. I was feeling excited to promote my book, meet attendees, and give away 8 copies over the next few days. I was feeling grateful!

Landed in Toronto, let’s go!

To celebrate my book launch and successfully striking out on my own as an independent Postgres and Rails consultant, I hosted a happy hour gathering of Postgres Fans. We had a good turnout, conversations, and new and strengthened connections. We didn’t talk about Postgres much, but the event was a success! 🙌

Party time, Postgres fans Happy Hour at The Queen & Beaver Public House

This was my first time hosting an event like this. I was able to sponsor a round of drinks and appetizers thanks to the success in book sales and my consulting business.

It felt great to bring folks together and I appreciated everyone coming out!

Afterwards, most of the group walked over to the Shopify pre-registration event to get badges, hang out, and grab dinner.

Pre-registration party by Shopify, with John and Jesper

Conference Day 1: Thursday

In the kick-off, I appreciated the overview of the Rails Foundation activities, and was excited to see my GitHub handle up on screen briefly along with some friends! In the last year, I made some contributions to the Rails Guides, particularly in the Active Record area. It was a nice surprise to be mentioned!

Have to zoom way in 🔍 but honored to see my GitHub handle listed as a contributor to Rails Guides, an important effort lead by awesome people like @Ridhwana_K pic.twitter.com/LMrcF307ZN

— Andrew Atkinson (@andatki) September 27, 2024

Session: DHH Keynote

David’s keynote was good. It was a bit of storytelling and individual positions and perspectives, with occasional splashes of tech stuff mixed in. Entertaining! I always appreciate the live demos too.

David announced Rails 8 Beta 1 is now shipping, and even 8.1 was on the horizon. The technical work over the last year made this possible. Earlier this summer, I wrote for AppSignal about new database related functionality in Ruby on Rails 7.2.

I left the keynote with a call to action to be more competent in our roles as software engineers. Although it wasn’t said exactly, I wondered how much of this message was in response to the rise of AI-assisted programming, which has raised the floor perhaps on what’s expected from a modern professional software engineer.

Session: Postgres at Instacart

I was happy this session made it onto the agenda, given that PostgreSQL is the most popular database used by Rails developers, and this was the only Postgres related session at RailsWorld.

Attendance was good. Mostafa took on a challenging task, covering advanced concepts in a short period of time, and did a great job. A highlight for me was meeting Mostafa afterwards. I didn’t know Mostafa was a contributor to PgCat, which was cool to find out. Mostafa talked about different approaches they’d used at Instacart over the years from vertically scaling, to replication and routing read queries to replicas, and forms of database sharding.

Note from me: Remember that Active Record natively supports automatic Read and Write query routing, and Horizontal Sharding.

Mostafa covered the Makara gem, and briefly the benefits of the PgCat connection pooler over pgbouncer.

Mostafa from Instacart. Scaling Postgres, replication, sharding, PgCat.

Session: Fireside Chat DHH, Matz, Tobi

This was interesting to see. I knew Matz didn’t normally attend Ruby on Rails conferences, so it was exciting to see him attending. I should have taken the opportunity to ask for a picture, but did see plenty of others walking around getting pictures with Matz. Great for the community to see the heroes in person!

Event: Rails Startups With Evil Martians and Whop

Whop and Evil Martians sponsored a social event called Rails Startups. This was a fun event!

I appreciated being invited by Irina and had the chance to meet Jack from Whop and a bunch of other Ruby community folks like Kir, Miles, Yaroslov from SupeRails, and John.

Conference Day 2: Friday

On Day 2, the highlight and emphasis for me was Book Signing, handing out books, and talking Postgres + Rails!

Book Signing at Rails World 2024

Although it was meant to be a 1 hour session from 12pm - 1pm over the lunch period, I was able to meet a lot of people before and after, to generally chat about the book, Postgres, Rails, and hung out for much longer, missing even more sessions (but totally worth it)!

Since Day 2 was filled with meeting people, I’d like to leave you with more pictures. I got more selfies and pictures here than I’d normally do, and I’m really thankful for that. The pictures help me remember all these amazing people in the community, and it’s an honor to have spent a bit of time with them.

I got a signed copy of High Performance PostgreSQL for Rails from @andatki himself! I had read early versions of the book and can comfortably place this book in the “classics” shelf of my library. This is a must-read! #RailsWorld pic.twitter.com/zUY35TvEiA

— Emmanuel Hayford (@siaw23) October 2, 2024

Book giveaway winners, Wojciech, Matt

Book giveaway winners, Allison

Book giveaway winners, Bart

Book winners: Ridhwana

More Pics and Selfies

With Jesper

With Emmanuel

With Dan

With Julia and Jorge

While waiting for a car to the airport, I had a serendipitous opportunity to join Craig Kerstiens from Crunchy Data for a shared ride. Obviously we talked about Postgres!

With Craig

Rails World 2024 was a blast. Thank you to everyone involved in putting this event on. If we met at Rails World, I hope I’ve reached out to you on social media by now, but if not, please drop me a line. Let’s stay in touch.

Capping off Ruby events in 2024

I’ve been privileged to attend a lot of Ruby events and be a guest on podcasts this year, to promote my consulting services and my book. Rails World sort of capped it off for me, as I don’t have plans for any more Ruby events in 2024. I’m thinking about 2025 though!

2024 was my first year striking out on my own as an independent consultant. Thank you to all the clients, colleagues, and friends that have helped me make it work. I’ve got a lot to be thankful for, and I’m excited to keep it going in 2025!

I’d like to briefly look back at the year by sharing some links to past podcasts, conferences, and other content. Thanks for taking a look.

• Madison+ Ruby 2024 Conference Recap
• SaaS on Rails on PostgreSQL — POSETTE 2024
• 🎙️ IndieRails Podcast — Andrew Atkinson - The Postgres Specialist
• Top Five PostgreSQL Surprises from Rails Devs
• 🎙️ Ship It! Podcast — PostgreSQL with Andrew Atkinson
• RailsConf 2024 Conference — The Long Goodbye
• Mastering PostgreSQL for Rails: An Interview with Andy Atkinson
• Sin City Ruby 2024
• Rails + Postgres Postgres.FM 086 — Extended blog post edition! 🎙️
• Maintainable Podcast — Maintainable…Databases? 🎙️
• Remote Ruby — Unleashing the Power of Postgres with Andrew Atkinson 🎙️
• The Rails Changelog — #014: PostgreSQL for Rails Developers with Andrew Atkinson 🎙️