Design a unique ID generator
Sortable, 64-bit IDs across thousands of machines with no coordination - the Snowflake layout of timestamp, machine ID and sequence bits.
9 min read
Every row in every table needs an identifier. Getting one is trivial with a single database and surprisingly hard the moment there are two, because the properties you want, unique, sortable by time and cheap to mint, pull against each other.
Auto-increment works until there are two databases
One database hands out 1, 2, 3, and the problem is solved. Split it into two shards and each half keeps counting on its own, which means both of them issue ID 1.
The obvious escapes each give something up. A random UUID never collides, but it sorts into nonsense, and that wrecks index locality: every insert lands in a different page of the B-tree. A central ticket server hands out perfectly ordered numbers, and puts a single point of failure on every write in the system. Uniqueness without coordination is easy, and so is sortability. Getting both at once is the design problem.
Put time in the high bits and identity in the middle
Snowflake builds both properties into the bits of a 64-bit number: a sign bit, a 41-bit millisecond timestamp, 5 bits of datacenter, 5 of machine, and a 12-bit counter for IDs within the same millisecond. Because the timestamp is the most significant part, a later ID is always a larger number.
Each field prevents one kind of collision. Time separates milliseconds. The machine and datacenter bits, assigned once at startup and never shared, separate two machines minting in the same millisecond. The sequence separates IDs within one machine's millisecond, up to 4,096 of them. Nobody coordinates anything at mint time. The 41 timestamp bits last about 69 years from the chosen epoch, which is a deadline rather than a guarantee.
It depends on a clock that only moves forwards
All of that rests on two assumptions: time never goes backwards, and no machine needs more than 4,096 IDs in a millisecond. Production breaks both, and the failures are very different.
When an NTP correction steps the clock back, carrying on is the dangerous choice: the generator would reuse timestamps it has already used and mint IDs that already exist. A duplicate primary key is a corruption bug that surfaces weeks later, so real implementations remember the last timestamp and simply refuse to issue until the clock passes it. A brief write pause on one node is the far better failure.
The modern alternatives are the same idea re-standardised. ULID and UUIDv7 put a timestamp in the high bits and randomness below, trading machine IDs for enough random bits that collisions are negligible. The choice is not about elegance; it is about which failure you can live with.
The short version
- Per-shard counters collide; random UUIDs do not sort; ticket servers are a single point of failure.
- Snowflake packs timestamp, machine and sequence into 64 bits: unique and time-sortable, no coordination.
- 4,096 IDs per machine per millisecond; beyond that, requests wait a millisecond.
- If the clock steps back, block rather than risk duplicate IDs.