Design a chat system

The server has something for you, and cannot tell you

HTTP is a client asking and a server answering. A message arriving for you is an event on the server, and there is no way for the server to start that conversation. The naive fix is for the client to ask, over and over.

A busy user might receive twenty messages an hour. Polling every two seconds is 1,800 requests an hour to collect those twenty, so about 99% of the traffic is the client asking “anything yet?” and being told no. Each of those pays full price (TLS, authentication, a database read), and you still have up to two seconds of delay. Stretch the interval and the request rate drops, along with the product. No setting makes polling both cheap and instant.

Hold the connection open and let the server push

A WebSocket is one connection, opened once, that either side can write to. The chat server holds one per connected client and simply writes new messages down it. That makes the chat server the one genuinely stateful part of the system, and raises a new question: what if the recipient is not connected?

Holding the message in memory would lose it when that server restarts, and chat systems are not allowed to lose messages. So the delivery path forks. If the socket is open, relay the message now. If it is closed, write it to a message store and hand the wake-up to the notification system. When the phone reconnects, it pulls everything it missed.

Whose clock decides what was said first?

Two people in a group chat, connected to two different chat servers, send moments apart. Everyone must see the messages in the same order, and the two servers do not agree on what time it is.

NTP keeps clocks close, not identical, and skew of a few hundred milliseconds is plenty to put an answer above its question. The fix is not better clocks; it is to stop using time as the order. One authority per channel hands out a counter, and messages sort by that.

The counter only needs to be ordered within a conversation, a far weaker and cheaper guarantee than a global order. It doubles as the cursor a reconnecting client uses to ask for “everything after message 4,182”, so ordering and catch-up turn out to be the same mechanism.

A connection the server can write to, a fallback for when it cannot, and an order that does not depend on clocks. Everything else in a chat product is a CRUD app.