It worked at 12: what broke at 500 (and it wasn't Lambda)

That's how many bots got into the game when I sent 500. And then I sent 1,000 - you know, cos why the hell not.

Doubling the demand added two players. That isn't a box running out of room. That's a ceiling, and something in my design was holding it up.

Part 1 covered what Lambda Web Functions are and why I built a bingo app to find out. Fair warning: this one's a bit more technical. It's less about web functions and more about designing something that scales, or, more honestly, how I fixed my own half-arsed first attempt. This is the bit where it fell over, and what it took to fix it properly.

So what happened

The stack was the one from part 1: a bot script running from home in London, through CloudFront, to a web function endpoint in Ireland, backed by one on-demand DynamoDB table. The endpoint allowed 64 requests in flight per environment and up to 20 environments.

The bots create a game, then every one of them tries to join at once. No ramp, no politeness. It's the stampede from part 1, on purpose, because that's what a real all-hands looks like when someone drops the link in the chat.

The Lambda log had 229 errors in under a second and a half, one per failed join, and every one of them said the same thing:

{"msg":"request-failed","path":"/api/games/3CH7XG/players","error":"ThrottlingException",
 "detail":"Throughput exceeds the current capacity of your table or index. DynamoDB is
 automatically scaling your table or index so please try again shortly. If exceptions
 persist, check if you have a hot key: ..."}

DynamoDB told me what was wrong in the error message. "Check if you have a hot key." I did.

Which bit broke

Was it Lambda ? No. But a stack has a lot of places to point fingers, so I went through it layer by layer.

That last row has two bugs in it. The hot key is the big one. The smaller one is that DynamoDB said "slow down and try again" and my app passed that on to the player as a 500, which means "this is broken, go away". A bot doesn't care. A person on their phone in a meeting gives up.

Why one key was always going to do this

DynamoDB spreads a table over partitions, chosen by each item's partition key. On-demand mode scales the table for you, but a single partition still has a limit of about 1,000 write units a second. If every write lands on one key, the table's size doesn't matter. You're still queuing at one door.

And I'd put everything behind that door. Every item for a game lived under GAME#<id>: the META record with the game's state and player count, plus every player's card and marks.

A join was a transaction over two of those items. Add one to the player count, but only if the game isn't full, and write the new player:

new TransactWriteCommand({
  TransactItems: [
    { Update: { Key: gameKey(gameId),
        UpdateExpression: 'ADD playerCount :one',
        ConditionExpression: '... AND (attribute_not_exists(playerCount) OR playerCount < :max)' } },
    { Put: { Item: { ...playerKey(gameId, playerId), name, avatar, card } } },
  ],
})

Transactions cost double, so each join cost about 4 write units, all on the same partition. 1,000 divided by 4 is about 250 joins a second, per game, however many bots you throw at it. I got 271 and 273.

The SDK retries a throttled write, three attempts by default, and a stampede burns through those quickly. What came out the other end was a ThrottlingException. My join code was only expecting a TransactionCanceledException (the game's full, or the player ID's taken), so it didn't recognise the error, and the server turned anything it didn't recognise into a 500.

Marks lived on the same key too. If the joins had got through, 1,000 people dabbing squares would have hit the same wall a minute later. I never saw that happen, because no game got past its lobby.

Twelve players never came near any of this, which is why every test passed. That's the dangerous kind of green, and it's why you stress test a new pattern before production does it for you.

The fix I didn't use

The quick fix was more retries. Turn maxAttempts up, switch the SDK to adaptive retry mode, and let it back off until the writes get in.

It's the "have you tried turning it off and on again ?" of DynamoDB. It feels like doing something. But every retry is the same write knocking on the same door, so the queue gets longer rather than shorter, players wait longer to join, and the marking phase walks into the same wall.

Retries are still in the code (five attempts, standard mode) as defence in depth. There's a comment beside them saying the key layout is what keeps a game under the limits.

The fix I did use

The player count only existed to enforce the cap, and the transaction only existed to keep the count and the players in step. Get rid of the single counter and the transaction goes with it.

Each game now spreads over 20 shard keys, GAME#<id>#S#00 to S#19. Each shard holds a seat counter, an equal share of the game's seats and its share of the players.

A join shuffles the shards and tries to take a seat with one conditional update:

new UpdateCommand({
  Key: seatKey(gameId, shard),
  UpdateExpression: 'ADD seats :one SET expiresAt = if_not_exists(expiresAt, :exp)',
  ConditionExpression: 'attribute_not_exists(seats) OR seats < :cap',
})

If that shard is full, the condition fails and it tries the next. If all 20 are full, the game's full and you get a 409. Once it has a seat, a plain put writes the player under that shard's key. The first character of the player ID says which shard they're on, so finding a player later needs no lookup.

The cap is still strict. No shard can go over its share of the seats, so no game can go over its cap, and there's no transaction anywhere in the join. The random order matters too: a burst of joins spreads over all 20 seat counters instead of piling onto shard 00 until it fills.

There's one fiddly bit. If the player write fails after the seat's been taken, the seat has to go back. But a failed write isn't proof that nothing was written, because an earlier SDK attempt might have landed. So the seat only goes back once a consistent read shows there's no player. If that read fails as well, the seat stays taken until the game expires. A game that's one seat short is fine. A game that goes over its cap isn't.

The rest moved with it:

• Marks are an ADD to the player's own item, which now lives on its shard's key, so a room full of people dabbing spreads over 20 keys.
• The winner is still one conditional update on META: the game has to be running and have no winner yet. The first valid claim wins and everyone else gets a 409, however many people shout bingo in the same second.
• Throttles now come back as a 503 with Retry-After: 1, and the browser waits and tries again. I haven't seen one live, because nothing throttled on the new layout. I tested it locally against a stub that throttles on demand.

The same layout fixed something on the director screen. It wanted to show every player's card, and the obvious way to do that is to send the full player list in every update. That's every list to every client: with 500 players, 500 rows sent to each of 500 screens, every time someone dabs. Now every client gets the same small state frame whatever the player count, each player's own row goes down their stream separately, and only the director gets the full roster.

500, again

That's a capture run. The director grows from one tile to 500, then somebody wins before I could get to 1,000. Bingo's quick when 500 people are listening for the same phrases.

The proof runs used the same, unchanged bot script as the runs that failed, so it's a straight before and after. I also raised the endpoint's caps to 100 environments and 5,000 requests a second for the big runs.

Every failed join in the old run was DynamoDB saying no, not the endpoint. The thing I fixed was where the data lived.

One more thing: 1,000

That's 1,000 bots joining one game, the director growing to 1,000 mini-cards, then the whole board starting at once. 24 environments, no throttles, no errors.

The proof run used the same unchanged script that got 273 in on the old layout. This time it was 1,000 of 1,000, across 16 environments, with zero throttles.

Over the whole session, across every run, the endpoint took 26,274 requests, peaking at 5,314 a minute. There were zero 5xx responses.

1,000 is a ceiling I chose (20 shards of 50 seats), not a limit of the layout. Every environment streaming a game reads it once a second, all 20 shards of it. More players means more open streams, which means more environments, and each of them is reading more players. So reads grow with roughly the square of the player count. Over the session that came to 28,368 write units and 329,065 read units: the reads outweighed the writes more than ten to one.

In money, that's still not much. DynamoDB came to under half a cent per 1,000-player game, and about $0.06 for the whole session, all 13 runs. By my rough maths Lambda's share is about the same again, but that part's a guess until the bill lands.*

*These are estimates from the capacity each run used, at US East (N. Virginia) rates, which is all the pricing pages list; Ireland's are usually a little higher. I'll update the post with the real figures once my bill reflects them.

Raising the cap is a one-line change. Paying for it is a different conversation. And if your all-hands has more than 1,000 people in it, bingo isn't your biggest problem.

One aside on load testing. My first two 1,000-player runs fell over, and it wasn't the stack. It was my machine at home: one home connection holding 1,000 open streams to one CloudFront edge, opening more connections for every mark, and one dropped connection crashed the bot script. The server side was clean, with every join landed and zero errors. The same unchanged script, run from a small EC2 instance, passed first time. Your load generator needs capacity too.

Where that leaves web functions

Web functions made the compute side of a stampede boring, which is exactly what you want. Lambda added environments as the streams arrived and answered every request it was given. The bit that broke was the bit that's always been mine: deciding where the writes land.

Lambda scaled fine; the data model didn't.

Part 1 covers what web functions are and where I'd use one. The code, sharded layout included, is on GitHub. Same deal as before: I designed it and broke it; Claude Code did a lot of the typing and most of the log trawling.