I always knew PostgreSQL was the database of choice if you’re serious about building any scalable AI app.

I know most of you want to build AI apps that can scale to millions of users, and if that’s the case, you need to learn how to use PostgreSQL efficiently or at least understand the architecture design mindset for such an app.

When I came across this article on OpenAI is scaling a single PostgreSQL instance to handle millions of queries per second for 800 million ChatGPT users, I knew this was a goldmine for our use cases.

One of the most common questions I get asked is about technology choices when building large AI applications that won’t crash on the first heavy load.

OpenAI’s database load grew by more than 10x over the past year, and they had to push PostgreSQL to its absolute limits to keep ChatGPT running smoothly.

They’re now serving millions of queries per second with a single primary instance and nearly 50 read replicas spread across multiple regions.

There are a lot of things that go into designing such a system, and they won’t all fit in this article, but at least this will point you in the right direction and help you build the right mindset.

The best part about this guide is that the principles apply whether you’re building for 1,000 users or 800 million users. The scaling strategies, optimization techniques, and architectural decisions OpenAI made can be adapted to your specific needs.

Let’s find out how you can replicate these patterns to fit your current needs or what you can learn to avoid the common pitfalls that bring down production databases.

Scaling Challenge

After ChatGPT launched, traffic grew at an unprecedented rate.

OpenAI had to scale fast, and they did what most teams would do in that situation. They increased instance sizes, added more read replicas, and implemented optimizations at both the application and database layers.

This worked well for a long time, but cracks started to form.

Single-Primary Architecture Problem

OpenAI runs a single-primary PostgreSQL setup, which means one writer handles all write operations. This might sound limiting for a service with 800 million users, but it works because their workload is primarily read-heavy.

The problem isn’t the architecture itself but what happens when things go wrong.

They experienced several high-severity incidents that followed the same pattern.

An upstream issue causes a sudden spike in database load, such as widespread cache misses from a caching layer failure, a surge of expensive queries saturating the CPU, or a write storm from a new feature launch.

When this happens, resource utilization climbs, query latency rises, and requests begin timing out.

Vicious Cycle Under Load

When requests time out, applications retry.

Those retries further amplify the load, triggering a vicious cycle that can degrade the entire ChatGPT and API services.

Cache failures lead to PostgreSQL overload, which causes slow responses, which trigger retries, which further increase the load. This is the scenario that brings down production databases.

PostgreSQL handles read-heavy workloads extremely well, but write-heavy workloads expose limitations in its multiversion concurrency control (MVCC) implementation.

When a query updates even a single field, PostgreSQL copies the entire row to create a new version.

Under heavy write loads, this results in significant write amplification and increased read amplification since queries must scan through multiple tuple versions to retrieve the latest one.

OpenAI’s solution was to migrate shardable write-heavy workloads to sharded systems like Azure Cosmos DB while keeping PostgreSQL unsharded with a single primary for their read-heavy workloads.

How Does this Apply to Your AI App Design?

The patterns OpenAI encountered aren’t unique to ChatGPT’s scale.

These same issues show up in applications serving 1,000 users, 10,000 users, or 50,000 users. The vicious cycle doesn’t care about your user count but your architecture.

So, how do you design your application to avoid these problems? Let me show you with a real project that most of you are either working on or will work on soon.

Your AI Chatbot Project

You are building an AI-powered customer support chatbot and designing it from the ground up using OpenAI’s lessons.

You’ve built the initial version. It’s working great in development and has even handled your first 1,000 users without breaking a sweat.

Your PostgreSQL setup looks reasonable:

• One database instance handles everything

• Direct connections from your application

• Basic caching for user sessions

• Standard queries to fetch conversation history

The application works, and the early adopters love it.

Now you land a big client with 50,000 support agents who will use your platform simultaneously during peak hours.

This is where the lessons from OpenAI’s guide are needed.

Your chatbot needs to do three main things constantly:

1. Fetch conversation history when agents open a chat (read-heavy)

2. Store new messages as conversations happen (moderate writes)

3. Query user context and previous interactions (read-heavy)

Within the first hour of launch, you start seeing timeouts.

Your monitoring dashboard shows:

• Database CPU at 95%

• Connection count is hitting the 5,000 limit

• Query response times jumping from 20ms to 2 seconds

• Cache hit rate dropping from 80% to 30%

Your Redis cache layer goes down for 3 minutes during a deployment.

Suddenly, every conversation history request hits PostgreSQL directly. What was 100 queries per second becomes 5,000 queries per second.

The database can’t handle the load. Queries start timing out. Your application retries those failed queries, which adds even more load.

What’s Happening?

Your conversation history table looks like this:

CREATE TABLE conversations (
    id SERIAL PRIMARY KEY,
    user_id INTEGER,
    agent_id INTEGER,
    message TEXT,
    created_at TIMESTAMP,
    metadata JSONB
);

CREATE INDEX idx_user_conversations ON conversations(user_id, created_at);

Every time an agent opens a chat, you run:

SELECT * FROM conversations 
WHERE user_id = ? 
ORDER BY created_at DESC 
LIMIT 50;

This query is fast when you have 1,000 users. But with 50,000 concurrent users, you’re running this query thousands of times per second.

Each query needs to scan the index, fetch rows, and return data. Under normal load, PostgreSQL handles this fine. Under spike load, it can’t keep up.

Connection Exhaustion

Your application opens a new database connection for every request.

With 5,000 concurrent connections, PostgreSQL hits its connection limit. New requests start failing with “too many connections” errors.

Your application’s connection pool settings:

pool_size = 50  # connections per app instance
max_overflow = 100  # additional connections when needed

With 50 application instances, you potentially create 7,500 connections. PostgreSQL can’t handle it.

Write Amplification

Your chatbot also writes every new message to the database:

INSERT INTO conversations (user_id, agent_id, message, created_at, metadata)
VALUES (?, ?, ?, NOW(), ?);

With 50,000 agents handling an average of 5 conversations simultaneously, you’re looking at 250,000 active conversations.

If each conversation generates 10 messages per minute, that’s 2.5 million writes per minute or roughly 41,000 writes per second.

PostgreSQL’s MVCC system creates a new row version for every write. Old versions become dead tuples that need to be cleaned up by autovacuum.

You start looking at sharding solutions, NoSQL databases, or managed services that promise infinite scale.

But OpenAI showed there’s another way.

The problem isn’t PostgreSQL, but how you’re using it.

Let’s redesign this architecture using the lessons from OpenAI’s guide and build something that scales.

Architecture Solution

OpenAI’s solution centers on one primary PostgreSQL instance with nearly 50 read replicas distributed across multiple geographic regions.

The architecture looks simple on paper, but making it work at this scale required extensive optimizations across multiple layers.

Here’s how we redesign the AI chatbot platform using OpenAI’s scaling principles.

We’re keeping the single-primary PostgreSQL architecture but adding critical layers that make it work at scale.

Each layer solves a specific problem from the previous section.

Layer 1: PgBouncer for Connection Pooling

Instead of your application connecting directly to PostgreSQL, all connections go through PgBouncer first.

Your 10,000 application connections get pooled into just 100 PostgreSQL connections. Connection setup time drops from 50ms to 5ms, and you never hit the connection limit.

Layer 2: Cache Locking

The critical feature that prevents a thundering herd during cache failures.

When your cache goes down, only one request per cache key hits PostgreSQL. All other requests wait for the cache to be repopulated instead of overwhelming the database.

Layer 3: Read/Write Split

95% of your chatbot queries are reads (fetching conversation history). Only 5% are writes (storing new messages).

All reads go to replicas. Only writes touch the primary. This keeps your primary free to handle write spikes without being saturated by read traffic.

Layer 4: Rate Limiting

Application-level rate limiting prevents any single user from overwhelming your system.

You set limits like 100 conversation fetches per minute per user and 30 new messages per minute per user. Even if someone tries to abuse your API, they can’t take down your database.

Layer 5: Regional Replicas

For a global chatbot, you deploy read replicas in each major region. Users read from their nearest replica, reducing latency from 200ms to 20ms.

How the System Handled Failures

• Cache failure: Cache locking prevents database overload. Only one request per key hits the database. Performance degrades but doesn’t crash.

• Replica failure: Traffic automatically routes to healthy replicas. There is no manual intervention needed.

• Primary failure: Hot standby promotes to primary automatically. Downtime is 30-60 seconds while read traffic continues uninterrupted.

I’ll be covering the complete implementation of this architecture in the upcoming AI Build & Deploy Series, where we’ll build this chatbot step by step with:

• Complete Docker setup with PgBouncer, PostgreSQL, and Redis

• Python code for the database router (read/write split)

• Cache locking implementation that prevents thundering herd

• FastAPI endpoints with rate limiting

• Load testing scripts to verify it works under pressure

• Monitoring dashboards to track what matters

The complete code, including error handling, connection pooling configuration, cache implementation, and deployment setup, will be available in the series.

Final Thoughts

OpenAI's approach to scaling PostgreSQL shows it can scale further than most people realize when you apply the right strategies.

It demonstrates that with proper architecture, query optimization, connection pooling, caching, and workload isolation, a single-primary PostgreSQL instance can serve hundreds of millions of users.

These principles work at 1,000 users, 100,000 users, or 10 million users.

Your database architecture doesn’t need to be perfect from day one, but it should be designed to evolve as your application grows.

If you’re building an AI application and wondering whether PostgreSQL can handle your scale, the answer is probably yes, as long as you build with these principles in mind.

AI Software Engineer

Let’s Build The Coding Future Together

It’s said that 12 months from now, AI will replace software engineers and will write all the code.

But one thing will remain constant — you will be there, whether AI replaces software engineers or not, your career will be intact as a software engineer.

More important is what you will become since the change is inevitable, but I’m bullish on what’s ahead.

I hit 10,000 followers on Medium, and most of you have been reading my content there. The engagement has been incredible, but I’ve felt the constraint for a while now.

Medium limits how deep I can go on technical subjects.

The platform works well for overviews and introductions, but when you ask detailed questions in the comments or request step-by-step implementations, I hit a wall.

That’s why we’re here on Substack.

This newsletter gives me the space to go as deep as needed without artificial constraints.

I can write comprehensive guides, include detailed code examples, and provide the kind of technical depth that helps you implement these tools in your workflow.

Every week, I’ll publish one deep technical guide covering AI tools, coding trends, and software engineering workflows. Think RAG implementations, tool integration strategies, MCP deep dives, and practical examples you can use immediately.

On top of that, you’ll get daily updates when something important drops. AI moves fast, and I know your time is valuable, so I’ll break down new releases, model updates, and feature announcements as they happen.

The goal is to keep you informed and save you time digging through release notes and technical blogs.

I test everything before writing about it. You’ll get my real experience with these tools.

Newsletter Topics

This newsletter covers four main areas that matter most to developers working with AI.

1. AI Coding Tools & Trends

New model releases, coding tools, and frameworks as they launch.

I break down what’s useful versus what’s just hype, test the tools in real projects, and share what works.

2. AI Software Engineering Workflows

How AI fits into your actual development process.

We’re talking about productivity gains, workflow automation, and practical integration strategies that don’t require rebuilding your entire stack.

3. AI Deep Technical Guides

Weekly deep dives on subjects like RAG implementations, tool integration patterns, and MCP server architectures.

These are hands-on guides with code examples and real implementations you can use immediately.

4. AI Simplified Research

I take research papers and technical blogs that would take hours to digest and break them down into actionable insights.

The goal is to extract what matters for developers and skip the academic overhead.

Every piece of content here serves one purpose — help you build better software with AI tools that are evolving faster than anyone can keep up with on their own.

The content is technical but accessible. I write for software engineers, AI engineers, freelancers, and tech enthusiasts who want substance over surface-level summaries.

How it Works

The newsletter runs on two tracks.

Weekly Deep Dives

Every week, one comprehensive technical guide.

These are detailed explorations of tools, implementations, and workflows that require more than a quick overview.

These guides take time to write because I test everything first. I run the tools in real projects, document what works and what doesn’t, then break it down in a way that makes sense for developers at different skill levels.

Daily Updates

When something important drops, you’ll hear about it fast.

New model releases, major feature updates, or significant changes in the AI coding space get covered the same day or within 24 hours. These are shorter, focused posts that break down release notes into what you actually need to know.

The goal is to save you time. Instead of reading through technical blogs and official documentation, you get the essential information and practical implications in a few minutes.

My Approach

I want to write about actual insights instead of theoretical overviews.

The goal isn’t to hype every new release but to give you honest assessments based on real usage or technical review.

This is a two-way conversation.
Your questions and the topics you want covered shape what I write about next, so don’t hesitate to reach out with what you’re working on or struggling with.

This newsletter is built for developers and technical professionals working with AI.

If you’re a software engineer exploring AI coding tools, an AI engineer building production systems, a freelancer trying to stay competitive, or a tech enthusiast who wants to understand how these tools work, this is for you.

Let’s Get Started

The first deep technical guide drops soon.

We’ll cover RAG implementations, tool integration patterns, and MCP architecture with code examples and practical use cases you can implement in your projects. It’s going to be detailed, technical, and useful.

In the meantime, you’ll start seeing daily updates as important releases happen. AI doesn’t wait for publishing schedules, so neither will we.

What I Need From You

This newsletter works best when it’s a conversation. If there are specific tools you want me to test, topics you’re struggling with, or areas where you need deeper guidance, let me know.

Reply to this email or drop a comment with what you’re working on or what you want to see covered. Your questions and interests directly shape what gets published here.

Support This Work

If you find value in these guides and updates, consider pledging your support. It helps me dedicate more time to testing tools, writing detailed tutorials, and keeping the content quality high.

You can pledge support here on Substack, which gives you early access to my books, upcoming AI coding course, and the ability to request specific topics you want covered.

Whether you’re here as a free subscriber or a paying supporter, you’re part of this community.

We’re navigating this together, and the goal is to make sure you come out ahead.

Welcome to the AI Software Engineer Newsletter

Let’s Build The Coding Future Together