好文集ChatGPT Plus¥119

How to Master System Design for Building with AI (Full Course)

文章目录

I'm going to show you how to make an AI agent think in system design before it writes code.

because the agent already types faster than you. what it lacks is a set of decisions about where data lives, what happens under load, and what happens when a part fails.

those decisions are system design.

if the decisions live only in your head, the agent guesses. if they live in a file and in tests, the agent can check its own work against them.

that's the whole move... turn each concept into a rule the agent reads and a test the agent must fail once on purpose.

these are the system design concepts an agent must think through. each one is something your agent must reason about, write down, and check.

one note before the plan.

every number in a worked example is illustrative unless a source is linked.

every test outcome is an expected outcome, not a result i measured.

here's the plan:

  • why prompting alone does not hold a design

  • how to approach a design

  • performance vs scalability

  • latency vs throughput

  • availability vs consistency

  • DNS, CDN, load balancer, reverse proxy

  • the application layer

  • the database

  • the cache

  • background jobs and retries

  • communication

  • security basics the agent must not skip

  • how to encode all of it for an agent

  • worked design: a feed, with and without the concepts

  • real systems as reference cards

  • where this method breaks

  • your first week

  • the law


why prompting alone does not hold a design

an agent solves the request in front of it.

ask for a feed and you get a feed. it works for one user on one laptop with one database and no failures.

the gaps appear later:

  • a webhook arrives twice

  • a second server starts

  • a dependency hangs for thirty seconds

  • someone needs a restore

none of that was in the request, so none of it is in the code.

a prompt cannot fix this alone.

the agent reads a prompt once. the next session starts without it.

three things persist between sessions:

  • SYSTEM.md holds the design decisions and the numbers behind them

  • MODULES.md holds one contract per module: what it owns, what it promises, how it fails

  • red tests hold the checks, and they run in continuous integration (CI), which runs every check on each proposed change

a red test is a test that you make fail on purpose once.

you inject the fault it targets and watch it go red. then you fix the code and watch it go green.

this step matters.

a new test that never failed has not shown that it can see its fault.

so the rule for the whole article is simple:

before you trust a new test, make it fail for the fault it is meant to catch.

keep the claim narrow.

this method gives you a concrete way to review the agent's work.

the tests catch the failures you define. they do not catch the failures you forget.

research on agent context files is also mixed:

that is why the file alone is not the method.

the file guides the agent. the tests check the behavior.

every concept below gets the same treatment: plain meaning, the decision the agent must make, a SYSTEM.md rule, and a red test.


how to approach a design

i use four steps. they work for an agent as well as for a person.

  1. outline use cases, constraints, and assumptions

  2. create a high level design with the main components

  3. design the core components in detail

  4. scale the design by finding and fixing bottlenecks

the failure to watch for is a start at step 3. the agent picks a framework and writes handlers before it knows who uses the system.

so make step 1 a written artifact. the agent answers these questions in SYSTEM.md before it writes code:

  • who uses it?

  • how do they use it?

  • how many users are there?

  • what does the system do?

  • what are the inputs and outputs?

  • how much data do we expect?

  • how many requests per second (RPS) do we expect?

  • what is the read to write ratio?

the agent may not know many of these answers. that is fine.

it must label each one.

  • MEASURED means you have data for it

  • TARGET means you chose it as a goal

  • GUESS means nobody knows yet

a GUESS is honest. a number with no label is a hidden guess.

back-of-the-envelope math

back-of-the-envelope math is a rough estimate done by hand. it tells you which problems are real at your size.

here is my conversion guide. it is easy to remember:

  • a month has about 2.5 million seconds

  • 1 request per second is about 2.5 million requests per month

  • 40 requests per second is about 100 million requests per month

  • 400 requests per second is about 1 billion requests per month

here it is on a paste site:

  • 10 million paste writes per month is about 4 writes per second

  • 100 million reads per month is about 40 reads per second

it also sizes storage.

at about 1.27 KB per paste, 10 million pastes a month is about 12.7 GB of new content per month. over three years that is about 450 GB.

those are assumptions, not measurements.

the method is the point.

for sizes, think in powers of two:

  • 2^10 is about a thousand (1 KB)

  • 2^20 is about a million (1 MB)

  • 2^30 is about a billion (1 GB)

  • 2^40 is about a trillion (1 TB)

an estimate only needs the right order of magnitude.

traffic is not even across the day. peak traffic is higher than the average.

choose a peak factor from your traffic data. until you have data, mark the factor GUESS.

concurrency is not a pool size

Little's law connects three numbers. the average number of requests in flight equals the arrival rate times the average time each request spends in the system.

so 1,000 requests per second at 200 ms each gives 200 requests in flight on average.

that is the concurrency your design must support. it is not your connection pool size.

to size a pool, the agent must do more work:

  1. divide the concurrency across the instances you run

  2. find where requests wait (database, model API, disk, lock)

  3. add headroom for bursts above the average

  4. load-test the limit and record the result as MEASURED

here is the intake prompt. paste it before any build.

before you write code, write the "intake" section of SYSTEM.md.

1. list the use cases in scope and out of scope.
2. answer: users, usage pattern, data size, requests per second,
   read to write ratio, peak factor.
3. label every number MEASURED, TARGET, or GUESS.
4. show the back-of-the-envelope math for requests per second,
   storage per month, and storage after 3 years.
5. compute average in-flight requests at peak with Little's law.
   do not call that number a pool size.
6. draw the high level design as a list of components and the
   calls between them.
7. stop and wait for my review. do not write code yet.

to practice the intake, run it on designs you already know:

  • a paste site

  • a timeline

  • a web crawler

  • a personal finance tracker

  • a social graph

  • a key-value cache for search

  • a sales rank by category

  • a web app that grows to millions of users


performance vs scalability

here is the split.

a performance problem means the system is slow for a single user.

a scalability problem means the system is fast for a single user but slow under heavy load.

a service is scalable if added resources give a proportional increase in performance.

the agent must classify a slow complaint before it fixes anything. the fixes are different.

a quadratic loop is a performance problem. more servers do not fix it.

a single in-process lock is a scalability problem. a faster query does not fix it.

the rule for SYSTEM.md:

---


## performance vs scalability
- every slowness ticket states one of: "slow for one user" or
  "slow under load". attach the measurement.
- the app tier is stateless. sessions, uploads, and locks live in
  shared stores, never in process memory.
- a fix for "slow under load" includes a load test before and after.

the red test:

test: scale_out_raises_throughput
  run the load test against 1 app instance, record successful
  requests per second (exclude 401 and other auth failures)
  run the same load test against 2 app instances
  assert: successful RPS with 2 instances >= 1.6x successful RPS
          with 1 (threshold is an example; set yours from data)
  break: keep sessions in process memory -> requests on the
         second instance fail auth; raw throughput can stay high
         from fast 401s while successful RPS drops (RED)

latency vs throughput

latency is the time to perform one action.

throughput is the number of actions per unit of time.

aim for maximal throughput with acceptable latency. the agent must define "acceptable" as a number.

published tables of latency numbers help with orders of magnitude. memory is faster than disk, and a call inside one data center is faster than a call across regions.

treat those tables as historical rules of thumb. the agent measures its own system and labels the result MEASURED.

averages hide the slow tail. so use percentiles.

p50 means 50% of requests finish at or below this latency. p99 means 99% of requests finish at or below this latency.

tail latency grows when one request waits on many calls. the request is only as fast as its slowest dependency.

the failure to watch for is an external call with no timeout. one hung dependency then holds a worker until the worker pool is empty.

the rule:

---


## latency
- every endpoint has a p99 TARGET in SYSTEM.md.
- every external call has a timeout shorter than the endpoint budget.
- if a non-critical dependency times out, the endpoint returns a
  degraded response. it does not hang.

the red test:

test: slow_dependency_does_not_block_reads
  configure the fake link-preview service to hang for 30 seconds
  load the home timeline endpoint, which shows link previews,
  at TARGET peak
  assert: p99 stays under the endpoint TARGET
  assert: no worker waits longer than the configured timeout
  break: remove the client timeout -> workers stall and p99 climbs
         past TARGET (RED)

availability vs consistency

this is the CAP theorem. CAP stands for consistency, availability, and partition tolerance.

here are the terms:

  • consistency means every read receives the most recent write or an error

  • availability means every request receives a response, which may not hold the most recent write

  • partition tolerance means the system keeps working when the network splits its parts

networks fail, so a distributed system must tolerate partitions. during a partition, it must choose between consistency and availability.

CP means the system picks consistency during a partition.

some requests time out or return errors. it fits data that needs atomic reads and writes.

AP means the system picks availability during a partition.

reads can return stale data. it fits data that can be eventually consistent.

the agent must make this choice per data type, not per system. one app holds both kinds.

here is how i would sort a typical product. these are my defaults, not a law.

  • money, plan status, permissions, inventory: consistent

  • like counts, view counts, follower counts, feed order: can be eventual

consistency patterns

i use three patterns.

  • weak consistency: after a write, reads may or may not see it. examples are memcached and realtime uses such as voice calls and games.

  • eventual consistency: after a write, reads see it after a delay. data is copied asynchronously. examples are DNS and email.

  • strong consistency: after a write, reads see it. data is copied synchronously. examples are file systems and relational databases.

the failure to watch for here is read-your-writes. a user posts, the app reads from a replica that lags, and the new post is missing.

replicas lag. the delay can grow under load or failure.

the rule:

---


## consistency
- each table or collection is labeled CONSISTENT or EVENTUAL.
- CONSISTENT data is read from the primary.
- after a user writes, that user's next reads of the same data go
  to the primary for a TARGET window (example: 10 seconds).

the red test:

test: author_sees_own_post_under_replica_lag
  inject 5 seconds of lag on the read replica
  user A creates a post
  user A loads their own profile timeline at once
  assert: the new post is present
  break: route the author's reads to the replica -> the post is
         missing (RED)

availability patterns

there are two complementary patterns: failover and replication.

failover moves traffic to a standby when the active server fails.

  • active-passive: the active and passive servers exchange heartbeats. if the heartbeat stops, the passive server takes over. downtime depends on whether the standby is hot (running) or cold (must start).

  • active-active: both servers take traffic and share the load.

failover has costs. it adds hardware and complexity.

data can be lost if the active server fails before new writes reach the standby.

replication keeps copies of data on more than one server. the database section covers it.

availability in numbers

availability is measured in nines. here are the downtime budgets:

  • 99.9% (three nines): about 8h 45m per year, about 43m 50s per month

  • 99.99% (four nines): about 52m 36s per year, about 4m 23s per month

parts in a chain multiply. two independent parts at 99.9% in sequence give about 99.8%.

parts in parallel can raise availability. if failures are independent and failover works, two parallel parts at 99.9% give about 99.9999%.

shared failures break that estimate. shared power, shared network, shared data, and a shared bad deploy all fail together.

recovery: RPO and RTO

availability also covers recovery from data loss.

  • RPO (recovery point objective) is how much recent data you can afford to lose

  • RTO (recovery time objective) is how long a recovery can take

a daily backup alone cannot meet a 5-minute RPO. a restore from last night can lose almost a full day.

to meet a short RPO, archive the database's write-ahead log (WAL) continuously.

the WAL is the log of every change. you restore a base backup, then replay the WAL to a chosen point.

GitLab's 2017 outage shows why the drill matters. GitLab.com was not using WAL archiving.

an engineer accidentally removed data from the primary while trying to rebuild replication.

several backup methods had failed. a six-hour-old snapshot allowed a partial restore (https://about.gitlab.com/blog/postmortem-of-database-outage-of-january-31/).

the rule:

---


## recovery
- RPO: 5 minutes (TARGET). RTO: 30 minutes (TARGET).
- daily base backups plus continuous WAL archiving.
- backups go to a separate account. the app and the agent cannot
  delete them.
- a restore drill runs on a schedule and its result is logged.

the red test:

test: restore_meets_rpo
  restore the latest base backup to a scratch database
  replay archived WAL to a recovery point 5 minutes before "now"
  assert: no committed write older than the recovery point is lost
  assert: total recovery time is under the RTO
  break: disable WAL archiving -> the newest recovery point is the
         base backup and misses the RPO (RED)

DNS, CDN, load balancer, reverse proxy

these four sit between the user and your code. the agent often configures them instead of writing them.

faults hide in that configuration.

DNS

the domain name system (DNS) turns a name like example.com into an IP address.

resolvers cache each answer for its TTL (time to live). a change can take up to the TTL to reach users, and some resolvers hold records longer.

the agent must not assume a DNS change is instant.

  • inspect the real TTL on each record before a move

  • lower the TTL ahead of a planned change

  • keep the old target alive until the old TTL has passed

CDN

a content delivery network (CDN) is a set of servers near users that store and serve copies of your content.

there are two kinds.

  • a push CDN receives content when you upload it. you control expiry.

  • a pull CDN fetches content from your server on the first request, then caches it until the TTL ends.

the CDN is a shared cache. that is the danger.

if a private response enters a shared cache, the next user can receive it.

the agent must sort every response into public or private before it sets cache headers.

the rule:

---


## edge caching
- the CDN caches only responses marked public.
- any response that depends on the viewer (auth, follows, plan)
  sends Cache-Control: private, no-store.
- the CDN config never ignores the Authorization header or the
  session cookie for authenticated routes.

the red test. note that it proves A can read the post first.

a test that only checks B can pass for the wrong reason.

test: followers_only_post_not_served_from_shared_cache
  user A follows the author and requests a followers-only post
  assert: A receives 200 and the post body
  user B does not follow the author and requests the same URL
  through the CDN
  assert: B receives 403 or 404 and no part of the body
  break: configure the CDN to cache this route and ignore
         authorization -> B receives A's cached body (RED)

load balancer

a load balancer spreads incoming requests across several servers.

it also stops sending traffic to servers that fail health checks.

there are two levels.

  • layer 4 balancing looks at transport data such as IP address and port

  • layer 7 balancing looks at application data such as headers, URL path, and cookies

load balancers enable horizontal scaling, which means adding more servers instead of bigger ones.

horizontal scaling needs stateless servers.

if a server keeps sessions in memory, a user who lands on another server loses the session.

a load balancer can also become a single point of failure. run more than one, or use a managed one.

reverse proxy

a reverse proxy is a web server that sits in front of your app servers. it can terminate TLS (the encryption on HTTPS), compress responses, serve static files, and hide the backend servers.

here is the line:

  • a load balancer is useful when you have several servers

  • a reverse proxy is useful even with one server

the rule for both:

---


## edge and balancing
- at least 2 app instances behind a load balancer with health checks.
- app instances are stateless (see performance vs scalability).
- TLS terminates at the proxy or load balancer. internal hops that
  cross a network boundary are also encrypted.

the red test:

test: kill_one_instance_under_load
  run the load test at TARGET peak against 2 instances
  kill one instance mid-test
  assert: error rate stays under the TARGET budget
  assert: the load balancer removes the dead instance within the
          health check window
  break: store sessions in process memory -> users on the killed
         instance lose sessions and get 401 (RED)

the application layer

i recommend a separate application layer, also called a platform layer:

  • the web layer handles requests

  • the application layer and its workers do the work

  • each layer scales on its own

microservices take this further. each service is small, deployed on its own, and owns one job.

the split has costs. microservices add architecture, deployment, and operations work.

watch for two opposite failures:

  • the agent writes one giant file

  • it splits into many services on day one

many services need service discovery.

each service registers its location in a registry such as Consul, etcd, or ZooKeeper. other services look it up there.

health checks remove dead instances.

the middle path is a modular monolith.

it is one deployable app with strict module boundaries inside it, plus a separate worker process.

split a module into a service only for a named reason. examples are a different scaling need, a different owner, or a different failure domain.

the MODULES.md contract is the boundary. it is covered in the encode section.

the rule:

---


## application layer
- one web process, one worker process, one shared database.
- modules talk through their public interface only.
- a new service needs an ADR (architecture decision record) that
  names the reason for the split.

the red test:

test: module_boundaries_hold
  run the import checker over the codebase
  assert: no module imports another module's internal files
  break: import billing/internal/plans from feed -> checker
         fails (RED)

the database

the database holds the source of truth. the agent must treat it with more care than any other part.

relational databases

a relational database stores data in tables with a fixed schema. it supports ACID transactions.

  • atomicity: a transaction applies fully or not at all

  • consistency: a transaction moves the data from one valid state to another

  • isolation: concurrent transactions behave as if they ran one after another

  • durability: a committed transaction survives a crash

there are six ways to scale a relational database. three matter on day one.

  • primary-replica replication: the primary takes writes and copies them to replicas, which serve reads

  • denormalization: store redundant copies of data to avoid expensive joins on reads, at the cost of more complex writes

  • SQL tuning: tighten the schema, add good indexes, avoid expensive joins, partition large tables

primary-replica has a known cost. if the primary fails, a replica must be promoted, and writes in flight can be lost.

the other three matter when one primary cannot hold the writes.

  • primary-primary replication: two primaries both take writes. it needs conflict resolution, and it adds write latency or loosens consistency. use it only with a clear reason.

  • federation: split databases by function, such as users, posts, and billing. each database gets less load, but joins across databases get harder.

  • sharding: split one dataset across databases by key. use consistent hashing, which moves only some keys when the shard count changes, or a versioned shard map.

each move needs an ADR that names the write rate that forced it.

NoSQL

NoSQL is a broad name for databases that are not relational. there are four kinds:

  • key-value store: looks up a value by key, often from memory

  • document store: stores documents such as JSON, queried by key or content

  • wide column store: groups columns into families, built for very large data

  • graph database: stores nodes and the edges between them

SQL or NoSQL

each has its reasons:

  • SQL fits structured data, strict schemas, joins, and transactions

  • NoSQL fits flexible schemas, very large data, and very high write rates

keep the choice narrow. SQL and NoSQL products make different tradeoffs.

check the specific database's transactions, scaling model, schema rules, and consistency guarantees.

watch for two failures:

  • the agent picks whatever the last tutorial used

  • it ships a hot query with no index

my default for an agent build is one relational database until a named access pattern needs something else. that is a heuristic, not a law.

the rule:

---


## database
- one relational database is the source of truth.
- every hot query path has an index. the query plan is checked in CI.
- schema changes go through migrations. each migration is reviewed
  and has a tested rollback or a forward fix plan.
- the agent's credentials cannot run destructive statements on
  production (DROP, TRUNCATE, DELETE without WHERE).
- a new datastore needs an ADR that names the access pattern.

the red test:

test: hot_queries_use_index
  seed the test database with a production-sized row count (GUESS)
  run EXPLAIN on each query tagged "hot"
  assert: no full table scan on tables above the size threshold
  break: drop the index on posts(author_id, created_at) -> the
         timeline query plan shows a full scan (RED)

the cache

a cache stores the result of expensive work. it trades fresh data for speed.

a cache can sit in five places:

  • client caching: in the browser or app

  • CDN caching: at the edge

  • web server caching: in the reverse proxy

  • database caching: inside the database

  • application caching: in an in-memory store such as memcached or Redis, between the app and the database

application caching works at two levels.

  • query-level caching stores the result of a query. it is hard to invalidate, because one changed row can affect many cached queries.

  • object-level caching stores an assembled object such as a user profile or a post. it is easier to invalidate when the object changes.

i prefer object-level caching for that reason.

when to update the cache

this is the decision the agent skips.

there are four patterns, and SYSTEM.md must name one.

  • cache-aside: the app checks the cache. on a miss, it reads the database, then writes the result to the cache. data can be stale until the TTL ends. each miss costs extra trips.

  • write-through: the app writes to the cache, and the cache writes to the database at once. reads are fast and fresh. writes are slower, and the cache can fill with data nobody reads.

  • write-behind: the app writes to the cache, and the database update happens later. writes are fast. data can be lost if the cache fails first.

  • refresh-ahead: the cache refreshes hot entries before they expire. it helps only if you can predict which entries will be read.

two more faults matter for agents.

a stampede happens when a hot key expires and many requests miss at once.

they all hit the database together.

a cache that becomes the only copy of data is no longer a cache.

a flush then loses data.

the rule:

---


## cache
- the database is the source of truth. the cache is disposable.
- default pattern: cache-aside, object-level, with a TTL on every key.
- every write path that changes a cached object deletes or updates
  that key.
- write-behind is not allowed for data we cannot lose.
- hot keys use single-flight loading: one request loads, the
  others wait for that result.

the red test:

test: hot_key_miss_loads_once
  expire the cache entry for a hot post
  send 100 concurrent reads for that post
  assert: the database receives exactly 1 load query for it
  break: remove the single-flight lock -> the database receives
         many load queries (RED)

background jobs and retries

some work is too slow to do inside a request. the answer is a queue.

  1. the app publishes a job to the queue and tells the user the job is accepted

  2. a worker takes the job, does the work, and marks it complete

the user is not blocked.

the work happens in the background.

  • a message queue holds and delivers messages

  • a task queue receives tasks with their data, runs them, and returns results

task queues often support scheduling.

do not overuse queues.

cheap calculations and realtime flows can be simpler without a queue, because a queue adds delay and complexity.

duplicates are normal

some queues can deliver a message twice, and some brokers can lose messages.

products change. check your queue's documented delivery behavior and design for duplicates.

many queues deliver at least once. at-least-once means a message can arrive more than once, but it should not be lost.

so every consumer must be idempotent.

idempotent means running it twice has the same effect as running it once.

local idempotency is the easy part.

store the event ID and the local state change in one transaction, with a unique constraint on the event ID.

a duplicate then fails the insert and does nothing.

external side effects are the hard part. a payment, an email, or a push notification leaves your system.

a local unique row cannot stop a duplicate external call.

picture the crash:

  1. the provider accepts the call

  2. the worker dies before it records the result

  3. the retry has no record and calls again

there are two safe options:

  • send the same idempotency key on every attempt, to a provider that deduplicates by that key

  • if the provider cannot deduplicate, record the attempt as UNCERTAIN before the call, and reconcile with the provider before any retry

back pressure

a queue can grow faster than workers drain it.

a queue larger than memory causes disk reads and slower performance.

back pressure limits the queue size. when the queue is full, new requests get HTTP 503, which means "server busy, try again later".

clients retry with exponential backoff, which increases the delay after each failure.

add jitter, which is a small random change to each delay. jitter stops all clients from retrying at the same moment.

retries need a limit. after the last attempt, move the job to a dead-letter queue, which holds failed jobs for a human or a repair script.

the rule:

---


## background jobs
- slow or external work runs in the worker, not in the request.
- every consumer is idempotent. event ID + local state change are
  written in one transaction with a unique constraint on event ID.
- external calls send the same idempotency key on every attempt.
  if the provider cannot deduplicate, mark the attempt UNCERTAIN
  and reconcile before retrying.
- retries: max 5 attempts (example), exponential backoff with
  jitter, then the dead-letter queue.
- each queue has a max depth. when full, enqueue returns 503 with
  Retry-After.
- queue lag has an alert threshold (GUESS until measured).

the red tests:

test: payment_event_applies_once
  deliver the same "payment succeeded" event 5 times
  assert: the user has 1 plan change and 1 ledger entry
  break: remove the unique constraint on event ID -> a second
         delivery applies another plan change and ledger entry
         (RED)
test: provider_call_applies_once
  configure the fake payment provider to deduplicate by key
  crash the worker after the provider accepts a charge and
  before the local ack
  retry with the same idempotency key
  assert: the provider records exactly 1 operation
  break: generate a new idempotency key on each attempt -> the
         provider records 2 or more operations (RED)
test: billing_reconciles_uncertain_before_retry
  configure the fake provider to accept a charge, then hang so
  the caller marks the attempt UNCERTAIN
  attempt a retry before reconcile completes
  assert: no second provider call until reconcile finishes
  assert: after reconcile, the ledger matches the provider
  break: retry immediately on timeout without reconcile -> the
         provider records 2 operations (RED)
test: full_queue_sheds_load
  set the notification queue max depth to 1,000 (example)
  stop the workers and enqueue 1,500 jobs
  assert: jobs 1,001 to 1,500 receive 503 with Retry-After
  assert: after workers restart, the first 1,000 jobs complete
  break: remove the max depth -> jobs 1,001 to 1,500 are accepted
         and no 503 is returned (RED)

communication

agents need a small part of networking.

they need to know which calls can be retried, and what each call does on failure.

HTTP

HTTP (hypertext transfer protocol) is the request and response protocol of the web.

a client sends a request with a verb and a path. the server returns a status and a body.

here are the verbs, and which ones are idempotent:

  • GET reads a resource. it is idempotent and safe.

  • PUT creates or replaces a resource. it is idempotent.

  • PATCH partly updates a resource. treat it as not idempotent.

  • DELETE removes a resource. it is idempotent.

  • POST creates a resource or triggers a process. it is not idempotent.

that list drives retry rules.

a client can retry a GET. a client that retries a POST can create a duplicate, unless the server accepts an idempotency key.

TCP and UDP

  • TCP (transmission control protocol) delivers packets in order and resends lost ones. it fits web and API traffic

  • UDP (user datagram protocol) can lose or reorder packets. it fits realtime voice, video, and games

the agent seldom needs to choose.

RPC and REST

RPC (remote procedure call) makes a call to another machine look like a local function call. it fits internal calls where performance matters.

REST (representational state transfer) models the API as resources acted on with HTTP verbs. it fits public APIs.

the rule:

---


## communication
- public API: REST over HTTPS. internal calls: REST or RPC, chosen
  per module in MODULES.md.
- every outbound call has a timeout, a retry policy, and a flag:
  SAFE_TO_RETRY or NEEDS_IDEMPOTENCY_KEY.
- every POST that creates a resource accepts an Idempotency-Key
  header and stores the key with the result.

the red test:

test: create_post_retry_creates_once
  send POST /posts with Idempotency-Key K
  drop the response before the client receives it
  resend POST /posts with the same key K
  assert: exactly 1 post exists, and both responses return its ID
  break: ignore the Idempotency-Key header -> 2 posts exist (RED)

security basics the agent must not skip

security starts with four basics:

  • encrypt data in transit and at rest

  • sanitize user input to prevent XSS and SQL injection

  • use parameterized queries to prevent SQL injection

  • use the principle of least privilege

  • XSS (cross-site scripting) puts a script into a page that another user loads

  • SQL injection changes a database query through user input

  • least privilege means each part gets only the access it needs

for agent builds, i add four items.

  • secrets never enter the repository, the prompt, or the logs

  • the agent's credentials are separate from production credentials

  • backups live where the app and the agent cannot delete them

  • every public endpoint has a rate limit per user, per API key, and per IP

the backup item comes from incidents.

in July 2025, a user reported that a Replit agent deleted production data during a code freeze.

Replit confirmed the deletion and said a rollback restored the data (https://replit.com/blog/doubling-down-on-our-commitment-to-secure-vibe-coding).

the lesson is not "agents are dangerous". the lesson is that access is a design decision.

write it down and test it.

rate limiting is an edge control. it caps how many requests one caller can send in a time window.

over the limit, the caller gets HTTP 429, which means "too many requests", with Retry-After.

the agent must encode it, because a public endpoint with no limit invites scraping, credential stuffing, and runaway bills.

this matters more when an endpoint calls a paid model API.

put the limit at the edge or the load balancer, and keep the counters in a shared store. a counter in process memory resets per instance, so two instances double the real limit.

the rule:

---


## security
- TLS on every external connection. storage encrypted at rest.
- all queries are parameterized. no string-built SQL.
- user content is escaped on output. rich text passes an allowlist
  sanitizer.
- secrets come from the secret store. a scanner runs in CI.
- the agent's database role: read-only on production. write access
  only on dev and scratch databases.
- backups: separate account, write-once retention, no delete
  permission for app or agent roles.
- public endpoints: rate limit per user, key, and IP (TARGET).
  counters live in a shared store. over limit returns 429.

the red tests:

test: injection_payload_returns_nothing_extra
  search posts with the input: ' OR '1'='1
  assert: the response contains only posts that match the literal
          text, which is none
  break: build the search query with string concatenation -> the
         response returns every post (RED)
test: agent_role_cannot_destroy
  connect to a production-like database with the agent's DB role
  attempt DROP TABLE posts
  assert: DROP is denied
  with the agent's storage credentials, attempt to delete a
  backup object
  assert: the backup delete is denied
  break: grant the DB role ownership of the schema, and grant
         the storage credentials delete on the backup bucket ->
         DROP succeeds on the scratch copy and the backup object
         is deleted (RED)

how to encode all of it for an agent

you now have a rule and a test for each concept.

the agent needs them in three places, and it needs a loop that uses them.

SYSTEM.md

SYSTEM.md is the design file at the root of the repository. the agent reads it at the start of every session.

keep it short. it holds decisions and numbers, not explanations.

the explanations live in ADRs, one per decision.

here is a starter block. copy it, then replace every value with your own.

the values shown are examples.

# SYSTEM.md
---


## intake
- users: 50,000 registered (GUESS)
- peak RPS, reads: 120 (GUESS, peak factor 10 on the daily average)
- peak RPS, writes: 5 (GUESS; posts + likes + follows at peak)
- read to write ratio: about 100:1, timeline reads to posts (GUESS)
- data growth: about 4 GB per year of post text (GUESS)
- average in-flight requests at peak: about 12 (Little's law).
  this is not a pool size. pools are sized by load test.
---


## targets
- home timeline p99: 300 ms (TARGET)
- availability: 99.9% per month (TARGET)
- RPO: 5 minutes. RTO: 30 minutes. (TARGET)
---


## data
- source of truth: one Postgres primary.
- CONSISTENT: users, follows, plans, ledger, post visibility.
- EVENTUAL: like counts, view counts, timeline order.
- authors read their own writes from the primary for 10 seconds.
- backups: daily base backup + continuous WAL archive, separate
  account, no delete permission for app or agent roles.
---


## edge
- CDN caches public responses only.
- viewer-dependent responses: Cache-Control: private, no-store.
- 2+ stateless app instances behind a load balancer.
---


## cache
- cache-aside, object-level, TTL on every key.
- writes delete the affected keys.
- hot keys use single-flight loading.
---


## jobs
- notifications, fan-out, and payment events run in the worker.
- consumers are idempotent. external calls carry an idempotency key.
- retries: bounded, exponential backoff with jitter, dead-letter queue.
- queues have a max depth. full queue returns 503.
---


## calls
- every outbound call has a timeout and a retry flag.
- POST create endpoints accept Idempotency-Key.
---


## security
- parameterized queries only. output escaping on user content.
- secrets from the secret store. scanner in CI.
- agent role: read-only on production.
- rate limits on public endpoints. shared counters. 429 over limit.
---


## rules -> tests
- each rule above names its red test in tests/red/.
- a rule with no test is marked UNTESTED and listed in the PR.

MODULES.md

MODULES.md holds one contract per module. a contract tells the agent what the module owns, what it promises, and how it fails.

every module has an easy part and a hard part. the contract names the hard part, so the agent spends its care there.

here is one contract. it is the module with the most external risk in the worked design.

---


## module: billing

owns
- tables: plans, subscriptions, ledger_entries, payment_events
- the only code that calls the payment provider

public interface
- start_checkout(user_id, plan) -> checkout_url
- handle_payment_event(event) -> applied | duplicate
- charge_with_provider(intent) -> ok | failed | uncertain
- current_plan(user_id) -> plan

consistency
- CONSISTENT. reads of current_plan go to the primary.

promises
- handle_payment_event makes no outbound call. it cannot return
  uncertain. a payment event changes plan state at most once.
- charge_with_provider is the only path that calls the provider.
- each provider call sends the same idempotency key on every attempt.
- if the provider result is unknown, charge_with_provider returns
  uncertain. reconcile with the provider before any retry.

failure behavior
- provider timeout: 10 s (example). charge_with_provider returns
  uncertain. do not retry until reconcile finishes.
- provider down: checkout returns 503 with Retry-After. plan state
  does not change.

the hard part
- the crash between "provider accepted" and "we recorded it".

red tests
- payment_event_applies_once
- provider_call_applies_once
- billing_reconciles_uncertain_before_retry

not allowed
- other modules read billing tables directly.
- the agent changes this contract without an ADR.

the red test list

the eight red tests to write first. each one is named for the fault it catches.

the article defines more elsewhere.

tests/red/
  followers_only_post_not_served_from_shared_cache
    fault: a private response enters the shared CDN cache
  payment_event_applies_once
    fault: a redelivered payment event upgrades or credits twice
  restore_meets_rpo
    fault: backups cannot reach the recovery point (no WAL archive)
  author_sees_own_post_under_replica_lag
    fault: the author reads a stale replica after a write
  hot_key_miss_loads_once
    fault: a cache stampede on a hot key
  full_queue_sheds_load
    fault: an unbounded queue with no back pressure
  pool_holds_at_burst
    fault: pools sized to average concurrency with no headroom
  agent_role_cannot_destroy
    fault: the agent's credentials can drop data or delete backups

the pool test is new here. it checks the Little's law trap directly.

test: pool_holds_at_burst
  run the load test at TARGET peak, then a burst at 2x peak
  for 60 seconds (example)
  add 100 ms of injected latency on the database
  assert: no request waits for a database connection longer than
          the pool wait budget (example: 50 ms)
  assert: p99 stays under the endpoint TARGET during the burst
  break: set each instance's pool to the average in-flight count
         divided by instances, with no headroom -> pool waits
         exceed the budget during the burst (RED)

the agent loop

the files do nothing if the agent skips them. so the loop is written down too.

  1. read SYSTEM.md and the MODULES.md entry for the module you will touch

  2. state which rules the change affects

  3. write or update the red test for each affected rule

  4. inject the fault and confirm the test goes red

  5. write the code until the test goes green

  6. run the full suite

  7. if a decision changed, write an ADR and update SYSTEM.md

  8. open the PR with the red and green output pasted in

the prompt that starts each session:

you are working in a repo with SYSTEM.md and MODULES.md.

before you change code:
1. read SYSTEM.md and the MODULES.md entry for each module you touch.
2. list the SYSTEM.md rules this change affects.
3. for each rule, name its red test in tests/red/. if none exists,
   write one.
4. inject the fault the test targets. run it. paste the RED output.
5. then write the code. run it. paste the GREEN output.

rules:
- do not change a MODULES.md contract without writing an ADR.
- do not raise a number from GUESS to MEASURED without a run log.
- do not call average in-flight requests a pool size.
- if you cannot test a rule, mark it UNTESTED in the PR description.

worked design: a feed, with and without the concepts

this is a worked design, not an experiment. i did not run both versions and measure them.

the "without" column is the shape i would expect from a bare prompt. the "with" column is the design the concepts push the agent toward.

the product is a small community feed. users post, follow each other, read a home timeline, mark posts followers-only, get push notifications, and can pay for a pro plan.

the bare prompt

build a community feed app. users can post, follow people, and see
a home timeline of posts from people they follow. posts can be
public or followers-only. send a push notification when someone you
follow posts. add a paid pro plan with the payment provider.

what the bare prompt can produce

this is a plausible shape, not a recorded output.

  • the home timeline is one query that joins follows and posts on every request

  • the followers-only check lives in the handler, and the route shares the CDN rule of public pages

  • the post endpoint sends push notifications inline, one call per follower

  • the payment webhook upgrades the user each time it arrives

  • one database, with a nightly dump on the same account

  • the agent uses the database owner credentials

each item works in a demo.

each item maps to a concept above.

  • the timeline join is a scalability problem. it is fast at 10 follows and slow at 1,000.

  • the CDN rule is an edge caching problem. one misconfiguration exposes followers-only posts.

  • inline notifications are a latency problem. one slow provider slows every post.

  • the webhook is an idempotency problem. a redelivery upgrades or credits twice.

  • the nightly dump is an RPO problem. it can lose almost a day.

  • owner credentials are a least-privilege problem.

the concept-aware prompt

build a community feed app. follow SYSTEM.md and MODULES.md.

before any code:
1. write the intake section with labeled numbers.
2. write MODULES.md entries for: accounts, graph, posts, timeline,
   notifications, billing.
3. write the red tests in tests/red/ and paste their RED output.

design constraints:
- home timeline reads come from a precomputed timeline per user,
  built by fan-out on write in the worker.
- for authors above a follower threshold (GUESS), skip fan-out and
  merge their posts at read time.
- followers-only posts are never cached in a shared cache.
- notifications and fan-out are background jobs with bounded
  retries and a dead-letter queue.
- payment events are idempotent with provider-side deduplication.
- backups: base backup + continuous WAL archive, separate account.
- the agent role is read-only on production.

the intake math

the numbers are GUESS values for a small community. the method matters, not the values.

  • 50,000 users, each loading the home timeline 20 times a day: 1,000,000 reads per day

  • 1,000,000 reads over 86,400 seconds: about 12 reads per second on average

  • peak factor 10 (GUESS): about 120 reads per second at peak

  • 10,000 posts per day: about 0.12 posts per second on average

  • peak factor 10 on posts: about 1.2 posts per second at peak

  • likes and follows add the rest of the write budget. starter peak

writes of 5 RPS is posts + likes + follows (GUESS)

  • 100 followers per author on average (GUESS): about 1,000,000 timeline inserts per day, about 12 per second on average

  • post text at about 1 KB: about 10 MB per day, about 3.7 GB per year

  • timeline cache: 50,000 users times 800 entries times 17 bytes per entry is about 680 MB

the 17-byte entry stores a post ID (8 bytes), a user ID (8 bytes), and one meta byte.

Little's law at peak: 120 reads per second times 0.1 seconds average time in system is about 12 requests in flight. across 2 instances, that is about 6 each, on average.

that average is the floor, not the pool size.

the pool test sets the real number.

the design the concepts produce

  1. the client calls a reverse proxy, which terminates TLS and forwards to 2 stateless app instances behind a load balancer.

  2. the posts module writes the post to Postgres, the source of truth. it enqueues a fan-out job with the post ID.

  3. the worker reads the author's followers and inserts the post ID into each follower's timeline list in the in-memory store.

  4. the insert is keyed by post ID, so a redelivered job does not add a duplicate entry.

  5. the worker enqueues one notification job per batch of followers. the notification provider call carries an idempotency key.

  6. the timeline module reads the timeline list, then loads post objects through cache-aside with single-flight loading.

  7. followers-only posts pass a visibility check against the graph module, and the response is private, no-store.

  8. the billing module handles payment events as in its contract.

  9. Postgres takes a daily base backup and archives WAL continuously to a separate account.

fan-out on write has a known limit. it is fast unless the author has millions of followers.

the prompt handles that with a read-time merge for large authors.

the timeline cache is EVENTUAL. if it is lost, the worker rebuilds it from Postgres.

a cache flush slows reads but changes no answers.

expected red test outcomes

each test in the red test list has a break line. the expected outcome is RED with the break applied and GREEN with the design above.

these are specifications.

they become evidence only when you run them and paste the output.

in this worked design, the model is the same on both sides. what changes is the set of decisions the agent must write down and test.


real systems as reference cards

these cards come from my reference list of real-world architectures. each card names one move and links it to a rule above.

the cards are not proof that the method works. they show that the concepts are the same ones large systems faced.

card 1: memcache at Facebook

source: Scaling Memcache at Facebook, NSDI 2013 (https://cs.uwaterloo.ca/~brecht/courses/854-Emerging-2014/readings/key-value/fb-memcached-nsdi-2013.pdf)

  • the paper describes memcache as a look-aside cache in front of the databases, which is the cache-aside pattern

  • the paper describes leases, which address stale sets and thundering herds on hot keys

  • links to: the cache rules, hotkeymissloadsonce

card 2: Dynamo

source: Dynamo: Amazon's Highly Available Key-value Store, SOSP 2007 (http://www.read.seas.harvard.edu/~kohler/class/cs239-w08/decandia07dynamo.pdf).

  • Dynamo chooses availability for writes, with the shopping cart as the example, and accepts eventual consistency

  • it stores each key on N replicas. a read waits for R replicas and a write waits for W replicas

  • conflicting versions are reconciled later, sometimes by the application

  • R plus W greater than N does not alone guarantee strong consistency. failures and sloppy quorums still allow stale reads

  • links to: the CONSISTENT and EVENTUAL labels in SYSTEM.md

card 3: Spanner

source: Spanner: Google's Globally-Distributed Database, OSDI 2012 (http://research.google.com/archive/spanner-osdi2012.pdf)

  • Spanner offers strongly consistent transactions across data centers

  • it uses TrueTime, a clock API with a bounded uncertainty, and commits wait out that uncertainty

  • the move: strong consistency across regions is possible, and it costs commit latency

  • links to: the availability vs consistency choice. pay the latency only for CONSISTENT data

card 4: the Google File System

source: The Google File System, SOSP 2003 (http://static.googleusercontent.com/media/research.google.com/zh-CN/us/archive/gfs-sosp2003.pdf)

  • GFS stores files as large chunks, replicates each chunk across servers, and keeps metadata on a single master

  • it is built for large files and append-heavy workloads

  • the move i take from it is my reading, not a GFS rule. keep large blobs in object storage and metadata in the database

  • links to: the database rules and the storage math in intake

card 5: Dapper

source: Dapper, a Large-Scale Distributed Systems Tracing Infrastructure (http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36356.pdf)

  • Dapper traces a request across services by passing trace context with each call

  • it samples traces to keep overhead low

  • the move for an agent: pass a request ID through every call and every queued job. then a fan-out job can be traced back to the post that caused it

  • links to: background jobs, and the agent loop step that pastes red and green output


where this method breaks

this method has limits.

name them in your SYSTEM.md too.

  • a test catches only the fault you defined. if you chose the wrong invariant, the test passes and the system is still wrong.

  • a staging load test is not production traffic. treat its numbers as MEASURED in staging only.

  • a provider can behave differently from your fake. check the provider's documented idempotency and delivery behavior.

  • a managed platform may already handle some of these rules. then the rule is "verify the platform setting", not "build it".

  • a small app may not need every rule on day one. start with the rules that protect data and money.

  • the requirements can be wrong. a perfect design for the wrong product is still the wrong product.

  • context files guide the agent. they do not guarantee obedience. the tests are the check.

  • product behavior changes. check current docs for any product, queue, or database this article names.


your first week

the goal for the week is a repo where the agent reads the design, and the main faults have red tests.

  1. day 1: run the intake prompt on your current project. label every number MEASURED, TARGET, or GUESS.

  2. day 2: write SYSTEM.md from the starter block. delete every rule that does not apply. keep it under two pages.

  3. day 3: write MODULES.md entries for your two riskiest modules. riskiest means money, private data, or external calls.

  4. day 4: write the red tests for those two modules. inject each fault. paste the RED output into the PR.

  5. day 5: write the code until each test is GREEN. paste the GREEN output.

  6. day 6: run the restore drill once. replay WAL to a recovery point. record the real RPO and RTO.

  7. day 7: run one load test with a burst. size the pools from the result. move those numbers from GUESS to MEASURED.

after day 7, add one rule and one red test each time the agent makes a new class of mistake.


the law

the agent does not know your design until it can make a test fail for it.


全文完
查看原文