Backend engineer roadmap
All fifteen stages, from what HTTP does to choosing a database, the hard parts of distributed systems, and designing a system in an interview. Start here if you are not sure which path fits.
Core and recommended topics. Optional ones stay hidden; they pay off after you have run a few services in production.
- Core: needed for most jobs of this kind
- Recommended: worth learning early
- Optional: learn it when your work needs it
Stage 1
The internet and HTTP
Most confusion about backends starts here. Before you can debug a 502 or choose between gRPC and REST, you need a clear picture of what happens between a client and your server: DNS, then TCP, then TLS, then HTTP, and the response coming back the same way.
Core: How the internet works
How a packet gets from your laptop to a server on another network, passing through routers and several independently run networks on the way.
CodexNetworking GuideInternet timeline: how it grew Outside linkHigh Performance Browser Networking (free book) Outside linkwhat is the internet (Cloudflare) Outside linkSubmarine cable map
Core: DNS
How a name becomes an IP address through a chain of servers and caches. It runs before every request, causes its share of outages, and gets blamed for many more.
GuideHow DNS works CodexDNS SimulatorDNS resolution Outside linkDNS records explained (Cloudflare) Outside linkimplement DNS in a weekend (Julia Evans) Outside linkdig command tutorial
Core: IP addressing
IPv4 and IPv6 addresses, subnets and CIDR notation. IPv4 addresses ran out years ago, which is why NAT is everywhere and IPv6 keeps spreading.
CodexIP addressing GuideBGP & routing CodexRouting protocols Outside linkIPv6 articles (Cloudflare blog) Outside linkCIDR calculator
Core: HTTP
Methods, status codes and headers. Every backend service speaks HTTP all day, and knowing it well saves time whenever something breaks.
GuideHow HTTP works SimulatorHTTP flow Outside linkMDN HTTP reference Outside linkHTTP status codes (MDN) Outside linkevolution of HTTP (Mozilla)
Recommended: HTTP/2
Many requests share one TCP connection as separate streams. That removes head-of-line blocking between requests, but one lost TCP packet still stalls every stream.
SimulatorHTTP/2 streams Outside linkhttp2 explained (free book) Outside linkRFC 9113: HTTP/2 Outside linkHTTP/2 vs HTTP/1.1 (Cloudflare)
Recommended: HTTP/3 and QUIC
HTTP over QUIC, which runs on UDP. Each stream recovers from packet loss on its own, and a returning client can send data in its first packet.
SimulatorHTTP/3 + QUIC CodexQUIC Outside linkHTTP/3 explained (free book) Outside linkwhat is HTTP/3 (Cloudflare)
Core: HTTPS and TLS
The handshake that agrees on keys and the certificate chain that proves who the server is. Encryption without that proof would not stop someone pretending to be the server.
GuideHow HTTPS works CodexTLS Outside linkwhat is TLS (Cloudflare) Outside linkIllustrated TLS 1.3 handshake Outside linkTest your TLS: SSL Labs
Core: CDN basics
Servers in many cities that cache your content close to users. Usually the first thing a request reaches, and often the reason repeat visits are fast.
GuideHow CDNs work Outside linkwhat is a CDN (Cloudflare) Outside linkVCL & edge logic (Fastly)
Stage 2
OS and networking foundations
Processes, threads, file descriptors and memory. When a service misbehaves in the middle of the night, the people who can fix it usually know what the kernel is doing below the language runtime.
Core: Processes
The kernel's unit of isolation. Each process has its own address space, file descriptors and place in the scheduler, and every server you run is one or more of them.
CodexProcesses Outside linkThree Easy Pieces (free book) (OSTEP) Outside linkBeej: Unix IPC guide
Core: Threads
Threads share their process's memory but have their own stacks. They are cheaper than processes, and that shared memory is behind the hardest bugs you will debug.
CodexThreads GuideThread pools Outside linkLinux Programming Interface (book)
Recommended: Scheduling
How the kernel picks what runs next on a busy CPU, and how schedulers trade fairness against throughput and latency. It starts to matter once your cores are saturated.
CodexScheduling internals SimulatorCPU scheduler Outside linkLinux CFS: kernel docs
Core: Memory management
Virtual memory, paging and the allocator. Knowing this layer is what lets you find a memory leak in an afternoon instead of a week.
CodexMemory management CodexVirtual memory SimulatorVirtual memory GuideMemory allocation Outside linkWhat every programmer should know about memory (Drepper)
Core: I/O: blocking, non-blocking, async
Blocking, non-blocking and asynchronous I/O, and the kernel interfaces behind them: epoll on Linux, kqueue on BSD and macOS, and the newer io_uring. Your runtime has already chosen one.
CodexI/O internals Outside linkThe C10K problem (Kegel) Outside linkio_uring by example Outside linkWhat is epoll: explained
Recommended: File systems
Inodes, directories, journaling and fsync. A database's promise of durability depends on this layer, as the 2018 PostgreSQL fsync problem showed.
CodexFile systems Outside linkLWN: kernel filesystems Outside linkPostgres fsync gate (postmortem)
Core: IPC: pipes, sockets, shared memory
How two processes on the same machine talk to each other: pipes, Unix sockets and shared memory.
Core: Synchronization primitives
Locks, semaphores, atomics and RCU, and the deadlocks and races that come from using them slightly wrong.
CodexSynchronization SimulatorMutex & deadlock Outside linkPreshing: lockless programming Outside linkMechanical Sympathy (Martin Thompson)
Core: System calls
The boundary between your program and the kernel. Every read, write and epoll_wait crosses it, and strace shows you each call as it happens.
CodexSystem calls SimulatorSyscall journey Outside linkLinux syscall table Outside linkstrace tutorial
Core: TCP and UDP
TCP gives you ordered, reliable bytes at the cost of a handshake and head-of-line blocking. UDP just sends packets. Many network bugs turn out to live at this layer.
GuideHow TCP works SimulatorTCP handshake SimulatorTCP congestion control SimulatorTCP vs UDP CodexUDP Outside linkBeej: Network programming guide
Recommended: Sockets and terminal basics
Berkeley sockets, signals, file descriptors and permissions: the everyday tools for working with the OS from a shell.
CodexSockets Outside linkMIT: The Missing Semester Outside linkexplainshell.com
Stage 3
Version control
The basic commands take an afternoon. Understanding commits as a graph, and the difference between rebase, cherry-pick and revert, is what you need on the day a release has to be rolled back.
Core: Git fundamentals
Commits, branches, merges and rebase, and the commit graph they all work on.
GuideHow Git works Outside linkPro Git (free book) Outside linkAtlassian Git tutorials Outside linkLearn Git Branching (interactive)
Core: Branching strategies
Trunk-based development, Git Flow and GitHub Flow. The choice shapes how your CI and releases work.
Outside linktrunkbaseddevelopment.com Outside linkGitHub Flow Outside linkGit Flow: original (Driessen)
Recommended: Rebase vs merge
A linear history or one that keeps every branch point. Mostly a question of what makes changes easier to review.
Outside linkrebase vs merge (Atlassian) Outside linkPro Git: rewriting history
Recommended: Conflict resolution
Three-way merges, rerere, and how to resolve a conflict calmly instead of reaching for git reset.
Outside linkGit rerere docs Outside linkOh Shit, Git!?! Outside linkDangit, Git!?!
Core: Code review practices
What a useful review looks like: tone, scope, and what to leave for a later change.
Outside linkcode review guide (Google) Outside linkConventional Comments Outside linkThoughtbot: code review guide
Recommended: Conventional commits and semantic versioning
A commit message format that tools can read, and version numbers that tell users whether an upgrade can break them. Cheap to adopt.
Outside linkConventional Commits Outside linkSemantic Versioning (SemVer)
Stage 4, pick one
Pick a language
Backend jobs ask for fluency in one language, not five. Go is a sensible default. Rust gives the strongest safety guarantees and takes longer to learn. Node.js makes sense when you share code with a web frontend, and Java, Python and C# run a large share of company backends. Pick one you can defend in an interview; the others can come later.
Core: Go
Simple to learn, quick to compile and good at concurrency. A common choice for backend services and infrastructure tools.
CodexGo CodexGo internals CodexGoroutines and channels SimulatorGoroutine scheduler Outside linkA Tour of Go Outside linkEffective Go Outside linkGo by Example Outside linkGo 100 mistakes (book site)
Core: JavaScript / TypeScript
Node.js services, the event loop and async/await. Useful when the same team also writes the web frontend.
CodexJavaScript CodexEvent loop SimulatorEvent loop CodexV8 engine internals Outside linkjavascript.info Outside linkEloquent JavaScript Outside linkTypeScript handbook Outside linkTotal TypeScript (Matt Pocock)
Recommended: Rust
Ownership and borrowing let the compiler rule out data races and many memory errors. Harder to learn, and worth it for systems work.
CodexRust CodexBorrow checker internals CodexAsync runtime Outside linkThe Rust Book Outside linkRustlings: exercises Outside linkRust by Example Outside linkJon Gjengset: Crust of Rust (YouTube)
Recommended: Python
Used for data work, scripts, and services built with FastAPI or Django.
Outside linkPython tutorial: official Outside linkReal Python Outside linkFluent Python (book site) Outside linkFastAPI docs Outside linkDjango tutorial
Recommended: Java / Kotlin
The JVM is a mature, fast runtime, and Spring runs a large share of company backends.
Outside linkJava: Oracle tutorials Outside linkEffective Java (book) Outside linkBaeldung: Spring guides Outside linkKotlin docs
Stage 5
Design principles and patterns
The vocabulary code reviews use. Knowing SOLID means understanding what a reviewer means when they say a change breaks the open-closed principle, not reciting the five letters. The same goes for KISS, YAGNI, DRY, the twelve-factor app and the classic design patterns.
Core: SOLID
Single responsibility, open-closed, Liskov substitution, interface segregation and dependency inversion.
Outside linkSOLID (Wikipedia) Outside linkUncle Bob: The Principles of OOD Outside linkStackify: SOLID examples
Core: KISS, YAGNI, DRY
Keep it simple, don't build what you don't need yet, and don't repeat yourself. Three short rules that prevent a lot of bad code.
Outside linkKISS (Wikipedia) Outside linkYAGNI (Wikipedia) Outside linkDRY (Wikipedia) Outside linkMartin Fowler: YAGNI
Core: Design patterns
Factory, Strategy, Observer, Adapter and the rest of the classic catalogue, and when each one helps.
Outside linkpatterns (Refactoring Guru) Outside linkDesign Patterns (GoF): reference Outside linkSourceMaking: patterns
Core: Refactoring
Small changes that keep behaviour the same and make the next change easier.
Outside linkrefactoring catalog (Martin Fowler) Outside linkRefactoring Guru: refactoring
Recommended: Clean architecture / hexagonal
Keep frameworks and databases at the edges so the core logic can be tested on its own.
Outside linkclean architecture (Uncle Bob) Outside linkHexagonal: Alistair Cockburn
Recommended: Domain-driven design
Bounded contexts, a shared language with the business, and aggregates. Useful once the domain gets complicated.
Outside linkDDD reference (Evans, free PDF) Outside linkMicrosoft Learn: DDD basics Outside linkAwesome DDD
Core: Twelve-Factor App
A short list of rules for services that are easy to deploy and run, such as keeping configuration in the environment. Still a good checklist.
Outside link12factor.net Outside linkBeyond 12-factor (Central)
Recommended: Software architecture patterns
Layered, event-driven, microkernel and CQRS: patterns for whole systems rather than single classes.
Outside linkMark Richards: architecture styles (free O'Reilly) Outside linkcloud architecture patterns (Microsoft)
Stage 6
APIs and protocols
REST is the default, and most systems also use at least one of gRPC, GraphQL or WebSockets. Each suits a different problem. Knowing what each costs in latency, bytes on the wire and work to operate tells you when to use it.
Core: REST
Resources, HTTP methods and status codes. The usual choice for public APIs.
CodexREST Outside linkREST API design (Microsoft) Outside linkREST API tutorial Outside linkRoy Fielding's thesis (chapter 5)
Core: JSON
The most common format on the wire. Know the spec, its traps such as large numbers, and the binary alternatives.
Core: gRPC
Schema-first RPC over HTTP/2, with streaming. A common choice between internal services.
CodexgRPC SimulatorgRPC vs REST Outside linkgrpc.io: official docs Outside linkgRPC core concepts
Core: Protocol Buffers
The schema language and binary format behind gRPC. Smaller and faster than JSON, and designed so schemas can change safely.
CodexProtobuf SimulatorJSON vs Protobuf Outside linkprotobuf.dev: official Outside linkProtobuf language guide
Recommended: GraphQL
One endpoint where the client asks for exactly the fields it needs. Good when many clients need different shapes of the same data.
CodexGraphQL Outside linkgraphql.org: learn Outside linkApollo: odyssey course Outside linkGraphQL: official spec
Core: WebSockets
Long-lived, two-way connections, for chat, presence and anything the server pushes.
GuideHow WebSockets work GuideRealtime communication CodexWebSockets and SSE Outside linkRFC 6455: WebSocket protocol
Recommended: Server-Sent Events
One-way streaming from server to client over plain HTTP. Often enough when you don't need WebSockets.
Core: OpenAPI / Swagger
A schema for REST APIs that tools turn into clients, mocks and documentation. Easiest to adopt at the start.
Outside linkOpenAPI Initiative Outside linkSwagger: getting started Outside linkSpotlight: OpenAPI guide
Recommended: API versioning
Versions in the path, a header or a date. The choice matters once outside users depend on your API.
Recommended: Webhooks
Your server calls a URL the client gave you when something happens. Stripe, GitHub and Slack all use them.
CodexWebhooks Outside linkwebhooks guide (Stripe) Outside linkStandard Webhooks
Core: API authentication
API keys, OAuth tokens, mutual TLS and signed requests, and where each fits.
CodexAuth in API design Outside linkwhat is mTLS (Cloudflare) Outside linksigned request signing (SigV4) (AWS)
Core: Idempotency at the API layer
Idempotency keys let a client retry a request without it taking effect twice. Stripe's API is the usual example.
CodexIdempotence in distributed systems Outside linkidempotency keys (Stripe) Outside linkBrandur: idempotency keys
Recommended: API best practices
Pagination, filtering, error formats and rate-limit headers.
CodexAPI best practices Outside linkZalando RESTful API guidelines Outside linkMicrosoft REST API guidelines
Stage 7
Databases
The choice that is hardest to undo. Know the range from a single Postgres instance to a sharded, distributed store, and the signs that it is time to move along it. Move too early and you add complexity for nothing; move too late and the migration happens under pressure.
Core: Relational fundamentals
Schemas, keys, joins and normalisation. Most applications run on this model, usually in Postgres or MySQL.
CodexDatabases CodexChoosing a database Outside linkofficial tutorial (PostgreSQL) Outside linkSQLZoo: interactive SQL
Core: SQL and joins
Inner, left, right, full and cross joins. Picking the wrong one gives you a correct answer to a different question.
SimulatorSQL JOIN Outside linkSQL for Smarties (Celko): book Outside linkMode: SQL tutorial
Core: Indexes
B-trees, covering indexes and partial indexes, and EXPLAIN ANALYZE to see whether a query uses them. Most slow queries are missing an index or not using one.
GuideDatabase indexing SimulatorB-tree CodexB-tree Outside linkUse The Index, Luke!
Core: Transactions and ACID
Atomicity, consistency, isolation and durability. Each one costs some latency or throughput, and knowing the cost makes trade-off discussions easier.
SimulatorACID GuideTransactions Outside linkDesigning Data-Intensive Applications: ch. 7 Outside linkconcurrency control (PostgreSQL)
Core: Isolation levels
From read uncommitted to serializable. Each level prevents some anomalies and allows others, and most databases default to a weaker level than people assume.
SimulatorIsolation levels CodexIsolation levels Outside linkconsistency models (Jepsen)
Recommended: MVCC
How Postgres lets readers and writers work without blocking each other by keeping old row versions, and why that makes vacuuming necessary.
CodexMVCC Outside linkPostgres: MVCC chapter Outside linkBruce Momjian: MVCC unmasked
Core: WAL and crash recovery
Write the change to a log and flush it before touching the data files. Almost every durable database works this way.
Recommended: Storage engines (B-tree vs LSM)
B-trees favour reads and LSM trees favour writes. The engine decides which workloads your database handles well.
SimulatorStorage engine CodexLSM-tree Outside linkDatabase Internals (Petrov): book
Recommended: Page cache and buffer pool
How the database's buffer pool and the operating system's page cache share memory, and when they get in each other's way.
CodexPage cache Outside linkLinux page cache: kernel docs Outside linkPostgres shared buffers: wiki
Recommended: Query planner
How the database chooses a plan, how to read EXPLAIN output, and why it sometimes ignores your index.
CodexQuery planner SimulatorSQL query execution Outside linkPostgres: using EXPLAIN Outside linkPEV: explain visualiser
Core: When to use NoSQL
Key-value, document, wide-column and graph stores. Each handles one access pattern very well and others badly, so choose by access pattern first.
Outside linkNoSQL design (AWS) Outside linkCassandra: modeling tutorial
Recommended: Distributed SQL
Spanner, CockroachDB, YugabyteDB and TiDB give you transactions across many machines, at the cost of slower writes than a single Postgres. Worth it once one machine is not enough.
Recommended: Search engines
Inverted indexes and BM25 ranking, in Elasticsearch, OpenSearch and Solr, and lighter tools such as Meilisearch and Typesense.
Outside linkElasticsearch: guide Outside linkApache Solr: docs Outside linkMeilisearch: quick start Outside linkLucene: fundamentals
Stage 8
Caching
Fast systems cache at several layers. The skills here are choosing the layer, deciding what a TTL should be, and surviving the moment a popular key expires and thousands of requests reach the database at once.
Core: Caching strategies
Cache-aside, read-through, write-through and write-behind, and how to choose between them.
GuideHow caching works Outside linkcaching strategies (AWS) Outside linkcaching guidance (Microsoft)
Core: Eviction policies
LRU, LFU, ARC and TinyLFU, and the workloads each one handles best.
SimulatorCache eviction SimulatorLRU cache Outside linkCaffeine (Java): TinyLFU Outside linkcache replacement policies (Wikipedia)
Core: Redis
The most widely used in-memory store, with strings, sets, sorted sets and streams, and why running commands on one thread is fast enough.
GuideHow Redis works SimulatorRedis operations Outside linkredis.io: documentation Outside linkTry Redis: interactive Outside linkdata structures (Redis)
Core: CDN
Caching at the edge: points of presence, cache headers, and how long invalidation takes.
GuideHow CDNs work Outside linkhow a CDN works (Cloudflare) Outside linkedge platform (Fastly) Outside linkAWS CloudFront: docs
Core: HTTP cache headers
Cache-Control, ETag and Vary, which tell every cache between your server and the user what it may keep.
Outside linkHTTP caching (MDN) Outside linkRFC 9111: HTTP caching Outside linkweb.dev: HTTP cache
Core: Stampedes and invalidation
Request coalescing, randomised expiry and negative caching, for the two problems behind most cache incidents: stale data and stampedes.
SimulatorThundering herd Outside linkInstagram: solving cache stampedes
Stage 9
Web security
This stage won't make you a security specialist. The aim is to know enough about the OWASP Top 10, OAuth, CORS, JWTs and basic cryptography to avoid shipping the common mistakes.
Core: OWASP Top 10
The best-known list of web application vulnerabilities, updated every few years.
Outside linkOWASP Top 10: official Outside linkOWASP Cheat Sheet Series Outside linkPortSwigger Web Security Academy
Core: Hashing and password storage
bcrypt, scrypt and Argon2, salts and work factors. Never invent your own scheme.
Outside linkpassword storage cheat sheet (OWASP) SimulatorPassword hashing Outside linkRFC 9106: Argon2 Outside linkbcrypt (Wikipedia)
Recommended: Symmetric and asymmetric cryptography
AES, RSA and elliptic curves, and which kind you use to encrypt and which to sign.
Outside linkwhat is encryption (Cloudflare) Outside linkCrypto 101 (Laurens Van Houtven) Outside linkCryptopals challenges
Recommended: TLS in more depth
Cipher suites, certificate validation, mutual TLS and what changed in TLS 1.3.
CodexTLS Outside linkSSL Labs: server test Outside linkillustrated TLS 1.3 Outside linkserver-side TLS guidelines (Mozilla)
Core: OAuth 2.0
The framework behind most API access tokens and, with OpenID Connect, most "sign in with" buttons. Its flows, its tokens and its common mistakes.
GuideHow OAuth works CodexAuth in API design Outside linkRFC 6749: OAuth 2 Outside linkoauth.net: overview Outside linkAaron Parecki: OAuth tutorial
Core: OpenID Connect
The identity layer on top of OAuth: ID tokens, the userinfo endpoint and the usual "sign in with Google" flow.
GuideHow OIDC works Outside linkOIDC: official spec Outside linkAuth0: OIDC intro
Core: JWT
Signed tokens that carry claims. Convenient, and easy to get wrong, especially around expiry and revocation.
SimulatorJWT lifecycle Outside linkjwt.io: debugger Outside linkRFC 7519: JWT
Core: CORS
Why a browser blocks some cross-origin requests, and how a server allows them.
SimulatorCORS preflight Outside linkCORS (MDN) Outside linkweb.dev: CORS
Core: CSRF, XSS, SQL injection
The three classic web vulnerabilities. Each has a well-known fix that teams still skip.
Outside linkCSRF cheat sheet (OWASP) Outside linkOWASP: XSS cheat sheet Outside linkOWASP: SQL injection prevention Outside linkPortSwigger: XSS
Core: Security headers
CSP, HSTS, X-Frame-Options and Permissions-Policy: a few response headers that block whole classes of attack.
Outside linksecurity headers (MDN) Outside linksecurityheaders.com: scanner Outside linkweb.dev: CSP
Recommended: mTLS
Both client and server present certificates. A common way to secure traffic between internal services.
Outside linkwhat is mTLS (Cloudflare) Outside linkSPIFFE/SPIRE: workload identity
Core: Rate limiting
Token buckets, leaky buckets and fixed or sliding windows. Usually the first protection in front of a public API.
SimulatorRate limiter PlaybookRate limiter: playbook Outside linkrate limiters (Stripe)
Stage 10
Testing
Habits matter more than the framework. Two help most: write a failing test before you fix a bug, and give integration tests more weight than unit tests. Unit tests catch the bugs you thought of; integration tests catch the ones users find.
Core: Test pyramid
Many unit tests, fewer integration tests and a few end-to-end tests: the mix that finds the most bugs for the time CI spends.
Outside linkVocke (Practical Test Pyramid) Outside linktest pyramid revisited (Honeycomb)
Core: Unit tests
Tests of one piece of code on its own, fast enough to run on every save.
CodexTesting in Go CodexTesting in Rust Outside linkMartin Fowler: UnitTest Outside linkKent Beck: Test-Driven Development (book)
Core: Integration tests
Tests against a real database and real network calls, which find the bugs unit tests miss.
Outside linkintegration test (Martin Fowler) Outside linkTestcontainers: docs Outside linkPostgres testing patterns
Recommended: End-to-end
Tests of a whole user flow. Slow and fragile, but worth having for the paths that must not break.
Outside linkofficial docs (Playwright) Outside linkCypress: docs Outside linkSelenium: docs
Core: Mocking and test doubles
Stubs, spies, mocks and fakes, what each is for, and when using one hides a real bug.
Outside linktest doubles (Martin Fowler) Outside linkMartin Fowler: mocks aren't stubs
Recommended: Property-based testing
The framework generates inputs from rules you write, and finds cases you would not have thought of.
Outside linkHypothesis (Python) Outside linkQuickCheck: original (Haskell) Outside linkPropEr (Erlang)
Recommended: Contract testing
Pact and Spring Cloud Contract check that services agree on their APIs before they are deployed together.
Outside linkofficial docs (Pact) Outside linkMartin Fowler: contract test
Core: Load testing
k6, Vegeta and Locust, to find out where the system falls over before your users do.
CodexLoad testing Outside linkk6: docs Outside linkLocust: docs Outside linkVegeta: repo
Stage 11
Containers and orchestration
Docker is everywhere and Kubernetes runs much of production. You can be productive with both in a few weeks. Knowing how they work (namespaces, cgroups, the kubelet and the scheduler) is what helps on the day a pod won't start and the logs say nothing useful.
Core: How containers work
Namespaces, cgroups and layered file systems, and why a container is not a virtual machine.
GuideHow containers work SimulatorContainer layers Outside linkDocker: overview Outside linkLiz Rice: containers from scratch (talk)
Core: Dockerfile best practices
Layer caching, multi-stage builds and minimal base images, which can shrink an image a long way.
Outside linkDocker: Dockerfile best practices Outside linkDistroless images (Google) Outside linkSnyk: Docker security best practices
Recommended: Container registries
Docker Hub, GitHub Container Registry, ECR and Artifact Registry. Docker Hub's pull limits catch many teams by surprise.
Outside linkDocker Hub: docs Outside linkGitHub Container Registry Outside linkOCI image spec
Core: Kubernetes basics
Pods, deployments, services and ingress: enough to deploy and run an application.
CodexKubernetes CodexK8s architecture Outside linkKubernetes: official docs Outside linkKubernetes the Hard Way (Kelsey Hightower)
Recommended: The Kubernetes control plane
The API server, etcd, controllers, the scheduler and the kubelet, which together keep the cluster in the state you declared.
CodexControllers CodexScheduler CodexKubelet CodexAPI server Codexetcd
Recommended: Kubernetes networking and ingress
Services, network policies, Ingress and the Gateway API, with CNI plugins doing the routing.
CodexK8s networking GuideK8s networking primer Outside linkKubernetes Gateway API Outside linklearnk8s: networking guides
Core: Helm and Kustomize
Templates and overlays: two ways to manage Kubernetes YAML without copying it everywhere.
Outside linkHelm: official Outside linkKustomize: docs Outside linkHelm best practices
Core: CI/CD
GitHub Actions, GitLab CI, Argo CD and Flux: the pipeline from a commit to production.
Outside linkGitHub Actions: docs Outside linkGitLab CI: docs Outside linkArgoCD: docs Outside linkFlux: docs
Recommended: Infrastructure as code
Terraform, Pulumi and OpenTofu, so infrastructure can be reviewed and rebuilt from files.
Outside linkTerraform: docs Outside linkOpenTofu (Terraform fork) Outside linkPulumi: docs Outside linkCrossplane: docs
Stage 12
Scaling out
The techniques for handling more traffic than one machine can: load balancing, sharding, replication, autoscaling and serving from the edge. Each solves one problem and brings new ones, and knowing those trade-offs matters more than knowing the tools.
Core: Scaling out vs up
A bigger machine or more machines, and when each makes sense.
Core: Load balancing
Layer 4 and layer 7, round robin and least connections, sticky sessions and health checks.
GuideHow load balancing works SimulatorLoad balancer CodexLoad balancing
Recommended: Web servers
Nginx, Apache and Caddy, which still sit in front of a large share of the web.
Outside linkNginx: official docs Outside linkApache HTTPD: docs Outside linkCaddy: docs Outside linkEnvoy proxy: docs
Core: Reverse proxies and API gateways
The layer in front of your services that routes, limits and authenticates requests.
GuideAPI gateway GuideReverse proxy Outside linkKong gateway: docs Outside linkwhat is an API gateway (Cloudflare)
Core: Sharding and partitioning
Splitting a database across machines when one is no longer enough. Hard to undo, so worth planning.
CodexSharding SimulatorDatabase sharding SimulatorConsistent hashing Outside linkVitess: docs
Core: Replication
Read replicas, single-leader, multi-leader and leaderless replication, and what each costs.
CodexReplication Outside linkDDIA: chapter 5 (replication) Outside linkPostgres replication: docs
Recommended: Autoscaling
Reactive and predictive scaling, scaling in safely, and cold starts.
GuideHow autoscaling works SimulatorAutoscaling Outside linkautoscaling concepts (AWS) Outside linkK8s HPA: docs
Recommended: Service discovery
How services find each other when instances come and go: DNS, registries such as Consul, or a service mesh.
Core: Capacity planning
Little's law, a little queueing theory and rough estimates, to turn a request rate into a number of machines.
Stage 13
Distributed systems
Here backend work becomes less about being correct and more about staying correct when parts fail: consensus, replication, ordering and idempotence. Senior interviews usually touch several of these topics.
Core: CAP and PACELC
During a network partition a replicated system has to choose between consistency and availability. PACELC adds the trade-off when there is no partition: latency against consistency.
CodexCAP and PACELC SimulatorCAP theorem Outside linkBrewer's original keynote Outside linkDaniel Abadi: PACELC
Core: Consensus and Raft
How a group of servers agrees on the next entry in a shared log. Raft was designed to be easier to understand and implement than Paxos.
CodexConsensus SimulatorRaft Outside linkraft.github.io: visualisation CodexLeader election
Core: Quorum reads and writes
With N replicas, writing to W and reading from R where R + W > N means every read overlaps the latest write. Used in Dynamo-style stores such as Cassandra.
Core: Two-phase commit and sagas
Two-phase commit makes a change across services atomic but blocks if the coordinator fails. Sagas replace it with a series of local steps and compensating actions.
Codex2PC and sagas Outside linkmicroservices.io: Saga pattern
Core: Idempotence
The property that makes a retry safe. APIs usually provide it with idempotency keys, as Stripe's does.
CodexIdempotence Outside linkBrandur: idempotency keys Outside linkdesigning idempotent APIs (Stripe)
Recommended: Time and clocks
Why agreeing on "now" is hard across machines: Lamport clocks, vector clocks, and Google's TrueTime, which uses GPS and atomic clocks to bound clock error.
CodexTime and clocks Outside linkTrueTime: Spanner whitepaper Outside linkclocks talk (Martin Kleppmann)
Recommended: Gossip protocols
Spreading information by having each node pass it to a few random others. Used by Cassandra, and by Consul through SWIM.
Recommended: Failure detectors
How a cluster decides a node is down, including the phi accrual detector used by Cassandra and Akka.
CodexFailure detectors SimulatorSplit-brain Outside linkPhi-accrual paper (Hayashibara)
Recommended: CRDTs
Data types whose copies can be changed independently and always merge to the same result. Used in collaborative editing.
Outside linkcrdt.tech: index of resources Outside linkCRDTs talk (Martin Kleppmann)
Core: Backpressure and retries
Slowing producers down, retrying with backoff and jitter, and stopping one slow component from bringing down the rest.
CodexBackpressure and retries SimulatorCircuit breaker SimulatorRetry strategy Outside linkexponential backoff & jitter (AWS)
Core: Message queues
Kafka for a durable log, RabbitMQ for flexible routing, SQS for a managed queue, NATS for low latency and Pulsar for multi-tenant setups.
GuideHow message queues work GuideKafka, as a river Outside linkofficial docs (Kafka) Outside linkRabbitMQ: tutorials Outside linkNATS: docs Outside linkApache Pulsar: docs
Recommended: Microservices patterns
Sagas, the transactional outbox, CQRS, event sourcing and service meshes, and the problem each one solves.
Outside linkmicroservices.io: patterns Outside linkMicroservices.io: Saga pattern Outside linkOutbox pattern (Confluent) Outside linkBuilding Microservices (Sam Newman)
Core: Async architecture
Events, queues and delivery guarantees: at most once, at least once, and effectively once with idempotence.
Recommended: Distributed tracing
OpenTelemetry, Jaeger and Zipkin, for following one request across many services.
Outside linkOpenTelemetry: official Outside linkJaeger: docs
Stage 14
Observability
Logs, metrics and traces, plus two methods: USE for resources and RED for requests. Together they cover most of what an on-call engineer needs. The harder part is adding the instrumentation before the incident, so the data is there when you need it.
Core: Logs, metrics and traces
Logs say what happened, metrics say how much, and traces show the path a request took.
Outside linkAchieving Observability (free book) (Honeycomb) Outside linkCNCF Observability Whitepaper
Core: RED method
Rate, errors and duration: the three numbers to track for every service that handles requests.
Core: USE method
Utilisation, saturation and errors, checked for every resource. The counterpart to RED.
Core: SLI / SLO / SLA
Service level indicators, objectives and agreements: what you measure, what you aim for, and what you promise.
Outside linkGoogle SRE book: SLOs Outside linkGoogle SRE workbook: SLOs Outside linkSLO guide (Datadog)
Core: Prometheus and Grafana
The most common open-source metrics stack: PromQL, exporters and recording rules.
Outside linkPrometheus: official Outside linkGrafana: docs Outside linkPromQL tutorial: Robust Perception
Core: OpenTelemetry
The vendor-neutral standard for traces, metrics and logs, with automatic instrumentation for many languages.
Outside linkOpenTelemetry: official Outside linkCNCF: OpenTelemetry intro
Recommended: Log aggregation
Collecting logs in one place with the ELK stack, Loki or OpenSearch. Usually the cheapest improvement to debugging.
Outside linkElastic Stack: guide Outside linkGrafana Loki: docs Outside linkVector: log pipeline
Recommended: APM and error tracking
Hosted tools such as Datadog, New Relic and Sentry that show slow requests and errors as they happen.
Outside linkSentry: official docs Outside linkAPM docs (Datadog) Outside linkAPM docs (New Relic) Outside linkdocs (Honeycomb)
Recommended: Profiling
pprof, perf and flame graphs, for finding where a program spends its time.
CodexProfiling Outside linkflame graphs (Brendan Gregg) Outside linkpprof: docs
Recommended: Latency budgets
Percentiles such as p50 and p99, and splitting a response-time target across the parts of a request.
CodexLatency budgets Outside linkLatency numbers every programmer should know
Stage 15
System design
Knowing the components is different from designing with them. You get better by working through common problems out loud, such as a chat service, a news feed, a URL shortener and object storage, and defending each choice. Designing Data-Intensive Applications is the book to read first.
Core: The design framework
Clarify the scope, estimate the load, define the API and the data model, sketch the design, then go deeper where it matters. Practise until you can do it in 45 minutes without thinking about the steps.
Outside linkSystem Design Primer (donnemartin) Outside linkByteByteGo: system design newsletter
Core: Capacity planning
Little's law, a little queueing theory and rough estimates, to turn a request rate into a number of machines.
Core: Worked problem: Chat
A chat system like Discord: a fleet of WebSocket servers, message storage, presence, delivery guarantees and fan-out to groups.
PlaybookChat playbook GuideHow WebSockets work Outside linkbillions of messages (Discord)
Core: Worked problem: News feed
Fan-out on write makes reads fast and writes expensive; fan-out on read does the opposite; accounts with millions of followers break both. Most real systems combine the two.
PlaybookNews feed playbook Outside linkTwitter timelines at scale (2013) Outside linkTAO paper (Facebook social graph)
Core: Worked problem: URL shortener
Short keys, avoiding collisions, a CDN in front and fast redirects. Simple until the traffic gets large.
Core: Worked problem: Object storage (S3)
Storing huge amounts of data durably: erasure coding instead of full copies, separate metadata and data paths, multipart uploads and background repair.
PlaybookObject storage playbook Outside linkAndy Warfield: Building S3 (re:Invent 2023)
Recommended: More worked problems
A key-value store, a rate limiter, notifications, a web crawler, typeahead, top-K, a job scheduler, ride matching, a pastebin and event ingestion.
PlaybookPlaybook hub PlaybookKV store PlaybookRate limiter PlaybookTypeahead PlaybookTop-K PlaybookWeb crawler PlaybookDistributed scheduler
Recommended: Annotated classics
The papers behind ideas that are now everywhere: Dynamo, Spanner, MapReduce, Bigtable, GFS and Lamport's paper on time and clocks.
Recommended: Engineering blogs to follow
Write-ups from companies about how their systems work and how they failed.
Outside linkcase studies (High Scalability) Outside linkNetflix Tech Blog Outside linkUber Engineering Outside linkLinkedIn Engineering Outside linkMeta Engineering Outside linkCloudflare blog Outside linkAWS Builders' Library
Core: The book to read first
Designing Data-Intensive Applications by Martin Kleppmann covers storage, replication, partitioning and stream processing in one readable book.
Outside linkDDIA (Martin Kleppmann) Outside linkSoftware Engineering at Google (free) Outside linkGoogle SRE book (free) Outside linkGoogle SRE workbook (free)
Recommended: Practice mock interviews
Reading about a design is not the same as defending it. Practise with someone else, take turns, and talk through your reasoning out loud.
Outside linkPramp: free mock interviews Outside linkInterviewing.io: peer mocks Outside linkSystem Design Interview vol. 1 + 2 (Alex Xu)
After this roadmap
- System design playbookDesign problems worked from start to finish, such as a chat service, a news feed and a URL shortener.
- SimulatorsModels of Raft, consistent hashing, caches, load balancers and B-trees that run in the browser.
- CodexLonger study pages on databases, networking, operating systems, distributed systems and Kubernetes. Many topics above link into it.