jq

The question it answers

The question is "how do I read, filter, and reshape JSON at the terminal?" — and it comes up constantly because JSON quietly became the lingua franca of operational tooling. kubectl get pods -o json hands you the cluster's view of the world. aws ec2 describe-instances hands you a thousand lines of nested reservations. gh api, docker inspect, journalctl -o json, every webhook payload, every structured log line — all JSON. The format itself is simple (the JSON page covers why it won), but simple to parse is not the same as easy to read. A human scanning eighty screens of output for the one pod stuck in Pending is doing a machine's job badly.

The classic unix answer would be grep and awk, and people do try. It goes wrong immediately: JSON does not live on lines. A field can be wrapped, an array can be one element or one thousand, a string can contain the exact text you were grepping for in a place that means something else. Text tools see characters; the data has structure, and the question you are asking ("which items have status Pending?") is a question about that structure. jq is awk for JSON: a small language where you describe the shape of what you want, and it walks the structure for you. Filters compose with a pipe exactly the way shell commands do — the same habit that makes find and xargs click makes jq click, except the pipe carries JSON values instead of lines of text.

What it is not: a streaming log processor for terabytes (it can do it, but that is not the daily job), a JSON editor for files you own (just edit them), or something you need to learn completely. The language is deep — it has functions, recursion, regular expressions, even a module system — and you can ignore nearly all of it. Nine constructs cover what working engineers actually type. Learn those, learn the one mental model underneath, and the rest of the manual becomes something you look up twice a year.

The filters that matter

A jq invocation is jq 'PROGRAM' file.json, or more often some-command | jq 'PROGRAM'. The program is a filter: it takes a JSON value in and produces JSON values out. These are the nine you will type every week.

Two of these deserve a sentence more before the walkthrough. .[] is the hinge of the whole language: it takes one array and emits each element as its own value, turning a document into a stream, and everything downstream of it runs once per element without you writing a loop. And -r is the flag that separates "looks right" from "works": without it, a string comes out as "api-7f9cd" with quotes baked in, and the moment you pass that to kubectl delete pod or use it in a filename, the quotes come along and break things. If the output is leaving jq-land for shell-land, you want -r.

One document, three questions, one pipe at a time

The way to learn jq is not to memorise the table above; it is to grow a filter against a real document, one pipe at a time, checking the output at each step. Here is the document we will use for the rest of the page — a trimmed kubectl pod list, saved to a file so we can iterate against it cheaply. (The full output of kubectl get pods -o json runs to hundreds of fields per pod; this keeps the four we care about.)

$ kubectl get pods -n prod -o json > pods.json
$ jq '.' pods.json          # '.' is the identity filter — pretty-print and nothing else
{
  "items": [
    {
      "metadata": { "name": "api-7f9cd", "namespace": "prod" },
      "spec": { "nodeName": "node-a" },
      "status": { "phase": "Running", "containerStatuses": [{ "restartCount": 0 }] }
    },
    {
      "metadata": { "name": "worker-x2x4j", "namespace": "prod" },
      "spec": { "nodeName": null },
      "status": { "phase": "Pending", "containerStatuses": [{ "restartCount": 0 }] }
    },
    {
      "metadata": { "name": "cache-0", "namespace": "prod" },
      "spec": { "nodeName": "node-b" },
      "status": { "phase": "Running", "containerStatuses": [{ "restartCount": 14 }] }
    }
  ]
}

Question 1 — what pods are there?

Start at the top and descend. .items gives you the array. Add [] and the array becomes three separate objects — notice the output below is not one array, it is three values printed one after another. Then descend into each one. Three keystrokes of growth, checking the shape at every step:

$ jq '.items | length' pods.json     # sanity check: how many?
3
$ jq '.items[]' pods.json            # explode: three objects, no surrounding array
{ "metadata": { "name": "api-7f9cd", ... }
{ "metadata": { "name": "worker-x2x4j", ... }
{ "metadata": { "name": "cache-0", ... }
$ jq '.items[].metadata.name' pods.json
"api-7f9cd"
"worker-x2x4j"
"cache-0"

That last line is the pattern you will type a thousand times: explode the list, pick the field. .items[].metadata.name is shorthand for .items[] | .metadata.name — the pipe is implicit between a [] and whatever follows it.

Question 2 — which pods are not Running?

Take the working filter from question 1 and insert a select between the explode and the extract. select(cond) passes its input through untouched when the condition is true and emits nothing when it is false — a sieve in the middle of the pipe:

$ jq '.items[] | select(.status.phase != "Running")' pods.json
{
  "metadata": { "name": "worker-x2x4j", "namespace": "prod" },
  "spec": { "nodeName": null },
  "status": { "phase": "Pending", "containerStatuses": [{ "restartCount": 0 }] }
}
$ jq -r '.items[] | select(.status.phase != "Running") | .metadata.name' pods.json
worker-x2x4j

Note the order of operations in your head as you build it: first run the version without the final extraction, so you can see the whole object that survived and confirm the sieve kept the right things. Then add the extraction, then add -r because this output is destined for eyes or for another command, not for more JSON machinery. Building it in that order is what makes jq feel like a conversation instead of a puzzle.

Question 3 — a small report: name, node, restarts

The third move is reshaping. The document has forty fields per pod; you want three, with your own names on them. Object construction does it — wrap the fields you want in { } and give each a key:

$ jq '.items[] | {name: .metadata.name, node: .spec.nodeName, restarts: .status.containerStatuses[0].restartCount}' pods.json
{ "name": "api-7f9cd",    "node": "node-a", "restarts": 0 }
{ "name": "worker-x2x4j", "node": null,     "restarts": 0 }
{ "name": "cache-0",      "node": "node-b", "restarts": 14 }
$ jq -r '.items[] | [.metadata.name, .spec.nodeName // "unscheduled", .status.containerStatuses[0].restartCount] | @tsv' pods.json
api-7f9cd	node-a	0
worker-x2x4j	unscheduled	0
cache-0	node-b	14

Two new pieces appear in the second command. // "unscheduled" is the alternative operator: if the left side is null or false, use the right side instead — exactly what you want for the Pending pod whose nodeName is null. And @tsv takes an array and renders it as one tab-separated line, which combined with -r produces output that sort, column -t, awk, and spreadsheets all eat happily. That handoff — structure in jq, lines out to the rest of the shell toolbox — is where the tool earns its place.

Three production scenarios

"Which pods are unhealthy, across every namespace?"

A deploy went out twenty minutes ago and dashboards are amber. You want one terse list: every pod in the cluster that is not Running, with its namespace, so you can see whether the damage is one service or several. This is question 2 from above, widened to the whole cluster and made copy-pasteable:

$ kubectl get pods -A -o json | jq -r '
    .items[]
    | select(.status.phase != "Running" and .status.phase != "Succeeded")
    | .metadata.namespace + "/" + .metadata.name + "  " + .status.phase'
prod/worker-x2x4j  Pending
prod/ingest-b81lq  Pending
staging/migrate-job-4hxs  Failed

The + here is string concatenation — jq happily builds the exact line you want to read. One honest caveat: pod phase is coarser than the STATUS column kubectl get pods prints, which synthesises strings like CrashLoopBackOff from container-level state. For the deeper version, select on .status.containerStatuses[].state.waiting.reason. The kubectl cheat sheet keeps the ready-made variants; the point here is that you can build any of them yourself, one pipe at a time, the moment the canned one does not quite fit.

"Get this into a spreadsheet"

Finance wants the instance inventory, or a teammate wants the API response as a sheet. JSON-to-CSV is a one-liner once you know the shape: build an array per record, hand it to @csv, and use -r so the commas and quotes come out as real CSV rather than a JSON string containing CSV:

$ aws ec2 describe-instances | jq -r '
    .Reservations[].Instances[]
    | [.InstanceId, .InstanceType, .State.Name, .PublicIpAddress // "none"]
    | @csv' > inventory.csv
$ head -3 inventory.csv
"i-0b1c44e2a9f00d711","m5.large","running","54.210.8.13"
"i-0f8821cc3e0a9b2d4","t3.medium","stopped","none"
"i-09a3d5e7f6b1c8e02","m5.xlarge","running","3.92.141.77"

The double explode (.Reservations[].Instances[]) is the AWS-specific wrinkle — instances live inside reservations, a structure nobody remembers until jq '.' | head reminds them. The rest is the same three moves as the pod report: explode, reshape into an array, render. If you need a header row, emit it first: jq -r '["id","type","state","ip"], (.Reservations[].Instances[] | [...]) | @csv' — the comma between two filters means "run both against the input and emit both streams," which is the other composition operator the table above left out.

"Count the errors by type"

Your service writes structured logs, one JSON object per line, and something is wrong. The question is not "show me errors" — there are thousands — it is "what kinds of errors, and how many of each?" That is a group-and-count, and it needs one new flag: -s (slurp), which gathers all the input values into a single array so that array functions like group_by can see everything at once.

$ jq -s '
    map(select(.level == "error"))
    | group_by(.code)
    | map({code: .[0].code, count: length})
    | sort_by(-.count)' app.log
[
  { "code": "upstream_timeout", "count": 142 },
  { "code": "db_conn_reset",    "count": 31 },
  { "code": "bad_payload",      "count": 4 }
]

Read it stage by stage, because each line is one idea. map(select(...)) keeps only the error records (inside an array, map plays the role .[] | plays in a stream). group_by(.code) turns the flat array into an array of arrays, one bucket per distinct code. The second map collapses each bucket into a summary object — .[0].code grabs the code from the bucket's first element, length counts the bucket. sort_by(-.count) puts the biggest fire first. Forty characters of program, and a question that would have been a small Python script is answered at the prompt. The same pattern works on journalctl -o json output, where every field the journal knows becomes selectable — that pairing is covered in journalctl & dmesg.

Underneath: everything is a stream

The model that makes jq stop feeling like syntax soup is this: every filter is a function from one value to a stream of values — zero, one, or many. .name takes a value and emits exactly one. select emits one or zero. .[] emits as many as the array has elements. The pipe connects streams: every value the left side emits is fed, one at a time, through the right side, and the outputs are concatenated. There is no loop anywhere in your program because the looping is the pipe.

Several things that look like quirks fall directly out of this model. Why does jq '.items[]' print three top-level values with no commas and no brackets? Because the result is a stream of three values, not an array — and if you want an array back, you wrap the stream in [ ], which collects everything a filter emits: [.items[] | select(...)]. Why does map(f) exist when .[] | f seems to do the same thing? Because map(f) is literally defined as [.[] | f] — explode, transform, collect — a convenience for staying inside an array.

The model also explains the input side. By default jq runs your whole program once per input document, and a file of newline-delimited JSON log lines is many documents, so jq '.code' app.log emits one code per line with no further ceremony — streaming, constant memory, works on a multi-gigabyte log. -s changes the input contract: read everything, build one array, run the program once against it. You need that for any question that relates records to each other (group_by, sort_by, length across records), and you pay for it by holding the whole input in memory. The practical rule: stay streaming when each record can be judged alone, slurp when the question is about the collection. And -r, seen through this lens, is just the final print step choosing a different renderer: JSON encoding for machines downstream that speak JSON, raw text for everything else in the unix world, which speaks lines.

Pitfalls

The quoting wars. A jq program is full of characters the shell wants for itself — |, ", $, parentheses — so the rule is absolute: single-quote the program, always, and use double quotes inside it for jq's own strings. jq '.items[] | select(.phase == "Running")' survives the shell intact. The moment you need a shell variable inside the program, do not splice it into the text; pass it with --arg and refer to it as a variable:

$ ns="prod"
$ jq --arg ns "$ns" '.items[] | select(.metadata.namespace == $ns) | .metadata.name' pods.json
# not: jq ".items[] | select(.metadata.namespace == \"$ns\")"  — works until $ns contains a quote

--arg always delivers a string; use --argjson when the value is a number or boolean and you intend it to compare as one.

Null propagation hides typos. .a.b on an object with no a does not error — it returns null, quietly, because missing keys yield null and null.b is null again. The good half: deep lookups never blow up on sparse data. The bad half: misspell .metdata.name and you get a column of nulls instead of an error, and if a select compares against that null, everything silently fails the test and your output is empty for the wrong reason. When a filter returns nothing, suspect your spelling before you suspect the data. The error you do get — Cannot index string with "b" — appears when the value exists but is the wrong type, and the ? suffix (.a.b?) suppresses it for the rows where the shape varies.

Numbers are not strings. select(.status == 200) and select(.status == "200") match disjoint sets of records, and which one is right depends on what the producer emitted — some APIs send numbers, some send the same field as a string, some send both across versions. When a select that should obviously match comes back empty, print the raw field first (jq '.status' | sort | uniq -c on a sample) and look at whether the values wear quotes. jq will not coerce for you, which is the correct behaviour and also the cause of twenty silent minutes.

Scripts that cannot fail. jq exits 0 when the program runs, even if it produced null or nothing at all — so if kubectl ... | jq '...'; then happily takes the success branch on an empty result. In scripts, add -e: the exit status becomes 1 when the last output was null or false, which turns "did this query find anything?" into something if and set -e can act on. Pair it with -r and you have the two flags that make jq a well-behaved citizen of shell scripts rather than a pretty-printer that happens to be in one.

A drill you can run right now

Everything below is safe on any machine with jq installed: it reads nothing but text you type and creates one throwaway file in /tmp. The point is to feel the grow-one-pipe-at-a-time rhythm in your hands, on data small enough that every output is checkable by eye.

Step 1 — one object, growing lookups. Pipe a literal object through filters, adding one step each time, and predict the output before you press enter:

$ echo '{"name": "ada", "langs": ["lisp", "c", "ml"]}' | jq '.'
$ echo '{"name": "ada", "langs": ["lisp", "c", "ml"]}' | jq '.name'
"ada"
$ echo '{"name": "ada", "langs": ["lisp", "c", "ml"]}' | jq -r '.name'
ada                  # same value, no quotes — that is all -r does
$ echo '{"name": "ada", "langs": ["lisp", "c", "ml"]}' | jq '.langs[]'
"lisp"
"c"
"ml"
$ echo '{"name": "ada", "langs": ["lisp", "c", "ml"]}' | jq '{who: .name, n: (.langs | length)}'
{ "who": "ada", "n": 3 }

Step 2 — a stream of records, sieve and reshape. Write three records to a file, newline-delimited the way structured logs arrive, and ask it questions:

$ printf '%s\n' \
    '{"svc": "api",   "ok": true,  "ms": 12}' \
    '{"svc": "db",    "ok": false, "ms": 480}' \
    '{"svc": "cache", "ok": false, "ms": 3}' > /tmp/checks.json
$ jq -r 'select(.ok | not) | .svc' /tmp/checks.json
db
cache
$ jq -s 'group_by(.ok) | map({ok: .[0].ok, count: length})' /tmp/checks.json
[
  { "ok": false, "count": 2 },
  { "ok": true,  "count": 1 }
]
$ jq -s 'map(.ms) | add / length' /tmp/checks.json
165                  # yes, it does arithmetic too

Notice which commands needed -s and which did not. The first runs the program once per record — each record can be judged alone, so the stream stays a stream. The second and third ask questions about the collection, so the collection has to exist first. If that distinction feels obvious now, the stream model has landed.

Step 3 — real tools, if you have them. If there is a cluster in reach, run kubectl get pods -A -o json | jq '.items | length', then grow the unhealthy-pods filter from the scenario above one pipe at a time, checking the output at each stage. If there is an AWS account, do the same with aws ec2 describe-instances and the CSV recipe. And on any Linux box, journalctl -o json -n 50 | jq -r '._COMM' | sort | uniq -c counts which programs produced the last fifty journal entries — a question that takes one line because both halves, the journal and jq, agreed to speak JSON. These are read-only; the worst they can do is teach you something about your infrastructure.

Further reading

• The jq manual — long, but the Basic Filters and Builtin Operators sections cover everything on this page and reward a single slow read.
• jqplay — a browser playground: paste a document, type a filter, see the output live. The fastest way to build a hairy filter before committing it to a script.
• The official jq tutorial — the GitHub commits API walked through with the same grow-the-filter approach, in ten short steps.
• Semicolony — JSON — the format itself: why it won, where it creaks, and what the alternatives trade away.