Filters, null handling and exit codes, building JSON without injection, CSV/TSV/sh output, reading JSON into bash, NDJSON logs and real JSON from ip, lsblk and journalctl
▶Prefer to watch? This short video runs this page's main commands on a real Ubuntu 24.04 machine, with on-screen captions and no voiceover. Click to play, or watch it on YouTube.
jq in shell scripts: working with JSON safely
jq -r '.items[] | select(.state=="running") | .name' api.json
Almost every modern tool speaks JSON: cloud and container APIs, kubectl, docker, ip -j, lsblk -J, journalctl -o json, GitHub and monitoring APIs. The shell cannot parse JSON, and awk and sed must not be used for it (a regular expression cannot follow nested braces, quoted commas or escaped quotes). jq is the right tool: a small, fast program that reads JSON, applies a filter and prints the result. This page shows how to use it inside scripts: the filters you need every day, how null and missing values behave, what the exit codes mean, how to build JSON without injection bugs, how to hand values to bash without losing data, and how to read logs that have one JSON object per line.
Every example was run for real with jq 1.7 on Ubuntu 24.04 and bash 5.2. The sample data is a small API-style response (api.json, printed below); the last part uses real JSON from this machine (ip, lsblk, journalctl), with addresses and interface names changed. Timings are from a 200,000-line file.
The sample data
api.json is a typical API answer: a status, a count, a null "next_page" and a list of five servers, some with nested objects (owner), arrays (tags), a null owner and a name that contains quote characters on purpose:
$ cat api.json
{
"status": "ok",
"count": 5,
"next_page": null,
"items": [
{"id": 101, "name": "web1", "env": "prod", "cpu": 4, "mem_gb": 16, "tags": ["web","nginx"], "owner": {"name": "alice", "team": "platform"}, "state": "running", "price": 0.0832},
{"id": 102, "name": "web2", "env": "prod", "cpu": 4, "mem_gb": 16, "tags": ["web","nginx"], "owner": {"name": "alice", "team": "platform"}, "state": "stopped", "price": 0.0832},
{"id": 103, "name": "db1", "env": "prod", "cpu": 8, "mem_gb": 64, "tags": ["db"], "owner": {"name": "bob", "team": "data"}, "state": "running", "price": 0.4160},
{"id": 104, "name": "ci1", "env": "dev", "cpu": 2, "mem_gb": 8, "tags": [], "owner": null, "state": "running", "price": 0.0416},
{"id": 105, "name": "test \"quoted\" name", "env": "dev", "cpu": 2, "mem_gb": 8, "tags": ["tmp"], "owner": {"name": "carol", "team": "qa"}, "state": "running", "price": 0.0416}
]
}
$ jq --version
jq-1.7
First filters: fields, arrays, objects and the pipe
A jq program is a FILTER: it takes each JSON value from the input and produces zero or more values. The dot is the input itself; .name takes a field; .[] produces each element of an array (one output per element); and | passes the output of one filter into the next, like a shell pipe. Output is pretty-printed by default.
$ jq '.status, .count' api.json
"ok"
5
$ jq '.items | length' api.json
5
$ jq -r '.items[].name' api.json
web1
web2
db1
ci1
test "quoted" name
-r (raw output) prints strings without the surrounding JSON quotes: the first command printed "ok" with quotes (a JSON string), the third printed bare names. In scripts you almost always want -r when you take a string. A whole element, pretty-printed:
$ jq '.items[0]' api.json
{
"id": 101,
"name": "web1",
"env": "prod",
"cpu": 4,
"mem_gb": 16,
"tags": [
"web",
"nginx"
],
"owner": {
"name": "alice",
"team": "platform"
},
"state": "running",
"price": 0.0832
}
Build a new object from parts with {} and one output per line with -c (compact):
$ jq -c '.items[] | {id, name, owner: .owner.name}' api.json
{"id":101,"name":"web1","owner":"alice"}
{"id":102,"name":"web2","owner":"alice"}
{"id":103,"name":"db1","owner":"bob"}
{"id":104,"name":"ci1","owner":null}
{"id":105,"name":"test \"quoted\" name","owner":"carol"}
{id, name} is shorthand for {id: .id, name: .name}. Note how the name with quotes came out correctly escaped (\"), and how the missing owner of ci1 became null: jq never invents a value and never complains about a path through null.
Selecting, counting and grouping
select(condition) keeps the elements for which the condition is true. map(f) applies f to every element of an array; add sums an array; length counts:
$ jq -r '.items[] | select(.state=="running" and .env=="prod") | .name' api.json
web1
db1
$ jq '[.items[] | select(.cpu >= 4)] | length' api.json
3
$ jq '[.items[].price] | add' api.json
0.6656
$ jq '[.items[] | .mem_gb] | add / length' api.json
22.4
The square brackets around a filter collect its outputs into an array: [.items[] | select(...)] is "the array of matches", which you can then count or sum. Group and summarise with group_by (it needs an array, sorted by the key) and then build one object per group:
$ jq -c '.items | group_by(.env)[] | {env: .[0].env, n: length, cpu: (map(.cpu)|add)}' api.json
{"env":"dev","n":2,"cpu":4}
{"env":"prod","n":3,"cpu":16}
$ jq -c '[.items[] | {env, mem_gb}] | group_by(.env) | map({(.[0].env): (map(.mem_gb)|add)}) | add' api.json
{"dev":16,"prod":96}
Inside each group, .[0].env is the group's key and length is its size; the second form builds a single object whose keys are the group names. More selectors worth knowing: any(generator; condition) and all(...) answer yes/no questions, has("key") tests for a key, first(...) and limit(n; ...) stop early, and index("x") finds an element in an array:
$ jq -c '.items | (any(.[]; .state=="stopped")), (all(.[]; .cpu>=2))' api.json
true
true
$ jq -c '.items[0] | has("tags"), has("zone")' api.json
true
false
$ jq -c '[limit(2; .items[].name)], (first(.items[] | select(.cpu>=8)) | .name)' api.json
["web1","web2"]
"db1"
$ jq -c '.items | map(select(.tags|index("nginx"))) | map(.name)' api.json
["web1","web2"]
Null, missing keys and the // operator
Reading a key that does not exist, or going through a null, gives null and not an error:
$ jq '.next_page, .no_such_key, (.items[3].owner | type)' api.json
null
null
"null"
That is convenient and dangerous. The classic surprise is string interpolation, "\(...)", which turns a null into the four letters "null" instead of leaving the value out:
$ jq -r '.items[] | "\(.name): \(.owner.name)"' api.json
web1: alice
web2: alice
db1: bob
ci1: null
test "quoted" name: carol
ci1 printed the word "null" as if it were a name. If that went into a report, an email or a file name, nobody would notice. The alternative operator // gives a default when the left side is null or false (or produces nothing):
$ jq -r '.items[] | "\(.name): \(.owner.name // "unowned")"' api.json
web1: alice
web2: alice
db1: bob
ci1: unowned
test "quoted" name: carol
In a script, decide for every optional field what a missing value means, and write the default explicitly. (// replaces only null and false, so a value of 0 is kept, which is what you want for numbers.) To find the records that have the problem: jq -c '[.items[] | select(.owner == null) | .name]' api.json prints ["ci1"].
Exit codes, -e and bad input
By default jq exits 0 whenever it ran without an error, no matter what it printed, so "the filter found nothing" and "the filter found false" both look like success to the shell. The -e option (--exit-status) makes the last output decide: 0 if it was neither null nor false, 1 if it was null or false, and 4 if there was no output at all. That turns a jq filter into a condition for if:
$ jq -e '.items[] | select(.name=="db1") | .state=="running"' api.json >/dev/null; echo "[exit $?]"
[exit 0]
$ jq -e '.items[] | select(.name=="web2") | .state=="running"' api.json >/dev/null; echo "[exit $?]"
[exit 1]
$ jq -e '.items[] | select(.name=="nope")' api.json >/dev/null; echo "[exit $?]"
[exit 4]
$ jq -e '.status == "ok"' api.json; echo "[exit $?]"
true
[exit 0]
The answers: db1 is running (0), web2 is stopped so the comparison was false (1), no server with that name exists, so there was no output at all (4). A typical use after an API call is if curl -fsS "$url" | jq -e '.status == "ok"' >/dev/null; then ... fi. Remember that with set -o pipefail the exit status of curl also counts.
Bad input and bad filters are errors with status 5 (or 2 for usage mistakes):
$ printf '{"a": 1,}' | jq .; echo "[exit $?]"
jq: parse error: Expected another key-value pair at line 1, column 9
[exit 5]
$ jq '.items.name' api.json; echo "[exit $?]"
jq: error (at api.json:12): Cannot index array with string "name"
[exit 5]
$ jq '.items[0].cpu.x' api.json; echo "[exit $?]"
jq: error (at api.json:12): Cannot index number with string "x"
[exit 5]
The error says what went wrong (.items is an array, not an object; cpu is a number, not an object) and where. Note the difference from null: going through an absent key is fine, but indexing the WRONG TYPE is an error. Now the one that catches people: EMPTY input is not an error.
$ printf '' | jq .; echo "[exit $?]"
[exit 0]
No output, exit status 0. A script that downloads JSON, gets an empty body (a timeout, a proxy, a curl without -f) and pipes it to jq sees success and continues with nothing. Defend yourself: use curl -f (fail on HTTP errors), check that the result is not empty, and use jq -e so that "no output" becomes status 4. Never treat "jq printed nothing" as "all good".
Output formats: raw, CSV, TSV, shell, URL, base64
jq can format its output for other programs with the @ operators. They produce a STRING, so combine them with -r (otherwise the string is JSON-quoted again). On the sample data, with a name that contains quote characters:
$ jq -r '.items[] | [.id, .name, .cpu] | @csv' api.json
101,"web1",4
102,"web2",4
103,"db1",8
104,"ci1",2
105,"test ""quoted"" name",2
$ jq -r '.items[] | [.id, .name, .cpu] | @tsv' api.json
101 web1 4
102 web2 4
103 db1 8
104 ci1 2
105 test "quoted" name 2
$ jq -r '.items[] | "name=\(.name|@sh) cpu=\(.cpu)"' api.json
name='web1' cpu=4
name='web2' cpu=4
name='db1' cpu=8
name='ci1' cpu=2
name='test "quoted" name' cpu=2
$ jq -rn --arg q 'a b&c=d/é' '$q|@uri'
a%20b%26c%3Dd%2F%C3%A9
$ jq -rn '"hello"|@base64'; echo aGVsbG8= | jq -R '@base64d'
aGVsbG8=
"hello"
@csv quotes strings and doubles embedded quotes ("test ""quoted"" name"), which is exactly what spreadsheets expect; @tsv escapes tabs and newlines inside values with backslashes and leaves other quotes alone; @sh quotes a value so the shell reads it back as one word (use it with eval only when you must, see the next section); @uri percent-encodes for URLs; @base64 and @base64d encode and decode. A CSV with a header row: print the keys first, then the rows:
$ jq -r '(.items[0] | keys_unsorted | @csv), (.items[] | [.id,.name,.env,.cpu,.mem_gb,(.tags|join("|")),(.owner.name // ""),.state,.price] | @csv)' api.json | head -3
"id","name","env","cpu","mem_gb","tags","owner","state","price"
101,"web1","prod",4,16,"web|nginx","alice","running",0.0832
102,"web2","prod",4,16,"web|nginx","alice","stopped",0.0832
The header comes from the keys of the first record, and the rows follow (head -3 kept only the first three lines). Arrays inside a row need flattening (join), and optional values need a default, otherwise @csv prints null. The reverse direction (CSV to JSON) is not something jq does well: use jq -R to read lines and split them for simple cases, or Python/Miller for real CSV.
Building JSON safely: --arg, --argjson and injection
Never build JSON by pasting text together in the shell. A value that contains a quote or a backslash breaks it, and one that was written by someone else can add its own fields:
$ name='My "quoted" server'; echo "{\"name\": \"$name\", \"cpu\": 4}" | jq .
jq: parse error: Invalid numeric literal at line 1, column 21
[exit 5]
$ name='My "quoted" server'; jq -n --arg name "$name" --argjson cpu 4 '{name: $name, cpu: $cpu}'
{
"name": "My \"quoted\" server",
"cpu": 4
}
The first version produced broken JSON (the quote ended the string early); the second let jq do the quoting. -n means "no input, just run the program" (the filter builds the value from nothing). --arg NAME VALUE sets $NAME to a STRING; --argjson NAME JSON sets it to a parsed JSON value (a number, boolean, array or object); the difference matters:
$ jq -n --arg n 5 '{n: $n, type: ($n|type)}' -c; jq -n --argjson n 5 '{n: $n, type: ($n|type)}' -c
{"n":"5","type":"string"}
{"n":5,"type":"number"}
With --arg the number became the string "5"; with --argjson it stayed a number. Pass numbers and booleans with --argjson (or convert with tonumber), strings with --arg. For larger values, --slurpfile NAME FILE loads a JSON file into an array (every JSON value in the file is one element) and --rawfile NAME FILE loads a text file as a string, and --args passes the remaining command-line words as the array $ARGS.positional (these three options were not run here).
The same rule applies to the jq PROGRAM, and this is the injection bug of the jq world. If you splice a shell variable into the filter text, the value becomes CODE:
$ wanted='web1'; jq -r ".items[] | select(.name==\"$wanted\") | .id" api.json
101
$ wanted='x" or true or "'; jq -r ".items[] | select(.name==\"$wanted\") | .id" api.json | tr '\n' ' '; echo
101 102 103 104 105
$ wanted='x" or true or "'; jq -r --arg w "$wanted" '.items[] | select(.name==$w) | .id' api.json | wc -l
0
The value web1 worked in both styles. The hostile value turned the condition into name=="x" or true or "" and selected EVERY server (all five ids), while with --arg the same text was just a string that matched nothing (0 lines). If the value comes from a user, a file, an API or a tag, assume it can be hostile. Rule: the jq program in single quotes, every variable through --arg, --argjson or the environment ($ENV.NAME or env.NAME).
Getting values into bash
Three shapes of data come up, each with a safe way.
Many records, a few fields each: let jq print TAB-separated lines and read them with read:
$ cat readloop.sh
#!/bin/bash
set -euo pipefail
while IFS=$'\t' read -r id name state; do
printf 'id=%s name=[%s] state=%s\n' "$id" "$name" "$state"
done < <(jq -r '.items[] | [.id, .name, .state] | @tsv' api.json)
$ bash readloop.sh
id=101 name=[web1] state=running
id=102 name=[web2] state=stopped
id=103 name=[db1] state=running
id=104 name=[ci1] state=running
id=105 name=[test "quoted" name] state=running
The loop reads from a process substitution (< <(...)), so variables set inside it survive (the pipe-into-while trap from the Gotchas page). The name with quotes arrived intact. One caution about tabs. The shell treats a tab as "whitespace" in IFS and COLLAPSES several in a row, so an EMPTY field in the middle shifts everything left:
$ printf 'a\t\tc\n' | { IFS=$'\t' read -r x y z; echo "x=[$x] y=[$y] z=[$z]"; }
x=[a] y=[c] z=[]
$ printf 'a,,c\n' | { IFS=, read -r x y z; echo "x=[$x] y=[$y] z=[$z]"; }
x=[a] y=[] z=[c]
With a tab the empty middle field disappeared (y got c); with a comma it was kept. So give every optional field a default in jq ((.owner.name // "-")) so no column is ever empty, or choose a separator that is not whitespace, or read the output with awk -F'\t', which keeps empty fields (see the awk and sed page).
A list of values, into an array: one value per line with mapfile (bash 4 or newer; the names may contain spaces):
$ mapfile -t names < <(jq -r '.items[].name' api.json)
$ printf '%d names; the 5th is [%s]\n' "${#names[@]}" "${names[4]}"
5 names; the 5th is [test "quoted" name]
A value that might contain a newline would break the one-per-line idea. jq 1.7 can end each output with a NUL byte (--raw-output0), the one character that cannot appear in a string value, and bash reads it with read -d '':
$ while IFS= read -r -d '' n; do printf '[%s] ' "$n"; done < <(jq --raw-output0 '.items[].name' api.json); echo
[web1] [web2] [db1] [ci1] [test "quoted" name]
A single value: use -e so a missing value is an error, or // empty so it is an empty string, and always quote the result:
$ owner=$(jq -r '.items[3].owner.name // empty' api.json)
$ echo "owner of ci1: [${owner:-<none>}]"
owner of ci1: [<none>]
$ cpu=$(jq -er '.items[2].cpu' api.json) && echo "cpu of db1: $cpu"
cpu of db1: 8
$ if ! x=$(jq -er '.items[3].owner.name' api.json); then echo "jq -e failed (null) -> handled"; fi
jq -e failed (null) -> handled
Without -e and without a default, jq -r would have printed the four letters "null" for the missing owner, and the variable would hold the text null. Avoid eval "$(jq -r '... | @sh')": it works (the @sh output is quoted for the shell) but it executes whatever jq prints, so a slip in the quoting is a command injection. Prefer the read and mapfile forms above.
Transforming and updating
Everything so far read data; jq also rewrites it. The tools: map, to_entries / from_entries / with_entries (turn an object into a list of {key, value} and back), the update operators (=, |=, +=, *=), del, sort_by, unique, reduce, and the recursive helpers (paths, ..). Real runs:
$ jq -c '.items | map({name, monthly: (.price*730*100|round/100)})' api.json
[{"name":"web1","monthly":60.74},{"name":"web2","monthly":60.74},{"name":"db1","monthly":303.68},{"name":"ci1","monthly":30.37},{"name":"test \"quoted\" name","monthly":30.37}]
$ jq -c '[.items[] | select(.env=="prod")] | sort_by(-.mem_gb) | map(.name)' api.json
["db1","web1","web2"]
$ jq -c '.items | map(.owner.team // "none") | unique' api.json
["data","none","platform","qa"]
$ jq -c '.items[0] | .state = "stopped" | del(.tags, .owner, .price)' api.json
{"id":101,"name":"web1","env":"prod","cpu":4,"mem_gb":16,"state":"stopped"}
$ jq -c '.items |= map(.cpu *= 2) | [.items[].cpu]' api.json
[8,8,16,4,4]
$ jq -c '.items[0] | to_entries | map(select(.value|type=="number")) | from_entries' api.json
{"id":101,"cpu":4,"mem_gb":16,"price":0.0832}
$ jq -c '.items | map({(.name): .cpu}) | add' api.json
{"web1":4,"web2":4,"db1":8,"ci1":2,"test \"quoted\" name":2}
$ jq -c 'reduce .items[] as $i ({}; .[$i.env] += $i.cpu)' api.json
{"prod":16,"dev":4}
What they did, in order: the monthly cost per server (730 hours, rounded to cents); the prod servers by memory, biggest first (sort_by(-.mem_gb)); the distinct teams with a default for the missing owner; one record with a field changed and three removed; every cpu value doubled in place (|= applies a filter to the value at that path, *= multiplies it); the numeric fields of one record (to_entries turns {a:1} into [{key:"a",value:1}], from_entries goes back); an object keyed by server name (map to one-key objects, add merges them); and the total cpu per environment with reduce, jq's loop-with-accumulator. The last starts from {} and adds each server's cpu under its env key (a missing key counts as 0).
To walk a structure of unknown shape: paths lists the path to every value (paths(scalars) only the leaves; join(".") makes a dotted name), and .. visits every value:
$ jq -c '[.items[0] | paths(scalars) | join(".")?]' api.json
["id","name","env","cpu","mem_gb","tags.0","tags.1","owner.name","owner.team","state","price"]
$ jq -c '[.. | numbers] | length' api.json
21
$ jq -c '[paths(type=="number")] | length' api.json
21
(Both counts are the same: 21 numbers in the whole document, found two ways.) The trailing ? in join(".")? suppresses the error that join gives on numeric path elements such as the array index 0.
Strings, regular expressions, dates and numbers
$ jq -r '.items[] | .name | select(test("^web[0-9]$"))' api.json | tr '\n' ' '
web1 web2
$ jq -r '.items[0].name | sub("web";"app") | ascii_upcase' api.json
APP1
$ jq -rn '"a,b,,c" | split(",") | length, (. | map(select(. != "")) | join("-"))'
4
a-b-c
$ jq -rn '"host=web1 port=8080" | capture("host=(?<h>\\S+) port=(?<p>[0-9]+)") | .h, .p'
web1
8080
test(regex) is a boolean match (add "i" for case-insensitive), sub and gsub replace, split and join convert between a string and an array, and capture returns an object of named groups, so one line extracts fields from free text. Dates are seconds since 1970 inside jq; fromdate parses an ISO 8601 UTC time, todate prints one, and strftime formats it:
$ jq -c '"2026-10-10T08:15:00Z" | fromdate | (., todate, (strftime("%A")))' <<< null
1791620100
"2026-10-10T08:15:00Z"
"Saturday"
Numbers: jq 1.7 keeps the exact digits of a large integer that it only passes through, so ids and counters survive; arithmetic is ordinary double-precision floating point, with the usual rounding:
$ echo '{"id": 12345678901234567890, "f": 0.1}' | jq -c '., (.id|tostring), (.f+0.2)'
{"id":12345678901234567890,"f":0.1}
"12345678901234567890"
0.30000000000000004
The 20-digit id came out unchanged (older jq versions, 1.6 and before, converted such numbers to floating point and lost digits; that was not run here, so check your version if ids are large), and 0.1+0.2 shows the floating-point error you would see in any language: round when you print money (the monthly cost above used round/100). A value that is JSON stored inside a JSON string (common in logs and queue messages) is opened with fromjson, and tojson makes the reverse:
$ echo '{"payload": "{\"a\": 1, \"b\": [2,3]}"}' | jq -c '.payload | fromjson | .b[1]'
3
Logs with one JSON object per line (NDJSON)
Many services write one JSON object per line ("JSON lines" or NDJSON). jq reads such a file as a stream of separate values, one filter run per line, so it needs no special option:
$ cat app.ndjson
{"t":"08:01","lvl":"info","msg":"start"}
{"t":"08:02","lvl":"error","msg":"disk full","dev":"sda1"}
{"t":"08:03","lvl":"error","msg":"retry failed"}
{"t":"08:04","lvl":"info","msg":"stop"}
$ jq -c 'select(.lvl=="error") | {t, msg}' app.ndjson
{"t":"08:02","msg":"disk full"}
{"t":"08:03","msg":"retry failed"}
$ jq -r '[.t, .lvl, (.dev // "-"), .msg] | @tsv' app.ndjson
08:01 info - start
08:02 error sda1 disk full
08:03 error - retry failed
08:04 info - stop
To summarise across lines, either slurp the whole file into one array with -s, or keep the memory flat with -n and inputs and a reduce:
$ jq -s 'group_by(.lvl) | map({lvl: .[0].lvl, n: length}) | tojson' -r app.ndjson
[{"lvl":"error","n":2},{"lvl":"info","n":2}]
$ jq -s 'length' app.ndjson
4
(On the 200,000-line test file the streaming count is: jq -n -c 'reduce inputs as $l ({}; .[$l.lvl] += 1)' big.ndjson, which printed {"info":189978,"error":10022}; slurping builds the whole array in memory first, see Speed.) What about a log with some lines that are NOT JSON (a startup banner, a stack trace)? jq stops at the first one with a parse error. Read the file as raw text (-R) and parse each line yourself, with fromjson? silently skipping non-JSON, or turning it into an object of its own:
$ printf 'plain line\n{"ok":1}\n' | jq -c . ; echo "[exit ${PIPESTATUS[1]}]"
jq: parse error: Invalid numeric literal at line 1, column 6
[exit 5]
$ printf 'plain line\n{"ok":1}\n' | jq -R 'fromjson? // {raw: .}' -c
{"raw":"plain line"}
{"ok":1}
The plain parse failed on line 1 and exited with 5; the -R form kept going and wrapped the plain line as {"raw": "..."} (the ? after fromjson turns the error into "no output", and // then supplies the alternative).
Real JSON from the system: ip, lsblk, journalctl
Prefer the JSON output of system tools to parsing their text. ip -j, lsblk -J, journalctl -o json, docker inspect, kubectl -o json, systemctl show and many others have it. Interface names and addresses below were changed.
$ ip -j link | jq -r '.[] | [.ifname, .operstate, .mtu] | @tsv'
lo UNKNOWN 65536
wlp2s0 UP 1500
br-xxxxxxxxxxxx UP 1500
docker0 DOWN 1500
$ ip -j -4 addr | jq -r '.[] | select(.operstate=="UP") | .ifname + " " + (.addr_info | map(.local + "/" + (.prefixlen|tostring)) | join(","))'
wlp2s0 <ip>/22
br-xxxxxxxxxxxx <ip>/16
$ lsblk -J -o NAME,SIZE,TYPE | jq -r '.blockdevices[] | select(.type!="loop") | [.name,.type,.size] | @tsv'
nvme0n1 disk 931.5G
The first listed every interface with its state and MTU; the second the addresses of interfaces that are up (note the number-to-string conversion with tostring before joining text); the third the real disks without the loop devices. With journalctl -o json every entry is a JSON object with fields such as MESSAGE and PRIORITY. Two test messages were written with logger, an error and an info, and filtered by priority (jq compares numbers, so the PRIORITY text is converted with tonumber):
$ logger -t labjson -p user.err 'disk full on /var'; logger -t labjson -p user.info 'retrying'
$ journalctl -t labjson -o json -n 2 --no-pager | jq -r '[.SYSLOG_IDENTIFIER, .PRIORITY, .MESSAGE] | @tsv'
labjson 3 disk full on /var
labjson 6 retrying
$ journalctl -t labjson -o json --no-pager | jq -s 'map(select(.PRIORITY|tonumber <= 3) | .MESSAGE)' -c
["disk full on /var"]
Priority 3 is "err" and 6 is "info" (see the journalctl page); the second command kept the messages of priority 3 or worse. This is much safer than grepping the text output, because fields with spaces, quotes or newlines are handled by the JSON format itself.
Editing a JSON file: the redirect trap
jq does not edit files in place. The tempting one-liner destroys the file:
$ printf '{"a":1}\n' > tr.json; jq '.a=2' tr.json > tr.json; echo "after jq ... file > file: $(wc -c < tr.json) bytes"
after jq ... file > file: 0 bytes
The shell opens (and empties) the output file BEFORE it starts jq, so jq read an empty file and wrote nothing. The file is now 0 bytes and the data is gone. Write to a temporary file in the same folder and move it over the original, only if jq succeeded:
$ printf '{"a":1}\n' > tr.json; jq '.a=2' tr.json > tr.json.tmp && mv tr.json.tmp tr.json; cat tr.json
{
"a": 2
}
Because of the &&, a jq error leaves the original alone. (The sponge command from moreutils does the same job, jq ... file | sponge file; it was not installed here.) For configuration files that must stay valid, also validate the result (jq empty tr.json.tmp, or the program's own checker) before the mv, as in the safe_edit pattern on the awk and sed page, and keep a backup.
Speed
On a 200,000-line NDJSON file (9.2 MB, one small object per line):
471 ms jq -c 'select(.lvl=="error")' big.ndjson (streams: one line at a time)
641 ms jq -s 'map(select(.lvl=="error")) | length' (slurps everything into one array)
636 ms jq -r '[.user,.ms]|@tsv' big.ndjson (reformatting every line)
8 ms grep -c '"error"' big.ndjson (text only, for comparison)
1155 ms an empty bash while-read loop over the same file
Both ways gave the same count (10022 errors). So jq handles hundreds of thousands of records per second, and doing the same job in a bash loop would take far longer than jq. When the data is truly huge: stream instead of slurping (no -s; use -n with inputs and reduce), pre-filter the text with grep -F where the match is a plain string (grep found the lines in 8 ms) and send only those to jq, process with -c so the output is one line per record, and for a single multi-gigabyte JSON document use jq --stream. And do not call jq once per record in a shell loop: starting a process 200,000 times is far slower than one jq reading the whole file.
Common problems
parse error: Invalid numeric literal / Expected another key-value pair the input is not JSON: an HTML error page, a trailing comma, a log line, a BOM. Look at the first bytes (head -c 200); use curl -f so HTTP errors are not piped in
Cannot index array with string "x" the path expects an object but the value is an array: add .[] or an index: .items[].name
Cannot iterate over null .[] on a missing key: use .items[]? or (.items // [])[] to treat missing as empty
jq printed nothing and exited 0 the input was empty, or the filter matched nothing: use -e, check the input, and never treat silence as success
A field shows the word "null" in my output string interpolation or -r printed null: use // "default" or // empty, and jq -e when it must exist
The filter works in my terminal, not in the script a shell variable inside single quotes is not expanded (and inside double quotes it is spliced into the code): pass it with --arg or --argjson
A number became the string "5" --arg always makes a string: use --argjson, or tonumber
The file is empty after jq ... > same-file the shell truncated it first: write to a temp file and mv
Non-ASCII characters appear as é use -r for strings, and check the locale; jq output is UTF-8; --ascii-output forces escapes if you ever need them
Keys come out sorted only if you used -S (--sort-keys); by default jq keeps the input order
Large ids lose digits an old jq (1.6 or earlier) used floating point: upgrade, or treat ids as strings in the source
jq: command not found / flag not recognised not installed, or an old version: --raw-output0 and some functions need jq 1.7; check jq --version in the script header
Output order of an array is not what I expect group_by sorts by the key; use sort_by(...) to order explicitly, and unique_by to deduplicate
YAML input jq reads JSON only; a separate tool (yq) converts YAML (not installed in the lab)
Tips
Use jq for JSON, always: never grep, sed or awk a JSON document, not even for a "simple" field.
-r for strings you hand to the shell, -c for one record per line, -e when the answer must exist, -n when you build a value from nothing.
Pass every variable with --arg (strings) or --argjson (numbers, booleans, objects); keep the program in single quotes; never paste shell values into the filter.
Decide what null and missing mean for each field and write the default: // "unowned", // empty, // 0. Remember that interpolating null prints the word null.
Build JSON with jq, not with echo and quotes. A value with a quote or a backslash will break the string version.
Read into bash with a while read loop from a process substitution (tab-separated, with no empty fields), mapfile for lists, --raw-output0 and read -d '' for values that may contain newlines, and $(jq -er ...) for a single value.
Treat empty output and exit status 0 with suspicion: check the input and use jq -e.
Never redirect jq to its own input file: write a temp file in the same folder, validate it, and mv it over the original.
Stream big files (no -s), filter early with grep -F when a plain string will do, and keep jq out of per-record shell loops.
Prefer the JSON output of tools (ip -j, lsblk -J, journalctl -o json, docker inspect, kubectl -o json) to parsing their text.