jq Command Guide: Filter, Transform, and Validate JSON from APIs and Logs
jqJSONCLIAPI toolslog analysis

jq Command Guide: Filter, Transform, and Validate JSON from APIs and Logs

TToggle.top Editorial Team
2026-11-08
6 min read

A practical jq command guide for filtering API responses, reshaping nested JSON, handling nulls, producing CSV, and troubleshooting shell scripts.

jq is a command-line JSON processor for inspecting API responses, reshaping deployment output, extracting fields from logs, and validating payloads without opening a separate application. This guide covers the filters worth remembering, nested-array patterns, null handling, CSV output, shell quoting, and a practical review cycle for keeping recurring jq scripts reliable.

Overview

When JSON is small, a browser-based JSON formatter may be enough. When the data is part of a repeatable workflow, jq is more useful because it can read from files, standard input, pipelines, and API commands. It can select values, construct new objects, search arrays, perform calculations, and produce machine-readable output for another command.

The basic form is:

jq 'filter' input.json

A period represents the current input value, so this command formats a JSON document without changing its data:

jq '.' response.json

You can also pipe API output directly into jq:

curl -s https://example.test/api/items | jq '.items[] | {id, name}'

For sensitive API responses, prefer local processing and review the command before sending data to any online developer tool. The same principle applies to tokens, credentials, customer records, and internal logs. For a broader workflow, see how to choose online developer utilities safely.

What to track

Core filters and projections

Most jq work starts with selecting a field:

jq '.status' response.json
jq '.user.email' response.json
jq '.metadata["release-name"]' response.json

Use a pipe to move from one value to the next. For an array, .items[] emits each item separately:

jq '.items[] | .name' response.json
jq '.items[] | {name: .name, state: .status}' response.json

The second example creates a smaller object. This is often preferable to passing an entire API response into another command when only two fields matter.

Search and filter arrays

The select function keeps values that satisfy a condition:

jq '.items[] | select(.status == "failed")' response.json
jq '.deployments[] | select(.duration_seconds > 300) | .id' response.json

Combine conditions with and or or:

jq '.items[] | select(.environment == "production" and .healthy == false)' response.json

Nested objects and arrays

For data shaped like an array of deployments, each containing services, chain the array expansions:

jq '.deployments[] | .services[] | {deployment: .id, service: .name}' response.json

In that example, the inner object may not contain the deployment ID after the second pipe. Preserve the parent value by binding it with as:

jq '.deployments[] as $deployment | $deployment.services[] | {deployment: $deployment.id, service: .name}' response.json

When you need array indexes, use to_entries:

jq '.items | to_entries[] | {index: .key, value: .value}' response.json

Nulls, defaults, and missing fields

Missing fields and explicit null values are common in API responses. The alternative operator provides a fallback:

jq '.owner.name // "unassigned"' response.json

Use a type check when an API field may change shape:

jq '.labels | if type == "array" then join(",") else . // "" end' response.json

Be careful with the difference between a missing value, null, an empty string, and false. A filter designed around truthiness can accidentally discard a legitimate false value. When the distinction matters, test it explicitly.

Reshape JSON and produce CSV

Construct objects with braces and arrays with brackets:

jq '[.items[] | {id, name, status}]' response.json

For a spreadsheet-friendly result, select the columns in order and use @csv. The -r option writes raw strings instead of JSON-quoted strings:

jq -r '.items[] | [.id, .name, .status] | @csv' response.json

To include a header row, combine arrays:

jq -r '(["id", "name", "status"], (.items[] | [.id, .name, .status])) | @csv' response.json

For line-oriented output that another shell command can consume, use compact JSON:

jq -c '.items[]' response.json

For related encoding tasks, the guide to Base64, URL encoding, and hashing helps distinguish data transformation from cryptographic hashing.

Cadence and checkpoints

jq is a utility rather than a dashboard, but jq scripts often become recurring engineering workflow tools. Review them on a monthly or quarterly cadence when they feed deployment checks, incident summaries, test fixtures, or scheduled reports. Also review them immediately when an API response, log schema, or service version changes.

CheckpointWhat to verifyUseful jq pattern
Input shapeIs the expected object or array still present?jq 'type, keys'
Required fieldsDo identifiers and status fields still exist?jq '.items[] | {id, status}'
Null behaviorAre missing values handled deliberately?jq '.owner.name // "unassigned"'
Output contractDoes downstream tooling receive JSON, CSV, or raw text?jq -c or jq -r
Failure behaviorDoes the shell pipeline stop when validation fails?jq -e with an explicit condition

Keep a small fixture file with representative success, failure, empty, and null cases. Running the same filters against that fixture makes schema changes easier to detect than testing only against the latest live response.

How to interpret changes

A jq failure does not always mean the filter is wrong. First identify whether the input is malformed, the data shape changed, the shell altered the command, or the filter returned an unexpected result.

  1. Inspect the top-level type. Run jq 'type'. If the filter expects an array but the API now returns an object containing an array, update the path rather than adding more complex logic.
  2. Inspect keys before values. Run jq 'keys' for an object or jq '.[0] | keys' for the first array item. This quickly exposes renamed or nested fields.
  3. Check shell quoting. Put jq filters in single quotes in POSIX-style shells so characters such as $, brackets, and parentheses reach jq unchanged. Use double quotes inside the filter for JSON strings.
  4. Separate jq variables from shell variables. Pass external values with --arg instead of interpolating them into a filter:
name="worker"
jq --arg name "$name" '.items[] | select(.name == $name)' response.json
  1. Validate deliberately. A formatting command can succeed while a required field is absent. For a simple check, use:
jq -e 'has("request_id") and (.status == "ok")' response.json

Use the command's exit status in CI or a shell script, and make the expected condition clear to reviewers. If the input contains one JSON document per line, process it as newline-delimited JSON only when every line is valid JSON. A single malformed line can interrupt the workflow, so retain the original input and report the failing record when troubleshooting.

For configuration files, jq is appropriate for JSON. YAML and TOML require parsers that understand those formats; a general JSON formatter cannot safely validate them. See the workflow for YAML, JSON, and TOML configuration validation when a repository uses multiple configuration formats.

When to revisit

Return to this guide whenever a recurring jq command starts producing empty output, unexpected nulls, broken CSV columns, or a nonzero status in CI. A monthly or quarterly review is sensible for scripts tied to API integrations, deployment reports, and operational logs; event-driven review is more important when a provider changes its response schema or a team renames fields.

Use this short checklist during the review:

  • Save one current sample and compare its top-level type and keys with the fixture.
  • Test an ordinary record, an empty collection, a missing field, and a null field.
  • Confirm whether the consumer expects compact JSON, pretty JSON, raw text, or CSV.
  • Check shell quoting on the operating systems used by the team.
  • Run the command against malformed input to confirm that the failure is visible.
  • Document assumptions about field names, array structure, and required values beside the script.

Once the filter is stable, keep it small and composable. A clear sequence such as .items[] | select(...) | {id, status} is usually easier to maintain than one opaque expression. Pair jq with a JSON formatter for quick visual inspection, and use a dedicated API debugging workflow when the problem also involves authentication, URL encoding, or request construction.

Related Topics

#jq#JSON#CLI#API tools#log analysis
T

Toggle.top Editorial Team

Developer Tools Editor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.