Using curl with Jq 🤖

Usecase

  • You build or work with API’s on your daily job
  • You work with JSON a lot and you need a way to quickly manipulate it, transform, filter, find and/or do any other relevant operations

Why

  • You already have CURL on your computer
  • You don’t have to install or pay for anything
  • It’s light fast, consumes no resources
  • You’ll feel like a hacker with that green terminal theme 🤮
  • You’ve been using tools like Postman which were 99% overhead for what you actually needed
  • You can combine it with other bash tools you already use and simply pipe to output to whatever you want

Why not

  • You don’t have a computer
  • You are not yet brought to the planet to exist amongst the rest of us apes

Prerequisites

To establish some ground, let’s make sure we understand what CURL and Jq are. (Descriptions taken from the world wide web, links included)

What is CURL?

curl is a tool to transfer data from or to a server, using one of the supported protocols (HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, DICT, TELNET, LDAP or FILE).

What is Jq?

jq is a lightweight and flexible command-line JSON processor. If you are a command line addict, you will like the official description. jq is like sed for JSON data – you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text.

Get ready to rumble

The best way to start is by demonstrating why would one even consider using CURL with Jq. In this specific article, we will mainly focus on fetching JSON data and doing transformations on it.

We will use the Fake JSON Placeholder API to demonstrate how to do transformations using Jq. Before we get to it, let’s first learn about Pipes. If you are already familiar with the concept, feel free to skip it.

Pipes

In simple words, pipes enable us to pass the output of one program as the input to another one.

  someProgramThatReturnsSomething | someOtherProgram
  # Whatever 'someProgramThatReturnsSomething' returns will be passed to 'someOtherProgram'
  # The | in-between them is called a pipe operator

The reason why pipes are so powerful is that we can keep passing the data to the next program (function) as long as the previous one returns some data. This enables us to pipe the data through as many different tools (functions) as we want, and eventually get the desired ouput. Then we can either store this output as a file or do whatever we want with it. You can think of this like function composition in functional programming.

Transforming JSON with Jq

Now when we know what pipes are and how to use them, let’s finally start with Jq.

Since Jq provides various methods for transforming data, I will showcase the ones I most commonly use. Once you get the feel of how the Jq works, make sure you check the official documentation.

Filters

Filters are methods which are gonna be applied to our data. They are the essential part of Jq.

dot

Dot filter prettifies the JSON output by formatting it and adding some colors for enhanced readability. Try the example below in your terminal with and without Jq and you will notice the difference.

curl https://jsonplaceholder.typicode.com/users/1 | jq .

.key

Let’s learn how we can get a single property of an object. Let’s imagine we only want to get the username property

curl https://jsonplaceholder.typicode.com/users/1 | jq ".username"
#ouputs: "Bret"

You can also provide several object keys by separating them by a comma

curl https://jsonplaceholder.typicode.com/users/1 | jq ".id, .username"
#ouputs:
# 1
# "Bret"

Accessing sub-objects is as simple as you would do it in e.g. Javascript

curl https://jsonplaceholder.typicode.com/users/1 | jq ".company.name"
#ouputs:
# Romaguera-Crona

Keys

Keys filter is idential to Object.keys() method in Javascript. It returns all the object keys as a string array

curl https://jsonplaceholder.typicode.com/users/1 | jq "keys"
#ouputs:
# [
#  "address",
#  "company",
#  "email",
#  "id",
#  "name",
#  "phone",
#  "username",
#  "website"
]

That’s essentially it for this quick demonstration of jq. For many other filters, make sure to check the official documentation. In case you’ve liked what you read, good, I’m happy for you.