10 ms·
Mastering Jq: Part 1
- jbotz 6y agojq is to cloud and "Infrastructure as Code" software what grep and awk are to unix software. It's the glue that lets you combine the pieces into a system. It's an odd language though, and not that easy to master. Tutorials like this one are helpful and needed... Thanks, Tyler! However, there are quite a few typos in the examples... even the very first one is missing a quote and won't parse if cut-and-pasted as is.
- code-faster 6y agoThanks for flagging that, they've been fixed and retested, thank you jbotz!
- cle 6y agoI used to use shell scripts and jq a lot, but got tired of my scripts growing into monstrous mudballs of Bash hackery. I do my scripting mostly in Go now and it's much easier to structure my code and grow it over time. I sometimes use Python, but Go is more flexible for what I do. I can compile an exe and drop it onto a host and run it, without worrying about VM and library versioning (or in Bash's case, making sure jq is installed and all the Unix utils have compatible versions).
- thechao 6y agoHow’s go’s batteries-included story? I’ve got nigh-on 50k lines of Python2 that IT is about to boot. My choice is to port it to ... something. I was going to do Python3 but I’ve always thought that a memory-management-free systems-like language would’ve always been better.
- eat_veggies 6y agogo's standard library and docs are high quality and well designed. You can take a look here to see if it fits your needs: https://golang.org/pkg/ https://golang.org/pkg/
- cle 6y agoThere is not a language as batteries-included as Go, that I've found. The standard lib includes almost everything, including an excellent HTTP server and client (even an HTTP proxy), a test framework, JSON/XML/CSV libs, SQL driver, CLI parser, a template library, crypto, etc. The "go" command includes tooling like a package manager, unit test runner, code formatter, static analyzer, race condition checker, code coverage, profiler, cross-compiler, etc. I don't know what you mean by "memory-management-free" though, Go is still a garbage collected language.
- Evidlo 6y agoI'd say Python has more batteries than Go. Also I recently learned Go's CLI parser is pretty weak compared to argparse.
- cp9 6y agoymmv there but for what I'm looking for Go is usually more helpful than python straight out of the box, specifically because you don't have to reach for a third party library to make simple http requests, which is what I often end up using a scripting language for. I did just pick up argparse though and was very pleasantly surprised about how nice it is, go's `flags` module in the stdlib is unfortunately not quite as nice and is one of the things I reach for a third party library on. However, because it's pure go it still compiles down to a single binary which you can just scp around.
- closeparen 6y agoGo's error-oriented style makes it very painful to write long sequences of fallible operations, particularly if your organization has test coverage requirements. I'll write servers in Go all day long, but scripting is the last thing I'd use it for.
- otabdeveloper4 6y agoYou're not actually supposed to care about errors. It's the C tradition, YOLO. (bash doesn't care about errors either, by the way.)
- cle 6y agoThere is some unfortunate boilerplate involved for scripting, but the benefits dramatically outweight that minor annoyance, IMO. Also, as the script accretes complexity (which happens quite often for me), the disciplined error handling starts adding more value.
- closeparen 6y agoExplain the benefits. "if err != nil { return nil, err }" is implicit for every statement in languages with exceptions.
- cle 6y agoThat's only one way to handle an error. The benefit is that it's explicit, not implicit, and it's a psychological nudge to ask yourself "oh yeah this could produce an error, how should I handle it?". I can scan code and see where all the errors occur and how that particular component intends for each to be handled. And handling errors in concurrent code is consistent--still passed as values. Languages with implicit stack-unwinding exceptions have inconsistent ways to deal with the errors because, in concurrent programs, error handling doesn't end once the stack is unwound, because there are lots of concurrent stacks. You then usually catch the exception near the top of the stack and pass it to another stack...as a value. So yes, there is some boilerplate involved in that. But as a codebase grows, I appreciate that the language encourages me and others to be intentional about how errors are handled, and also provides a consistent mechanism for handling them in concurrent code (concurrency is a big reason I use Go to begin with).
- MattyRad 6y agoBy far the most useful jq operator I've used is recursive descent: `..`. It can take a positively heinous json blob and pick out the attributes/keys you care about. For example: Yikes: > wget -O - -q 'https://reddit.com/r/unixporn/new.json' https://reddit.com/r/unixporn/new.json' | jq I just want the links: > wget -O - -q 'https://reddit.com/r/unixporn/new.json' https://reddit.com/r/unixporn/new.json' | jq '..|.permalink? | select(.)'
- Arkanosis 6y agoI've been casually using jq for years, yet you just managed to put three things I didn't know about it and will use extensively from now on in a single one-liner. Thank you!
- hiram112 6y agoThank you. This will be added to my 'How To JQ' list, since I'm convinced now that I'll never fully understand its syntax enough to master it outright.
- mahalol 6y agoOh, would you mind sharing your list please?
- jolux 6y agoIs there stuff jq can do that I can't do with Invoke-RestMethod and manipulating objects in PowerShell?
- inshadows 6y agoIt can run without PowerShell.
- dragonwriter 6y agoThere's nothing in jq you can't do with any programming language with a JSON parser and serializer implemented. But I find jq more pleasant than most for lots of JSON-to-JSON transformations, and Powershell worse than not only jq but also Node or Python for that task.
- jolux 6y agoIt’s a one-liner in PowerShell to grab JSON from an endpoint and deserialize it.
- dragonwriter 6y ago> It’s a one-liner in PowerShell to grab JSON from an endpoint and deserialize it. Yes, and that’s great. I don't prefer jq for downloading JSON (which it doesn't do), but, as I said, for JSON-to-JSON transformations.)
- jolux 6y agoThat’s fair, it probably makes more sense for that usecase than PS. It just feels like yet another DSL to learn
- thatfunkymunki 6y agoI really feel like powershell is underrated in the silicon valley ecosystem. I just had my work computer switched from mac to win10 and am so glad to have powershell back at the handy.
- llimllib 6y agoThis example is weirdly spelled: echo '{"k1": [{"k2": [9]}]}' | jq '.k1 | .[0] | .k2 | .[0]' is equivalent to echo '{"k1": [{"k2": [9]}]}' | jq '.k1[0].k2[0]' I kept waiting for the author to explain this, but they did not
- code-faster 6y agoIt is a weird way to write it, I never use this notation for myself. When teaching though, I'd rather make the it more visually explicit that jq commands are a sequence of filters. With the former notation, a | explicitly tells the reader that it's the end of one filter and the start of a new one, the latter is more implicit.
- llimllib 6y agoThat's possible! I've never had to teach it, good point.
- encloser 6y agoWhen I first started using jq, all of the examples I was following used the short-form. It took me a while to figure out how to use the functions and do some advanced transforms because of this. Since this article is Part 1 of a series I suspect the author will cover this in a later article. From a teaching perspective it is probably best to start with the long-form.
- llimllib 6y agogood point, I hadn't considered that idea.
- JamesSwift 6y ago`jq` is up there with `nix` for me where I can sort of remember the DSL but always have to open the docs. I recently found https://jqplay.org/ https://jqplay.org/ and its a huge help. I can crank scripts out much quicker now that I'm iterating with that.
- yingw787 6y agohttps://jqterm.com https://jqterm.com is also a good one :)
- coding123 6y agoJq is quickly becoming one of the most useful tools in devops.
- ing33k 6y agoMhm. Can you please add some details on how you are using it ?
- code-faster 6y agoI would also be fascinated to hear how you're using it.
- coding123 6y agoGenerally we use it to manipulate JSON configuration files. So often we'll have a single Config file across many microservices. We just use the same config for simplicity. Then we deploy many different services that have access to that same config. Some of the docker containers know how to natively access the config file. Others, things were we might be running one-off jobs especially, are often built from outside software that has no idea how to read our config file. Instead of programming a harness around it, we just typically use jq to read the config elements we need and save them as variables in a script. Then we can pass those variables as needed to whatever command line arguments. Now, admittedly anywhere we have Node available, it pretty much negates a lot of it. But even with node, it's sometimes easier to just throw in a couple jq calls to extract data.
- ryanmccullagh 6y agoMy favorite way to use jq is as follows: `tail -f my-log-file.json | jq` This can be great if you've setup Nginx to log as JSON.
- kureikain 6y agoAnother good one about JQ: https://programminghistorian.org/en/lessons/json-and-jq#the-dot- https://programminghistorian.org/en/lessons/json-and-jq#the-...
- anarcat 6y agoI know I might be a dissenting opinion here, but I can never wrap my head around `jq`. I can manage `jq .`, `jq .foo` and `jq -r`, but beyond that, the DSL is just opaque to me. And every time I try to learn, I get lost in a maze of a manpage: https://manpages.debian.org/jq https://manpages.debian.org/jq I mean there are useful tips in there, but I rarely find what I need. I usually find it simpler to write a Python script (which ships with `json` because it's "batteries included") and operate on lists and dicts the normal way... It's longer, but at least I don't need to learn a new programming language (if we can call jq that...)
- jldugger 6y agoAFAICT, it's basically xpath applied to json. Which isn't helpful, but perhaps explains why it's so obtuse.
- de_watcher 6y agoAnd I still have no idea why jq is slow af.
- IshKebab 6y agoSeems pretty fast to me considering it is reading JSON. JSON is difficult to read quickly.
- lucideer 6y agoI know xpath pretty well, and cannot wrap my head around jq's DSL. It might have borrowed from elements of xpath's more obtuse expressions, but simple xpath seems much simpler (to me) than simple jq.
- kureikain 6y agoI cannot remember the syntax and have to search for it every time but it's very powerful. You basically give a DSL to process json. I can do map, filter and even math. The reason it's popular partly because it's available from both Centos/Debian based. Just use `yum` and `apt`. Most of time, you maybe limited with what have on a container. And `jq` may even pre-install by DevOps. Second is it play well with unix tool, pipe and friend.
- Ericson2314 6y agoThe author of jq is quite talented. I hope jq becomes a gateway drug.
- Rapzid 6y agoI don't know what it is, but anything more than moderately simple stuff gets super annoying for me with JQ. After getting super frustrated with the documentation while trying to accomplish stuff that should have been straight forward, I created jsling so I could pipe output through node for JavaScript one-liners. Anything moderately complicated that doesn't need to be portable; I just use that. EDIT: I should mention that by "moderately complicated" I mean stuff that starts to get into the realm of joins, correlated sub queries, and the like.
- pyuser583 6y agojq really is an awesome tool.
- ryannevius 6y agoOne of the most useful ways I've found to learn/test queries in `jq` is by using it as a semi-REPL via fzf with `--preview`: echo '' | fzf --print-query --preview 'cat example.json | jq {q}' Here's a rough example of it in action: https://asciinema.org/a/y4WGyqcz1wWdiyxDofdPXKtdC https://asciinema.org/a/y4WGyqcz1wWdiyxDofdPXKtdC
- ctippett 6y agoWow, thank you for sharing this. I had no idea you could do that with fzf.
- labelbias 6y agoJust recently used xq to convert and clean up a bunch of weird xml files to JSON. It was such a breeze. I love how nested lists inside objects can expand into a particular one with the .[] operator. For example: { a: [{b: 5}, {b: 3}], c: 5} can be transformed into: [{ b: 5, c: 5}, {b: 3, c: 5}] using jq '{c: .c, b: .a[].b}' For heavily nested XMLs I can get a nice flat output.
- code-faster 6y agoNice! I've found xml transforms is one of the best applications of jq (yes, jq, not xq). In a future post, I'll cover how to use jq not just for json and xml but any data format.
- ojhughes 6y agoWhen I find myself reaching for jq, that’s a signal that maybe this bash script should be implemented in a language such as python or go
- yiyus 6y agoAt first I thought this was about github.com/timestored/jq and was very confused.
- moreati 6y agoI wrote https://github.com/moreati/jq-filter https://github.com/moreati/jq-filter for using jq with as a filter in Ansible.
- macca321 6y agoI reckon JMESPath should be first choice in 2020, as it's properly specified, so you can find x-language implementations that behave the same.
- xonix 6y agoI myself am developing a very similar tool [1] with somewhat saner (IMHO!) DSL. In fact this is a side-project from a JS lib I developed long ago [2]. My approach uses very simple ideas and is heavily based on JS internally. This is simple once you understand it [3] (thus saner DSL) and gives the whole power of JS in your hands. I've also prepared a basic comparison jq vs jsqry based on some examples in article [4]. It's worth noting that currently the CLI tool [1] is written in Java using Graal VM polyglot native image compilation. Thus HUGE executable size of 95 MB (sic!) because it bundles JS engine. I'm considering rewriting this to QuickJS by Fabrice Bellard [5]. This should make it MUCH smaller. [1] https://github.com/jsqry/jsqry-cli https://github.com/jsqry/jsqry-cli [2] https://github.com/jsqry/jsqry https://github.com/jsqry/jsqry [3] https://jsqry.github.io/#filtering https://jsqry.github.io/#filtering [4] https://gist.github.com/xonixx/d6066e83ec0773df248141440b18e8e4 https://gist.github.com/xonixx/d6066e83ec0773df248141440b18e... [5] https://bellard.org/quickjs/ https://bellard.org/quickjs/
- maddyboo 6y agojq is honestly one of my favorite tools ever. Super powerful yet simple. I've written some relatively complex programs with it, e.g. this one [1] I just wrote to manipulate the output from neuron [2] to generate index files based on tags. It's probably overkill/not as efficient as it could be, but I wanted to be able to expand upon it later. I think once you grok the basics, which can be a bit confusing at first, you can accomplish some amazing things with the help of a few of the more advanced features like reduce, to_entries/from_entries, etc. I really wish it was more actively developed, I feel like it actually has potential to be a semi-general-purpose functional programming language. 1: https://gist.github.com/b50a047115d1dcf1f15c16a6f7b71e3c https://gist.github.com/b50a047115d1dcf1f15c16a6f7b71e3c 2: https://github.com/srid/neuron https://github.com/srid/neuron