5 ms·

Curl will now output JSON

120 points by logicalstack 7y ago

dTal 7y ago
I think if all command line tools outputted JSON, it would go a long way towards fixing the "ad-hoc parser problem" in scripts, and bring unixy shells around to a more PowerShell-like object-passing style. Passing plain text around is a great idea, but It's silly to restrict yourself to a single structural unit within that text (the "line"). We have better tech now.
- majkinetor 7y ago
  While powershell is a lot more then passing hashtable around, it would make a first good step. Accepting json object as parameter in "unix standard" should be done along.
- banger180 7y ago
  Yep this would be really great.
- ayyess 7y ago
  Have you seen jc[0] as an effort to wrap common utilities to output json? [0]: https://github.com/kellyjonbrazil/jc https://github.com/kellyjonbrazil/jc
  - phnofive 7y ago
    I hadn’t; cheers!
- nyberg 7y ago
  It's nice but I don't think json is such a great format for it. The overhead of parsing it is incredibly annoying along with being unable to pack binary data which is what unix tools can deal with to some extent
  - carlhjerpe 7y ago
    json is fast, according to https://github.com/xi-editor/xi-editor/blob/master/README.md#design-decisions https://github.com/xi-editor/xi-editor/blob/master/README.md... "I considered binary formats, but the actual improvement in performance would be completely in the noise. Using JSON considerably lowers friction for developing plug-ins, as it’s available out of the box for most modern languages, and there are plenty of the libraries available for the other ones." It doesn't have to be a one size fits all solution, there are edge cases where json wouldn't be the right choice, but for many and most things i could think of that i use daily I'd be happy with json. If it's small binary data i guess it could be base64 encoded. Querying network interfaces, directories, process information, systemd daemons or anything else, i see these things could do great if they just outputted json and my shell could treat it as an object. If a script is performance critical, maybe it's time to rewrite it in another language anyways?
  - rantwasp 7y ago
    i have 2 letters for you: jq any tool that outputs json + jq => instant scriptability win.
    - technofiend 7y ago
      Particularly if you prepend queries with try. jq likes to throw errors when unexpected things happen in your search or your data, and it may return "null". Using try instead returns an empty string. When walking JSON structures to validate data I code each validation separately instead of burying them in blocks of if-then-else logic. I think this makes it easier to maintain and more obvious when dealing with data that for example may be in a list or an array of lists. But yeah if you write golang scripts and want to shuffle data in and out of bash, jq is your #1 power tool.
    - de_watcher 7y ago
      Two letters, but then several lines to write an actual simple request (with some dances to battle the null and the exit code), and it's slow.
- Arnavion 7y ago
  libxo is doing a good job for FreeBSD. It would be nice if Linux also adapts that, or something similar, one day. Eg I have a script that needs output from netstat, and `netstat -bin --libxo json | jq ...` is a great way to get that info instead of scraping the human-readable output.
  - touisteur 7y ago
    Sorry, stupid question, but why not use /proc/net directly?
    - naniwaduni 7y ago
      /proc is a linuxism.
      - touisteur 7y ago
        Yes... But even on Linux I've been bitten so much by changes in 'standard' tools just on successive versions of centos, the last 10 years... ifconfig changing its output syntax (I know 'just use ip' well it changes sometimes too...), hostname sets a transient hostname now, hostnames with _ are now refused... netstat had some subtle changes some time ago too. free changed too. df, fricking df output format changed. So now, I just hit /proc and /sys, and document the hell out of it. Sure, new fields can appear there but they're new columns. The parser has to handle 'additional' columns. I trust Linus more with not-breaking-userland than the folks that make 'portable' tools. Glad I don't have to support other OSes...
  - kbrazil 7y ago
    It would be great to have libxo in linux! FYI, `jc netstat` also generates JSON output for `netstat`. (disclosure: I'm the author of `jc`) While I'm at it... check out `jtbl` on my github which prints the JSON or JSON Lines output to a table in your terminal. https://github.com/kellyjonbrazil/jc https://github.com/kellyjonbrazil/jc https://github.com/kellyjonbrazil/jtbl https://github.com/kellyjonbrazil/jtbl
- chatmasta 7y ago
  Maybe a good interim solution would be a “proxy command” that knows the output format of common commands, and how to convert them to json. So for example you could pipe the output of ls to this tool, and it would be smart enough to parse it and convert to json.
  - matt_kantor 7y ago
    https://github.com/kellyjonbrazil/jc https://github.com/kellyjonbrazil/jc was posted in another comment; it appears to be exactly that.
- de_watcher 7y ago
  It's slow and obtuse, the syntax of processing commands is too verbose, there are surprises with null values and return codes jq gives. It's not the answer for everything, it's just one of the tools. It feels cosy to huge amount of people because it's the product of web explosion, but at the core it's accidental and not that good.
pyuser583 7y ago
That’s nice.
MuffinFlavored 7y ago
Do you still need to pass it to `jq` for it to be pretty printed or no, I couldn't tell from reading the article?
- kuesji 7y ago
  yes, you still need 'jq' for pretty print.
quintonish 7y ago
Note that this is not titled, "Curl will now format JSON". This will "Make curl display information on stdout after a completed transfer"[1]. [1] https://curl.haxx.se/docs/manpage.html#-w https://curl.haxx.se/docs/manpage.html#-w
theMiddleBlue 7y ago
I'm testing it, maybe I'm doing it wrong but I don't understand well if it's possible to show only the json output (and not both json and response body). I'm doing it with a trick: curl -I -XGET -w '%{json}' https://www.google.com https://www.google.com | tail -1 | jq
- otterlicious 7y ago
  -o /dev/null
enriquto 7y ago
No need to panic! You can still pipe it to gron [0] and recover useable output. [0] https://github.com/TomNomNom/gron https://github.com/TomNomNom/gron
monkey26 7y ago
To bad this requires nightly.
rossmohax 7y ago
It is a step in the right direction, but still far from all the goodnes of PowerShell's Invoke-WebRequest . Not sure curl can do anything about it though.
omani 7y ago
beautiful. thank you for this.

120 points by logicalstack 7y ago

dTal 7y ago

I think if all command line tools outputted JSON, it would go a long way towards fixing the "ad-hoc parser problem" in scripts, and bring unixy shells around to a more PowerShell-like object-passing style. Passing plain text around is a great idea, but It's silly to restrict yourself to a single structural unit within that text (the "line"). We have better tech now.

majkinetor 7y ago
While powershell is a lot more then passing hashtable around, it would make a first good step. Accepting json object as parameter in "unix standard" should be done along.
banger180 7y ago
Yep this would be really great.
ayyess 7y ago
Have you seen jc[0] as an effort to wrap common utilities to output json? [0]: https://github.com/kellyjonbrazil/jc https://github.com/kellyjonbrazil/jc
- phnofive 7y ago
  I hadn’t; cheers!
nyberg 7y ago
It's nice but I don't think json is such a great format for it. The overhead of parsing it is incredibly annoying along with being unable to pack binary data which is what unix tools can deal with to some extent
- carlhjerpe 7y ago
  json is fast, according to https://github.com/xi-editor/xi-editor/blob/master/README.md#design-decisions https://github.com/xi-editor/xi-editor/blob/master/README.md... "I considered binary formats, but the actual improvement in performance would be completely in the noise. Using JSON considerably lowers friction for developing plug-ins, as it’s available out of the box for most modern languages, and there are plenty of the libraries available for the other ones." It doesn't have to be a one size fits all solution, there are edge cases where json wouldn't be the right choice, but for many and most things i could think of that i use daily I'd be happy with json. If it's small binary data i guess it could be base64 encoded. Querying network interfaces, directories, process information, systemd daemons or anything else, i see these things could do great if they just outputted json and my shell could treat it as an object. If a script is performance critical, maybe it's time to rewrite it in another language anyways?
- rantwasp 7y ago
  i have 2 letters for you: jq any tool that outputs json + jq => instant scriptability win.
  - technofiend 7y ago
    Particularly if you prepend queries with try. jq likes to throw errors when unexpected things happen in your search or your data, and it may return "null". Using try instead returns an empty string. When walking JSON structures to validate data I code each validation separately instead of burying them in blocks of if-then-else logic. I think this makes it easier to maintain and more obvious when dealing with data that for example may be in a list or an array of lists. But yeah if you write golang scripts and want to shuffle data in and out of bash, jq is your #1 power tool.
  - de_watcher 7y ago
    Two letters, but then several lines to write an actual simple request (with some dances to battle the null and the exit code), and it's slow.
Arnavion 7y ago
libxo is doing a good job for FreeBSD. It would be nice if Linux also adapts that, or something similar, one day. Eg I have a script that needs output from netstat, and `netstat -bin --libxo json | jq ...` is a great way to get that info instead of scraping the human-readable output.
- touisteur 7y ago
  Sorry, stupid question, but why not use /proc/net directly?
  - naniwaduni 7y ago
    /proc is a linuxism.
    - touisteur 7y ago
      Yes... But even on Linux I've been bitten so much by changes in 'standard' tools just on successive versions of centos, the last 10 years... ifconfig changing its output syntax (I know 'just use ip' well it changes sometimes too...), hostname sets a transient hostname now, hostnames with _ are now refused... netstat had some subtle changes some time ago too. free changed too. df, fricking df output format changed. So now, I just hit /proc and /sys, and document the hell out of it. Sure, new fields can appear there but they're new columns. The parser has to handle 'additional' columns. I trust Linus more with not-breaking-userland than the folks that make 'portable' tools. Glad I don't have to support other OSes...
- kbrazil 7y ago
  It would be great to have libxo in linux! FYI, `jc netstat` also generates JSON output for `netstat`. (disclosure: I'm the author of `jc`) While I'm at it... check out `jtbl` on my github which prints the JSON or JSON Lines output to a table in your terminal. https://github.com/kellyjonbrazil/jc https://github.com/kellyjonbrazil/jc https://github.com/kellyjonbrazil/jtbl https://github.com/kellyjonbrazil/jtbl
chatmasta 7y ago
Maybe a good interim solution would be a “proxy command” that knows the output format of common commands, and how to convert them to json. So for example you could pipe the output of ls to this tool, and it would be smart enough to parse it and convert to json.
- matt_kantor 7y ago
  https://github.com/kellyjonbrazil/jc https://github.com/kellyjonbrazil/jc was posted in another comment; it appears to be exactly that.
de_watcher 7y ago
It's slow and obtuse, the syntax of processing commands is too verbose, there are surprises with null values and return codes jq gives. It's not the answer for everything, it's just one of the tools. It feels cosy to huge amount of people because it's the product of web explosion, but at the core it's accidental and not that good.

pyuser583 7y ago

That’s nice.

MuffinFlavored 7y ago

Do you still need to pass it to `jq` for it to be pretty printed or no, I couldn't tell from reading the article?

kuesji 7y ago
yes, you still need 'jq' for pretty print.

quintonish 7y ago

Note that this is not titled, "Curl will now format JSON". This will "Make curl display information on stdout after a completed transfer"[1]. [1] https://curl.haxx.se/docs/manpage.html#-w https://curl.haxx.se/docs/manpage.html#-w

theMiddleBlue 7y ago

I'm testing it, maybe I'm doing it wrong but I don't understand well if it's possible to show only the json output (and not both json and response body). I'm doing it with a trick: curl -I -XGET -w '%{json}' https://www.google.com https://www.google.com | tail -1 | jq

otterlicious 7y ago
-o /dev/null

enriquto 7y ago

No need to panic! You can still pipe it to gron [0] and recover useable output. [0] https://github.com/TomNomNom/gron https://github.com/TomNomNom/gron

monkey26 7y ago

To bad this requires nightly.

rossmohax 7y ago

It is a step in the right direction, but still far from all the goodnes of PowerShell's Invoke-WebRequest . Not sure curl can do anything about it though.

omani 7y ago

beautiful. thank you for this.