8 ms·

Parsing JSON in Forty Lines of Awk

• teddyh 1y ago
  “except Unicode escape sequences”
  • Brian_K_White 1y ago
    The same author also made jawk which is more thorough. This is explicitly stated to be a cut down version.
• chaps 1y ago
  Awk is great and this is a great post. But dang, awk really shoots itself so much with its lack of features that it so desperately needs! Like: printing all but one column somewhere in the middle. It turns into long, long commands that really pull away from the spirit of fast fabrication unix experimentation. jq and sql both have the same problem :)
  • mauvehaus 1y ago
    ...And once you get away from the most basic, standard set of features, the several awks in existence have diverging sets of additional features.
    • chaps 1y ago
      Things are already like that, friend! We have mawk, gawk and nawk. But it's fun to think about how we could improve our ideal tooling if we had a time machine.
  • thrwwy9234 1y ago
    $ echo "one two three four five" | awk '{$3="";print}' one two four five
    • chaps 1y ago
      Oh dang, that's good.
      • shawn_w 1y ago
        As long as you don't mind the extra space in the middle.
        • fragmede 1y ago
          if you do: sed 's/ / /g'
          • lucb1e 1y ago
            Or sticking with awk, I have this bash alias to remove excess whitespace that is just: awk '{$1=$1};1'
            • fuzztester 1y ago
              what does the 1 at the end do? make awk print all lines? I'm a bit rusty with my awk.
              • thrwwy9234 1y ago
                It’s a condition. 1 is true-ish, so it’s a condition that is always true. The default action in awk is print, so it’s the same as: '{$1=$1}1{print}' Or the same as '{$1=$1}{print}' Since the default condition is true. But 1 is shorter than {print}.
        • chaps 1y ago
          Often times I don't! Entirely depends on what I'm doing. #1 thing off the top of my head is to remove That One Column that's a bajillion characters long that makes exploratory analysis difficult.
        • saghm 1y ago
  • SoftTalker 1y ago
    > awk really shoots itself so much with its lack of features that it so desperately needs Whence perl.
    • librasteve 1y ago
      or raku https://github.com/moritz/json/blob/master/lib/JSON/Tiny/Grammar.pm https://github.com/moritz/json/blob/master/lib/JSON/Tiny/Gra...
      • saghm 1y ago
        I suspect the rationale for Perl is that most Linux systems will probably have it installed already. Installing something you're familiar with is great when you can, but I'm guessing the awk script linked to here was picked more for its ubiquity than elegance.
        • chaps 1y ago
          Kinda, but not really. Of the infrastructures I've worked on, not a single one has been consistent in installing perl on 100% of hosts. The ones that get close are usually like that because one high up person really, really likes perl. And they send a lot of angry emails about perl not being installed. Within infrastructures where perl is installed on 95% of hosts, that 5% really bites you in the ass and leads to infrastructure rot very quickly. You're kinda stuck writing and maintaining two separate scripts to do the same thing.
          • SoftTalker 1y ago
            Same with Python. It's mostly available, but sometimes not. With Perl, I find that a base installation is almost always available, but many packages might not be.
            • chaps 1y ago
              I dunno about that. IME, python is much, much more universally installed on the hosts I've worked on. Sure, usually it's 2.7, but it's there! I've tended to work on rhel and debian hosts, with some fedora in the mix. (Once had a coworker reject a PR I wrote because I included a bash builtin in a deployment script. He said that python is more likely to be installed than bash, so we should not use bash. These debates are funny sometimes.)
  • jcynix 1y ago
    >awk really shoots itself so much with its lack of features that it so desperately needs! That's why I use Perl instead (besides some short one liners in awk, which in some cases are even shorter than the Perl version) and do my JSON parsing in Perl. This diff -rs a/ b/ | ask '/identical/ {print $4}' | xargs rm is one of my often used awk one liners. Unless some filenames contain e.g. whitespace, then it's Perl again
    • ptspts 1y ago
      This is much safer: xargs -d '\n' rm -f --
      • jcynix 1y ago
        Sure, but my example was just that and I actually use /identical$/ as the pattern. Sorry for the typo. And I use this "historic" one liner only when I know about the contents of both directories. As soon as I need a "safer" solution I use a Perl script and pattern matching, as I said.
        • poke646 1y ago
          In this case the Perl one-liner would be conceptually identical, the same length, but more performant (no calling out to rm): diff -rs a/ b/ | perl -ane '/identical$/ && unlink $F[3]'
      • jcynix 1y ago
        Just tried to use "-d" and learned that it's a GNUism which isn't available under MacOS, so it's not a portable solution. And neither was it available under BSD 4.3 when I learned about xargs the first time.
    • 8n4vidtmkvmk 1y ago
      I've been using perl instead of sed because PCRE is just better and it's the same regex that PHP uses which I've been coding in for nearly 20 years. I still don't actually know perl, but apparently Gemini does. It wrote a particularly crazy find and replace for me. Never got around to using or learning awk. Only time I see it come up is when you want to parse some tab delimited output
  • toddm 1y ago
    Us old UNIX guys would likely go for cut for this sort of task: cut -d " " -f1-2,4-5 file.txt where file.txt is: one two three four five and the return is: one two four five
• wutwutwat 1y ago
  Your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should.
• chubot 1y ago
  JSON is not a friendly format to the Unix shell — it’s hierarchical, and cannot be reasonably split on any character Yes, shell is definitely too weak to parse JSON! (One reason I started https://oils.pub https://oils.pub is because I saw that bash completion scripts try to parse bash in bash, which is an even worse idea than trying to parse JSON in bash) I'd argue that Awk is ALSO too weak to parse JSON The following code assumes that it will be fed valid JSON. It has some basic validation as a function of the parsing and will most likely throw an error if it encounters something strange, but there are no guarantees beyond that. Yeah I don't like that! If you don't reject invalid input, you're not really parsing --- OSH and YSH both have JSON built-in, and they have the hierarchical/recursive data structures you need for the common Python/JS-like API: osh-0.33$ var d = { date: $(date --iso-8601) } osh-0.33$ json write (d) | tee tmp.txt { "date": "2025-06-28" } Parse, then pretty print the data structure you got: $ cat tmp.txt | json read (&x) osh-0.33$ = x (Dict) {date: '2025-06-28'} Create a JSON syntax error on purpose: osh-0.33$ sed 's/"/bad/"' tmp.txt | json read (&x) sed: -e expression #1, char 9: unknown option to `s' sed 's/"/bad/"' tmp.txt | json read (&x) ^~~~ [ interactive ]:20: json read: Unexpected EOF while parsing JSON (line 1, offset 0-0: '') (now I see the error message could be better) Another example from wezm yesterday: https://mastodon.decentralised.social/@wezm/114758602660836154 https://mastodon.decentralised.social/@wezm/1147586026608361... YSH has JSON natively, but for anyone interested, it would be fun to test out the language by writing a JSON parser in YSH It's fundamentally more powerful than shell and awk because it has garbage-collected data structures - https://www.oilshell.org/blog/2024/09/gc.html https://www.oilshell.org/blog/2024/09/gc.html Also, OSH is now FASTER than bash, in both computation and I/O. This is despite garbage collection, and despite being written in typed Python! I hope to publish a post about these recent improvements
  • packetlost 1y ago
    I don't really buy that shell / awk is "too weak" to deal with JSON, the ecosystem of tools is just fairly immature as most of the shells common tools predate JSON by at least a decade. `jq` being a pretty reasonable addition to the standard set of tools included in environments by default. IMO the real problem is that JSON doesn't work very well at as a because it's core abstraction is objects. It's a pain to deal with in pretty much every statically typed non-object oriented language unless you parse it into native, predefined data structures (think annotated Go structs, Rust, etc.).
• ameliaquining 1y ago
  I feel like the takeaway here is that jq should probably be considered an indispensable part of the modern shell environment. Because without it, you can't sensibly deal with JSON in a shell script, and JSON is everywhere now. It's cool that you can write a parser in awk but the point of this kind of scripting language is to not have to do things like that.
• izabera 1y ago
  did something along those lines a while ago https://github.com/izabera/j https://github.com/izabera/j trying to keep an interface similar to jq
• latexr 1y ago
  > Another option is to use Python, which is ubiquitous enough that it can be expected to be installed on virtually every machine Not on macOS. You can do it easily by just invoking /usr/bin/python3, but you’ll get a (short and non-threatening) dialog to install the Xcode Command-Line Developer Tools. For macOS environments, JavaScript for Automation (i.e. JXA, i.e /usr/bin/osascript -l JavaScript) is a better choice. However, since Sequoia, jq is now installed by default on macOS.
• HappMacDonald 1y ago
  One day I wanted to use a TAP parser for the Test Anything Protocol. But I didn't want to be bogged down by dependencies.. so I didn't want to go nowhere near Python (and pyenv.. and anaconda.. and then probably having to dockerize that for some reason too..) nor nodeJS nor any of that. Found a bash shell script to parse TAP written by ESR of all people. That sounds fine, I thought. Most everywhere has bash, and there are no other dependencies. But it was slow. I mean.. painfully, ridiculously slow. Parsing like 400 lines of TAP took almost a minute. That's when I did some digging and learned about what awk really is. A scripting language that's baked into the POSIX suite. I had heard vaguely about it beforehand, but never realized it had more power than it's sed/ed/cut/etc brethren in the suite. So I coded up a TAP parser in awk and it went swimmingly, matched the ESR parser's feature set, and ran millions of lines in less than a second. Score! :D
  • theamk 1y ago
    For the record, "python-without-extra-dependencies" is a thing and a very nice one too. I always prefer it over awk. Highly recommend to everyone - plenty of "batteries" included, like json parser, basic http client and even XML parser, and no venv/conda required. Very good forward compatibility. Fast (compared to bash).
    • wintergrasp 1y ago
      Can you explain more or provide more information/links?
      • stephenlf 1y ago
        The Python Standard Library https://docs.python.org/3/library/index.html https://docs.python.org/3/library/index.html
    • HappMacDonald 1y ago
      That does sound compelling, but I frequently enough have to work on embedded systems (often running busybox) where python isn't installed and available R/W storage space is measured in tens or hundreds of Kb. I find that that is one more environment where awk scripting can get the job done, python/perl/php/etc just can't be introduced, bash can _sometimes_ get the job done if it doesn't have to spawn too many subprocesses, and C/other-compiled-options _might_ be able to help if I had some kind of build environment targeting the platform(s) in question and enough patience. I'll keep an eye out for python with no extra dependency options on the platforms that can handle that though.
• twoodfin 1y ago
  Contemplating this, it’s too bad the Unix scripting ecosystem never evolved a tripartite symbiosis of ‘file‘, ‘lex‘, and ‘yacc‘, or similar tools. That is, one tool to magically identify a file type, one to tokenize it based on that identification, one to correspondingly parse it. All in a streaming/pipe-friendly mode. Would fit right in, other than the Unix prejudice (nonsensical from Day 0) for LF-separated text records as the “one true format”.
  • kristopolous 1y ago
    I guess ... I don't think it really gets you much unless it's 1-pass streaming otherwise we're dealing with entire input buffers and then we're just back to files. You could argue using the vertical separator | is more syntactically graceful but then it's just a shell argument. There's quite a few radically different shells out there these days like xonsh, murex, and nushell so if simply arranging logic on the screen in a different syntax is what you're looking for then that's probably the way.
    • twoodfin 1y ago
      What I meant was something like the SAX streaming parse model disaggregated and broken into sniff / lex / parse phases all mediated by a modestly structured stream of the kind a tool like awk could process naturally.
      • kristopolous 1y ago
        Right okay I ran into a very similar limitation in Unix recently. It's really about multiplexed semantic routing. There's no great tools, abstractions, jargon or syscalls for it. I mean there's plenty for things like the networking stack, dbus, Wayland, and various audio stacks but they're all a pain to deal with. Nobody has really figured it out. The list inputs, list outputs, create bridges and taps, assign properties, which is what everybody does is complicated. I'm sure there's brilliant people that find this intuitive but for me it requires too much orchestration and feels brittle. People see this in microservices as well. There needs to be something as intuitive as the mouse drag (Raskin), Unix pipe (McIlroy) or drop down menu (Atkinson) to deal with this stuff. All these "obvious" things had to be invented. There's something dumb and obvious here nobody's cracked yet The primary problem is the concept of discovering such things has ossified and few people are experimenting on introducing new ones
• cAtte_ 1y ago
  just use nushell man
• deleted 1y ago
  [deleted]
• 1vuio0pswjnm7 1y ago
  I use flex. Faster than awk or python. It produces relatively seems to be available "everywhere" because it is a build requirement for so many software programs. For exampole, NetBSD toolchain includes it. It is a build requirement for Linux kernel.^1 I have even used it with Interix SFU on Windows before WSL existed. I do not use jq. Too complicated for me. Overkill. I created statically-linked program less than half the size of official statically-linked jq that is adequate for own needs. flex is a build requirement for jq. 1. https://www.kernel.org/doc/Documentation/admin-guide/quickly-build-trimmed-linux.rst https://www.kernel.org/doc/Documentation/admin-guide/quickly...
  • 1vuio0pswjnm7 1y ago
    Using C program generated with flex is faster than AWK or Python. Flex is required to compile the awk interpreter, as well as the jq interpreter. I use a custom scanner generated by flex to process JSON.