8 ms·

Awk: The Power and Promise of a 40-Year-Old Language

251 points by jangid 5y ago

asicsp 5y ago
HN discussion threads for some of the links mentioned in the article: * Using AWK and R to parse 25TB - https://news.ycombinator.com/item?id=20293579 https://news.ycombinator.com/item?id=20293579 * Command-line Tools can be 235x Faster than a Hadoop Cluster - https://news.ycombinator.com/item?id=17135841 https://news.ycombinator.com/item?id=17135841 * The State of the AWK - https://news.ycombinator.com/item?id=23240800 https://news.ycombinator.com/item?id=23240800 For awk alternative implementations, I'm keeping an eye on frawk [0]. Aims to be faster, supports csv, etc. [0] https://github.com/ezrosent/frawk https://github.com/ezrosent/frawk
- nmz 5y ago
  CSV is a complicated format but that does not mean awk is incapable of dealing with it. https://www.gnu.org/software/gawk/manual/html_node/Splitting-By-Content.html https://www.gnu.org/software/gawk/manual/html_node/Splitting... https://github.com/e36freak/awk-libs/blob/master/csv.awk https://github.com/e36freak/awk-libs/blob/master/csv.awk https://raw.githubusercontent.com/Nomarian/Awk-Batteries/master/Units/csv.awk https://raw.githubusercontent.com/Nomarian/Awk-Batteries/mas...
  - boogies 5y ago
    > CSV is a complicated format Surprisingly and unnecessarily so: > ["DSV"] is to Unix what CSV (comma-separated value) format is under Microsoft Windows and elsewhere outside the Unix world. CSV (fields separated by commas, double quotes used to escape commas, no continuation lines) is rarely found under Unix. > In fact, the Microsoft version of CSV is a textbook example of how not to design a textual file format. Its problems begin with the case in which the separator character (in this case, a comma) is found inside a field. The Unix way would be to simply escape the separator with a backslash, and have a double escape represent a literal backslash. This design gives us a single special case (the escape character) to check for when parsing the file, and only a single action when the escape is found (treat the following character as a literal). The latter conveniently not only handles the separator character, but gives us a way to handle the escape character and newlines for free. CSV, on the other hand, encloses the entire field in double quotes if it contains the separator. If the field contains double quotes, it must also be enclosed in double quotes, and the individual double quotes in the field must themselves be repeated twice to indicate that they don't end the field. > The bad results of proliferating special cases are twofold. First, the complexity of the parser (and its vulnerability to bugs) is increased. Second, because the format rules are complex and underspecified, different implementations diverge in their handling of edge cases. Sometimes continuation lines are supported, by starting the last field of the line with an unterminated double quote — but only in some products! Microsoft has incompatible versions of CSV files between its own applications, and in some cases between different versions of the same application (Excel being the obvious example here). — The Art of Unix Programming http://www.catb.org/~esr/writings/taoup/html/ch05s02.html http://www.catb.org/~esr/writings/taoup/html/ch05s02.html
    - AstroJetson 5y ago
      This is why I hate CSV files. Trying to reformat huge blocks of data is a job that Awk does well. The associative arrays let you build structures that let you do the heavy lifting. For record processing, Awk should be one of the first tools you look at.
    - Akronymus 5y ago
      > The latter conveniently not only handles the separator character, but gives us a way to handle the escape character and newlines for free. CSV, on the other hand, encloses the entire field in double quotes if it contains the separator. If the field contains double quotes, it must also be enclosed in double quotes, and the individual double quotes in the field must themselves be repeated twice to indicate that they don't end the field. I KNOW how CSV works, for the most part. And my brain still started tuning out/stopped building up the mental model.
    - nerdponx 5y ago
      The quoting also helps preserve embedded non-printable characters, newlines, etc. (yes, which can appear). One extension of the "Unix version" would be to impose a requirement like that in JSON, where all non-printable and/or non-ASCII characters must be written as an escape sequence like "\uXXXX" escape.
ketanmaheshwari 5y ago
My own shameless plug: https://ketancmaheshwari.github.io/posts/2020/05/24/SMC18-Data-Challenge-4.html https://ketancmaheshwari.github.io/posts/2020/05/24/SMC18-Da...
tyingq 5y ago
Gawk's ability to extend it with C code is interesting as well, and pretty straightforward. Here's the source for the fork() extension that ships with gawk...it's ~150 lines or so: https://git.savannah.gnu.org/cgit/gawk.git/tree/extension/fork.c https://git.savannah.gnu.org/cgit/gawk.git/tree/extension/fo... I was able to make a (terrible/joke/but-it-kinda-works) web server with gawk using the extensions that ship with it: https://gist.github.com/willurd/5720255#gistcomment-3143007 https://gist.github.com/willurd/5720255#gistcomment-3143007
- tgv 5y ago
  My opinion that belongs to me is as follows. This is how it goes. The next thing I'm going to say is my opinion. The C interop and name-spaces (also in gawk) is a bridge too far for me. By the time you need one of those, it's time to look for another language. Awk is just not enough of a language to write serious programs in. And I really like awk. It has enabled great scripting not only for log files, but also for dictionaries, back in the day when it was still hard to load one in memory. That is my opinion, it is mine, and belongs to me and I own it, and what it is too.
  - gompertz 5y ago
    It's good you're unapologetic. At the same time, these sort of features are what I love as they avoid me having to move onwards to something new, and start near ground zero. Living by the mantra "Do 2 things 1000 times, not 1000 things 2 times."
melling 5y ago
i no longer use it but Perl was always the better solution when one thought AWK was the answer. Perl will do those things where AWK really shines and if the problem got bigger, Perl was easier to deal with.
- tyingq 5y ago
  I found that to be the case many times as well. But awk also often outperforms Perl, especially mawk.
- Scarbutt 5y ago
  Yes but you can't learn perl as quickly as you can learn awk.
  - jfk13 5y ago
    Though you can learn just enough perl to do awk-like things fairly easily. And then grow from there as needed.
    - throwawayboise 5y ago
      IDK. On my OpenBSD system the awk man page is under 500 lines, and it pretty much covers the subject. I've tried to get started in Perl a few times, and just found it weird. It doesn't click. Awk is kind of weird too but it's so simple it doesn't matter. I'm sure I would eventually get Perl if I had to use it. But for me, awk and sed and shell scripting have covered my needs.
- coliveira 5y ago
  The problem is that awk is a very simple language, which you can learn in an afternoon. Perl is a very complex language, and is not used anymore, so you're just spending your time on something you'll rarely use.
  - forinti 5y ago
    If you work a lot with Linux, you can pretty much count on Perl and awk always being there. So it comes in quite handy to know them.
justin_oaks 5y ago
I only recently learned Awk enough to be useful. But I still don't reach for it when I probably should. What are the most common cases where you reach for Awk instead of some other tools? I recently used it to parse and recombine data from the OpenVPN status file. That file has a few differently formatted tables in the same file. Using Awk, I was able to change a variable as each table was encountered, this I could change the Awk program behavior by which table it was operating on.
- mellavora 5y ago
  try running this: awk '{cmd="rm " FILENAME; print cmd; system(cmd) }' file* best results if you do 'sudo' first ymmv
  - deleted 5y ago
    [deleted]
  - generalizations 5y ago
    At least add a /s to your comment. I like learning from the stuff people comment on here, and while there's an element of "that would be an important lesson" to what you posted, it's mostly just an unnecessary landmine.
- coliveira 5y ago
  Anything that is command line based and needs small changes to text input can be done with awk. It is a very competent language for scripts.
- chasil 5y ago
  Here is a script that I use to send SMTP mail, via the gawk networking extensions. I have a few different versions, but this is the most basic: #!/bin/gawk -f BEGIN { smtp="/inet/tcp/0/smtp.yourhost.com/25"; ORS="\r\n"; r=ARGV[1]; s=ARGV[2]; sbj=ARGV[3]; # /usr/local/bin/awkmail to from subj < in print "helo " ENVIRON["HOSTNAME"] |& smtp; smtp |& getline j; print j print "mail from: " s |& smtp; smtp |& getline j; print j if(match(r, ",")) { split(r, z, ",") for(y in z) { print "rcpt to: " z[y] |& smtp; smtp |& getline j; print j } } else { print "rcpt to: " r |& smtp; smtp |& getline j; print j } print "data" |& smtp; smtp |& getline j; print j print "From: " s |& smtp; ARGV[2] = "" # not a file print "To: " r |& smtp; ARGV[1] = "" # not a file if(length(sbj)) { print "Subject: " sbj |& smtp; ARGV[3] = "" } # not a file print "" |& smtp while(getline > 0) print |& smtp print "." |& smtp; smtp |& getline j; print j print "quit" |& smtp; smtp |& getline j; print j close(smtp) } # /inet/protocol/local-port/remote-host/remote-port This allows me to bypass the local MTA (if present). The message ID is also returned, which can be useful to log.
shp0ngle 5y ago
awk is fast and really useful. It's also generally unreadable.
- coliveira 5y ago
  I don't agree. Awk is very readable for people used to c-like languages like javascript. And it is much cleaner that Perl.
  - gpderetta 5y ago
    It is certainly more readable than sed for example.
    - throwawayboise 5y ago
      Yeah I use sed not infrequently but try to keep things simple. Anything more complicated than a "standard" sed one-liner (google it) I will start looking for something else.
vyuh 5y ago
"A good programmer uses the most powerful tool to do a job. A great programmer uses the least powerful tool that does the job." I believe this, and I always try to find the combination of simple and lightweight tools which does the job at hand correctly. Awk sometimes proves surprisingly powerful. Just look at the concision of this awk one liner doing a fairly complex job: zcat large.log.gz | awk '{print $0 | "gzip -v9c > large.log-"$1"_"$2".gz"}' # Breakup compressed log by syslog date and recompress. #awksome Taken from: https://mobile.twitter.com/climagic/status/614153897230397440 https://mobile.twitter.com/climagic/status/61415389723039744...
- dunefox 5y ago
  Ehh. Until the 'job' gets extended and then your simple tool makes it exponentially more complex and you have to rewrite it with the more powerful tool.
  - inanutshellus 5y ago
    Choosing a "good enough for the medium term with minimal effort now" is a winner in my book, even if it's likely to be rewritten in the long term.
    - selfhoster11 5y ago
      Exactly. I end up re-implementing my scripts if they outgrow the original scripting language anyway, because it's a good time to add proper argument and error handling, logging, etc.
  - Folcon 5y ago
    Surely that isn't a weakness of a simple tool? A 5 min job that probably won't get extended saving you from having to spend 20 mins coding something up is better than, feeling annoyed that you have spent the 20 mins coding up the original implementation and then extend it. Hopefully, you also get the benefit of additional knowledge on that future implementation as well. Why wouldn't this just be a net win? Unless you're talking about writing hack after hack after hack, eventually leaving yourself with some incomprehensible eldritch monstrosity, in which case, don't do that?
zeveb 5y ago
> Very few people still code with the legacies of the 1970s: ML, Pascal, Scheme, Smalltalk. Arguably, the software world would be better off if more people did code with those 1970s languages, than with the ones we are stuck with now. And that applies to Awk, too. As the author quotes Neil Ormos stating, Awk is well suited for personal computing, something which we have gotten further and further from at the same time as computers have become more distributed. At what point in history have such a large fraction of the human race had the ability to calculate to such an amazing order of magnitude, and at what point in history have such a large fraction of the same human race not bothered with calculation? Awk is a great tool precisely because it puts quite a lot of expressive power in the hands of an average user on a Unix system. Sure, on a Lisp machine or Smalltalk machine there really isn't the same need for Awk: the systems languages on such machines are safe enough and expressive enough to do what Awk does. But in the Unix context — which is basically what we're all living in, with even the VMS-derived Windows more-or-less adhering to the Unix model — Awk is a godsend. edit: correct typo
- gompertz 5y ago
  Oh man, you sound like a long lost friend. As someone who struggles to adopt really anything post ~1995 in the programming world, I couldn't agree more. I've worked for Fortune 100s my whole career; mostly in big data problem-spaces, before it ever was cool (if it even is now?), and I really feel all the problems people perceive today were solved all the way back to the 1960s (i.e. Snobol4). I understand for modern web and mobile contexts, sure there is new fancy tools for that; but as you said, in the personal computing space, the proper tools have existed for decades.
dekhn 5y ago
I've used Python almost my entire career, but started with out the UNIX tools. I never found awk interesting, then took a peek at it recently and understood: this was the pre-perl! it had scripting-language hash tables!
- Anon84 5y ago
  PERL was originally advertised as a replacement for “awk and sed”
  - dekhn 5y ago
    yep- and I went straight to perl after learning sed, and ignoring awk. awk looked even weirder than perl (I wasn't a big fan of the pattern matching style). In retrospect, I think awk is a massively underappreciated (for its time and context). I can't say I'd want to work with it regularly (same for perl; in the long run, I prefer variants of C style).
  - kamaal 5y ago
    First version of Perl was a replacement for C+awk+sed. These are days when things like GC, hashmaps, file operations etc were hard things on Unix.
phkahler 5y ago
I never use Awk until last year. I wanted to monitor an embedded device with little more than bustbox and python on it. There was quite a bit of information in the log files (I had already written a custom log file viewer with some highlighting) but I wanted to monitor in real-time. Somehow I decided to use Awk to monitor the tail of the log file and do realtime bar-graphs by generating appropriate cursor control sequences. In the end I had about 50 lines of Awk to upload to the board and run a command to pipe the log into it - very minimally invasive and very informative. Would recommend learning Awk with some kind of real-world use of your own. BTW it reminded me of using XSLT which I think is another often overlooked "good thing".
- cogman10 5y ago
  The biggest reason to learn AWK, IMO, is that it's on pretty much every single linux distribution. You might not have perl or python. You WILL have AWK. Only the most minimal of minimal linux systems will exclude it. Even busybox includes awk. That's how essential it's viewed.
  - michaelcampbell 5y ago
    I'm curious what linux distros don't have either some version of perl or python. I like awk, mind, but this is not necessarily (IME) a good argument for it.
    - kragen 5y ago
      Anything busybox-based. I'm not sure busybox awk is very complete, either.
    - cogman10 5y ago
      You'll find this a lot in the embedded space. As well, you'll see a bunch of docker images that don't have perl/python.
      - selfhoster11 5y ago
dugmartin 5y ago
My first and only real use of awk was around 1995. I was working at a new job doing embedded software work at GE and we had a lot of documentation in SGML, written/viewed using Interleaf. Interleaf was super slow on the HP-UX workstations we had and iirc search was even slower. I got the idea to convert all the SGML files into a single HTML file and I reached for awk as I had used it for some one-liners previously. I ended up writing an awk script that generated a frameset with one sidebar frame that was a treeish table of contents and the other frame the mondo html file with anchors for the table of contents. It loaded pretty fast in the HP-UX browser and search was really fast.
zeteo 5y ago
My company mandates Windows but Git Bash has been a backdoor into Unix tools and I've recently learned sed and awk to take full advantage of it. You need to think a bit about your one liners and they'll always feel very hacky, but sed/awk (with a bit of sort thrown in) are an amazingly powerful combination for dealing with all sorts of messy data dumps. In 10 minutes I can craft a one liner that replaces a 2 hours C# console app and runs just as fast. And, surprisingly, I often find it easier to go back months later and understand the messy looking one liner than the nicely formatted, well commented, unit tested console app.
torcete 5y ago
I use awk constantly in bioinformatics, for many of the file formats designed to store genomic data, awk is the easiest tool you can use for processing.
- jhbadger 5y ago
  There's even a version of awk specifically designed for bioinformatics that natively knows how to handle fasta, fastq, and sam files, among other formats. https://github.com/lh3/bioawk https://github.com/lh3/bioawk
- unemphysbro 5y ago
  I did the exact same thing! quickly looking at averages/errors, a simple awk one-liner will do.
mukundesh 5y ago
awk is great for data analysis - usually, I start with cut, then move to awk as complexity increases and finally to python.
jrochkind1 5y ago
My first job getting paid to program was in awk. Processing log files. In the middle of that job, my supervsior, you know what, we're doing increasingly complicated things with awk and it's getting increasingly hacky... I've heard that Perl is like awk but better, do you want to learn Perl and switch to that? And so we did. My thought then was there was little that was easier in awk than Perl, you could use Perl very much like awk if you wanted, you can even use the right command-line args to have Perl have an "implied loop" like awk... but then you can do a lot more with Perl too. I don't use Perl anymore. Or awk.
- linuxlizard 5y ago
  I think I remember reading somewhere Larry Wall was inspired to create Perl in order to combine awk+sed functionality. He was sick of awk+sed being almost powerful enough to do what he needed. (I can't find a reference to this though.)
linuxlizard 5y ago
I use awk to auto-generate C header files from other header files. I work with $vendor's huge complicated kernel driver codebase. I need small pieces of $vendor's interconnected header files in order to make kernel calls to their drivers without pulling in all their code.
nesuse 5y ago
There's a free awk course here for anyone interested https://www.udemy.com/course/awk-tutorial/ https://www.udemy.com/course/awk-tutorial/
arendtio 5y ago
Learning awk is actually pretty simple. For years I just used the '{print $2}' version to extract fields, but after reading some short book I felt pretty confident of having understood the basics. Sadly I don't remember which book it was, but this page looks like a good start: https://ferd.ca/awk-in-20-minutes.html https://ferd.ca/awk-in-20-minutes.html
- abecedarius 5y ago
  Likely the one by A, W, and K. https://news.ycombinator.com/item?id=13451454 https://news.ycombinator.com/item?id=13451454
  - arendtio 5y ago
    Yes, this looks like it. Thanks :-)
forinti 5y ago
sed is pretty ancient too. I've used it a lot with Docker to alter parameters during builds.
gompertz 5y ago
And let's not forget about the amazing commercial offering of Awk, known as Tawk (by Thompson Automation). To this day some features from Tawk cannot be found in Gawk.
- AstroJetson 5y ago
  Loved TWAK, but sadly they went out of business
SjorsVG 5y ago
I find it very unpleasant to read Awk code. It looks as bad as regex to me.
cb321 5y ago
When you have a standardized problem setting like the implicit loop in awk, n alternative to a whole new programming language is a simple < 100 lines of code program generator [1]. This design lets you retain easy access to large sets of pre-existing libraries as well as have a "compiled/statically typed" situation, if you want. It also leverages familiarity with your existing programming languages. I adapted a similar small program like this to emit a C program, but anything else is obviously pretty easy. Easy is good. Familiar is good. Interactivity-wise, with a TinyC/tcc fast running compiler backend my `rp` programs run sub-second from ENTER to completion on small data. Even with not optimizing tcc, they they still run faster than byte-compiled/VM interpreted mawk/gawk on a per input-byte basis. If you take the time to do an optimized build with gcc -O3/etc., they can run much faster. And I leave the source code around if you want to just use the program generator as a way to save keystrokes/get a fast start on a row processing program. Anyway, I'm not trying to start a language holy war, but just exhibit how if you rotate the problem (or your head looking at the problem) ever so slightly another answer exists in this space and is quite easy. :-) [1] https://github.com/c-blake/cligen/blob/master/examples/rp.nim https://github.com/c-blake/cligen/blob/master/examples/rp.ni...

251 points by jangid 5y ago

HN discussion threads for some of the links mentioned in the article: * Using AWK and R to parse 25TB - https://news.ycombinator.com/item?id=20293579 https://news.ycombinator.com/item?id=20293579 * Command-line Tools can be 235x Faster than a Hadoop Cluster - https://news.ycombinator.com/item?id=17135841 https://news.ycombinator.com/item?id=17135841 * The State of the AWK - https://news.ycombinator.com/item?id=23240800 https://news.ycombinator.com/item?id=23240800 For awk alternative implementations, I'm keeping an eye on frawk [0]. Aims to be faster, supports csv, etc. [0] https://github.com/ezrosent/frawk https://github.com/ezrosent/frawk

nmz 5y ago
CSV is a complicated format but that does not mean awk is incapable of dealing with it. https://www.gnu.org/software/gawk/manual/html_node/Splitting-By-Content.html https://www.gnu.org/software/gawk/manual/html_node/Splitting... https://github.com/e36freak/awk-libs/blob/master/csv.awk https://github.com/e36freak/awk-libs/blob/master/csv.awk https://raw.githubusercontent.com/Nomarian/Awk-Batteries/master/Units/csv.awk https://raw.githubusercontent.com/Nomarian/Awk-Batteries/mas...
- boogies 5y ago
  > CSV is a complicated format Surprisingly and unnecessarily so: > ["DSV"] is to Unix what CSV (comma-separated value) format is under Microsoft Windows and elsewhere outside the Unix world. CSV (fields separated by commas, double quotes used to escape commas, no continuation lines) is rarely found under Unix. > In fact, the Microsoft version of CSV is a textbook example of how not to design a textual file format. Its problems begin with the case in which the separator character (in this case, a comma) is found inside a field. The Unix way would be to simply escape the separator with a backslash, and have a double escape represent a literal backslash. This design gives us a single special case (the escape character) to check for when parsing the file, and only a single action when the escape is found (treat the following character as a literal). The latter conveniently not only handles the separator character, but gives us a way to handle the escape character and newlines for free. CSV, on the other hand, encloses the entire field in double quotes if it contains the separator. If the field contains double quotes, it must also be enclosed in double quotes, and the individual double quotes in the field must themselves be repeated twice to indicate that they don't end the field. > The bad results of proliferating special cases are twofold. First, the complexity of the parser (and its vulnerability to bugs) is increased. Second, because the format rules are complex and underspecified, different implementations diverge in their handling of edge cases. Sometimes continuation lines are supported, by starting the last field of the line with an unterminated double quote — but only in some products! Microsoft has incompatible versions of CSV files between its own applications, and in some cases between different versions of the same application (Excel being the obvious example here). — The Art of Unix Programming http://www.catb.org/~esr/writings/taoup/html/ch05s02.html http://www.catb.org/~esr/writings/taoup/html/ch05s02.html
  - AstroJetson 5y ago
    This is why I hate CSV files. Trying to reformat huge blocks of data is a job that Awk does well. The associative arrays let you build structures that let you do the heavy lifting. For record processing, Awk should be one of the first tools you look at.
  - Akronymus 5y ago
    > The latter conveniently not only handles the separator character, but gives us a way to handle the escape character and newlines for free. CSV, on the other hand, encloses the entire field in double quotes if it contains the separator. If the field contains double quotes, it must also be enclosed in double quotes, and the individual double quotes in the field must themselves be repeated twice to indicate that they don't end the field. I KNOW how CSV works, for the most part. And my brain still started tuning out/stopped building up the mental model.
  - nerdponx 5y ago
    The quoting also helps preserve embedded non-printable characters, newlines, etc. (yes, which can appear). One extension of the "Unix version" would be to impose a requirement like that in JSON, where all non-printable and/or non-ASCII characters must be written as an escape sequence like "\uXXXX" escape.

ketanmaheshwari 5y ago

My own shameless plug: https://ketancmaheshwari.github.io/posts/2020/05/24/SMC18-Data-Challenge-4.html https://ketancmaheshwari.github.io/posts/2020/05/24/SMC18-Da...

tyingq 5y ago

Gawk's ability to extend it with C code is interesting as well, and pretty straightforward. Here's the source for the fork() extension that ships with gawk...it's ~150 lines or so: https://git.savannah.gnu.org/cgit/gawk.git/tree/extension/fork.c https://git.savannah.gnu.org/cgit/gawk.git/tree/extension/fo... I was able to make a (terrible/joke/but-it-kinda-works) web server with gawk using the extensions that ship with it: https://gist.github.com/willurd/5720255#gistcomment-3143007 https://gist.github.com/willurd/5720255#gistcomment-3143007

tgv 5y ago
My opinion that belongs to me is as follows. This is how it goes. The next thing I'm going to say is my opinion. The C interop and name-spaces (also in gawk) is a bridge too far for me. By the time you need one of those, it's time to look for another language. Awk is just not enough of a language to write serious programs in. And I really like awk. It has enabled great scripting not only for log files, but also for dictionaries, back in the day when it was still hard to load one in memory. That is my opinion, it is mine, and belongs to me and I own it, and what it is too.
- gompertz 5y ago
  It's good you're unapologetic. At the same time, these sort of features are what I love as they avoid me having to move onwards to something new, and start near ground zero. Living by the mantra "Do 2 things 1000 times, not 1000 things 2 times."

melling 5y ago

i no longer use it but Perl was always the better solution when one thought AWK was the answer. Perl will do those things where AWK really shines and if the problem got bigger, Perl was easier to deal with.

tyingq 5y ago
I found that to be the case many times as well. But awk also often outperforms Perl, especially mawk.
Scarbutt 5y ago
Yes but you can't learn perl as quickly as you can learn awk.
- jfk13 5y ago
  Though you can learn just enough perl to do awk-like things fairly easily. And then grow from there as needed.
  - throwawayboise 5y ago
    IDK. On my OpenBSD system the awk man page is under 500 lines, and it pretty much covers the subject. I've tried to get started in Perl a few times, and just found it weird. It doesn't click. Awk is kind of weird too but it's so simple it doesn't matter. I'm sure I would eventually get Perl if I had to use it. But for me, awk and sed and shell scripting have covered my needs.
coliveira 5y ago
The problem is that awk is a very simple language, which you can learn in an afternoon. Perl is a very complex language, and is not used anymore, so you're just spending your time on something you'll rarely use.
- forinti 5y ago
  If you work a lot with Linux, you can pretty much count on Perl and awk always being there. So it comes in quite handy to know them.

justin_oaks 5y ago

I only recently learned Awk enough to be useful. But I still don't reach for it when I probably should. What are the most common cases where you reach for Awk instead of some other tools? I recently used it to parse and recombine data from the OpenVPN status file. That file has a few differently formatted tables in the same file. Using Awk, I was able to change a variable as each table was encountered, this I could change the Awk program behavior by which table it was operating on.

mellavora 5y ago
try running this: awk '{cmd="rm " FILENAME; print cmd; system(cmd) }' file* best results if you do 'sudo' first ymmv
- deleted 5y ago
  [deleted]
- generalizations 5y ago
  At least add a /s to your comment. I like learning from the stuff people comment on here, and while there's an element of "that would be an important lesson" to what you posted, it's mostly just an unnecessary landmine.
coliveira 5y ago
Anything that is command line based and needs small changes to text input can be done with awk. It is a very competent language for scripts.
chasil 5y ago
Here is a script that I use to send SMTP mail, via the gawk networking extensions. I have a few different versions, but this is the most basic: #!/bin/gawk -f BEGIN { smtp="/inet/tcp/0/smtp.yourhost.com/25"; ORS="\r\n"; r=ARGV[1]; s=ARGV[2]; sbj=ARGV[3]; # /usr/local/bin/awkmail to from subj < in print "helo " ENVIRON["HOSTNAME"] |& smtp; smtp |& getline j; print j print "mail from: " s |& smtp; smtp |& getline j; print j if(match(r, ",")) { split(r, z, ",") for(y in z) { print "rcpt to: " z[y] |& smtp; smtp |& getline j; print j } } else { print "rcpt to: " r |& smtp; smtp |& getline j; print j } print "data" |& smtp; smtp |& getline j; print j print "From: " s |& smtp; ARGV[2] = "" # not a file print "To: " r |& smtp; ARGV[1] = "" # not a file if(length(sbj)) { print "Subject: " sbj |& smtp; ARGV[3] = "" } # not a file print "" |& smtp while(getline > 0) print |& smtp print "." |& smtp; smtp |& getline j; print j print "quit" |& smtp; smtp |& getline j; print j close(smtp) } # /inet/protocol/local-port/remote-host/remote-port This allows me to bypass the local MTA (if present). The message ID is also returned, which can be useful to log.

shp0ngle 5y ago

awk is fast and really useful. It's also generally unreadable.

coliveira 5y ago
I don't agree. Awk is very readable for people used to c-like languages like javascript. And it is much cleaner that Perl.
- gpderetta 5y ago
  It is certainly more readable than sed for example.
  - throwawayboise 5y ago
    Yeah I use sed not infrequently but try to keep things simple. Anything more complicated than a "standard" sed one-liner (google it) I will start looking for something else.

vyuh 5y ago

"A good programmer uses the most powerful tool to do a job. A great programmer uses the least powerful tool that does the job." I believe this, and I always try to find the combination of simple and lightweight tools which does the job at hand correctly. Awk sometimes proves surprisingly powerful. Just look at the concision of this awk one liner doing a fairly complex job: zcat large.log.gz | awk '{print $0 | "gzip -v9c > large.log-"$1"_"$2".gz"}' # Breakup compressed log by syslog date and recompress. #awksome Taken from: https://mobile.twitter.com/climagic/status/614153897230397440 https://mobile.twitter.com/climagic/status/61415389723039744...

dunefox 5y ago
Ehh. Until the 'job' gets extended and then your simple tool makes it exponentially more complex and you have to rewrite it with the more powerful tool.
- inanutshellus 5y ago
  Choosing a "good enough for the medium term with minimal effort now" is a winner in my book, even if it's likely to be rewritten in the long term.
  - selfhoster11 5y ago
    Exactly. I end up re-implementing my scripts if they outgrow the original scripting language anyway, because it's a good time to add proper argument and error handling, logging, etc.
- Folcon 5y ago
  Surely that isn't a weakness of a simple tool? A 5 min job that probably won't get extended saving you from having to spend 20 mins coding something up is better than, feeling annoyed that you have spent the 20 mins coding up the original implementation and then extend it. Hopefully, you also get the benefit of additional knowledge on that future implementation as well. Why wouldn't this just be a net win? Unless you're talking about writing hack after hack after hack, eventually leaving yourself with some incomprehensible eldritch monstrosity, in which case, don't do that?

zeveb 5y ago

> Very few people still code with the legacies of the 1970s: ML, Pascal, Scheme, Smalltalk. Arguably, the software world would be better off if more people did code with those 1970s languages, than with the ones we are stuck with now. And that applies to Awk, too. As the author quotes Neil Ormos stating, Awk is well suited for personal computing, something which we have gotten further and further from at the same time as computers have become more distributed. At what point in history have such a large fraction of the human race had the ability to calculate to such an amazing order of magnitude, and at what point in history have such a large fraction of the same human race not bothered with calculation? Awk is a great tool precisely because it puts quite a lot of expressive power in the hands of an average user on a Unix system. Sure, on a Lisp machine or Smalltalk machine there really isn't the same need for Awk: the systems languages on such machines are safe enough and expressive enough to do what Awk does. But in the Unix context — which is basically what we're all living in, with even the VMS-derived Windows more-or-less adhering to the Unix model — Awk is a godsend. edit: correct typo

gompertz 5y ago
Oh man, you sound like a long lost friend. As someone who struggles to adopt really anything post ~1995 in the programming world, I couldn't agree more. I've worked for Fortune 100s my whole career; mostly in big data problem-spaces, before it ever was cool (if it even is now?), and I really feel all the problems people perceive today were solved all the way back to the 1960s (i.e. Snobol4). I understand for modern web and mobile contexts, sure there is new fancy tools for that; but as you said, in the personal computing space, the proper tools have existed for decades.

dekhn 5y ago

I've used Python almost my entire career, but started with out the UNIX tools. I never found awk interesting, then took a peek at it recently and understood: this was the pre-perl! it had scripting-language hash tables!

Anon84 5y ago
PERL was originally advertised as a replacement for “awk and sed”
- dekhn 5y ago
  yep- and I went straight to perl after learning sed, and ignoring awk. awk looked even weirder than perl (I wasn't a big fan of the pattern matching style). In retrospect, I think awk is a massively underappreciated (for its time and context). I can't say I'd want to work with it regularly (same for perl; in the long run, I prefer variants of C style).
- kamaal 5y ago
  First version of Perl was a replacement for C+awk+sed. These are days when things like GC, hashmaps, file operations etc were hard things on Unix.

phkahler 5y ago

I never use Awk until last year. I wanted to monitor an embedded device with little more than bustbox and python on it. There was quite a bit of information in the log files (I had already written a custom log file viewer with some highlighting) but I wanted to monitor in real-time. Somehow I decided to use Awk to monitor the tail of the log file and do realtime bar-graphs by generating appropriate cursor control sequences. In the end I had about 50 lines of Awk to upload to the board and run a command to pipe the log into it - very minimally invasive and very informative. Would recommend learning Awk with some kind of real-world use of your own. BTW it reminded me of using XSLT which I think is another often overlooked "good thing".

cogman10 5y ago
The biggest reason to learn AWK, IMO, is that it's on pretty much every single linux distribution. You might not have perl or python. You WILL have AWK. Only the most minimal of minimal linux systems will exclude it. Even busybox includes awk. That's how essential it's viewed.
- michaelcampbell 5y ago
  I'm curious what linux distros don't have either some version of perl or python. I like awk, mind, but this is not necessarily (IME) a good argument for it.
  - kragen 5y ago
    Anything busybox-based. I'm not sure busybox awk is very complete, either.
  - cogman10 5y ago
    You'll find this a lot in the embedded space. As well, you'll see a bunch of docker images that don't have perl/python.
    - selfhoster11 5y ago

dugmartin 5y ago

My first and only real use of awk was around 1995. I was working at a new job doing embedded software work at GE and we had a lot of documentation in SGML, written/viewed using Interleaf. Interleaf was super slow on the HP-UX workstations we had and iirc search was even slower. I got the idea to convert all the SGML files into a single HTML file and I reached for awk as I had used it for some one-liners previously. I ended up writing an awk script that generated a frameset with one sidebar frame that was a treeish table of contents and the other frame the mondo html file with anchors for the table of contents. It loaded pretty fast in the HP-UX browser and search was really fast.

zeteo 5y ago

My company mandates Windows but Git Bash has been a backdoor into Unix tools and I've recently learned sed and awk to take full advantage of it. You need to think a bit about your one liners and they'll always feel very hacky, but sed/awk (with a bit of sort thrown in) are an amazingly powerful combination for dealing with all sorts of messy data dumps. In 10 minutes I can craft a one liner that replaces a 2 hours C# console app and runs just as fast. And, surprisingly, I often find it easier to go back months later and understand the messy looking one liner than the nicely formatted, well commented, unit tested console app.

torcete 5y ago

I use awk constantly in bioinformatics, for many of the file formats designed to store genomic data, awk is the easiest tool you can use for processing.

jhbadger 5y ago
There's even a version of awk specifically designed for bioinformatics that natively knows how to handle fasta, fastq, and sam files, among other formats. https://github.com/lh3/bioawk https://github.com/lh3/bioawk
unemphysbro 5y ago
I did the exact same thing! quickly looking at averages/errors, a simple awk one-liner will do.

mukundesh 5y ago

awk is great for data analysis - usually, I start with cut, then move to awk as complexity increases and finally to python.

jrochkind1 5y ago

My first job getting paid to program was in awk. Processing log files. In the middle of that job, my supervsior, you know what, we're doing increasingly complicated things with awk and it's getting increasingly hacky... I've heard that Perl is like awk but better, do you want to learn Perl and switch to that? And so we did. My thought then was there was little that was easier in awk than Perl, you could use Perl very much like awk if you wanted, you can even use the right command-line args to have Perl have an "implied loop" like awk... but then you can do a lot more with Perl too. I don't use Perl anymore. Or awk.

linuxlizard 5y ago
I think I remember reading somewhere Larry Wall was inspired to create Perl in order to combine awk+sed functionality. He was sick of awk+sed being almost powerful enough to do what he needed. (I can't find a reference to this though.)

linuxlizard 5y ago

I use awk to auto-generate C header files from other header files. I work with $vendor's huge complicated kernel driver codebase. I need small pieces of $vendor's interconnected header files in order to make kernel calls to their drivers without pulling in all their code.

nesuse 5y ago

There's a free awk course here for anyone interested https://www.udemy.com/course/awk-tutorial/ https://www.udemy.com/course/awk-tutorial/

arendtio 5y ago

Learning awk is actually pretty simple. For years I just used the '{print $2}' version to extract fields, but after reading some short book I felt pretty confident of having understood the basics. Sadly I don't remember which book it was, but this page looks like a good start: https://ferd.ca/awk-in-20-minutes.html https://ferd.ca/awk-in-20-minutes.html

abecedarius 5y ago
Likely the one by A, W, and K. https://news.ycombinator.com/item?id=13451454 https://news.ycombinator.com/item?id=13451454
- arendtio 5y ago
  Yes, this looks like it. Thanks :-)

forinti 5y ago

sed is pretty ancient too. I've used it a lot with Docker to alter parameters during builds.

gompertz 5y ago

And let's not forget about the amazing commercial offering of Awk, known as Tawk (by Thompson Automation). To this day some features from Tawk cannot be found in Gawk.

AstroJetson 5y ago
Loved TWAK, but sadly they went out of business

SjorsVG 5y ago

I find it very unpleasant to read Awk code. It looks as bad as regex to me.

cb321 5y ago

When you have a standardized problem setting like the implicit loop in awk, n alternative to a whole new programming language is a simple < 100 lines of code program generator [1]. This design lets you retain easy access to large sets of pre-existing libraries as well as have a "compiled/statically typed" situation, if you want. It also leverages familiarity with your existing programming languages. I adapted a similar small program like this to emit a C program, but anything else is obviously pretty easy. Easy is good. Familiar is good. Interactivity-wise, with a TinyC/tcc fast running compiler backend my `rp` programs run sub-second from ENTER to completion on small data. Even with not optimizing tcc, they they still run faster than byte-compiled/VM interpreted mawk/gawk on a per input-byte basis. If you take the time to do an optimized build with gcc -O3/etc., they can run much faster. And I leave the source code around if you want to just use the program generator as a way to save keystrokes/get a fast start on a row processing program. Anyway, I'm not trying to start a language holy war, but just exhibit how if you rotate the problem (or your head looking at the problem) ever so slightly another answer exists in this space and is quite easy. :-) [1] https://github.com/c-blake/cligen/blob/master/examples/rp.nim https://github.com/c-blake/cligen/blob/master/examples/rp.ni...