7 ms·

GNU grep 2.12 changes behavior of recursion options, breaks existing scripts

119 points by cschramm 14y ago

buster 14y ago
Who thought it might be a good idea to break -r (most likely the most often used option) and use the old behaviour for -R? Wouldn't it be better to atleast let -r behave like ever and change -R? Anyone ever used -R here? :)
- leif 14y ago
  Yeah, for habitual reasons I usually use -R. No idea why.
  - lubutu 14y ago
    In POSIX -R is the recursion flag for ls, cp, rm, etc. So if you want to recurse, -R is probably a safer bet than -r. (ls -r just lists in reverse.)
    - leif 14y ago
      that's probably it
    - buster 14y ago
      Yeah, that may be it.. I'm wondering why they didn't just add a new option (aka --no-symlinks) or something
    - wildmXranat 14y ago
      Same here. I've been conditioned to use -R
    - deleted 14y ago
      [deleted]
    - a3_nm 14y ago
      Annoying exception: scp only accepts -r as recursion flag, not -R.
    - mturmon 14y ago
      Good point. Also, since you reminded me, chmod, chown, chgrp.
  - adavies42 14y ago
    ever work on a non-gnu box? (old solaris, *bsd, stock os x, etc.) there's a decent chance they don't have -r at all.
    - cschramm 14y ago
      *BSDs have (=> OS X probably too). Solaris, HP-UX and OpenServer have not. AIX is weird: > -r Searches directories recursively. By default, links to directories are followed. > -R Searches directories recursively. By default, links to directories are not followed.
      - rbanffy 14y ago
        Isn't the AIX behavior the one POSIX specifies?
        lmm 14y ago
        That would be kind of perverse given how cp behaves for -r and -R
        cschramm 14y ago
        Looking at MattJ100's link (http://pubs.opengroup.org/onlinepubs/009695399/utilities/grep.html http://pubs.opengroup.org/onlinepubs/009695399/utilities/gre...), it seems like POSIX grep actually does not have any recursion. Hence, only lubutu's comment ("-R is the recursion flag for ls, cp, rm, etc") is true, while _delirium ("-R is the POSIX-standard flag for recursive grep") is wrong, which means AIX is free to do whatever it wants with both the -r and the -R flags.
- freehunter 14y ago
  When I read the first post in the chain, I thought "well, it has to be a bug". Then I read the second post... wow, it was intentional. I've used -r and -R both in the same script to do the same thing, just depending on how I was feeling that day. Now I'm afraid to update. I don't think distros should be afraid to break compatibility with main if main is making a change that makes no sense. Breaking essential and classic *nix functions defeats the purpose of CLI utilities.
  - pif 14y ago
    > I don't think distros should be afraid to break compatibility with main if main is making a change that makes no sense. Hi, I don't agree. Different versions of a tool showing different behaviour for the same option is enough for me; the same version of the tool showing different behaviour for the same option _depending on the distribution_ ... I feel that's too much!
    - freehunter 14y ago
      Well, like the link states this already breaks compatibility with BSD grep. Combine that with breaking compatibility with previous version, breaking previous scripts, and breaking based on distro depending on the speed at which the package maintainers upgrade (if ever)... it's best IMO to just leave it the way it always was. It's going to be hard enough switching between a machine that only gets critical updates and a machine with the same distro but getting all updates. Grep has been around since 1973, is there any serious Unix scripter who feels it still needs more features?
- 0x0 14y ago
  Yeah, this sounds really misguided. If they change/add functionality, why not move that to a new option? Those who want the new way can use the new feature, and scripts won't act differently depending on which version is installed. New features should get new options; they should not move old features to new flags and put the new features on the old flags. Nuts.
  - rmc 14y ago
    Increasing the number of options and switches has downsides, it makes the software more complex to use.
    - 0x0 14y ago
      Hehe, you know what makes the software more complex to use? Having to grep in the output of "grep --version" before deciding which switch to pass to grep.. :)
      - michaelhoffman 14y ago
        You don't need to do that, just always use -R :)
        nitrogen 14y ago
        We'll get right on inventing that time machine so everyone can go back in time to the introduction of the -r flag and warn themselves ;).
        rbanffy 14y ago
        And note that it will make your code more compatible with other, POSIX-compliant, versions of grep.
- cschramm 14y ago
  I've always used -r before, since it saves a key press. ;) But I started using -R today, in the middle of a WTF event while inspecting some Python files, digging up a Debian bug report, inspecting the upstream change and posting this to HN. I still think it's not the best idea for Debian (or any other system) to break compatibility with upstream, since this will lead to different behaviors on different systems, not only depending on version and vendor (GNU or *BSD). But of course, it could also lead to the change being reverted, which would be welcome (guess nobody relies on the new behavior yet). The POSIX argument is definitely valid, but nevertheless GNU's grep has always supported -r and BSD versions do as well (although OpenBSD's man page reads "This implementation supports those options; however, their use is strongly discouraged."), so it just unnecessarily breaks existing stuff.
  - Evbn 14y ago
    I found some in house utilities using -R on their own interfaces. I thought it was weird. Now I know they were right.
- _delirium 14y ago
  -R is the POSIX-standard flag for recursive grep, so that would be worse to change imo. It's also the flag used for recursive grep by the BSDs (some of which do support '-r', but only as a deprecated historical option... OpenBSD calls it "strongly discouraged").
  - eschulte 14y ago
    Thanks for these important missing bits of information, in light of these I think the change sounds much more reasonable. /me updates rgrep alias to use -R
  - ineedtosleep 14y ago
    Currently -r and -R are essentially the same flags, correct? And this new change would make -R and -r two different operations?
- jasomill 14y ago
  I've always use -R, presumably because it's consistent with other commands like ls, rm, and cp (and consequently get annoyed by tools like scp that only support -r, and zip, that use -R for something else entirely); I wasn't even aware that -r was supported until I saw this.
- emmelaich 14y ago
  I've only used -R, because of it's posixness. Finger memory from adminning Solaris. Not to say that this is a good or bad change. -R matches the -R in chown/chgrp They should really introduce -r for those two to NOT follow symlinks. Following symlinks for those is bad.
js2 14y ago
Here's the change and it's justification: http://git.savannah.gnu.org/cgit/grep.git/commit/?id=c6e3ea61d9f08aa0128a0eb13d31a2fbad376f99 http://git.savannah.gnu.org/cgit/grep.git/commit/?id=c6e3ea6... Change -r to follow only command-line symlinks, and by default to read only devices named on the command line. This is a simple way to get a more-useful behavior when searching random directories; the idea is to use 'find' if you want something fancy. -R acts as before and gets a new alias --dereference-recursive. Personally I think breaking compatibility for this change was a poor decision.
- akkartik 14y ago
  That link doesn't work. I thought at first that it was incorrectly copied, but you can see it here: http://git.savannah.gnu.org/cgit/grep.git/log/?qt=grep&q=Change+-r http://git.savannah.gnu.org/cgit/grep.git/log/?qt=grep&q... And clicking on it says 'No repositories found'. Anybody else have this problem?
  - cschramm 14y ago
    Yes, seems like Savannah can't completely handle HN ;)
    - akkartik 14y ago
      The error message doesn't suggest this is a load issue. Also, wouldn't the entire app go down rather than a single link?
      - cschramm 14y ago
        If it's not a load issue and neither some intended limitation to prevent load issues, somebody pulled it on purpose. ...which I actually included in not being able to handle HN. ;)
  - js2 14y ago
    Alternate link: http://repo.or.cz/w/grep.git/commit/c6e3ea61d9f08aa0128a0eb13d31a2fbad376f99 http://repo.or.cz/w/grep.git/commit/c6e3ea61d9f08aa0128a0eb1... (Yea for the "D" in DVCS.) Also, the corresponding bug which has additional discussion: http://savannah.gnu.org/bugs/?17623 http://savannah.gnu.org/bugs/?17623
JoachimSchipper 14y ago
Remember: GNU is not UNIX.
m0skit0 14y ago
Backwards compatibility is an evil illness that sometimes must be broken. It's for the good of evolution. I praise engineers that make such decisions, even if they are unpopular.
- cturner 14y ago
  You'll feel differently when you inherit responsibility for some god-awful mess of shell and perl that nobody properly understands that used to Just Work but doesn't after an innocent upgrade.
  - deleted 14y ago
    [deleted]
- TillE 14y ago
  When there's a clear benefit, that's great. Feel free to scrap backwards compatibility when there's significant progress to be made in doing so. This feels a lot more like a "color of the bike shed" choice. There are use cases where the new behavior makes sense, sure, but there are also plenty of cases where the old way is better. This isn't an upgrade, it's a lateral move.
  - tinco 14y ago
    So we have to continue with an ugly bikeshed for the rest of eternity? I think it's very important that whenever it is decided that objectively one way of doing it is better than another way that eventually that way finds itself to be the way it is. You basically imply that this change is not big enough to warrant a break with backwards compatibility, but small discontinuities and hacks add up. If thinks like this aren't fixed every once in a while the system will be ridden with inconsistencies. Besides, even though it hurts when you've inherited some crazy unreadable code that utilizes some obsoleted functionality I think it is always positive for the code quality when the chaos monkey comes around and breaks something. edit: the downvote button is to indicate I am detrimental to the discussion, the reply button is for when you disagree with me :)
larrik 14y ago
The biggest problem I see is: who the heck would expect a .12 release to break compatibility in an ancient and solid piece of software?
- sneak 14y ago
  Why is grep being modified at all? It is clearly not broken.
  - Aglet 14y ago
    To quote a recent Coding Horror post, its bugs are now Common Law Features, and must be supported.
cschramm 14y ago
Another thought: Since rgrep is an alias for grep -r and grep -r has changed, the complete rgrep tool has changed and this can _not_ be fixed by using the "right" switch. Hence rgrep is useless for scripting now.
pooriaazimi 14y ago
This is a serious question: Why would you want to use plain old "grep" instead of "ack"[1]? Of course, other than the fact that it's on all machines. Why would you use it instrad of "ack" on your own machines? That's an honest question, I'm not starting a flamewar... The highlighting and filename/line# by default is the killer feature for me. Edit: Come on. Downvotes for this? Honestly... It's HN, not StackOverflow. You don't mark questions as "off-topic" unless they're trolling... [1]: http://betterthangrep.com http://betterthangrep.com
- audiodude 14y ago
  I love ack, I've never looked back.
- MattJ100 14y ago
  The main reason to use grep is portability/availability. While a modern improvement to grep (yes, I'm an ack user and fan too) you can't simply depend on it in portable scripts and such. grep is defined for POSIX after all: http://pubs.opengroup.org/onlinepubs/009695399/utilities/grep.html http://pubs.opengroup.org/onlinepubs/009695399/utilities/gre... I also suspect (without looking) that there are obscure (but occasionally useful) grep features that ack doesn't support.
- greyboy 14y ago
  That's a fine recommendation for machines that you solely work on, or have complete control over. However, it's good to be able to work with the standard toolset if you frequently work on a variety of remote machines (where it's quite common that you cannot install such things, due to permissions or policies, and want to get started working before attempting to download/install a bunch of custom binaries). Edit: MattJ100 made a more clear point while I was responding.
  - pooriaazimi 14y ago
    Thanks for the response. However, a nice thing about ack is that it's not a binary (necessarily) - it's a perl script and can be used without sysadmin permissions: http://betterthangrep.com/install/ http://betterthangrep.com/install/ The other points are quite understandable and correct though. It's always best to at least be familiar with standard tools, even if you want to use an slightly different version for yourself.
snorkel 14y ago
I also decided that sed no longer supports regexes because I said so. And ls -l will now shows a list of print jobs instead of files because maybe that's what you meant. ... oh and ping no longer supports IPv4 because I want everyone to adopt IPv6 immediately because I said so.
rwos 14y ago
It's funny how one of the relatively few design errors in Unix shells now indirectly comes back to haunt us. Recursing is built into pretty much every command that can handle multiple files - which very strongly suggests it should have been made a feature of glob (or the shell). I think it's a bit sad that those "big-picture" features in unix are treated as if the were written in stone.
- nvarsj 14y ago
  This is one of the great reasons to use zshell. :-) grep -in **/*txt "sometext"
deleted 14y ago
[deleted]

119 points by cschramm 14y ago

Who thought it might be a good idea to break -r (most likely the most often used option) and use the old behaviour for -R? Wouldn't it be better to atleast let -r behave like ever and change -R? Anyone ever used -R here? :)

leif 14y ago
Yeah, for habitual reasons I usually use -R. No idea why.
- lubutu 14y ago
  In POSIX -R is the recursion flag for ls, cp, rm, etc. So if you want to recurse, -R is probably a safer bet than -r. (ls -r just lists in reverse.)
  - leif 14y ago
    that's probably it
  - buster 14y ago
    Yeah, that may be it.. I'm wondering why they didn't just add a new option (aka --no-symlinks) or something
  - wildmXranat 14y ago
    Same here. I've been conditioned to use -R
  - deleted 14y ago
    [deleted]
  - a3_nm 14y ago
    Annoying exception: scp only accepts -r as recursion flag, not -R.
  - mturmon 14y ago
    Good point. Also, since you reminded me, chmod, chown, chgrp.
- adavies42 14y ago
  ever work on a non-gnu box? (old solaris, *bsd, stock os x, etc.) there's a decent chance they don't have -r at all.
  - cschramm 14y ago
    *BSDs have (=> OS X probably too). Solaris, HP-UX and OpenServer have not. AIX is weird: > -r Searches directories recursively. By default, links to directories are followed. > -R Searches directories recursively. By default, links to directories are not followed.
    - rbanffy 14y ago
      Isn't the AIX behavior the one POSIX specifies?
      - lmm 14y ago
        That would be kind of perverse given how cp behaves for -r and -R
      - cschramm 14y ago
        Looking at MattJ100's link (http://pubs.opengroup.org/onlinepubs/009695399/utilities/grep.html http://pubs.opengroup.org/onlinepubs/009695399/utilities/gre...), it seems like POSIX grep actually does not have any recursion. Hence, only lubutu's comment ("-R is the recursion flag for ls, cp, rm, etc") is true, while _delirium ("-R is the POSIX-standard flag for recursive grep") is wrong, which means AIX is free to do whatever it wants with both the -r and the -R flags.
freehunter 14y ago
When I read the first post in the chain, I thought "well, it has to be a bug". Then I read the second post... wow, it was intentional. I've used -r and -R both in the same script to do the same thing, just depending on how I was feeling that day. Now I'm afraid to update. I don't think distros should be afraid to break compatibility with main if main is making a change that makes no sense. Breaking essential and classic *nix functions defeats the purpose of CLI utilities.
- pif 14y ago
  > I don't think distros should be afraid to break compatibility with main if main is making a change that makes no sense. Hi, I don't agree. Different versions of a tool showing different behaviour for the same option is enough for me; the same version of the tool showing different behaviour for the same option _depending on the distribution_ ... I feel that's too much!
  - freehunter 14y ago
    Well, like the link states this already breaks compatibility with BSD grep. Combine that with breaking compatibility with previous version, breaking previous scripts, and breaking based on distro depending on the speed at which the package maintainers upgrade (if ever)... it's best IMO to just leave it the way it always was. It's going to be hard enough switching between a machine that only gets critical updates and a machine with the same distro but getting all updates. Grep has been around since 1973, is there any serious Unix scripter who feels it still needs more features?
0x0 14y ago
Yeah, this sounds really misguided. If they change/add functionality, why not move that to a new option? Those who want the new way can use the new feature, and scripts won't act differently depending on which version is installed. New features should get new options; they should not move old features to new flags and put the new features on the old flags. Nuts.
- rmc 14y ago
  Increasing the number of options and switches has downsides, it makes the software more complex to use.
  - 0x0 14y ago
    Hehe, you know what makes the software more complex to use? Having to grep in the output of "grep --version" before deciding which switch to pass to grep.. :)
    - michaelhoffman 14y ago
      You don't need to do that, just always use -R :)
      - nitrogen 14y ago
        We'll get right on inventing that time machine so everyone can go back in time to the introduction of the -r flag and warn themselves ;).
      - rbanffy 14y ago
        And note that it will make your code more compatible with other, POSIX-compliant, versions of grep.
cschramm 14y ago
I've always used -r before, since it saves a key press. ;) But I started using -R today, in the middle of a WTF event while inspecting some Python files, digging up a Debian bug report, inspecting the upstream change and posting this to HN. I still think it's not the best idea for Debian (or any other system) to break compatibility with upstream, since this will lead to different behaviors on different systems, not only depending on version and vendor (GNU or *BSD). But of course, it could also lead to the change being reverted, which would be welcome (guess nobody relies on the new behavior yet). The POSIX argument is definitely valid, but nevertheless GNU's grep has always supported -r and BSD versions do as well (although OpenBSD's man page reads "This implementation supports those options; however, their use is strongly discouraged."), so it just unnecessarily breaks existing stuff.
- Evbn 14y ago
  I found some in house utilities using -R on their own interfaces. I thought it was weird. Now I know they were right.
_delirium 14y ago
-R is the POSIX-standard flag for recursive grep, so that would be worse to change imo. It's also the flag used for recursive grep by the BSDs (some of which do support '-r', but only as a deprecated historical option... OpenBSD calls it "strongly discouraged").
- eschulte 14y ago
  Thanks for these important missing bits of information, in light of these I think the change sounds much more reasonable. /me updates rgrep alias to use -R
- ineedtosleep 14y ago
  Currently -r and -R are essentially the same flags, correct? And this new change would make -R and -r two different operations?
jasomill 14y ago
I've always use -R, presumably because it's consistent with other commands like ls, rm, and cp (and consequently get annoyed by tools like scp that only support -r, and zip, that use -R for something else entirely); I wasn't even aware that -r was supported until I saw this.
emmelaich 14y ago
I've only used -R, because of it's posixness. Finger memory from adminning Solaris. Not to say that this is a good or bad change. -R matches the -R in chown/chgrp They should really introduce -r for those two to NOT follow symlinks. Following symlinks for those is bad.

js2 14y ago

Here's the change and it's justification: http://git.savannah.gnu.org/cgit/grep.git/commit/?id=c6e3ea61d9f08aa0128a0eb13d31a2fbad376f99 http://git.savannah.gnu.org/cgit/grep.git/commit/?id=c6e3ea6... Change -r to follow only command-line symlinks, and by default to read only devices named on the command line. This is a simple way to get a more-useful behavior when searching random directories; the idea is to use 'find' if you want something fancy. -R acts as before and gets a new alias --dereference-recursive. Personally I think breaking compatibility for this change was a poor decision.

akkartik 14y ago
That link doesn't work. I thought at first that it was incorrectly copied, but you can see it here: http://git.savannah.gnu.org/cgit/grep.git/log/?qt=grep&q=Change+-r http://git.savannah.gnu.org/cgit/grep.git/log/?qt=grep&q... And clicking on it says 'No repositories found'. Anybody else have this problem?
- cschramm 14y ago
  Yes, seems like Savannah can't completely handle HN ;)
  - akkartik 14y ago
    The error message doesn't suggest this is a load issue. Also, wouldn't the entire app go down rather than a single link?
    - cschramm 14y ago
      If it's not a load issue and neither some intended limitation to prevent load issues, somebody pulled it on purpose. ...which I actually included in not being able to handle HN. ;)
- js2 14y ago
  Alternate link: http://repo.or.cz/w/grep.git/commit/c6e3ea61d9f08aa0128a0eb13d31a2fbad376f99 http://repo.or.cz/w/grep.git/commit/c6e3ea61d9f08aa0128a0eb1... (Yea for the "D" in DVCS.) Also, the corresponding bug which has additional discussion: http://savannah.gnu.org/bugs/?17623 http://savannah.gnu.org/bugs/?17623

JoachimSchipper 14y ago

Remember: GNU is not UNIX.

m0skit0 14y ago

Backwards compatibility is an evil illness that sometimes must be broken. It's for the good of evolution. I praise engineers that make such decisions, even if they are unpopular.

cturner 14y ago
You'll feel differently when you inherit responsibility for some god-awful mess of shell and perl that nobody properly understands that used to Just Work but doesn't after an innocent upgrade.
- deleted 14y ago
  [deleted]
TillE 14y ago
When there's a clear benefit, that's great. Feel free to scrap backwards compatibility when there's significant progress to be made in doing so. This feels a lot more like a "color of the bike shed" choice. There are use cases where the new behavior makes sense, sure, but there are also plenty of cases where the old way is better. This isn't an upgrade, it's a lateral move.
- tinco 14y ago
  So we have to continue with an ugly bikeshed for the rest of eternity? I think it's very important that whenever it is decided that objectively one way of doing it is better than another way that eventually that way finds itself to be the way it is. You basically imply that this change is not big enough to warrant a break with backwards compatibility, but small discontinuities and hacks add up. If thinks like this aren't fixed every once in a while the system will be ridden with inconsistencies. Besides, even though it hurts when you've inherited some crazy unreadable code that utilizes some obsoleted functionality I think it is always positive for the code quality when the chaos monkey comes around and breaks something. edit: the downvote button is to indicate I am detrimental to the discussion, the reply button is for when you disagree with me :)

larrik 14y ago

The biggest problem I see is: who the heck would expect a .12 release to break compatibility in an ancient and solid piece of software?

sneak 14y ago
Why is grep being modified at all? It is clearly not broken.
- Aglet 14y ago
  To quote a recent Coding Horror post, its bugs are now Common Law Features, and must be supported.

cschramm 14y ago

Another thought: Since rgrep is an alias for grep -r and grep -r has changed, the complete rgrep tool has changed and this can _not_ be fixed by using the "right" switch. Hence rgrep is useless for scripting now.

pooriaazimi 14y ago

This is a serious question: Why would you want to use plain old "grep" instead of "ack"[1]? Of course, other than the fact that it's on all machines. Why would you use it instrad of "ack" on your own machines? That's an honest question, I'm not starting a flamewar... The highlighting and filename/line# by default is the killer feature for me. Edit: Come on. Downvotes for this? Honestly... It's HN, not StackOverflow. You don't mark questions as "off-topic" unless they're trolling... [1]: http://betterthangrep.com http://betterthangrep.com

audiodude 14y ago
I love ack, I've never looked back.
MattJ100 14y ago
The main reason to use grep is portability/availability. While a modern improvement to grep (yes, I'm an ack user and fan too) you can't simply depend on it in portable scripts and such. grep is defined for POSIX after all: http://pubs.opengroup.org/onlinepubs/009695399/utilities/grep.html http://pubs.opengroup.org/onlinepubs/009695399/utilities/gre... I also suspect (without looking) that there are obscure (but occasionally useful) grep features that ack doesn't support.
greyboy 14y ago
That's a fine recommendation for machines that you solely work on, or have complete control over. However, it's good to be able to work with the standard toolset if you frequently work on a variety of remote machines (where it's quite common that you cannot install such things, due to permissions or policies, and want to get started working before attempting to download/install a bunch of custom binaries). Edit: MattJ100 made a more clear point while I was responding.
- pooriaazimi 14y ago
  Thanks for the response. However, a nice thing about ack is that it's not a binary (necessarily) - it's a perl script and can be used without sysadmin permissions: http://betterthangrep.com/install/ http://betterthangrep.com/install/ The other points are quite understandable and correct though. It's always best to at least be familiar with standard tools, even if you want to use an slightly different version for yourself.

snorkel 14y ago

I also decided that sed no longer supports regexes because I said so. And ls -l will now shows a list of print jobs instead of files because maybe that's what you meant. ... oh and ping no longer supports IPv4 because I want everyone to adopt IPv6 immediately because I said so.

rwos 14y ago

It's funny how one of the relatively few design errors in Unix shells now indirectly comes back to haunt us. Recursing is built into pretty much every command that can handle multiple files - which very strongly suggests it should have been made a feature of glob (or the shell). I think it's a bit sad that those "big-picture" features in unix are treated as if the were written in stone.

nvarsj 14y ago
This is one of the great reasons to use zshell. :-) grep -in **/*txt "sometext"

deleted 14y ago

[deleted]