12 ms·

Use Haskell for shell scripting

217 points by stefans 12y ago

Doji 12y ago
The tutorial does a great job of explaining why this is interesting: http://hackage.haskell.org/package/turtle-1.0.0/docs/Turtle-Tutorial.html http://hackage.haskell.org/package/turtle-1.0.0/docs/Turtle-... For example, the pwd function returns a FilePath type rather than a String: Prelude Turtle> :type pwd pwd :: IO Turtle.FilePath The datefile function is also typed: Prelude Turtle> :type datefile datefile :: Turtle.FilePath -> IO UTCTime So this really does seem to structure the data passed between commands, instead of the "stringly typing" unix shells have historically been known for.
- TazeTSchnitzel 12y ago
  Are those types just aliases of String?
  - marcosdumay 12y ago
    What do you mean by "alias"? Path carries String-like information, it can even be easily converted to and from strings. Yet, it's a strong type that won't let you write something like 'path </> file_contents' (although, with overloaded strings, you can do 'path </> "file_name"'). UTCTime is not String-like.
  - throwaway283719 12y ago
    No. For example, a FilePath is (after resolving a few other type aliases) data Root = RootPosix | RootWindowsVolume Char | RootWindowsCurrentVolume data FilePath = FilePath { pathRoot :: Maybe Root , pathDirectories :: [String] , pathBasename :: Maybe String , pathExtensions :: [String] }
tel 12y ago
But why "turtle"?
- strager 12y ago
  Turtle shell.
  - tel 12y ago
    And in retrospect that's quite obvious, hah! I spent the whole time trying to think how this was connected to LOGO.
dkarapetyan 12y ago
Who's the target audience of this exactly? I already see a language pragma, do notation, liftIO, parser combinators. Hamming has this great set of lectures on how he became a world renowned scientist and in one of the lectures he explains why Ada failed and other languages succeeded. The difference was that Ada was designed logically and most successful languages were designed psychologically. Even when government contracts mandated Ada people still wrote in Fortan and hand translated to Ada. You can watch the videos and take from it what you will. A minimal bash file is`#!/bin/bash`. A minimal turtle file is already way too long and logical. The set of videos: https://www.youtube.com/playlist?list=PL2FF649D0C4407B30 https://www.youtube.com/playlist?list=PL2FF649D0C4407B30.
- madawan 12y ago
  Would it be possible to create a /bin/turtle script which prepends those import and language statements? That way all the boilerplate that's needed would be "main = do ...", which seems acceptable imo.
- nandemo 12y ago
  Presumably the target is existing haskellers. Basically, "if you use Haskell, here's something to help you use it for shell scripts too". I don't think it's reasonable to assert that Ada "failed" (e.g. it runs on large passenger airplanes), but in any case that's kinda beside the point, TFA isn't primarily about Haskell evangelism/advocacy. > language pragma, do notation, liftIO, parser combinators. Arguably, all of this is within the reach of an intermediate-level Haskell programmer. OverloadedStrings is considered a basic pragma.
- pjmlp 12y ago
  Ada might have failed on OS, but that is just because few startups that based their workstation OS in UNIX succeeded in the market at large. C goes hand-in-hand with UNIX, so clearly no UNIX vendor would have it in their SDK and UNIX developers weren't willing to pay for tools. As history has shown, the moment UNIX vendors started doing "Home" and "Pro" editions, GCC got lots of help. As Ada talks at FOSDEM show, it is present everywhere where safety matters and its use has been slowly increasing since the Internet has shown how bad idea is to connect C code to the outside world. http://www.kb.cert.org/vuls/byupdate?open=&start=1&count=10 http://www.kb.cert.org/vuls/byupdate?open=&start=1&count=10
meekins 12y ago
no
akurilin 12y ago
This is awesome, I was actually looking for something like that out of sheer curiosity, but perhaps it'll make it into production at some point.
cies 12y ago
Thanks Gabriel Gonzalez! There is a comment on the blog post (by Chris Done) asking how it deals with piping. I really wonder about that too. Some related projects: - Joey Hess recently released a nice Haskell-to-sh compiler. I like this approach as the resulting sh scripts are runnable on pretty much every *nix. https://joeyh.name/blog/entry/shell_monad/ https://joeyh.name/blog/entry/shell_monad/ - Chris Done also released a lib to do shell stuff from Haskell, which build on the conduit library http://chrisdone.com/posts/shell-conduit http://chrisdone.com/posts/shell-conduit - Chris also wrote a shell in Haskell https://github.com/chrisdone/hell https://github.com/chrisdone/hell - Then there is Shelly by Greg Weber https://github.com/yesodweb/Shelly.hs https://github.com/yesodweb/Shelly.hs There are probably more...
- Gabriel439 12y ago
  You use `inproc` and `inshell` for piping. For example, here's the type of `inshell`: inshell :: Text -- Shell command -> Shell Text -- Standard input to feed command -> Shell Text -- Standard output produced by command I made one intentional simplification in the API, which was to not provide a way to capture standard error. It's definitely possible to provide such a utility, but I wanted to simplify things as much as possible in the first release before the slow onslaught of feature cruft begins. If there were such a utility, it would have this type: both :: Text -- Shell command -> Shell Text -- Standard input to feed command -> Shell (Either Text Text) ... and you could selectively listen to just stderr or stdout by taking advantage of the fact that pattern match failures short-circuit downstream commands: Left txt <- both -- only read stderr There is one more shell library that I know of: `process-streaming`. I actually didn't know about `shell_monad` (that's the one most similar in spirit to what I wrote). The main reason I rolled my own library is that this was written with the specific audience of people who didn't know any Haskell, but were comfortable with Python or Bash. My actual goal is to convince people internally at Twitter to use Haskell instead of Python for large scripts. I reviewed all those libraries (with the exception of shell_monad) to see if I felt comfortable marketing them to non-Haskell programmers and none of them felt like the right level of abstraction to me. I almost ended up going with Shelly, but in the process of polishing shelly for internal usage I found myself continually wrapping things with better names, different types, and providing missing features to get a single import umbrella, so I just stopped and asked: "why not just do this as a cohesive single library instead?". Also, `shelly` does not provide any `IO`-only commands: everything has to be wrapped in the `Sh` monad. As for the other libraries, `shell-conduit` was too complex for new users in my opinion and `hell` is not embedded within Haskell (it's a separate language), and I wanted to keep the features of Haskell. I still need some more time to review `shell_monad` to see if I made a mistake by ignoring it.
q3k 12y ago
I don't really see the point of this, apart from academic research values. POSIX shell is everywhere - your current Linux and OS X machines, old UNIX workstations, home routers, servers... Just drop in a file and it will probably run just fine, unless the author screwed something up completely. POSIX shell scripts are the perfect bootstrap mechanisms that will run almost anywhere regardless of architecture. Haskell, on the other hand, is rarely present in an operating system - if you absolutely, positively need a higher-level language for „shell scripting”, then you have a much higher chance of finding a Perl interpreter, or even Python. Heck, even getting ghc and its' basic ecosystem running has always proved to be a huge burden to me. Try sticking a `cabal install` in your CI flow, you'll see your job times increase by hours. Third, there's just the KISS aspect of it - if you're writing something that has logic so simple it can be stuck in a shell file, why not just write it in a shell file? You don't need category theory to get a few files installed...
- knivets 12y ago
  As far as I understand, you need interpreter only if you want to run file as a script, but if you compile beforehand, you can just use it, so no need for Haskell to be present in an operating system.
  - q3k 12y ago
    What's the point then? You just end up with opaque binary blobs that you have to first cross-compile to every possible OS and architecture... How is that scripting by any stretch of the definition?
- comex 12y ago
  Because shell is so deficient that even for "simple" things it is really easy to screw up - when whitespace or special characters in filenames cause some case you overlooked to screw up due to terrible quoting rules, when missing arguments cause [1], when you accidentally put bashisms in scripts labeled /bin/sh, when you suddenly have to do some basic text parsing (e.g. extracting capture groups from a regex) and have to either switch to perl or use some ugly bash extension that's incompatible between the version of bash OS X uses and the newer ones. So you might want to use a different language - even for purely/mostly personal use, in which case Haskell would be fine. [1] https://github.com/ValveSoftware/steam-for-linux/issues/3671 https://github.com/ValveSoftware/steam-for-linux/issues/3671
boothead 12y ago
For all the considerable awesomeness that Gabriel produces, I always think the best part is the *.Tutorial module he includes. I always learn a lot and it's always a great over view that puts the work in context. Everyone should do this!
S4M 12y ago
I don't know much about Haskell, but I thought it had some properties to isolate side effects, but the code he gives: main = do cd "/tmp" mkdir "test" output "test/foo" "Hello, world!" -- Write "Hello, world!" to "test/foo" stdout (input "test/foo") -- Stream "test/foo" to stdout rm "test/foo" rmdir "test" sleep 1 die "Urk!" Clearly doesn't (it creates a directory, writes in a file, removes that file and that directory all in one go without anything indicated by the function main. Is it because it's the main function of the program, or am I missing something?
- zoomerang 12y ago
  Haskell functions return side-effects using the IO type, with the boilerplate plumbing being hidden with monads and do-notation. "main" in Haskell by default has a return type of "IO ()", and any "IO" values returned by that function are executed by the runtime. The end result in this case is something that just looks and feels completely imperative. but if you were to try and call, say, the "rmdir" function inside another function that didn't have an IO return type, you'd get a compile error. (More specifically, you could technically call the function, you just couldn't return the "IO" value as a result, so it couldn't perform any actions).
  - tome 12y ago
    > IO type :)
- sbergot 12y ago
  You are right. In this case effects are not isolated. But in this particular script, there are no interesting things to move into a pure function. It does not mean that it wouldn't be the case in a more complex script. Like everything, you have to learn to balance your IO code and your pure code. A bit like learning when to factor something into a separate class, or leave it in a few statement/methods. If you write everything in IO & do notation, you don't get the main benefits of haskell. But if your code is more than 10 lines long, chances are that there will be useful pure functions in it.
mercurial 12y ago
I like the Pattern thing. However, it seems to me that you're going to quickly run into trouble if you need to even vaguely emulate shell scripting. Shell utilities live and die by their options. It's unfortunate Haskell supports neither named arguments nor default values. Which means that in order to emulate options, you would need to pass records to your "shell" utility, which, on top of being cumbersome, forces you to prefix every option in a way unique to your utility, since you cannot have two records with the same fields in the same namespace...
- gamegoblin 12y ago
  Quite a lot of libraries here: https://wiki.haskell.org/Command_line_option_parsers https://wiki.haskell.org/Command_line_option_parsers
  - mercurial 12y ago
    That's not the issue. The issue is that, if you want to simulate both "grep" and "grep -r", you need to different functions, or you need to have your "grep" function accept a record of parameters.
    - Hz8NSD 12y ago
      I saw those as regular parser that returns text but not just grep wrapper. Yes It has function named 'grep', but its not grep wrapper. It looks like `lstree` could be combined with `grep` function for emulate `grep -r`.
      - mercurial 12y ago
        Take lstree then. How would you give it an option giving the kind of ordering you want?
        barsoap 12y ago
        I guess you wouldn't, you'd have a sort function afterwards. ...for Unix' insistence on composability, the shell tools are often unnecessarily monolithic, probably because that's the only sane way if the only type you have in interconnect is `string`.
amelius 12y ago
I wonder why it uses the convention: stdout (input "test/foo") instead of: output stdout (input "test/foo") which would be expected considering the previous line.
barrkel 12y ago
OK. How do you easily fork to run a command in the background? How does setting up pipes work? What's the idiom for chdir'ing to a subdirectory such that you pop back out again when you're done (I'd use a subshell with (ch xxx; ...) in bash)? Getting into more tricky stuff, what's the equivalent of <() in bash? This doesn't really demonstrate anything that shell scripts are actually written for: orchestrating and composing other processes, and job control. If you wanted to leverage type checking for safety, it would be more interesting to typecheck the streams input and output by pipes.
- tinco 12y ago
  Did we read the same article? The entire 'streaming section' is about pipes and I/O redirects. Running a command in the background is just forkIO $ proc ..etc.., as in regular Haskell. Nothing tricky about it.
  - barrkel 12y ago
    The streaming section of the article has nothing about composing processes, that I could see; it appeared to be about treating the output of commands as input to Haskell lazy lists. I may have misread it, though. Here's a pattern that comes up fairly frequently for me: foo | fgrep -v -f <(cut -f 2 info.csv) | bar It uses the second column in info.csv as fixed strings to match inside lines in the output of foo, and filters them out, with the remaining lines going to bar. All 4 processes (foo, bar, fgrep, cut) run concurrently. Likely fgrep will block on cut sooner or later, but the point is that multiple communicating concurrent processes are set up using a fairly easy to use DSL. That's what a shell is, to me.
    - Gabriel439 12y ago
      There are two ways you can embed that within `turtle`. You can either embed each step as its own concurrent process, like this: -- Note, the flow is right-to-left, not left-to-right inshell "bar" (inshell "fgrep ..." (inshell "foo" empty)) Or you can just embed the entire thing within a single `inshell` command: inshell "foo | fgrep ... | bar" The reason this works is that the type of `inshell` is: inshell :: Text -- Command line -> Shell Text -- Standard input to feed -> Shell Text -- Standard output from command This leads you stream to any shell command's input and read the command's output also as a stream.
qznc 12y ago
Haskell is low on boilerplate? Yes, in general I would agree. Those scripts however, all have to be prefixed with "{-# LANGUAGE OverloadedStrings #-} import Turtle main = do". This is tedious boilerplate.
- sukilot 12y ago
  You could trivially have a wrapper program that added that to every script file before calling runhaskell.
fallat 12y ago
I've been pushing for alternative shell scripts for awhile now. I mostly stick to Python and Haskell now. It is great. Highly recommended.
chrisBob 12y ago
After learning Perl I started using it where some more educated people might recommend a proper shell script. My thinking is that using what you know is a whole lot more efficient than learning a new tool for a small job, even if some people think it is the right tool. I am sure it is no different for people familiar with Haskell.
- falcolas 12y ago
  Agreed. I look upon people who use PHP for shell scripting with a sigh and a shaken head, but I can't fault them for it. PHP works, PHP is quick to write, and for many tasks, PHP is sufficient. I hope Haskel can gain traction in this area, if only because options are always nice to have, and competition forces everyone to bring their best game.
- loudmax 12y ago
  I do a lot of shell scripting, and I'm not sure there is such a thing as a "proper" shell script. The shell just isn't a great programming language. Just about any modern scripting language is better, starting with Perl. But the shell has been the lingua franca of the Unix world for decades now. It's the one language that you can pretty much guarantee is on any Unix or Linux server, even pretty ancient ones. I don't doubt that Haskell isn't a better scripting language language than the shell, but you can't assume /usr/bin/env runhaskell is going to return anything on random Linux servers. Perl and Python, maybe, but Haskell isn't there yet.
  - klibertp 12y ago
    > you can pretty much guarantee is on any Unix or Linux server, even pretty ancient ones Well, yes and no. You can get reasonable compatibility with different Unix flavours if you stick to sh. Your script is not going to work on BSDs once you start using bash specific features, though. Fun fact: on FreeBSD bash does not live in /bin/bash, it's in /usr/local/bin/bash. Every time you write a shebang with /bin/bash hardcoded you're making your script harder to use there. Perl is everywhere almost by default and it's more compatible as it has just one implementation, without sh/bash/csh/ksh/tcsh/zsh madness. I'd say it's a good idea to use Perl instead of shell script for anything more complicated than a few lines of code if it's meant to be portable. (And I'm not Perl programmer at all).
joelthelion 12y ago
I want to see how you implement the pipe :)
falcolas 12y ago
Please forgive my lack of familiarity with the concurrent workings of Haskell, but since the Shell streams are based off []/IO, and not Concurrent.Chan, does this mean one turtle function has to complete (and write its results to memory) before the next turtle function can run? To me, magic bits of shell scripts which turtle would need to improve upon were it to replace said scripts are not the loop constructs, conditionals, or even the type system (even though it's completely lacking in bash), it is the ability to use pipes to link processes concurrently.
- joeyh 12y ago
  The streaming section shows some examples of combining turtle functions, this will be the same as shell pipes. There's also nothing stopping you from using forkIO to spark off a separate thread, and doing IO in multiple threads concurrently. Haskell's IO manager allows multiple threads doing concurrent IO in what looks like an imperative, one instruction after the other manner. Instead of async callbacks like you might expect from other languages.
npsimons 12y ago
Nice! Now I can add Haskell to my list of languages I can script with. I'm always on the lookout for new languages I can script with (or at least get closer to rapid prototyping) for easier learning, testing, problem solving, etc. I've got templates that I run against linters, style checkers, etc for many languages and it will be helpful to have even more options.
Klasiaster 12y ago
For me combing the best parts of bash and ipython is the way to go. Up to now this seems more comfortable to me than using subprocess in python or this haskell aproach which needs to be aware of every programme output to give what it promises. You can easily copy big parts of existing bash scripts and e.g. add error handling in the python way :) Even I think for loops/list comprehensions are betten than the strange bash syntax. And here a short example:: #!/usr/bin/env ipython3 # # 1. echo "#!/usr/bin/env ipython3" > scriptname.ipy # creates new ipy-file # # 2. chmod +x scriptname.ipy # make it executable # # 3. starting with line 2, write normal python or do some of # the ! magic of ipython, so that you can use shell commands # within python and even assign their output to a variable via # var = !cmd1 | cmd2 | cmd3 # enjoy ;) # # 4. run via ./scriptname.ipy - if it fails with recognizing % and ! # but parses raw python fine, please check again for the .ipy suffix which must be there! # # ugly example, please go and find more in the wild files = !ls *.* | grep "y" for file in files: !echo $file | grep "p" # sorry for this nonsense example ;) # it's even possible to access the output of a command by outputvariable.s, .p or .n # see file:///usr/share/doc/ipython-doc/html/interactive/reference.html#system-shell-access Better take a look here, it's more complete: https://blog.safaribooksonline.com/2014/02/12/using-shell-commands-effectively-ipython/ https://blog.safaribooksonline.com/2014/02/12/using-shell-co...
- codygman 12y ago
  Oh, I'm going to have to see if I can use Turtle with IHaskell tomorrow! 0: http://gibiansky.github.io/IHaskell/ http://gibiansky.github.io/IHaskell/ 1: https://registry.hub.docker.com/u/gregweber/ihaskell/ https://registry.hub.docker.com/u/gregweber/ihaskell/
agumonkey 12y ago
It's not closely related but still, it reminded me of the wonderful https://pypi.python.org/pypi/sh https://pypi.python.org/pypi/sh to write 'shell' script in python with very low boilerplate.

217 points by stefans 12y ago

Doji 12y ago

The tutorial does a great job of explaining why this is interesting: http://hackage.haskell.org/package/turtle-1.0.0/docs/Turtle-Tutorial.html http://hackage.haskell.org/package/turtle-1.0.0/docs/Turtle-... For example, the pwd function returns a FilePath type rather than a String: Prelude Turtle> :type pwd pwd :: IO Turtle.FilePath The datefile function is also typed: Prelude Turtle> :type datefile datefile :: Turtle.FilePath -> IO UTCTime So this really does seem to structure the data passed between commands, instead of the "stringly typing" unix shells have historically been known for.

TazeTSchnitzel 12y ago
Are those types just aliases of String?
- marcosdumay 12y ago
  What do you mean by "alias"? Path carries String-like information, it can even be easily converted to and from strings. Yet, it's a strong type that won't let you write something like 'path </> file_contents' (although, with overloaded strings, you can do 'path </> "file_name"'). UTCTime is not String-like.
- throwaway283719 12y ago
  No. For example, a FilePath is (after resolving a few other type aliases) data Root = RootPosix | RootWindowsVolume Char | RootWindowsCurrentVolume data FilePath = FilePath { pathRoot :: Maybe Root , pathDirectories :: [String] , pathBasename :: Maybe String , pathExtensions :: [String] }

tel 12y ago

But why "turtle"?

strager 12y ago
Turtle shell.
- tel 12y ago
  And in retrospect that's quite obvious, hah! I spent the whole time trying to think how this was connected to LOGO.

dkarapetyan 12y ago

Who's the target audience of this exactly? I already see a language pragma, do notation, liftIO, parser combinators. Hamming has this great set of lectures on how he became a world renowned scientist and in one of the lectures he explains why Ada failed and other languages succeeded. The difference was that Ada was designed logically and most successful languages were designed psychologically. Even when government contracts mandated Ada people still wrote in Fortan and hand translated to Ada. You can watch the videos and take from it what you will. A minimal bash file is`#!/bin/bash`. A minimal turtle file is already way too long and logical. The set of videos: https://www.youtube.com/playlist?list=PL2FF649D0C4407B30 https://www.youtube.com/playlist?list=PL2FF649D0C4407B30.

madawan 12y ago
Would it be possible to create a /bin/turtle script which prepends those import and language statements? That way all the boilerplate that's needed would be "main = do ...", which seems acceptable imo.
nandemo 12y ago
Presumably the target is existing haskellers. Basically, "if you use Haskell, here's something to help you use it for shell scripts too". I don't think it's reasonable to assert that Ada "failed" (e.g. it runs on large passenger airplanes), but in any case that's kinda beside the point, TFA isn't primarily about Haskell evangelism/advocacy. > language pragma, do notation, liftIO, parser combinators. Arguably, all of this is within the reach of an intermediate-level Haskell programmer. OverloadedStrings is considered a basic pragma.
pjmlp 12y ago
Ada might have failed on OS, but that is just because few startups that based their workstation OS in UNIX succeeded in the market at large. C goes hand-in-hand with UNIX, so clearly no UNIX vendor would have it in their SDK and UNIX developers weren't willing to pay for tools. As history has shown, the moment UNIX vendors started doing "Home" and "Pro" editions, GCC got lots of help. As Ada talks at FOSDEM show, it is present everywhere where safety matters and its use has been slowly increasing since the Internet has shown how bad idea is to connect C code to the outside world. http://www.kb.cert.org/vuls/byupdate?open=&start=1&count=10 http://www.kb.cert.org/vuls/byupdate?open=&start=1&count=10

meekins 12y ago

akurilin 12y ago

This is awesome, I was actually looking for something like that out of sheer curiosity, but perhaps it'll make it into production at some point.

cies 12y ago

Thanks Gabriel Gonzalez! There is a comment on the blog post (by Chris Done) asking how it deals with piping. I really wonder about that too. Some related projects: - Joey Hess recently released a nice Haskell-to-sh compiler. I like this approach as the resulting sh scripts are runnable on pretty much every *nix. https://joeyh.name/blog/entry/shell_monad/ https://joeyh.name/blog/entry/shell_monad/ - Chris Done also released a lib to do shell stuff from Haskell, which build on the conduit library http://chrisdone.com/posts/shell-conduit http://chrisdone.com/posts/shell-conduit - Chris also wrote a shell in Haskell https://github.com/chrisdone/hell https://github.com/chrisdone/hell - Then there is Shelly by Greg Weber https://github.com/yesodweb/Shelly.hs https://github.com/yesodweb/Shelly.hs There are probably more...

Gabriel439 12y ago
You use `inproc` and `inshell` for piping. For example, here's the type of `inshell`: inshell :: Text -- Shell command -> Shell Text -- Standard input to feed command -> Shell Text -- Standard output produced by command I made one intentional simplification in the API, which was to not provide a way to capture standard error. It's definitely possible to provide such a utility, but I wanted to simplify things as much as possible in the first release before the slow onslaught of feature cruft begins. If there were such a utility, it would have this type: both :: Text -- Shell command -> Shell Text -- Standard input to feed command -> Shell (Either Text Text) ... and you could selectively listen to just stderr or stdout by taking advantage of the fact that pattern match failures short-circuit downstream commands: Left txt <- both -- only read stderr There is one more shell library that I know of: `process-streaming`. I actually didn't know about `shell_monad` (that's the one most similar in spirit to what I wrote). The main reason I rolled my own library is that this was written with the specific audience of people who didn't know any Haskell, but were comfortable with Python or Bash. My actual goal is to convince people internally at Twitter to use Haskell instead of Python for large scripts. I reviewed all those libraries (with the exception of shell_monad) to see if I felt comfortable marketing them to non-Haskell programmers and none of them felt like the right level of abstraction to me. I almost ended up going with Shelly, but in the process of polishing shelly for internal usage I found myself continually wrapping things with better names, different types, and providing missing features to get a single import umbrella, so I just stopped and asked: "why not just do this as a cohesive single library instead?". Also, `shelly` does not provide any `IO`-only commands: everything has to be wrapped in the `Sh` monad. As for the other libraries, `shell-conduit` was too complex for new users in my opinion and `hell` is not embedded within Haskell (it's a separate language), and I wanted to keep the features of Haskell. I still need some more time to review `shell_monad` to see if I made a mistake by ignoring it.

q3k 12y ago

I don't really see the point of this, apart from academic research values. POSIX shell is everywhere - your current Linux and OS X machines, old UNIX workstations, home routers, servers... Just drop in a file and it will probably run just fine, unless the author screwed something up completely. POSIX shell scripts are the perfect bootstrap mechanisms that will run almost anywhere regardless of architecture. Haskell, on the other hand, is rarely present in an operating system - if you absolutely, positively need a higher-level language for „shell scripting”, then you have a much higher chance of finding a Perl interpreter, or even Python. Heck, even getting ghc and its' basic ecosystem running has always proved to be a huge burden to me. Try sticking a `cabal install` in your CI flow, you'll see your job times increase by hours. Third, there's just the KISS aspect of it - if you're writing something that has logic so simple it can be stuck in a shell file, why not just write it in a shell file? You don't need category theory to get a few files installed...

knivets 12y ago
As far as I understand, you need interpreter only if you want to run file as a script, but if you compile beforehand, you can just use it, so no need for Haskell to be present in an operating system.
- q3k 12y ago
  What's the point then? You just end up with opaque binary blobs that you have to first cross-compile to every possible OS and architecture... How is that scripting by any stretch of the definition?
comex 12y ago
Because shell is so deficient that even for "simple" things it is really easy to screw up - when whitespace or special characters in filenames cause some case you overlooked to screw up due to terrible quoting rules, when missing arguments cause [1], when you accidentally put bashisms in scripts labeled /bin/sh, when you suddenly have to do some basic text parsing (e.g. extracting capture groups from a regex) and have to either switch to perl or use some ugly bash extension that's incompatible between the version of bash OS X uses and the newer ones. So you might want to use a different language - even for purely/mostly personal use, in which case Haskell would be fine. [1] https://github.com/ValveSoftware/steam-for-linux/issues/3671 https://github.com/ValveSoftware/steam-for-linux/issues/3671

boothead 12y ago

For all the considerable awesomeness that Gabriel produces, I always think the best part is the *.Tutorial module he includes. I always learn a lot and it's always a great over view that puts the work in context. Everyone should do this!

S4M 12y ago

I don't know much about Haskell, but I thought it had some properties to isolate side effects, but the code he gives: main = do cd "/tmp" mkdir "test" output "test/foo" "Hello, world!" -- Write "Hello, world!" to "test/foo" stdout (input "test/foo") -- Stream "test/foo" to stdout rm "test/foo" rmdir "test" sleep 1 die "Urk!" Clearly doesn't (it creates a directory, writes in a file, removes that file and that directory all in one go without anything indicated by the function main. Is it because it's the main function of the program, or am I missing something?

zoomerang 12y ago
Haskell functions return side-effects using the IO type, with the boilerplate plumbing being hidden with monads and do-notation. "main" in Haskell by default has a return type of "IO ()", and any "IO" values returned by that function are executed by the runtime. The end result in this case is something that just looks and feels completely imperative. but if you were to try and call, say, the "rmdir" function inside another function that didn't have an IO return type, you'd get a compile error. (More specifically, you could technically call the function, you just couldn't return the "IO" value as a result, so it couldn't perform any actions).
- tome 12y ago
  > IO type :)
sbergot 12y ago
You are right. In this case effects are not isolated. But in this particular script, there are no interesting things to move into a pure function. It does not mean that it wouldn't be the case in a more complex script. Like everything, you have to learn to balance your IO code and your pure code. A bit like learning when to factor something into a separate class, or leave it in a few statement/methods. If you write everything in IO & do notation, you don't get the main benefits of haskell. But if your code is more than 10 lines long, chances are that there will be useful pure functions in it.

mercurial 12y ago

I like the Pattern thing. However, it seems to me that you're going to quickly run into trouble if you need to even vaguely emulate shell scripting. Shell utilities live and die by their options. It's unfortunate Haskell supports neither named arguments nor default values. Which means that in order to emulate options, you would need to pass records to your "shell" utility, which, on top of being cumbersome, forces you to prefix every option in a way unique to your utility, since you cannot have two records with the same fields in the same namespace...

gamegoblin 12y ago
Quite a lot of libraries here: https://wiki.haskell.org/Command_line_option_parsers https://wiki.haskell.org/Command_line_option_parsers
- mercurial 12y ago
  That's not the issue. The issue is that, if you want to simulate both "grep" and "grep -r", you need to different functions, or you need to have your "grep" function accept a record of parameters.
  - Hz8NSD 12y ago
    I saw those as regular parser that returns text but not just grep wrapper. Yes It has function named 'grep', but its not grep wrapper. It looks like `lstree` could be combined with `grep` function for emulate `grep -r`.
    - mercurial 12y ago
      Take lstree then. How would you give it an option giving the kind of ordering you want?
      - barsoap 12y ago
        I guess you wouldn't, you'd have a sort function afterwards. ...for Unix' insistence on composability, the shell tools are often unnecessarily monolithic, probably because that's the only sane way if the only type you have in interconnect is `string`.

amelius 12y ago

I wonder why it uses the convention: stdout (input "test/foo") instead of: output stdout (input "test/foo") which would be expected considering the previous line.

barrkel 12y ago

OK. How do you easily fork to run a command in the background? How does setting up pipes work? What's the idiom for chdir'ing to a subdirectory such that you pop back out again when you're done (I'd use a subshell with (ch xxx; ...) in bash)? Getting into more tricky stuff, what's the equivalent of <() in bash? This doesn't really demonstrate anything that shell scripts are actually written for: orchestrating and composing other processes, and job control. If you wanted to leverage type checking for safety, it would be more interesting to typecheck the streams input and output by pipes.

tinco 12y ago
Did we read the same article? The entire 'streaming section' is about pipes and I/O redirects. Running a command in the background is just forkIO $ proc ..etc.., as in regular Haskell. Nothing tricky about it.
- barrkel 12y ago
  The streaming section of the article has nothing about composing processes, that I could see; it appeared to be about treating the output of commands as input to Haskell lazy lists. I may have misread it, though. Here's a pattern that comes up fairly frequently for me: foo | fgrep -v -f <(cut -f 2 info.csv) | bar It uses the second column in info.csv as fixed strings to match inside lines in the output of foo, and filters them out, with the remaining lines going to bar. All 4 processes (foo, bar, fgrep, cut) run concurrently. Likely fgrep will block on cut sooner or later, but the point is that multiple communicating concurrent processes are set up using a fairly easy to use DSL. That's what a shell is, to me.
  - Gabriel439 12y ago
    There are two ways you can embed that within `turtle`. You can either embed each step as its own concurrent process, like this: -- Note, the flow is right-to-left, not left-to-right inshell "bar" (inshell "fgrep ..." (inshell "foo" empty)) Or you can just embed the entire thing within a single `inshell` command: inshell "foo | fgrep ... | bar" The reason this works is that the type of `inshell` is: inshell :: Text -- Command line -> Shell Text -- Standard input to feed -> Shell Text -- Standard output from command This leads you stream to any shell command's input and read the command's output also as a stream.

qznc 12y ago

Haskell is low on boilerplate? Yes, in general I would agree. Those scripts however, all have to be prefixed with "{-# LANGUAGE OverloadedStrings #-} import Turtle main = do". This is tedious boilerplate.

sukilot 12y ago
You could trivially have a wrapper program that added that to every script file before calling runhaskell.

fallat 12y ago

I've been pushing for alternative shell scripts for awhile now. I mostly stick to Python and Haskell now. It is great. Highly recommended.

chrisBob 12y ago

After learning Perl I started using it where some more educated people might recommend a proper shell script. My thinking is that using what you know is a whole lot more efficient than learning a new tool for a small job, even if some people think it is the right tool. I am sure it is no different for people familiar with Haskell.

falcolas 12y ago
Agreed. I look upon people who use PHP for shell scripting with a sigh and a shaken head, but I can't fault them for it. PHP works, PHP is quick to write, and for many tasks, PHP is sufficient. I hope Haskel can gain traction in this area, if only because options are always nice to have, and competition forces everyone to bring their best game.
loudmax 12y ago
I do a lot of shell scripting, and I'm not sure there is such a thing as a "proper" shell script. The shell just isn't a great programming language. Just about any modern scripting language is better, starting with Perl. But the shell has been the lingua franca of the Unix world for decades now. It's the one language that you can pretty much guarantee is on any Unix or Linux server, even pretty ancient ones. I don't doubt that Haskell isn't a better scripting language language than the shell, but you can't assume /usr/bin/env runhaskell is going to return anything on random Linux servers. Perl and Python, maybe, but Haskell isn't there yet.
- klibertp 12y ago
  > you can pretty much guarantee is on any Unix or Linux server, even pretty ancient ones Well, yes and no. You can get reasonable compatibility with different Unix flavours if you stick to sh. Your script is not going to work on BSDs once you start using bash specific features, though. Fun fact: on FreeBSD bash does not live in /bin/bash, it's in /usr/local/bin/bash. Every time you write a shebang with /bin/bash hardcoded you're making your script harder to use there. Perl is everywhere almost by default and it's more compatible as it has just one implementation, without sh/bash/csh/ksh/tcsh/zsh madness. I'd say it's a good idea to use Perl instead of shell script for anything more complicated than a few lines of code if it's meant to be portable. (And I'm not Perl programmer at all).

joelthelion 12y ago

I want to see how you implement the pipe :)

falcolas 12y ago

Please forgive my lack of familiarity with the concurrent workings of Haskell, but since the Shell streams are based off []/IO, and not Concurrent.Chan, does this mean one turtle function has to complete (and write its results to memory) before the next turtle function can run? To me, magic bits of shell scripts which turtle would need to improve upon were it to replace said scripts are not the loop constructs, conditionals, or even the type system (even though it's completely lacking in bash), it is the ability to use pipes to link processes concurrently.

joeyh 12y ago
The streaming section shows some examples of combining turtle functions, this will be the same as shell pipes. There's also nothing stopping you from using forkIO to spark off a separate thread, and doing IO in multiple threads concurrently. Haskell's IO manager allows multiple threads doing concurrent IO in what looks like an imperative, one instruction after the other manner. Instead of async callbacks like you might expect from other languages.

npsimons 12y ago

Nice! Now I can add Haskell to my list of languages I can script with. I'm always on the lookout for new languages I can script with (or at least get closer to rapid prototyping) for easier learning, testing, problem solving, etc. I've got templates that I run against linters, style checkers, etc for many languages and it will be helpful to have even more options.

Klasiaster 12y ago

For me combing the best parts of bash and ipython is the way to go. Up to now this seems more comfortable to me than using subprocess in python or this haskell aproach which needs to be aware of every programme output to give what it promises. You can easily copy big parts of existing bash scripts and e.g. add error handling in the python way :) Even I think for loops/list comprehensions are betten than the strange bash syntax. And here a short example:: #!/usr/bin/env ipython3 # # 1. echo "#!/usr/bin/env ipython3" > scriptname.ipy # creates new ipy-file # # 2. chmod +x scriptname.ipy # make it executable # # 3. starting with line 2, write normal python or do some of # the ! magic of ipython, so that you can use shell commands # within python and even assign their output to a variable via # var = !cmd1 | cmd2 | cmd3 # enjoy ;) # # 4. run via ./scriptname.ipy - if it fails with recognizing % and ! # but parses raw python fine, please check again for the .ipy suffix which must be there! # # ugly example, please go and find more in the wild files = !ls *.* | grep "y" for file in files: !echo $file | grep "p" # sorry for this nonsense example ;) # it's even possible to access the output of a command by outputvariable.s, .p or .n # see file:///usr/share/doc/ipython-doc/html/interactive/reference.html#system-shell-access Better take a look here, it's more complete: https://blog.safaribooksonline.com/2014/02/12/using-shell-commands-effectively-ipython/ https://blog.safaribooksonline.com/2014/02/12/using-shell-co...

codygman 12y ago
Oh, I'm going to have to see if I can use Turtle with IHaskell tomorrow! 0: http://gibiansky.github.io/IHaskell/ http://gibiansky.github.io/IHaskell/ 1: https://registry.hub.docker.com/u/gregweber/ihaskell/ https://registry.hub.docker.com/u/gregweber/ihaskell/

agumonkey 12y ago

It's not closely related but still, it reminded me of the wonderful https://pypi.python.org/pypi/sh https://pypi.python.org/pypi/sh to write 'shell' script in python with very low boilerplate.