7 ms·

Haskell improves log processing 4x over Python

114 points by jmintz 16y ago

andrewcooke 16y ago
The work sounds very cool (and they are hiring), but (only) a factor of 4 speedup over Python is (to repeat a phrase from elsewhere today) like boasting that you're the tallest midget ;o)
- jamwt 16y ago
  Hi, article author here. It's important to note that this particular job is largely bound on a.) I/O and b.) format serialization tasks. Both Python's BSON and JSON libraries are mature and have their critical sections written in C, so a speedup of 4x is still noteworthy. The Haskell version, on the other hand, is pure Haskell.
  - andrewcooke 16y ago
    Neat - thanks.
  - deleted 16y ago
    [deleted]
- jbellis 16y ago
  Agreed. Even where you can optimize the hot code in C, Python is no speed demon. Cassandra's java stress test can push out about 10x as many ops/s as the python one, even though Thrift C extension for Python is quite good. /still a Python fan
  - jamwt 16y ago
    Yeah, Haskell is roughly as fast as Java. I imagine, with the tuning I allude to in the blog post, we could restore about a 10x improvement, even on a single core. After all, 10x was about what we had with raw Redis ops/s before the serialization libraries got involved. /also still a python fan :-)
    - j_baker 16y ago
      I think it's difficult to compare Java to Haskell. Java is JIT compiled and trades startup time for optimized performance. Haskell is compiled to native code and has a much faster startup time, but the optimizer doesn't have as good information because it runs when the code is compiled rather than at runtime. In general, Java tends to be better for a long-running process while Haskell might be better for a one-off job (not to say that one couldn't do the other without problem though).
      - orblivion 16y ago
        > Haskell is compiled to native code and has a much faster startup time, but the optimizer doesn't have as good information because it runs when the code is compiled rather than at runtime. I'm not that familiar with JIT, but wouldn't compiling everything to native code ahead of time be at least as good as compiling parts that turn out to be slow, just in time? Is it about what sorts of optimizations to use, which you don't know until runtime? Also, would this situation change given the fact that they're switching to LLVM?
        jasonwatkinspdx 16y ago
        There are many in-lining opportunities that depend on knowing what branches are followed in the program at run time. A JIT has better access to this information, and so can be more aggressive in leveraging it. I'm not familiar with how common these techniques are in production JVM's, but as they're becoming common in javascript implementations I'd assume it's equally common.
        Tuna-Fish 16y ago
        JIT has strictly more information than an ahead-of-time compiler running without good profiling data (that should be based on the real workload, not a dev test). JITs can notice things like paths never taken and compile the code into a set of sanity checks and a single completely inlined tight codepath for the common case, and when sanity checks fail, fall back to normal code. JIT does, however, use cycles at runtime to do the actual compile. While I have seen benchmarks where JIT wins over AoT, I'd still bet that the balance favors traditional compilers.
Peaker 16y ago
Sounds great. I'm a very big Haskell fan. I'd love to point people to this when trying to convey some advantages of Haskell. To make it more compelling, can you expand some on the downsides and maybe obstacles you encountered? The thing I'm unsure about, is how difficult it would be for (very) talented developers to just jump in. We have really talented developers, and everyone is super time-constrained, so many are wary of diving into a language as different as Haskell. Was it hard for your developers to figure Haskell out? Did your previous use of Scala help? How long did it take them to dive into Scala?
- jamwt 16y ago
  I would say the two real barriers to writing effective Haskell projects are a.) "getting" monads, and b.) understanding the implications of laziness, especially with regard to space leaks and unconsumed thunks. Everything else isn't that big of a deal. It's all much easier to digest, though, even for "really talented developers", if they have some experience with another functional language first. OCaml is a nice stepping stone before digging into the abstractions involved in understanding Haskell's powerful type system. Scala is good too, but having the object stuff mixed in there can lead you to rely on some patterns that aren't going to be available in a non-OOP language. I think the scheme/clojure path isn't bad either, but it's probably ideal to spend some time in the "statically typed" wing of the functional universe before going to Haskell.
  - grav1tas 16y ago
    I agree. I dove into Haskell without doing any of the prior, and it was like running into a brick wall. However, persistence paid off in my case, but I do wonder how I would have handled it if I would have spent time with OCaml beforehand.
  - samstokes 16y ago
    Could you say more about why "getting" monads was needed? I came to Haskell with no understanding of monads, started writing code, and eventually used my knowledge of Haskell to learn about monads. Not understanding monads just meant I was lacking a useful design pattern, and found certain API docs confusing, but it didn't stop me from writing reasonable code in most circumstances. On the other hand what you describe in your (awesome) blog post is a more significant Haskell project than any I've worked on, so I'd be interested to hear your experience. I've not really written my own monad, or properly looked into monad transformer stacks, and I'm aware that I could probably clean up a lot of code using them - is that the sort of thing you mean?
ynniv 16y ago
Are the logs being read from disk? In my experience, python is highly optimized for reading (possibly compressed) files from disk. If your infrastructure keeps logs in memory, python will lose this advantage and compete on computational performance where Haskell has the advantage. This is important for those of us who grind logs on disk and might be considering a language switch.
- jamwt 16y ago
  Nope, this is a process that BLPOP's logs from some Redis queue, does some processing on them, then writes them to disk.
- enneff 16y ago
  What do you mean by optimized? Python makes the same read and write syscalls everyone else does. What you're probably observing is Python's slow code generation being masked by the inherent slowness of I/O.
  - ynniv 16y ago
    Python makes the same read and write syscalls everyone else does Except, when python's pants are on, it makes gold records. I haven't looked to see if there are any explicit optimizations, but your statement is ridiculous; an effective IO strategy can have an enormous effect on performance.
    - enneff 16y ago
      I'm sorry, what? Just being disagreeable is not an answer. What does "IO strategy" mean? You are being incredibly vague and unhelpful. Reading data from a file handle into a buffer is a trivial operation. It's what you do with that data afterwards that is important. In C (or Go) you have complete control over what happens next. As for Python, I don't know what happens, but I don't see how it could possibly be more efficient than any other sane language.
      - ynniv 16y ago
aristus 16y ago
Awesome work. If you haven't heard about Tim Bray's WideFinder challenge, it was really interesting. http://tartarus.org/james/diary/2008/06/17/widefinder-final-results http://tartarus.org/james/diary/2008/06/17/widefinder-final-...
Locke1689 16y ago
The author is mostly write about the usage cases of Haskell, but simply "systems" is a bit misleading because there are certain performance characteristics of lazy programs which make them bad choices for some systems programs. Any type of real-time system, for example, can suffer unpredictable performance in critical sections, which is pretty undesirable.
- awj 16y ago
  Not to argue the example, but Python's garbage collection disqualifies it for real-time systems as well. In fact, I'm having a hard time find a "system" task for which Python (as a language) is qualified by Haskell is not.
  - Locke1689 16y ago
    Python is not a systems programming language.
    - danieldk 16y ago
      Maybe not, but it is certainly used as one. Everything from package managers (yum) to operating system installers (Anaconda) have been written in Python. Besides that, the grandparent is right: possibly every situation where Python was used as a systems programming language, Haskell could fit in (and more).
      - Locke1689 16y ago
        I think you're confused to what "systems" programming entails. User-space packaging with a bunch of scripts is not systems programming. OS installers is not systems programming. To give you an example of what is systems programming, I have helped developed operating systems kernels, virtual machine monitors, and distributed networked systems. All of these things would be considered systems programming. See http://en.wikipedia.org/wiki/System_programming http://en.wikipedia.org/wiki/System_programming for more information.
kordless 16y ago
I'd be interested in hearing more about how the author is using the resulting data set. Doing extractions at event generation time can be very useful if you know what you are after in advance, but not so good for adhoc analysis. Any reason why you didn't use Hadoop for this, then run batch jobs to extract summaries?
- jamwt 16y ago
  Yeah, the whole pipeline is actually quite more faceted than can be deduced from this summary. This stage actually just persists the events into a consolidated transaction log. Then, there are secondary processes that scan these transaction logs (in batch) and distribute data into various databases for system, business, and user analytics. I can't go into too much detail there, but the actual digesting and reporting side is more involved.
  - kordless 16y ago
    I'd like to hear more about the use case if you have time, and can talk about it. I'm kordless at loggly dot com.

114 points by jmintz 16y ago

The work sounds very cool (and they are hiring), but (only) a factor of 4 speedup over Python is (to repeat a phrase from elsewhere today) like boasting that you're the tallest midget ;o)

jamwt 16y ago
Hi, article author here. It's important to note that this particular job is largely bound on a.) I/O and b.) format serialization tasks. Both Python's BSON and JSON libraries are mature and have their critical sections written in C, so a speedup of 4x is still noteworthy. The Haskell version, on the other hand, is pure Haskell.
- andrewcooke 16y ago
  Neat - thanks.
- deleted 16y ago
  [deleted]
jbellis 16y ago
Agreed. Even where you can optimize the hot code in C, Python is no speed demon. Cassandra's java stress test can push out about 10x as many ops/s as the python one, even though Thrift C extension for Python is quite good. /still a Python fan
- jamwt 16y ago
  Yeah, Haskell is roughly as fast as Java. I imagine, with the tuning I allude to in the blog post, we could restore about a 10x improvement, even on a single core. After all, 10x was about what we had with raw Redis ops/s before the serialization libraries got involved. /also still a python fan :-)
  - j_baker 16y ago
    I think it's difficult to compare Java to Haskell. Java is JIT compiled and trades startup time for optimized performance. Haskell is compiled to native code and has a much faster startup time, but the optimizer doesn't have as good information because it runs when the code is compiled rather than at runtime. In general, Java tends to be better for a long-running process while Haskell might be better for a one-off job (not to say that one couldn't do the other without problem though).
    - orblivion 16y ago
      > Haskell is compiled to native code and has a much faster startup time, but the optimizer doesn't have as good information because it runs when the code is compiled rather than at runtime. I'm not that familiar with JIT, but wouldn't compiling everything to native code ahead of time be at least as good as compiling parts that turn out to be slow, just in time? Is it about what sorts of optimizations to use, which you don't know until runtime? Also, would this situation change given the fact that they're switching to LLVM?
      - jasonwatkinspdx 16y ago
        There are many in-lining opportunities that depend on knowing what branches are followed in the program at run time. A JIT has better access to this information, and so can be more aggressive in leveraging it. I'm not familiar with how common these techniques are in production JVM's, but as they're becoming common in javascript implementations I'd assume it's equally common.
      - Tuna-Fish 16y ago
        JIT has strictly more information than an ahead-of-time compiler running without good profiling data (that should be based on the real workload, not a dev test). JITs can notice things like paths never taken and compile the code into a set of sanity checks and a single completely inlined tight codepath for the common case, and when sanity checks fail, fall back to normal code. JIT does, however, use cycles at runtime to do the actual compile. While I have seen benchmarks where JIT wins over AoT, I'd still bet that the balance favors traditional compilers.

Peaker 16y ago

Sounds great. I'm a very big Haskell fan. I'd love to point people to this when trying to convey some advantages of Haskell. To make it more compelling, can you expand some on the downsides and maybe obstacles you encountered? The thing I'm unsure about, is how difficult it would be for (very) talented developers to just jump in. We have really talented developers, and everyone is super time-constrained, so many are wary of diving into a language as different as Haskell. Was it hard for your developers to figure Haskell out? Did your previous use of Scala help? How long did it take them to dive into Scala?

jamwt 16y ago
I would say the two real barriers to writing effective Haskell projects are a.) "getting" monads, and b.) understanding the implications of laziness, especially with regard to space leaks and unconsumed thunks. Everything else isn't that big of a deal. It's all much easier to digest, though, even for "really talented developers", if they have some experience with another functional language first. OCaml is a nice stepping stone before digging into the abstractions involved in understanding Haskell's powerful type system. Scala is good too, but having the object stuff mixed in there can lead you to rely on some patterns that aren't going to be available in a non-OOP language. I think the scheme/clojure path isn't bad either, but it's probably ideal to spend some time in the "statically typed" wing of the functional universe before going to Haskell.
- grav1tas 16y ago
  I agree. I dove into Haskell without doing any of the prior, and it was like running into a brick wall. However, persistence paid off in my case, but I do wonder how I would have handled it if I would have spent time with OCaml beforehand.
- samstokes 16y ago
  Could you say more about why "getting" monads was needed? I came to Haskell with no understanding of monads, started writing code, and eventually used my knowledge of Haskell to learn about monads. Not understanding monads just meant I was lacking a useful design pattern, and found certain API docs confusing, but it didn't stop me from writing reasonable code in most circumstances. On the other hand what you describe in your (awesome) blog post is a more significant Haskell project than any I've worked on, so I'd be interested to hear your experience. I've not really written my own monad, or properly looked into monad transformer stacks, and I'm aware that I could probably clean up a lot of code using them - is that the sort of thing you mean?

ynniv 16y ago

Are the logs being read from disk? In my experience, python is highly optimized for reading (possibly compressed) files from disk. If your infrastructure keeps logs in memory, python will lose this advantage and compete on computational performance where Haskell has the advantage. This is important for those of us who grind logs on disk and might be considering a language switch.

jamwt 16y ago
Nope, this is a process that BLPOP's logs from some Redis queue, does some processing on them, then writes them to disk.
enneff 16y ago
What do you mean by optimized? Python makes the same read and write syscalls everyone else does. What you're probably observing is Python's slow code generation being masked by the inherent slowness of I/O.
- ynniv 16y ago
  Python makes the same read and write syscalls everyone else does Except, when python's pants are on, it makes gold records. I haven't looked to see if there are any explicit optimizations, but your statement is ridiculous; an effective IO strategy can have an enormous effect on performance.
  - enneff 16y ago
    I'm sorry, what? Just being disagreeable is not an answer. What does "IO strategy" mean? You are being incredibly vague and unhelpful. Reading data from a file handle into a buffer is a trivial operation. It's what you do with that data afterwards that is important. In C (or Go) you have complete control over what happens next. As for Python, I don't know what happens, but I don't see how it could possibly be more efficient than any other sane language.
    - ynniv 16y ago

aristus 16y ago

Awesome work. If you haven't heard about Tim Bray's WideFinder challenge, it was really interesting. http://tartarus.org/james/diary/2008/06/17/widefinder-final-results http://tartarus.org/james/diary/2008/06/17/widefinder-final-...

Locke1689 16y ago

The author is mostly write about the usage cases of Haskell, but simply "systems" is a bit misleading because there are certain performance characteristics of lazy programs which make them bad choices for some systems programs. Any type of real-time system, for example, can suffer unpredictable performance in critical sections, which is pretty undesirable.

awj 16y ago
Not to argue the example, but Python's garbage collection disqualifies it for real-time systems as well. In fact, I'm having a hard time find a "system" task for which Python (as a language) is qualified by Haskell is not.
- Locke1689 16y ago
  Python is not a systems programming language.
  - danieldk 16y ago
    Maybe not, but it is certainly used as one. Everything from package managers (yum) to operating system installers (Anaconda) have been written in Python. Besides that, the grandparent is right: possibly every situation where Python was used as a systems programming language, Haskell could fit in (and more).
    - Locke1689 16y ago
      I think you're confused to what "systems" programming entails. User-space packaging with a bunch of scripts is not systems programming. OS installers is not systems programming. To give you an example of what is systems programming, I have helped developed operating systems kernels, virtual machine monitors, and distributed networked systems. All of these things would be considered systems programming. See http://en.wikipedia.org/wiki/System_programming http://en.wikipedia.org/wiki/System_programming for more information.

kordless 16y ago

I'd be interested in hearing more about how the author is using the resulting data set. Doing extractions at event generation time can be very useful if you know what you are after in advance, but not so good for adhoc analysis. Any reason why you didn't use Hadoop for this, then run batch jobs to extract summaries?

jamwt 16y ago
Yeah, the whole pipeline is actually quite more faceted than can be deduced from this summary. This stage actually just persists the events into a consolidated transaction log. Then, there are secondary processes that scan these transaction logs (in batch) and distribute data into various databases for system, business, and user analytics. I can't go into too much detail there, but the actual digesting and reporting side is more involved.
- kordless 16y ago
  I'd like to hear more about the use case if you have time, and can talk about it. I'm kordless at loggly dot com.