4 ms·
For the every day when I want to make embarrassingly parallel operations in Python go fast I find joblib to be a pretty good solution. It doesn't work for every
by tiger10guy 12y ago
For the every day when I want to make embarrassingly parallel operations in Python go fast I find joblib to be a pretty good solution. It doesn't work for everything, but it's quick and simple where it does work.
https://pythonhosted.org/joblib/ https://pythonhosted.org/joblib/
- shogunmike 12y agoI was going to discuss Parallel Python (http://www.parallelpython.com/ http://www.parallelpython.com/) in the next article - have you used that? How does it compare to joblib?
- tiger10guy 12y agoI haven't used that, but it looks interesting. After a brief look it seems like they both submit jobs to Python interpreters started up in other processes. Parallel Python (PP) seems to have a clunkier API, but also more functionality. I think the biggest advantage is that it can distribute jobs over a cluster instead of just different cores on the same machine. I might look into PP if I need to do things on a cluster, but I think I'll still stick with joblib when I'm on one machine. That's just my first impression. I'd be interested to read your blog post.
- harlowja 12y agoFeel free to checkout http://docs.openstack.org/developer/taskflow/ http://docs.openstack.org/developer/taskflow/ (it has a similar set of concepts to both of the mentioned libraries here, joblib and parallel python).