5 ms·

Do you think that Python libraries or a comprehensive development push will overcome barriers to entry for statistical process platforms? What if the greater ad

Do you think that Python libraries or a comprehensive development push will overcome barriers to entry for statistical process platforms? What if the greater adoption of these techniques is more limited by communities of users than the developers?

wesm 16y ago
Depends on what you're referring to by "statistical process platforms". Python has been quite popular in the machine learning communities for several years, with some pure Python libraries (e.g. nltk, scikits.learn), and many wrappers to non-Python libraries (Orange, Shogun, and I'm sure some others). One issue at the moment is that there's an R package for just about anything you would want to do. But the issue with R is its software development tools (which are not very good) and its ability to integrate with other systems. As far as mainstream frequentist statistics in Python, there's still a ways to go. scikits.statsmodels has made a lot of headway the last couple years in implementing standard linear (regression) modeling classes and econometrics tools, with some time series analysis and other things thrown in (I'm actually actively working on adding time series analysis functionality to that library). My general view is that given how friendly a language Python is for scientific programmers and the vibrancy of the community (in terms of new tools being built, etc.), Python could become a permanent fixture for statistical computing. Add on top of that things like integrated distributed computing inside IPython and projects like PyCUDA (which could make high performance statistical computing much more accessible), it's looking very good. But there's a slight chicken-and-egg problem at the moment. I should also point out that R is sort of a frankenstein of half-baked object oriented ideas and remnants of the Splus era. So if you want to design extensible classes for statistical models, it can be quite challenging.
- chillaxn 16y ago
  Not being a scientific programmer, I'm not exposed to the vibrancy of the community that you refer to. Would you recommend any websites where I could follow some discussions on the topic to get a better understand of the available community resources? I will look into Python and its libraries for the research I perform.
  - wesm 16y ago
    Take a look at some of the really cool stuff at the last few SciPy conferences: http://conference.scipy.org/proceedings/SciPy2009/ http://conference.scipy.org/proceedings/SciPy2009/ http://conference.scipy.org/scipy2010/schedule.html http://conference.scipy.org/scipy2010/schedule.html also, consider joining the mailing lists I listed above (numpy-discussion, scipy-user). Other smaller lists are: pystatsmodels, lists for Cython: www.cython.org and the associated Sage project.
deleted 16y ago
[deleted]
hyperbovine 16y ago
I have often wondered when something Python-ish will come along to compete with R. I like the power of R, and there is no better choice if you require an esoteric statistical estimator, but there are also a lot of downsides. It's difficult to debug, the interpreter seems flaky, and parts of the SPLUS syntax feels dated. Also, I know it claims to be object oriented, but I have never really understood the OO system in R. So much more time and effort is being poured into improving the Python runtime, that it seems silly not to try to build something on top of it. I am starting a stats PhD in the fall and have about nine months to kill between now and then. I'm seriously considering devoting all that time to building something like this.
- chillaxn 16y ago
  I also have some time to kill before finishing my degree. As an entrepreneur, I'm looking for supportive platforms that can incorporate empirical research in rapid development. For instance, I really treasure the rapid development capable in lisp (particularly PLT Scheme for home-brewed libraries). My market research methodology is focussed around measurables and "surveys". Do you know of any resources where one could learn about statistical data processing in a rapid development environment? Perhaps more specifically: smooth dataflow between processes? Best of luck on your PhD.
- wesm 16y ago
  I'm actually working on a stats PhD (in the early stages) and I've made it my goal to build a lot of Python software as I go along. So if you have the inclination I'd recommend giving it a shot. When you consider the wealth of tools out there for both high level (NumPy-based) and lower-level computation (e.g. Cython, for speeding up algorithms), and software development (especially interactive debugging and testing), it's a fairly compelling proposition (for me, at least). But long story short is that people are working on making Python more amenable to applied statistics work. And the more people working toward that goal, the faster we'll get there. So I would recommend: join numpy-discussion and scipy-user mailing lists, explore projects out there-- last few SciPy conferences are a decent place to start.
  - hyperbovine 16y ago

wesm 16y ago

Depends on what you're referring to by "statistical process platforms". Python has been quite popular in the machine learning communities for several years, with some pure Python libraries (e.g. nltk, scikits.learn), and many wrappers to non-Python libraries (Orange, Shogun, and I'm sure some others). One issue at the moment is that there's an R package for just about anything you would want to do. But the issue with R is its software development tools (which are not very good) and its ability to integrate with other systems. As far as mainstream frequentist statistics in Python, there's still a ways to go. scikits.statsmodels has made a lot of headway the last couple years in implementing standard linear (regression) modeling classes and econometrics tools, with some time series analysis and other things thrown in (I'm actually actively working on adding time series analysis functionality to that library). My general view is that given how friendly a language Python is for scientific programmers and the vibrancy of the community (in terms of new tools being built, etc.), Python could become a permanent fixture for statistical computing. Add on top of that things like integrated distributed computing inside IPython and projects like PyCUDA (which could make high performance statistical computing much more accessible), it's looking very good. But there's a slight chicken-and-egg problem at the moment. I should also point out that R is sort of a frankenstein of half-baked object oriented ideas and remnants of the Splus era. So if you want to design extensible classes for statistical models, it can be quite challenging.

chillaxn 16y ago
Not being a scientific programmer, I'm not exposed to the vibrancy of the community that you refer to. Would you recommend any websites where I could follow some discussions on the topic to get a better understand of the available community resources? I will look into Python and its libraries for the research I perform.
- wesm 16y ago
  Take a look at some of the really cool stuff at the last few SciPy conferences: http://conference.scipy.org/proceedings/SciPy2009/ http://conference.scipy.org/proceedings/SciPy2009/ http://conference.scipy.org/scipy2010/schedule.html http://conference.scipy.org/scipy2010/schedule.html also, consider joining the mailing lists I listed above (numpy-discussion, scipy-user). Other smaller lists are: pystatsmodels, lists for Cython: www.cython.org and the associated Sage project.

deleted 16y ago

[deleted]

hyperbovine 16y ago

I have often wondered when something Python-ish will come along to compete with R. I like the power of R, and there is no better choice if you require an esoteric statistical estimator, but there are also a lot of downsides. It's difficult to debug, the interpreter seems flaky, and parts of the SPLUS syntax feels dated. Also, I know it claims to be object oriented, but I have never really understood the OO system in R. So much more time and effort is being poured into improving the Python runtime, that it seems silly not to try to build something on top of it. I am starting a stats PhD in the fall and have about nine months to kill between now and then. I'm seriously considering devoting all that time to building something like this.

chillaxn 16y ago
I also have some time to kill before finishing my degree. As an entrepreneur, I'm looking for supportive platforms that can incorporate empirical research in rapid development. For instance, I really treasure the rapid development capable in lisp (particularly PLT Scheme for home-brewed libraries). My market research methodology is focussed around measurables and "surveys". Do you know of any resources where one could learn about statistical data processing in a rapid development environment? Perhaps more specifically: smooth dataflow between processes? Best of luck on your PhD.
wesm 16y ago
I'm actually working on a stats PhD (in the early stages) and I've made it my goal to build a lot of Python software as I go along. So if you have the inclination I'd recommend giving it a shot. When you consider the wealth of tools out there for both high level (NumPy-based) and lower-level computation (e.g. Cython, for speeding up algorithms), and software development (especially interactive debugging and testing), it's a fairly compelling proposition (for me, at least). But long story short is that people are working on making Python more amenable to applied statistics work. And the more people working toward that goal, the faster we'll get there. So I would recommend: join numpy-discussion and scipy-user mailing lists, explore projects out there-- last few SciPy conferences are a decent place to start.
- hyperbovine 16y ago