8 ms·

Why Is NumPy Only Now Getting Funded?

358 points by numfocusfnd 9y ago

js8 9y ago
"The problem of sustainability for open source scientific software projects is significant." Yeah, William Stein can tell these stories too: http://sagemath.blogspot.cz/2015/09/funding-open-source-mathematical.html http://sagemath.blogspot.cz/2015/09/funding-open-source-math...
- kronos29296 9y ago
  Yeah, every open source scientific software seems to have an uphill task before it gets recognized. Sagemath more so than others because of its scale and having a huge code base. Having to integrate multiple open source projects isn't helping either. But its a fine project and one that I use. For a collection of other projects it works better than most other individual projects. "Every successful science library has the ashes of an academic career in the making" - Same guy
rjbwork 9y ago
We have this problem in the .NET world. Accord.NET is written by a brilliant academic and programmer. It's well written, and has a good API, but it is largely the effort of this one dude, with minor contributions from a smattering of other fellows. Again, it is great in general, but it has bugs and rough edges here or there, and a lot of people don't trust it for production. I wish there was a way for people to be properly compensated for building and maintaining such vital scientific and mathematical computing software.
- devoply 9y ago
  That's because people don't actually value free things, until they come to depend on them, and then there is no way for that to continue unless they pay. And even then they will pay the bare minimum to keep it going. That's the whole psychology behind open source. Companies could pay for it and those using it can more than afford to do so, but it's free, and apparently it's going to stay that way so why not ride that gravy train?
  - deleted 9y ago
    [deleted]
- hiram112 9y ago
  .Net had always had a problem with open source because Microsoft implemented APIs and kept them closed source. Users ended up paying for it via licensing, MSDN, and Visual Studio. I often see a lot of really mice Java frameworks with a .Net clone which gets much less attention (Hibernate, some of the Spring stuff, etc). Whole I understand Microsoft is going their best to open up .Net, I'm hesitant to believe they'll ever get to the point where original implementations are done in it, and then ported to other langs - it just has a deservedly bad rep.
  - rjbwork 9y ago
    I started programming C# professionally in 2013...the ecosystem has grown and been improved by leaps and bounds since then. I'm hopeful, personally. I think one thing that keeps C# things from being implemented elsewhere is that there are just very different styles between even C# and Java. Some fancy libraries will do lots of reflectiony things, build syntax trees, and compile on the fly for performance, allowing complex configuration to be simplified and get fast code at runtime. There's also a big culture of LINQ/functional style, and fluent interfaces in libraries, etc. which doesn't seem to be present in something like the Java community (at least until streams), whereas more dynamic/weakly typed languages don't have, for obvious reasons, a culture of strong typing and letting the compiler do the work for you. Then when you go to the hardcore functional languages like Haskell, ML, etc. you don't have a culture of OO much, not to mention they just aren't used much in production environments. So, you see a lot of things from elsewhere ported and brought to C#/.NET, but not a lot of things moving from C# to other ecosystems.
santaclaus 9y ago
It blows my mind that NumPy is just getting funding. How did the Eigen (used in TensorFlow, among other things) folks keep it going?
- ajtulloch 9y ago
  A few of the Eigen core devs (Benoit Jacob, Benoit Steiner, ...) work at Google.
mattfrommars 9y ago
Ok, so the problem seems to be 'lack of maintainer' or could be stretched to contributer. The article later linked to https://www.slideshare.net/NadiaEghbal/consider-the-maintainer https://www.slideshare.net/NadiaEghbal/consider-the-maintain... which kind of reminded me a problem which I'm facing. After getting through basic of "Introduction to Computer Science Using Python" and forever pending goal to become a "Python Developer", is anyone here who is experience in Python willing to be my mentor? In return, free Python labor. :)
- pc2g4d 9y ago
  Ask me questions and I'll do my best to respond
  - mattfrommars 9y ago
    Want to develop a Github profile which could be very attractive to Python related job. How do I get started on it? People say "do something new and solve an existing problem with Python" But is EVERYONE solving a new problem which hasn't come before to get a Python Job? I just want to become a potential for Python Junior Dev job.
    - pc2g4d 9y ago
      I think the basic idea is to show through your profile that you're a competent developer. The project you pick doesn't necessarily need to do anything new and amazing, it just needs to be your work. You could make a task-list app (which has been done a million times) but if you do it well then you've achieved your goal. Of course, the best outcome would be to make something that other people want to use. This would get attention and further validation. So if you find a project you're passionate about---the "I built it because nobody had yet" thing---then that would be best. Good luck!
- webmaven 9y ago
  What sort of things ("Python development" is a bit nebulous) are you most interested in?
jordigh 9y ago
Also a problem for Octave. Remember this? https://news.ycombinator.com/item?id=13603575 https://news.ycombinator.com/item?id=13603575 It also stings a little when people say that it's completely obsolete because Matlab itself is "legacy" and we should all be abandoning the language, Octave included... and yet, even though I like numpy and Python and matplotlib and Julia and R, I still find myself reaching for Octave whenever I need a quick visualisation of some data.
- in9 9y ago
  Really? Why is that? In my mind, it is tough to beat ggplot
  - jordigh 9y ago
    Partly it's more familiarity with Octave than ggplot, but partly also that the Octave syntax is just about the easiest there is out there. Type something, immediately see some graphs. Type some more, gradually modify that graph. The simplicity is a big selling point. Julia's syntax comes pretty close, but I'm not as familiar with it yet. Remember that Matlab's and Octave's biggest audience consists of not-a-programmer programmers, that is, people who consider using Matlab or Octave as being something different than using a "real" programming language. I call them not-a-programmer because that's what I've often heard them call themselves, even as they write Matlab or Octave code.
  - mattkrause 9y ago
    Matlab/octave also include everything but the kitchen sink, especially if you're on an academic license. You can certainly replicate this with a python stack, but it involves a fair number of external libraries.
- maliker 9y ago
  You might like matplotlib for visualization. It's a Python clone of Matlab's plotting tools. And it's compatible with numpy if you need to do linear algebra or operate on big datasets.
  - 9y ago
jeremynixon 9y ago
This is still an important problem for numerical compute in other languages. It's a struggle to do data analysis and write machine learning applications in Scala, Java, C++, etc. due to a lack of Numpy / Pandas style ease of use and functionality.
- deleted 9y ago
  [deleted]
projectramo 9y ago
Q: Are we conflating two issues? Is there a difference between the "sole developer problem" and the "lack of funding" problem. I mean, even if a project finds funding, does it follow that it will attract more talented developers? One way to distinguish the two issues is to look at for-profit software. In the cases where there is one primary developer, do they find it easy to keep the software going when the person retires? I ask this because, I think, beyond the very real monetary issue, there is a question of how development works. Do we need one very talented individual who does the lion's share of the lifting?
- webmaven 9y ago
  > even if a project finds funding, does it follow that it will attract more talented developers? It depends on what the funding is used for. Outreach in various forms (better documentation, presenting at conferences, a nicer website, moderating community venues from mailing lists to chat channels, triaging and grooming issues and PRs, etc.) takes time and effort (and therefore needs to be funded) as well.
  - williamstein 9y ago
    For SageMath, what funding we’ve got has been mostly used for organizing many “Sage Days” workshop coding sprints – we’ve had nearly 100 now over the last decade! They have made an absolutely tremendous difference in attracting the over 600 people who have contributed. But also key design decision, e.g., 100% test coverage, helped too. I clearly remember the moment – in a big discussion at a Sage Days (funded by the Clay Math Institute) – that Craig Citro argued for 100% doctest coverage, and we all decided to do it. Without funding, those Sage Days wouldn’t have happened, and the community would not have developed. I also remember a Sage Days that we did jointly with Numpy at Enthought, where discussions and work we did together grew into many important things later. Funding is ridicuously absolutely critical to growing certain types of open source projects. In another direction, a lot of our infrastructure (e.g., build, testing, etc.,) is hosted on a big computer bought with a grant.
ArneBab 9y ago
Another project which is easy to overlook: Think about how many scientists use Emacs for most of their development and writing. But there is (to my knowledge) not even a single paid developer working on it. ( http://gnu.org/s/emacs http://gnu.org/s/emacs )
mtmail 9y ago
If your open source project needs funding there is https://opencollective.com/opensource https://opencollective.com/opensource (currently waiting, I'm not affiliated)
- SwellJoe 9y ago
  Look at the numbers before assuming this is a solution to funding Open Source. I'm not saying it's a bad thing, just that the amounts it's bringing in for most projects is effectively like buying the developer a cup of coffee. It's not a sustainable sum for someone to live on, in almost all cases. Right now, Webpack has $99k; which is respectable. That's a decent annual salary for a full-time developer that lives anywhere other than Silicon Valley or NYC. But, it falls off a cliff after that. Next is MochaJS at 16k; that's two months salary, maybe three or four in cheap places. After the top twenty projects, you're closer to nothing than you are to something. The time it takes to set it up would be worth more than it'll return (there are lots of $0 projects in the list). Maybe it'll improve with time. I dunno how long this project has been going on, and it looks really well done. It's just that the numbers are abysmal. I hope it'll improve. I wish we had a way to fund our software without having to sell a commercial version...but, it's so far from realistic, we can't really even consider it. I've been making my living from OSS for ~20 years now, and it's the great curse and tragedy of the thing that because the software is free, there's an incredibly pervasive belief that tiny sums are sufficient to keep a project afloat. We tried, very early on (~14 years ago now), to crowd-fund some major enhancements to our software...we raised a total of about $15,000 (this was long before crowd-funding was a thing), which to a lot of people seemed like a lot of money, and it caused a lot of folks to feel extraordinarily entitled to specific results (e.g. because they contributed $200, they assumed it would mean we'd develop a big feature they were the only person asking for, and that would take days or even weeks to build and test). But, for two developers working full-time on something, $15,000 is practically nothing. Most developers just don't have the freedom to trade in a full-time job paying market rates for a sub-poverty annual wage. Realistically, we couldn't commit full-time to the project until we started a business based on it that brought in predictable, recurring revenue. And, we couldn't effectively do that without providing things that the OSS version didn't have. OSS is a really tough way to make a living, is what I'm trying to say, and I don't know a lot of people who do it successfully without being employed by someone that does a lot of non-OSS stuff, too.
largote 9y ago
Because it's an important tool for Machine Learning, which makes money from that (of which there's plenty going around right now) flow into it.
ssivark 9y ago
There is a long-standing problem in open source software, which is that there is no "business model" associated with funneling resources to people putting significant effort into it. Setting up a consulting business to monetize software creates the perverse incentive to make software harder to use, but there seem to be some examples where this model has worked out reasonably. Open source projects are typically started by people working in the field, who have a strong urge to scratch some itch. Even if we find a way to find money for them to work full-time, they often don't have the desire to "productize" software, or to create/nurture/govern an organization around bringing together different stakeholders who might be able to use, or contribute to the software. (We got really lucky with Linus+Linux)
danjoc 9y ago
>the entire scientific Python stack was essentially relying upon on the “free time” work of only about 30 people—and no one had funding! 30 people? I remember a time when a certain fruit company would enter a field, literally hire all 30 of those guys, and put them behind closed doors. Then in 2 years they'd dominate the field for the next decade. Are these guys turning down offers? Or is the fruit company that poorly managed now?
- mschaef 9y ago
  I don't know who you're talking about, but Orange Micro went out of business in 2004. ( https://en.wikipedia.org/wiki/Orange_Micro https://en.wikipedia.org/wiki/Orange_Micro )
  - JPKab 9y ago
    Not sure if you're being sarcastic, but he means Apple.
    - danjoc 9y ago
      I think he knows. Mike has a 5 digit /. ID :)
      - mschaef 9y ago
        Heh... I knew my Slashdot ID was pretty low (~32K), but had no idea I'd been around here that long. Now I feel old. :-) Speaking of old, I used Reddit as my main news source for a while, and I remember /r/programming having a post celebrating 65,536 users. Now it's over 10 times that, and there are more than three subreddits.
    - mschaef 9y ago
      It was more an attempt at light humor than sarcasm. :-) (Not to mention I thought it was hard to imagine anybody around here being unaware of the most capitalized company in the world. Aside from maybe Saudi Aramco.)
- 9y ago
carapace 9y ago
tl;dr: > "And if you’d like to take action to contribute to project sustainability, consider becoming a NumFOCUS member today." https://www.numfocus.org/community/donate/ https://www.numfocus.org/community/donate/
mschaef 9y ago
It's surprising to me that people are surprised by this. Even setting aside the fact that the people that can do this work are few in number, the vast majority of people need a way to support themselves and their family. If the number of people that have these skills is low, the subset that is both altruistic enough to donate them for a sufficient period of time and personally able to do so must be vanishingly small. (And the negative feedback a lot of OSS maintainers receive doesn't help.) Companies have the same issue... there has to be a fairly direct connection between an expenditure (paying developers) and a return on that investment. That can be a very difficult argument to make.
anigbrowl 9y ago
Because capitalism is an inherently exploitative economic paradigm?
alannallama 9y ago
This is exactly the problem Open Collective[0] exists to solve. Often times, there are people who want to financially support a given open source project, but there is no channel by which to do so. Creating the financial channel is the first step toward a much needed culture change where the assumption is you will support the open source you rely on, especially if you're making money off it. [0] http://www.opencollective.com http://www.opencollective.com
travisoliphant 9y ago
Original NumPy author here. I have a lot to say on this topic, given that it has literally consumed my life over the past 20 years. You can go here for some thoughts about some of this: http://technicaldiscovery.blogspot.com/ http://technicaldiscovery.blogspot.com/ There are several articles there that relate but in particular http://technicaldiscovery.blogspot.com/2012/10/continuum-and-open-source.html http://technicaldiscovery.blogspot.com/2012/10/continuum-and... and http://technicaldiscovery.blogspot.com/2017/02/numfocus-past-and-future.html http://technicaldiscovery.blogspot.com/2017/02/numfocus-past... I knew what I was getting into when I wrote NumPy. I knew there was not a clear way to support my family by releasing open source software, and I knew I was risking my academic career. I did it because I believed in the wider benefit of ideas that can be infinitely shared once created and the need for software infrastructure to be open-source --- especially to empower the brightest minds to create. I did it because others had done it before me and I loved using the tools they created. I hoped I would inspire others to share what they could. There have been a lot of people who have helped over the years. From employers willing to allow a few hours here and there to go to the project, to community members willing to spend nights and weekends with you making things work, to investors (at Continuum) willing to help you build a business centered on Open Source. There are many people who are helping to fix the problem. In 2012, I had two ideas as to how to help. Those who know me will not be surprised to learn that I pursued both of them. One was the creation of NumFOCUS that is working as a non-profit to improve things. The second was the creation of Continuum (http://www.continuum.io http://www.continuum.io) to be a company that would work to find a way to pay people to work on Open Source full-time. We have explored several business models and actually found three that work pretty well for us. One we are growing with investors, a second we are continuing with, and another we are actually in the process of helping others get started with and ramping down on ourselves. Along the way, I've learned that open source is best described in the business world as "shared R&D". To really take advantage of that R&D you need to particpate in it. We call our group that does that our "Community Innovation" group. We have about 35 people in that group now all building open-source software funded via several mechanisms. We are looking for people to help us continue this journey of growing a company that resonantly contributes significantly to Open Source as part of its mission. If you are interested, contact me --- I am easy to track down via email.
Q6T46nT668w6i3m 9y ago
I believe I’m one of (or near) the “top 30” contributors (I’ve made substantial contributions to all of the aforementioned packages), and I’m funded to write scientific software. I’m extremely fortunate. Unfortunately, like so many things, I suspect it has everything to do with pedigree (e.g. my lab, my institution, my peers, etc.) rather than my (or my coworkers) exact contributions. In fact, I don’t know if any of my lab’s grants has ever explicitly mentioned our contributions to one of the discussed packages. However, this could change. I’m extremely encouraged, for example, by the comments from new institutions like OpenAI or the Chan Zuckerberg Initiative about the necessity of funding software.
deleted 9y ago
[deleted]
kem 9y ago
I appreciate this article being posted, and have the utmost respect for NumPy developers. The urgency and discrepancy between use of certain important open-source libraries, and their support, is bewildering sometimes. As I was thinking about it, though, I'm not surprised NumPy hasn't been funded before. The reasons why say a lot about biases in memory. It wasn't that long ago that the sorts of things NumPy does were seen as fairly niche, and in the domain of statistics or engineering. It's only with relatively recent interest in AI and DL that this has been seen as within the purview of Silicon Valley-comp sci-type business, as opposed to EE or something different. I still am kind of a little disoriented--the other day, looking through our university's course catalog, I realized that certain topics that would have been taught in the stats or psychology departments are now being seen as the territory of comp sci. Statisticians have written excoriations about being treated as if they don't exist, as comp sci blithely barrels forward, reinventing the wheel. I'm not meaning to take sides with these issues, only pointing out that I think the world we live in was very different not so long ago. It might seem puzzling that NumPy hasn't had more funding, but I think that's in part because what it's most profitably used for now wasn't really seen as much more than academic science fiction not too long ago. The other part of it too, is that until relatively recently, if you were to do numerical heavy lifting, you'd almost certainly be expected to do that in C/C++ or maybe Fortran. There's a tension in numerical computing, between the performance and expressiveness that's needed, and Python is on one end of that continuum, far from the end that is traditionally associated with complex numerical computing. Sure, you had things like MATLAB with Python in the same functional role, but those were largely seen as teaching tools, or something that engineers did for one-off projects, having learned to do that in school (I still think the use of python in ML derives from the use of Python as a teaching tool in uni). I'm not trying to knock Python or NumPy or anything, just kind of trying to convey a different perspective, which is that I can remember a time not too long ago when the use of Python in numerics was seen as primarily didactic in nature, or for limited circumscribed applications. FWIW, it seems to me Python is kind of on a path similar to what happened with javascript, which was treated as kind of an ancillary helper language on the web, until Google started pushing its limits. Then there was browser wars 2.0, and huge efforts put into javascript, and it became a main player in network computing. To me, there's a similar trend with Python: it really kind of existed as a language for prototyping and scripting tasks, and now finds itself in a different role than it has been used for traditionally, and projects in that area are getting an influx of money accordingly. What I see happening is (1) a blossoming diversity of numerical computing communities (Haskell, Python, Julia, Kotlin, Scala, Rust, Go, etc.), due to competition and variation in application scenarios and preferences, (2) a huge influx of resources being put into Python to make it more performant, or (3) people jumping ship from Python into one of those other platforms to get more bang-for-the buck [or (4) some combination of all of these.]
Radim 9y ago
Is the idea that "foundational work" (in any field) can be done without "huge sacrifices" widely accepted? It sounds a tad unrealistic to me, unsupported by history. It's as if people want to have it both ways: Create innovative SW, but also don't take risks or make sacrifices. Offer software "for free" (and belligerently oppose even something like GPL), but also get paid (preferably by the government, so the people actually footing the bill have no say in it) and be long-term sustainable. What's next: get paid, but also don't pay income taxes? Give away project control, but also keep it? :) All understandable desires, but a little schizophrenic. Disclaimer: I am a big fan of open source and NumPy in particular. I mentor students and OSS newcomers, I even pay one full-time dev to work only on OSS. It's just that I try not to kid myself about where the time&money comes from and where it goes, and I try not to have random people pay for my hobbies. Extremely relevant previous HN conversion on this topic: https://news.ycombinator.com/item?id=14551355 https://news.ycombinator.com/item?id=14551355
- coldtea 9y ago
  Who said people want to do "foundational work"? The just want to write some specific software they enjoy writing together. The question rather is: after said software has been proven valuable and widely used by companies, and when there remain some hard and/or non-enjoyable parts to be done to complete it/enhance it (specific drivers, documentation, advanced features, etc), why don't any of the users donate some money to help?
  - Radim 9y ago
    That's an easy one -- because they don't have to :-) In some cases, "paying extra" without being asked to might even be compromising your fiduciary duties. I feel the open source movement has spoilt its "user base" a little. There's very little expectation of reciprocity. Sometimes even outright hostility toward OSS modes that try work with reciprocity (check out the linked HN thread above). This mindset might be hard to reverse at this point. As with all "communal sharing" experiments, the true crunch time comes as the economic reality sets in.
  - dagw 9y ago
sandGorgon 9y ago
Because developers generally don't know (or don't like) the outreach necessary to fundraise. For example in numpy case - https://github.com/numpy/numpy.org/issues/9 https://github.com/numpy/numpy.org/issues/9 That's a request in March 2017 to add a donation button to the website. Im not sure that 6 months back, if Numpy was legally structured to receive larger funding. I posted a similar comment (with many more replies) in the context of Octave and it's funding https://news.ycombinator.com/item?id=13604564 https://news.ycombinator.com/item?id=13604564 Tl;Dr Don't ask for donations - instead sell "gratitude-ware" There are tons of people who WANT to support these projects, but you have to make it easy and accountable to do that. The best example that I usually give is Sidekiq. @mperham is awesome that way "This is exactly why I disclosed my revenue: people won't know there's a successful path forward unless it's disclosed. I want more OSS developers to follow my lead and build a bright future for themselves based on their awesome software." In fact, I believe there's a start-up to be done here. "Stripe Atlas for Open Source software"
- numfocusfnd 9y ago
  NumPy joined NumFOCUS as a fiscally sponsored project in 2015, so it's been eligible to receive donations as a non-profit (because of NumFOCUS' nonprofit status) since then.
teekert 9y ago
I tried at work to get some money to the maintainer of iRAP, an RNA sequencing analysis pipeline we depend on heavily at the moment. But business sees this as wasted money, it's there for free, why not take it? Reading this, I think I'm going to double down on my efforts again. We get so much value out of a huge pile of FOSS software, we should be donating. Meanwhile we have spend piles of money on Matlab for years and we aren't even allowed to run Linux on our Laptops if we wanted to.
- Vinnl 9y ago
  I think someone on here recently suggested to a project that they provide an "enterprise subscription" or something like that with minor benefits to make it easier for businesses to justify (or maybe it made administration easier?) to "donate". You could suggest that to the maintainer, and then use that to convince business that you need that.
wodenokoto 9y ago
The "Every successful science library has the ashes of an academic career in the making" quote has been mentioned several times in the comments, so I thought I would give a plea to everybody who works in academia to help the people who build the foundational tools of your research by citing them in your papers: https://www.scipy.org/citing.html https://www.scipy.org/citing.html
- torrent-of-ions 9y ago
  Sad to say that I haven't cited these in papers that I've published so far. Much software is taken for granted, especially libraries. I don't often see papers mention SciPy specifically let alone add a citation. The author of GNU Parallel resorted to a message on stderr asking academics to cite it. But this isn't even the whole story. As we all know, designing and implementing a program or library is often just the first step. Someone then has to maintain that code. There is absolutely no incentive for anyone to maintain these. Institutions will fund new software because they expect a paper to be published about it. But nobody is paying anyone to maintain this stuff.
prewett 9y ago
I thought that Enthought sponsored a lot of NumPy development, kind of like as a corporate caretaker or something, is that not the case?
- acqq 9y ago
  I'd also like to read about what Enthought actually did. Anybody knows the details and willing to tell?
abousara 9y ago
Each country have its own taxes to sustain fundamentals, same thing must apply in software industry. Software engineers might be paid +100k$ while developing over open source languages/frameworks or libraries. That's is not fair, it is like riding a starving horse. A French point of view (after all, France invented VAT...) would suggest to introduce a taxe on software engineers salaries (1% ?) and redistribute this fund on most used languages/frameworks/libraries and use a part to sustain a new projects.
- adrianN 9y ago
  How about instead of taxing the workers we tax the companies that profit from open source efforts.
  - dagw 9y ago
    How would you measure how much of the profits a company makes come from open source efforts?
    - adrianN 9y ago
      You don't, you just use a fixed percentage of whatever business tax they pay to fund open source development.
      - dagw 9y ago
        How are you going to separate out companies that profit from open source software, from those that don't? Would the fixed percentage be same for a company that is built 100% on open source software as it would for one where a handful of people in one department occasionally use VS Code to write some python scripts? If you want to fund open source using tax money, then fund open source using tax money. Trying to implement a special Open Source Tax and apply it only to those that "profit from" open source seems both counterproductive and virtually impossible.
        adrianN 9y ago

358 points by numfocusfnd 9y ago

js8 9y ago

"The problem of sustainability for open source scientific software projects is significant." Yeah, William Stein can tell these stories too: http://sagemath.blogspot.cz/2015/09/funding-open-source-mathematical.html http://sagemath.blogspot.cz/2015/09/funding-open-source-math...

kronos29296 9y ago
Yeah, every open source scientific software seems to have an uphill task before it gets recognized. Sagemath more so than others because of its scale and having a huge code base. Having to integrate multiple open source projects isn't helping either. But its a fine project and one that I use. For a collection of other projects it works better than most other individual projects. "Every successful science library has the ashes of an academic career in the making" - Same guy

rjbwork 9y ago

We have this problem in the .NET world. Accord.NET is written by a brilliant academic and programmer. It's well written, and has a good API, but it is largely the effort of this one dude, with minor contributions from a smattering of other fellows. Again, it is great in general, but it has bugs and rough edges here or there, and a lot of people don't trust it for production. I wish there was a way for people to be properly compensated for building and maintaining such vital scientific and mathematical computing software.

devoply 9y ago
That's because people don't actually value free things, until they come to depend on them, and then there is no way for that to continue unless they pay. And even then they will pay the bare minimum to keep it going. That's the whole psychology behind open source. Companies could pay for it and those using it can more than afford to do so, but it's free, and apparently it's going to stay that way so why not ride that gravy train?
- deleted 9y ago
  [deleted]
hiram112 9y ago
.Net had always had a problem with open source because Microsoft implemented APIs and kept them closed source. Users ended up paying for it via licensing, MSDN, and Visual Studio. I often see a lot of really mice Java frameworks with a .Net clone which gets much less attention (Hibernate, some of the Spring stuff, etc). Whole I understand Microsoft is going their best to open up .Net, I'm hesitant to believe they'll ever get to the point where original implementations are done in it, and then ported to other langs - it just has a deservedly bad rep.
- rjbwork 9y ago
  I started programming C# professionally in 2013...the ecosystem has grown and been improved by leaps and bounds since then. I'm hopeful, personally. I think one thing that keeps C# things from being implemented elsewhere is that there are just very different styles between even C# and Java. Some fancy libraries will do lots of reflectiony things, build syntax trees, and compile on the fly for performance, allowing complex configuration to be simplified and get fast code at runtime. There's also a big culture of LINQ/functional style, and fluent interfaces in libraries, etc. which doesn't seem to be present in something like the Java community (at least until streams), whereas more dynamic/weakly typed languages don't have, for obvious reasons, a culture of strong typing and letting the compiler do the work for you. Then when you go to the hardcore functional languages like Haskell, ML, etc. you don't have a culture of OO much, not to mention they just aren't used much in production environments. So, you see a lot of things from elsewhere ported and brought to C#/.NET, but not a lot of things moving from C# to other ecosystems.

santaclaus 9y ago

It blows my mind that NumPy is just getting funding. How did the Eigen (used in TensorFlow, among other things) folks keep it going?

ajtulloch 9y ago
A few of the Eigen core devs (Benoit Jacob, Benoit Steiner, ...) work at Google.

mattfrommars 9y ago

Ok, so the problem seems to be 'lack of maintainer' or could be stretched to contributer. The article later linked to https://www.slideshare.net/NadiaEghbal/consider-the-maintainer https://www.slideshare.net/NadiaEghbal/consider-the-maintain... which kind of reminded me a problem which I'm facing. After getting through basic of "Introduction to Computer Science Using Python" and forever pending goal to become a "Python Developer", is anyone here who is experience in Python willing to be my mentor? In return, free Python labor. :)

pc2g4d 9y ago
Ask me questions and I'll do my best to respond
- mattfrommars 9y ago
  Want to develop a Github profile which could be very attractive to Python related job. How do I get started on it? People say "do something new and solve an existing problem with Python" But is EVERYONE solving a new problem which hasn't come before to get a Python Job? I just want to become a potential for Python Junior Dev job.
  - pc2g4d 9y ago
    I think the basic idea is to show through your profile that you're a competent developer. The project you pick doesn't necessarily need to do anything new and amazing, it just needs to be your work. You could make a task-list app (which has been done a million times) but if you do it well then you've achieved your goal. Of course, the best outcome would be to make something that other people want to use. This would get attention and further validation. So if you find a project you're passionate about---the "I built it because nobody had yet" thing---then that would be best. Good luck!
webmaven 9y ago
What sort of things ("Python development" is a bit nebulous) are you most interested in?

jordigh 9y ago

Also a problem for Octave. Remember this? https://news.ycombinator.com/item?id=13603575 https://news.ycombinator.com/item?id=13603575 It also stings a little when people say that it's completely obsolete because Matlab itself is "legacy" and we should all be abandoning the language, Octave included... and yet, even though I like numpy and Python and matplotlib and Julia and R, I still find myself reaching for Octave whenever I need a quick visualisation of some data.

in9 9y ago
Really? Why is that? In my mind, it is tough to beat ggplot
- jordigh 9y ago
  Partly it's more familiarity with Octave than ggplot, but partly also that the Octave syntax is just about the easiest there is out there. Type something, immediately see some graphs. Type some more, gradually modify that graph. The simplicity is a big selling point. Julia's syntax comes pretty close, but I'm not as familiar with it yet. Remember that Matlab's and Octave's biggest audience consists of not-a-programmer programmers, that is, people who consider using Matlab or Octave as being something different than using a "real" programming language. I call them not-a-programmer because that's what I've often heard them call themselves, even as they write Matlab or Octave code.
- mattkrause 9y ago
  Matlab/octave also include everything but the kitchen sink, especially if you're on an academic license. You can certainly replicate this with a python stack, but it involves a fair number of external libraries.
maliker 9y ago
You might like matplotlib for visualization. It's a Python clone of Matlab's plotting tools. And it's compatible with numpy if you need to do linear algebra or operate on big datasets.
- 9y ago

jeremynixon 9y ago

This is still an important problem for numerical compute in other languages. It's a struggle to do data analysis and write machine learning applications in Scala, Java, C++, etc. due to a lack of Numpy / Pandas style ease of use and functionality.

deleted 9y ago
[deleted]

projectramo 9y ago

Q: Are we conflating two issues? Is there a difference between the "sole developer problem" and the "lack of funding" problem. I mean, even if a project finds funding, does it follow that it will attract more talented developers? One way to distinguish the two issues is to look at for-profit software. In the cases where there is one primary developer, do they find it easy to keep the software going when the person retires? I ask this because, I think, beyond the very real monetary issue, there is a question of how development works. Do we need one very talented individual who does the lion's share of the lifting?

webmaven 9y ago
> even if a project finds funding, does it follow that it will attract more talented developers? It depends on what the funding is used for. Outreach in various forms (better documentation, presenting at conferences, a nicer website, moderating community venues from mailing lists to chat channels, triaging and grooming issues and PRs, etc.) takes time and effort (and therefore needs to be funded) as well.
- williamstein 9y ago
  For SageMath, what funding we’ve got has been mostly used for organizing many “Sage Days” workshop coding sprints – we’ve had nearly 100 now over the last decade! They have made an absolutely tremendous difference in attracting the over 600 people who have contributed. But also key design decision, e.g., 100% test coverage, helped too. I clearly remember the moment – in a big discussion at a Sage Days (funded by the Clay Math Institute) – that Craig Citro argued for 100% doctest coverage, and we all decided to do it. Without funding, those Sage Days wouldn’t have happened, and the community would not have developed. I also remember a Sage Days that we did jointly with Numpy at Enthought, where discussions and work we did together grew into many important things later. Funding is ridicuously absolutely critical to growing certain types of open source projects. In another direction, a lot of our infrastructure (e.g., build, testing, etc.,) is hosted on a big computer bought with a grant.

ArneBab 9y ago

Another project which is easy to overlook: Think about how many scientists use Emacs for most of their development and writing. But there is (to my knowledge) not even a single paid developer working on it. ( http://gnu.org/s/emacs http://gnu.org/s/emacs )

mtmail 9y ago

If your open source project needs funding there is https://opencollective.com/opensource https://opencollective.com/opensource (currently waiting, I'm not affiliated)

SwellJoe 9y ago
Look at the numbers before assuming this is a solution to funding Open Source. I'm not saying it's a bad thing, just that the amounts it's bringing in for most projects is effectively like buying the developer a cup of coffee. It's not a sustainable sum for someone to live on, in almost all cases. Right now, Webpack has $99k; which is respectable. That's a decent annual salary for a full-time developer that lives anywhere other than Silicon Valley or NYC. But, it falls off a cliff after that. Next is MochaJS at 16k; that's two months salary, maybe three or four in cheap places. After the top twenty projects, you're closer to nothing than you are to something. The time it takes to set it up would be worth more than it'll return (there are lots of $0 projects in the list). Maybe it'll improve with time. I dunno how long this project has been going on, and it looks really well done. It's just that the numbers are abysmal. I hope it'll improve. I wish we had a way to fund our software without having to sell a commercial version...but, it's so far from realistic, we can't really even consider it. I've been making my living from OSS for ~20 years now, and it's the great curse and tragedy of the thing that because the software is free, there's an incredibly pervasive belief that tiny sums are sufficient to keep a project afloat. We tried, very early on (~14 years ago now), to crowd-fund some major enhancements to our software...we raised a total of about $15,000 (this was long before crowd-funding was a thing), which to a lot of people seemed like a lot of money, and it caused a lot of folks to feel extraordinarily entitled to specific results (e.g. because they contributed $200, they assumed it would mean we'd develop a big feature they were the only person asking for, and that would take days or even weeks to build and test). But, for two developers working full-time on something, $15,000 is practically nothing. Most developers just don't have the freedom to trade in a full-time job paying market rates for a sub-poverty annual wage. Realistically, we couldn't commit full-time to the project until we started a business based on it that brought in predictable, recurring revenue. And, we couldn't effectively do that without providing things that the OSS version didn't have. OSS is a really tough way to make a living, is what I'm trying to say, and I don't know a lot of people who do it successfully without being employed by someone that does a lot of non-OSS stuff, too.

largote 9y ago

Because it's an important tool for Machine Learning, which makes money from that (of which there's plenty going around right now) flow into it.

ssivark 9y ago

There is a long-standing problem in open source software, which is that there is no "business model" associated with funneling resources to people putting significant effort into it. Setting up a consulting business to monetize software creates the perverse incentive to make software harder to use, but there seem to be some examples where this model has worked out reasonably. Open source projects are typically started by people working in the field, who have a strong urge to scratch some itch. Even if we find a way to find money for them to work full-time, they often don't have the desire to "productize" software, or to create/nurture/govern an organization around bringing together different stakeholders who might be able to use, or contribute to the software. (We got really lucky with Linus+Linux)

danjoc 9y ago

>the entire scientific Python stack was essentially relying upon on the “free time” work of only about 30 people—and no one had funding! 30 people? I remember a time when a certain fruit company would enter a field, literally hire all 30 of those guys, and put them behind closed doors. Then in 2 years they'd dominate the field for the next decade. Are these guys turning down offers? Or is the fruit company that poorly managed now?

mschaef 9y ago
I don't know who you're talking about, but Orange Micro went out of business in 2004. ( https://en.wikipedia.org/wiki/Orange_Micro https://en.wikipedia.org/wiki/Orange_Micro )
- JPKab 9y ago
  Not sure if you're being sarcastic, but he means Apple.
  - danjoc 9y ago
    I think he knows. Mike has a 5 digit /. ID :)
    - mschaef 9y ago
      Heh... I knew my Slashdot ID was pretty low (~32K), but had no idea I'd been around here that long. Now I feel old. :-) Speaking of old, I used Reddit as my main news source for a while, and I remember /r/programming having a post celebrating 65,536 users. Now it's over 10 times that, and there are more than three subreddits.
  - mschaef 9y ago
    It was more an attempt at light humor than sarcasm. :-) (Not to mention I thought it was hard to imagine anybody around here being unaware of the most capitalized company in the world. Aside from maybe Saudi Aramco.)
9y ago

carapace 9y ago

tl;dr: > "And if you’d like to take action to contribute to project sustainability, consider becoming a NumFOCUS member today." https://www.numfocus.org/community/donate/ https://www.numfocus.org/community/donate/

mschaef 9y ago

It's surprising to me that people are surprised by this. Even setting aside the fact that the people that can do this work are few in number, the vast majority of people need a way to support themselves and their family. If the number of people that have these skills is low, the subset that is both altruistic enough to donate them for a sufficient period of time and personally able to do so must be vanishingly small. (And the negative feedback a lot of OSS maintainers receive doesn't help.) Companies have the same issue... there has to be a fairly direct connection between an expenditure (paying developers) and a return on that investment. That can be a very difficult argument to make.

anigbrowl 9y ago

Because capitalism is an inherently exploitative economic paradigm?

alannallama 9y ago

This is exactly the problem Open Collective[0] exists to solve. Often times, there are people who want to financially support a given open source project, but there is no channel by which to do so. Creating the financial channel is the first step toward a much needed culture change where the assumption is you will support the open source you rely on, especially if you're making money off it. [0] http://www.opencollective.com http://www.opencollective.com

travisoliphant 9y ago

Original NumPy author here. I have a lot to say on this topic, given that it has literally consumed my life over the past 20 years. You can go here for some thoughts about some of this: http://technicaldiscovery.blogspot.com/ http://technicaldiscovery.blogspot.com/ There are several articles there that relate but in particular http://technicaldiscovery.blogspot.com/2012/10/continuum-and-open-source.html http://technicaldiscovery.blogspot.com/2012/10/continuum-and... and http://technicaldiscovery.blogspot.com/2017/02/numfocus-past-and-future.html http://technicaldiscovery.blogspot.com/2017/02/numfocus-past... I knew what I was getting into when I wrote NumPy. I knew there was not a clear way to support my family by releasing open source software, and I knew I was risking my academic career. I did it because I believed in the wider benefit of ideas that can be infinitely shared once created and the need for software infrastructure to be open-source --- especially to empower the brightest minds to create. I did it because others had done it before me and I loved using the tools they created. I hoped I would inspire others to share what they could. There have been a lot of people who have helped over the years. From employers willing to allow a few hours here and there to go to the project, to community members willing to spend nights and weekends with you making things work, to investors (at Continuum) willing to help you build a business centered on Open Source. There are many people who are helping to fix the problem. In 2012, I had two ideas as to how to help. Those who know me will not be surprised to learn that I pursued both of them. One was the creation of NumFOCUS that is working as a non-profit to improve things. The second was the creation of Continuum (http://www.continuum.io http://www.continuum.io) to be a company that would work to find a way to pay people to work on Open Source full-time. We have explored several business models and actually found three that work pretty well for us. One we are growing with investors, a second we are continuing with, and another we are actually in the process of helping others get started with and ramping down on ourselves. Along the way, I've learned that open source is best described in the business world as "shared R&D". To really take advantage of that R&D you need to particpate in it. We call our group that does that our "Community Innovation" group. We have about 35 people in that group now all building open-source software funded via several mechanisms. We are looking for people to help us continue this journey of growing a company that resonantly contributes significantly to Open Source as part of its mission. If you are interested, contact me --- I am easy to track down via email.

Q6T46nT668w6i3m 9y ago

I believe I’m one of (or near) the “top 30” contributors (I’ve made substantial contributions to all of the aforementioned packages), and I’m funded to write scientific software. I’m extremely fortunate. Unfortunately, like so many things, I suspect it has everything to do with pedigree (e.g. my lab, my institution, my peers, etc.) rather than my (or my coworkers) exact contributions. In fact, I don’t know if any of my lab’s grants has ever explicitly mentioned our contributions to one of the discussed packages. However, this could change. I’m extremely encouraged, for example, by the comments from new institutions like OpenAI or the Chan Zuckerberg Initiative about the necessity of funding software.

deleted 9y ago

[deleted]

kem 9y ago

I appreciate this article being posted, and have the utmost respect for NumPy developers. The urgency and discrepancy between use of certain important open-source libraries, and their support, is bewildering sometimes. As I was thinking about it, though, I'm not surprised NumPy hasn't been funded before. The reasons why say a lot about biases in memory. It wasn't that long ago that the sorts of things NumPy does were seen as fairly niche, and in the domain of statistics or engineering. It's only with relatively recent interest in AI and DL that this has been seen as within the purview of Silicon Valley-comp sci-type business, as opposed to EE or something different. I still am kind of a little disoriented--the other day, looking through our university's course catalog, I realized that certain topics that would have been taught in the stats or psychology departments are now being seen as the territory of comp sci. Statisticians have written excoriations about being treated as if they don't exist, as comp sci blithely barrels forward, reinventing the wheel. I'm not meaning to take sides with these issues, only pointing out that I think the world we live in was very different not so long ago. It might seem puzzling that NumPy hasn't had more funding, but I think that's in part because what it's most profitably used for now wasn't really seen as much more than academic science fiction not too long ago. The other part of it too, is that until relatively recently, if you were to do numerical heavy lifting, you'd almost certainly be expected to do that in C/C++ or maybe Fortran. There's a tension in numerical computing, between the performance and expressiveness that's needed, and Python is on one end of that continuum, far from the end that is traditionally associated with complex numerical computing. Sure, you had things like MATLAB with Python in the same functional role, but those were largely seen as teaching tools, or something that engineers did for one-off projects, having learned to do that in school (I still think the use of python in ML derives from the use of Python as a teaching tool in uni). I'm not trying to knock Python or NumPy or anything, just kind of trying to convey a different perspective, which is that I can remember a time not too long ago when the use of Python in numerics was seen as primarily didactic in nature, or for limited circumscribed applications. FWIW, it seems to me Python is kind of on a path similar to what happened with javascript, which was treated as kind of an ancillary helper language on the web, until Google started pushing its limits. Then there was browser wars 2.0, and huge efforts put into javascript, and it became a main player in network computing. To me, there's a similar trend with Python: it really kind of existed as a language for prototyping and scripting tasks, and now finds itself in a different role than it has been used for traditionally, and projects in that area are getting an influx of money accordingly. What I see happening is (1) a blossoming diversity of numerical computing communities (Haskell, Python, Julia, Kotlin, Scala, Rust, Go, etc.), due to competition and variation in application scenarios and preferences, (2) a huge influx of resources being put into Python to make it more performant, or (3) people jumping ship from Python into one of those other platforms to get more bang-for-the buck [or (4) some combination of all of these.]

Radim 9y ago

Is the idea that "foundational work" (in any field) can be done without "huge sacrifices" widely accepted? It sounds a tad unrealistic to me, unsupported by history. It's as if people want to have it both ways: Create innovative SW, but also don't take risks or make sacrifices. Offer software "for free" (and belligerently oppose even something like GPL), but also get paid (preferably by the government, so the people actually footing the bill have no say in it) and be long-term sustainable. What's next: get paid, but also don't pay income taxes? Give away project control, but also keep it? :) All understandable desires, but a little schizophrenic. Disclaimer: I am a big fan of open source and NumPy in particular. I mentor students and OSS newcomers, I even pay one full-time dev to work only on OSS. It's just that I try not to kid myself about where the time&money comes from and where it goes, and I try not to have random people pay for my hobbies. Extremely relevant previous HN conversion on this topic: https://news.ycombinator.com/item?id=14551355 https://news.ycombinator.com/item?id=14551355

coldtea 9y ago
Who said people want to do "foundational work"? The just want to write some specific software they enjoy writing together. The question rather is: after said software has been proven valuable and widely used by companies, and when there remain some hard and/or non-enjoyable parts to be done to complete it/enhance it (specific drivers, documentation, advanced features, etc), why don't any of the users donate some money to help?
- Radim 9y ago
  That's an easy one -- because they don't have to :-) In some cases, "paying extra" without being asked to might even be compromising your fiduciary duties. I feel the open source movement has spoilt its "user base" a little. There's very little expectation of reciprocity. Sometimes even outright hostility toward OSS modes that try work with reciprocity (check out the linked HN thread above). This mindset might be hard to reverse at this point. As with all "communal sharing" experiments, the true crunch time comes as the economic reality sets in.
- dagw 9y ago

sandGorgon 9y ago

Because developers generally don't know (or don't like) the outreach necessary to fundraise. For example in numpy case - https://github.com/numpy/numpy.org/issues/9 https://github.com/numpy/numpy.org/issues/9 That's a request in March 2017 to add a donation button to the website. Im not sure that 6 months back, if Numpy was legally structured to receive larger funding. I posted a similar comment (with many more replies) in the context of Octave and it's funding https://news.ycombinator.com/item?id=13604564 https://news.ycombinator.com/item?id=13604564 Tl;Dr Don't ask for donations - instead sell "gratitude-ware" There are tons of people who WANT to support these projects, but you have to make it easy and accountable to do that. The best example that I usually give is Sidekiq. @mperham is awesome that way "This is exactly why I disclosed my revenue: people won't know there's a successful path forward unless it's disclosed. I want more OSS developers to follow my lead and build a bright future for themselves based on their awesome software." In fact, I believe there's a start-up to be done here. "Stripe Atlas for Open Source software"

numfocusfnd 9y ago
NumPy joined NumFOCUS as a fiscally sponsored project in 2015, so it's been eligible to receive donations as a non-profit (because of NumFOCUS' nonprofit status) since then.

teekert 9y ago

I tried at work to get some money to the maintainer of iRAP, an RNA sequencing analysis pipeline we depend on heavily at the moment. But business sees this as wasted money, it's there for free, why not take it? Reading this, I think I'm going to double down on my efforts again. We get so much value out of a huge pile of FOSS software, we should be donating. Meanwhile we have spend piles of money on Matlab for years and we aren't even allowed to run Linux on our Laptops if we wanted to.

Vinnl 9y ago
I think someone on here recently suggested to a project that they provide an "enterprise subscription" or something like that with minor benefits to make it easier for businesses to justify (or maybe it made administration easier?) to "donate". You could suggest that to the maintainer, and then use that to convince business that you need that.

wodenokoto 9y ago

The "Every successful science library has the ashes of an academic career in the making" quote has been mentioned several times in the comments, so I thought I would give a plea to everybody who works in academia to help the people who build the foundational tools of your research by citing them in your papers: https://www.scipy.org/citing.html https://www.scipy.org/citing.html

torrent-of-ions 9y ago
Sad to say that I haven't cited these in papers that I've published so far. Much software is taken for granted, especially libraries. I don't often see papers mention SciPy specifically let alone add a citation. The author of GNU Parallel resorted to a message on stderr asking academics to cite it. But this isn't even the whole story. As we all know, designing and implementing a program or library is often just the first step. Someone then has to maintain that code. There is absolutely no incentive for anyone to maintain these. Institutions will fund new software because they expect a paper to be published about it. But nobody is paying anyone to maintain this stuff.

prewett 9y ago

I thought that Enthought sponsored a lot of NumPy development, kind of like as a corporate caretaker or something, is that not the case?

acqq 9y ago
I'd also like to read about what Enthought actually did. Anybody knows the details and willing to tell?

abousara 9y ago

Each country have its own taxes to sustain fundamentals, same thing must apply in software industry. Software engineers might be paid +100k$ while developing over open source languages/frameworks or libraries. That's is not fair, it is like riding a starving horse. A French point of view (after all, France invented VAT...) would suggest to introduce a taxe on software engineers salaries (1% ?) and redistribute this fund on most used languages/frameworks/libraries and use a part to sustain a new projects.

adrianN 9y ago
How about instead of taxing the workers we tax the companies that profit from open source efforts.
- dagw 9y ago
  How would you measure how much of the profits a company makes come from open source efforts?
  - adrianN 9y ago
    You don't, you just use a fixed percentage of whatever business tax they pay to fund open source development.
    - dagw 9y ago
      How are you going to separate out companies that profit from open source software, from those that don't? Would the fixed percentage be same for a company that is built 100% on open source software as it would for one where a handful of people in one department occasionally use VS Code to write some python scripts? If you want to fund open source using tax money, then fund open source using tax money. Trying to implement a special Open Source Tax and apply it only to those that "profit from" open source seems both counterproductive and virtually impossible.
      - adrianN 9y ago