13 ms·

Kill Your Dependencies

409 points by twampss 11y ago

diggan 11y ago
As everything, I think a bit of balance is needed. You're doing a quick MVP to demonstrate that your idea is working? Fuck it, just throw in dependencies for everything, just care about solving the problem you're trying to solve and proving/disproving your point. Once you verified it, then go and kill your dependencies. But don't do it just because you want to do it. If in the end the users doesn't benefit from you optimizing your dependencies, why do it? (Speaking from a product side rather than a OSS project used by other projects) Not sure KILL ALL DEPENDENCIES is helpful, but I'm not sure that MAKE EVERYTHING A DEPENDENCY is helpful either so...
- badloginagain 11y ago
  Ruby is great for prototyping, because its so easy to get things up and running. The big thing is transitioning to any kind of final production code. The rules for clean code apply as much for Ruby as it does for any other language. But its a good post, it can easily be something you overlook, due to gems being so damn convenient.
- st3v3r 11y ago
  That'd be good advice if those MVPs didn't so often become the actual product themselves. If industry and management understood that these things were proof of concepts, and realized that the actual product is going to have to be rewritten, then I'd agree with you.
- fixermark 11y ago
  s/kill/understand, which is useful advice for software engineering in general. As time approaches infinity, the number of magic "I use this package and it does something in my code, and then it all just works" dependencies you pull in should approach 0.
- rhapsodyv 11y ago
  Perfect. Get things done fast. Prove your product. If it succeed, you will have time to tune every aspect and invent your own wheel that fit your needs. But until that, RAM is a lot cheaper than your own time writing from scratch your version of things that are very stable and largely used. But, the advice is really important for gem writters. As a gem author, I think you really need think a little more about our dependencies as you do with our public interface.
peterwwillis 11y ago
Perl apps have thousands upon thousands of dependencies. It's intentional - reused code in CPAN means less downloading, more efficient code, and less bugs as the codebase gets refined. An app that relies mostly on dependencies is essentially an app with free support by hundreds of developers. That's the case with CPAN anyway; I don't know how Ruby people do things. Bugs happen, though. If you see a bug in a dependency, it is your job to report it at the very least, if not make an attempt to fix it. Without this community of people helping to improve a common codebase, we'd all be writing everything from scratch, and progress would move a lot slower.
rileymat2 11y ago
"The mime-types gem recently optimized its memory usage and saved megabytes of RAM. Literally every Rails app in existence can benefit from this optimization because Rails depends on the mime-types gem transitively: rails -> actionmailer -> mail -> mime-types." It seems like this could also be cast as a major success for "semi" standard dependencies.
EGreg 11y ago
I was just thinking about this today. But from the point of view of growinga community around a platform! Would you want to have one namespace for "official" modules and heavily influence everyone to use them? That's centralization (of governance). But, it's not centralization of a process that requires high availability. So the "drawback" is only that you centralize control and can make certain guarantees to developers on your platform. When you're starting an ecosystem, you can choose a "main namespace" as yum, npm etc. does or you can choose the more "egalitarian" convention of "Vendor/product" as github and Composer do. I think, in the end, the latter leads to a lot more proliferation of crap, and as the articls said, multiple versions of everything existing side-by-side. I have to deal with these issues when designing our company's platform (http://qbix.com/platform http://qbix.com/platform) and I think that having a central namespace is good. The platform installer etc. will make it super easy to download and install the "official" plugins. You can distribute your own "custom" plugins but the preferred way to contribute to the community would be to check what's already there first and respect that. If you REALLY want to make an alternative to something, make it good enough that the community's admins protecting the namespace will allow it into the namespace. Otherwise, promote it yourself, or fork the whole platform.
simonw 11y ago
Another benefit to minimizing your dependencies is security. The less external packages you are using (especially packages without active, security-conscious maintainers) the less likely you are to suffer a surprise vulnerability due to something deep down in your dependency hierarchy. This goes for client-side JavaScript too. XSS holes are one of the worst web app vulnerabilities out there and could easily be introduced accidentally by a simple mistake in a library. And this stuff is incredibly hard to audit these days thanks to the JavaScript community's cultural trend towards deeply nested dependencies.
- riffraff 11y ago
  but otoh, if you try to reinvent something instead of using a tried & true library, you might as well just add new bugs. I.e. I'd 100% use libxml to sanitize xml rather than trying and reimplementing xml parsing myself. As always, trade offs.
  - fixermark 11y ago
    Yep. OpenSSL has major security issues encountered on a relatively regular basis. Do not do your users the disservice of rolling your own SSL implementation. ;)
yoz-y 11y ago
> No code runs faster than no code. > No code has fewer bugs than no code. > No code uses less memory than no code. > No code is easier to understand than no code. The dependencies you decide to implement yourself in a minimal fashion are code though. I generally agree with the article, but in the end It Depends™
- bpicolo 11y ago
  And are generally worse tested, worse supported, and you have to maintain it on your own
ninjakeyboard 11y ago
I'm not 100% sure I agree with this as stated. Sure if the functionality is in core lib, use it but... it depends... Consider these three statements: - No code runs faster than no code. - No code has fewer bugs than no code. - No code is easier to understand than no code. For a language like scala where there is no json processing in the standard lib, if there is a json library that is battle tested, then by removing my own json code and leaning on that well tried and tested code for serialization/de-serialization, I've removed a whole bunch of code from my own library. The whole point of having modules as abstractions is to keep concerns neatly tucked in their own places to to increase re-use. By subscribing to the idea that my module should implement all of the functionality it needs, we're loosing the benefits of modularization. I just went through this exercise myself in a library I maintain - I removed my own json code and put a library in. I removed a bunch of code and made the whole thing simpler by leaning on that abstraction.
- skewart 11y ago
  I think the point of the article is: use external libraries thoughtfully. I don't think he was suggesting not using them at all. Your example sounds like a situation where a dependency certainly makes sense.
- falcolas 11y ago
  > ... I've removed a whole bunch of code from my own library You removed a bunch of code you understood, and added a bunch more code you don't understand, along with whatever technical debt, edge cases, and performance issues which are lingering in that library. Adding a library is never removing code from your project, it's adding code you don't yet understand to your project. It can still be a net win, but it's not less code for you to maintain.
  - cyphar 11y ago
    > > ... I've removed a whole bunch of code from my own library > You removed a bunch of code you understood, and added a bunch more code you don't understand, along with whatever technical debt, edge cases, and performance issues which are lingering in that library. Not all code you've written is good code. Hell, not all code you've written you actually understand. Libraries and dependencies make sense in many cases. Don't write yet another JSON parsing library unless you really need to. > Adding a library is never removing code from your project, it's adding code you don't yet understand to your project. It can still be a net win, but it's not less code for you to maintain. It's referencing code that you don't maintain. If the maintainer is bad, use a different library.
Eric_WVGG 11y ago
“Oh, I thought you said ‘dependents’” — Abraham
mschuster91 11y ago
This is not just true for Ruby but also for the entire npm ecosystem. I wonder how much traffic could be saved by optimizing npm packages... probably on terabyte scale at github alone, methinks.
allendoerfer 11y ago
To me, this seems more like an argument for optimizing beyond your own stack. Don't kill your own dependencies. Your app uses to much memory? Improve a dependency, you have now improved other peoples apps, too. Your app uses to much dependencies in total? Try to get all your first-level dependencies to standardize on the best http-client. (Which he is partially doing with his post.) Dependencies may have problems, but shared problems are better than problems only you have.
- drunkenazi 11y ago
  Couldn't agree more. I think the title is unreasonably one sided, and saying "be part of the solution" is equally one sided. Dependencies are great for the reasons you specified, and I saw nothing in that article suggesting otherwise. The part that feels the worst to read is: > Can I implement the required minimal functionality myself? Own it. This is largely a judgement call; "can I" and "minimal functionality" are subject to change based on many external circumstances. "Own it" also seems to imply owning it not as a dependency, based on the context, but rather as a part of a monolithic whole. It is also interesting that the sidekiq product makes use of gem dependencies. At top level 5 without platform dependencies, which (mostly due to rails) expands out to many more. The message should not be to "kill your dependencies", because that mindset is outdated and slow. So tired of hearing about how bad dependencies or scripting languages are. Would be much more excited to hear about how to contribute to open source dependencies, and how to write efficient scripts.
- sargas 11y ago
  I agree 100% with this. I used to bring in dependencies with the "don't reinvent the wheel" mentality. Then I realized how much trust I'm giving to the authors of all dependencies I pull in. Now I tend to do my best to understand the dependencies I bring so I can improve them if I can. The only problem I find with this decision is when I make an improvement/fix a bug on a dependency, and the project is either inactive or the authors don't give a crap about your work.
  - allendoerfer 11y ago
    The only problem I find with this decision is when I make an improvement/fix a bug on a dependency, and the project is either inactive or the authors don't give a crap about your work. True, but I think a temporary fork, which will eventually be merged back in, is still better than your own code with its own bugs.
dec0dedab0de 11y ago
It sounds like this is advocating NIH syndrome. If a library is going to make my job easier I'm going to use it, unless there is a very specific benefit of doing it myself.
pcwalton 11y ago
A lot of apps (old-timey Windows apps, for example) have this philosophy, leading them to reinvent things like crypto and image decoding. Naturally, this leads to tons of bugs, including security bugs. I would revise this to: Don't bring in more code than you need. But if the choice is between writing something yourself and using someone else's well-tested, heavily-used library, always go for the latter.
- edgyswingset 11y ago
  > Don't bring in more code than you need. I see it as a sliding scale. If I'm parsing 1 string with the same date format into 1 object, I'm not going to pull in some general purpose time parsing library - I'll write the 10 lines of code myself, a few unit tests, and be happy. If in the future I start having to deal with different date strings and some need to do more than just throw up a single date on a page somewhere, I'll get a date/time library.
  - amelius 11y ago
    You could also use the third party library for your unit tests.
    - deleted 11y ago
      [deleted]
- jvehent 11y ago
  This is pretty much what Rob Pike advocates in Go: "A little copying is better than a little dependency." http://go-proverbs.github.io/ http://go-proverbs.github.io/
  - pcwalton 11y ago
    I pretty squarely disagree with Rob Pike on that one. Copying is how you introduce bugs and insulate yourself from upstream bugfixes. I'm suggesting that you should try to remove code first, add a dependency on well-trusted code if that doesn't work, and only copy/reinvent as a last resort.
tarr11 11y ago
Looks like the mime-types upgrade has some sort of hard dependency on Rails 5? I seem to be stuck on 2.99
- Rafert 11y ago
  Its the other way around actually: ActionMailer 4.2 depends on mail ~> 2.5, >= 2.5.4, so you'll get 2.6.3 now. That version depends on mime-types <3, >= 1.16, so you'll get 2.99. The big change in mime-types 3 is using the columnar store by default, which is where the memory savings come from. It's opt-in from mime-types 2.6 onwards because it's a breaking change. Mail and afaik most other gems have opted in already.
brightball 11y ago
This should be one of the perks of Go since the compiler won't let you include anything that you aren't using.
- jerf 11y ago
  That's an orthogonal issue. You won't accidentally bring in something completely unrelated because you forgot to remove it from the "include"s, but nothing technically stops you from having a deep dependency chain. Culturally the Go community is aware of the issue, though. Still, it isn't hard to bring in a chain accidentally. I have a program than needs to do a query against the local LDAP system to extract members of a specified group. The LDAP library brings in five more libraries for parsing all the various bits of LDAP. Since this isn't C, I'm a bit less nervous about pulling in, say, a BER decoding library, because at least Go is generally memory-safe, but, still, that's a somewhat large stack for such a simple query. (Traditionally in C, you might as well just expect any library that decodes anything remotely binary-esque will have buffer overflows. C is a DSL for writing buffer overflows.) And yet, I'd be insane to try to implement some sort of just-barely-minimal LDAP client to do it myself. Looking at my local godoc instance's full set of packages that have gotten pulled in one way or another is still sort of intimidating. Some of them are cases where I'm just pulling in a subdir and got an entire large repo (the golang experimental repos do that a lot), but, still, I've got a lot of stuff in there. If you're a go programmer and you haven't run godoc locally and had a look at the packages page, have a look. You may be surprised.
gravypod 11y ago
Yes, this is the sort of thing that scares me away from Ruby. I'm worried this sort of "screw it just add a library" is going to spread further in my language of choice: Java. In my time doing open source programming on the side, I've found that it has become more common with the advent of things like mvn and gradle to just slather on layers to your stack even for the simple tasks. Need a function to turn a byte buffer into a string? Download these 3 Apache commons libraries and their dependencies. I understand if you are relying on a large portion of a library and you need to use it, but why bring an entire library in for one function.
- mbrock 11y ago
  There are ideas floating around that make it appealing to do just that. For example, the commons library might be considered "battle-tested," and who really knows what could happen with your own custom byte-buffer-to-string function? Maybe you missed something? Maybe there is some "best practice" that you didn't follow? Maybe the commons library is optimized? And writing your own thing doesn't add business value. Developer time is more expensive than dependencies. And so on. Me, I very often prefer to write things myself, in a way that can get labelled as NIH. My inclination is based on bad experiences with trying to debug external libraries. Sometimes I look at open source library code and find staggering complexity that I have no need for. Yes, maybe the library is great, but if its combinatorial size is 10,000 times the functionality that we need, then depending on its correctness becomes scary to me. And when I need to customize it, due to some requirements alteration, I will find it difficult and tedious. Black-box type libraries for isolated complicated tasks like codecs and crypto I will happily use. Otherwise, I'm a fan of the "design patterns" approach to reuse, which is all about learning from others, but without creating reusable formal abstractions in library form. So if you teach me how to write an URL router, I can then use your insights without depending on your code base, and I can adapt the idea so it fits my application perfectly.
  - gravypod 11y ago
    I agree completely. I'm not the one who will sit down and attempt to implement my own RSA, or hashing algorithms. It's like pornography. I don't know how to define it, but I definitely know when I see it. There are correct times to use libraries. But pulling a 15meg for some simple functionality is not a good practice in my opinion.
Animats 11y ago
Avoid shims. There are lots of libraries that just put one interface on top of another interface. They don't do much actual work. Pulling in shims, especially if they pull in lots of other stuff you're not using, should be avoided. If the dependency does real work you'd otherwise have to code, then use it.
- dawnerd 11y ago
  cough Mongoose. Been moving away from it on my projects. While it does provide a nice interface, it just creates more work down the road.
jeffdavis 11y ago
Your app/library inherits the technical debt of all its dependencies. There's a natural tension between code reuse and avoiding dependencies. If you can avoid a big dependency by writing a couple hundred lines of low-maintenance code, its probably worth it.
BinaryIdiot 11y ago
This is a great read that can be applied to node.js very much. I've seen apps that include 10, maybe 20 dependencies but when you flatten out the full dependency tree? Thousands. It's incredible and if one of those dependencies screws up semantic versioning or just screws up in general it can be a nightmare to debug and fix. This is why every 1.0 product I work on I include every dependency that speeds up my development. In 2.0 the first things to do is prune all unnecessary dependencies and start minor rewrites when a dependency can be done in house (yeah yeah reinventing the wheel is a problem but most npm dependencies are small and many can be recreated internally without issue). This is even more important if you're creating a library / module. My msngr.js library uses zero dependencies and yet can make http calls in node and the browser because it was easy to implement the minimal solution I needed without bringing in dependencies to support a single way of calling http.
- nextos 11y ago
  This sadly also happens in some Linux repositories that add too many dependencies to a few key packages. On NixOS, last time I tried, installing mutt ended up bringing python as well.
  - davexunit 11y ago
    In Nix, it's very easy to make a minimal mutt variant by removing the python input.
    - nextos 11y ago
      Yes, but these things shouldn't be happening. Default builds of packages should be kept more minimal. I understand it's hard with Nix philosophy, and things are improving with different package outputs. For the record, python was pulled indirectly via the gnupg dependency I think.
      - davexunit 11y ago
        It's a difference in philosophy. I think packages by default should be full-featured, and minimal variants can be created for those that want it.
sksixk 11y ago
what's the point? none of them are realistic. "no code" - well, it's there for a reason. "own it" - do i really want to write my own minimal implementation? i understand that dependency is a pita but this post doesn't provide anything worthwhile.
jowiar 11y ago
I find dependencies to be a very good indicator for how my code should be modularized. That is, rather than pulling a boatload of dependencies into "the application", pull a couple dependencies into a module, and then depend on the module. It makes it very easy for dependencies to be a "well, it gets the job done for now, and I can reimplement that myself if that changes" sort of thing.
- edvinbesic 11y ago
  I found this approach to work well for me as well. It has payed off many times. I try to wrap most of my dependencies so that if I later feel like I need to pick some low-hanging fruit I can implement some of the functionality internally while maintaining the original api.
  - jowiar 11y ago
    Also, this is probably the single biggest difference for me when working with a static vs. dynamically typed language. With Static Typing, there's a "translate the dependency's types into the application's types" step that pretty much screams "put a seam here!". With Dynamic Typing, it's a bit less loud.
    - Aeolos 11y ago
      That's pretty much a C/C++ problem though. I don't think I've ever had to translate types in C# or F#.
      - jowiar 11y ago
        I was referring to that, with Scala, I like to avoid leaking the innards of a JSON serialization library in the code for an API client, instead returning domain objects. Whereas with, JS or Python, the initial approach is to sling around the blob of JSON.
laumars 11y ago
This article would be more accurately written as "prefer the standard library over 3rd party solutions" since all the examples given still required dependencies, but ones that are shipped as part of the language runtime (Ruby in this case). However when discussing languages with no specific standard library or languages who's standard library is missing feature y, then it's quite understandable to use a 3rd party battle tested dependency. In fact I'd go further and say it would be advisable to use a respected 3rd party library when dealing with code which handles security or other complex concepts with high failure rates.
- 50CNT 11y ago
  Matter of fact, sometimes it's better to be using a respectable 3rd party library. Requests vs. urllib2 in Python springs to mind, and I'm sure there's more examples.
spullara 11y ago
This is a huge mistake if applied without care. Building things from scratch necessarily will introduce more bugs, more maintenance costs and leave you with a codebase that suffers from a lack of maturity.
- edvinbesic 11y ago
  I don't think the sentiment is 'implement everything from scratch', but rather that if something exists natively try to use that instead of pulling in other dependencies, like in the http client example.
- Chris_Newton 11y ago
  Building things from scratch necessarily will introduce more bugs, more maintenance costs and leave you with a codebase that suffers from a lack of maturity. Unfortunately, in some programming language ecosystems where having many small and transitive dependencies on modules from an non-curated repository is common, none of those three things is necessarily true. Code reuse is not a trivial problem, and you always have to weigh the benefits against the costs and risks to decide whether it’s worth it. If we’re depending on GitHub repositories with a dozen files and three subdirectories just to provide some simple functionality that any junior programmer could implement directly in five lines of code, we’ve probably lost the plot. On the other hand, if we have a full in-house implementation of encryption algorithms we use to throw sensitive customer data around between the browser and our servers, we’ve also probably lost the plot.
liveoneggs 11y ago
mojolicious does a great job with this, supporting optional dependencies as progressive enhancement (installing EV will speed you up, but you don't necessarily need it) http://mojolicious.org/ http://mojolicious.org/
greggman 11y ago
This reminds me of an example I ran into yesterday. I haven't used webpack yet but I saw a question on SO of someone wanting to use some package called glslify. I thought I'd take a look and maybe learn webpage in the process. From the description all glslify does is look for files with the extensions .glsl, .frag, and .vert and lets you get their contents with `content = require(filename)`. Sounds like it would be at most 10-30 lines of code. Nope npm install --save glslify-loader webpack-glsl-test@1.0.0 /Users/gregg/temp/webpack-glsl-test └─┬ glslify-loader@1.0.2 └─┬ glslify@2.3.1 ├─┬ bl@0.9.5 │ └─┬ readable-stream@1.0.33 │ ├── core-util-is@1.0.2 │ ├── isarray@0.0.1 │ └── string_decoder@0.10.31 ├─┬ glsl-resolve@0.0.1 │ ├── resolve@0.6.3 │ └── xtend@2.2.0 ├─┬ glslify-bundle@2.0.4 │ ├─┬ glsl-inject-defines@1.0.3 │ │ └── glsl-token-inject-block@1.0.0 │ ├── glsl-token-defines@1.0.0 │ ├── glsl-token-depth@1.1.2 │ ├─┬ glsl-token-descope@1.0.2 │ │ ├── glsl-token-assignments@2.0.1 │ │ └── glsl-token-properties@1.0.1 │ ├── glsl-token-scope@1.1.2 │ ├── glsl-token-string@1.0.1 │ └── glsl-tokenizer@2.0.2 ├─┬ glslify-deps@1.2.5 │ ├── events@1.1.0 │ ├─┬ findup@0.1.5 │ │ ├── colors@0.6.2 │ │ └── commander@2.1.0 │ ├── graceful-fs@4.1.3 │ ├── inherits@2.0.1 │ └─┬ map-limit@0.0.1 │ └─┬ once@1.3.3 │ └── wrappy@1.0.1 ├── minimist@1.2.0 ├── resolve@1.1.7 ├─┬ static-module@1.3.0 │ ├─┬ concat-stream@1.4.10 │ │ ├── readable-stream@1.1.13 │ │ └── typedarray@0.0.6 │ ├─┬ duplexer2@0.0.2 │ │ └── readable-stream@1.1.13 │ ├─┬ escodegen@1.3.3 │ │ ├── esprima@1.1.1 │ │ ├── estraverse@1.5.1 │ │ ├── esutils@1.0.0 │ │ └─┬ source-map@0.1.43 │ │ └── amdefine@1.0.0 │ ├─┬ falafel@1.2.0 │ │ ├── acorn@1.2.2 │ │ ├── foreach@2.0.5 │ │ └── object-keys@1.0.9 │ ├─┬ has@1.0.1 │ │ └── function-bind@1.0.2 │ ├── object-inspect@0.4.0 │ ├─┬ quote-stream@0.0.0 │ │ ├── minimist@0.0.8 │ │ └─┬ through2@0.4.2 │ │ └─┬ xtend@2.1.2 │ │ └── object-keys@0.4.0 │ ├── shallow-copy@0.0.1 │ ├─┬ static-eval@0.2.4 │ │ └─┬ escodegen@0.0.28 │ │ ├── esprima@1.0.4 │ │ └── estraverse@1.3.2 │ └─┬ through2@0.4.2 │ └─┬ xtend@2.1.2 │ └── object-keys@0.4.0 ├── through2@0.6.5 └── xtend@4.0.1 > 4 meg of source files --- update: I think maybe I misunderstood the description. glslify actually parses GLSL and re-writes it in various ways so maybe this is a bad example. I've seen other though. Like 40k+ lines of deps for an ANSI color library or 200k+ lines deps and native node plugins for launching a browser from node.
mwcampbell 11y ago
A large number of dependencies is only a problem in environments that aren't amenable to per-function static linking or tree-shaking. These include dynamically typed languages like Python, Ruby, and JavaScript (except when using the Google Closure Compiler in advanced mode), but also platforms like the JVM and .NET when reflection is allowed. Where static linking or tree-shaking is feasible, the run-time impact of bringing in a large library but only using a small part of it is no more than the impact of rewriting the small part you actually use. Edit: Dart is an interesting case. It has dynamic typing, but it's static enough that tree-shaking is feasible. Seth Ladd's blog post about why tree-shaking is necessary [1] makes the same point that I'm making here. [1]: http://blog.sethladd.com/2013/01/minification-is-not-enough-you-need.html http://blog.sethladd.com/2013/01/minification-is-not-enough-...
- shoover 11y ago
  Yeah, but it seems like you listed most of the platforms most people actually use, all in the X column. How do we get from 100MB Electron deployment and fat, partially used jars and dlls to this magical tree-shaking world?
  - mwcampbell 11y ago
    You're right; most of the languages and platforms used for developing applications don't have reliable support for tree-shaking or fine-grained static linking. But there's hope for some of them. When building Android applications, it's common to process the JVM bytecode with ProGuard before converting it to Dex bytecode. ProGuard includes a tree-shaking step. Sometimes it's necessary to tell ProGuard about specific classes or class members that it should leave alone, if they're accessed dynamically (e.g. using java.lang.reflect or Class.forName). But it's still better than nothing. Likewise, .NET applications for the Windows Store are compiled to native code using .NET Native, and that compilation includes a tree-shaking step. This introduces some limitations on the use of reflection. I'm guessing similar limitations will apply to the native compilation option of .NET Core. As for JavaScript, Google's Closure Compiler can do tree-shaking. But I don't know if the Closure Compiler's advanced mode works with any of the popular JavaScript libraries or frameworks, or just Google's Closure Library.
gtrubetskoy 11y ago
Your programs shouldn't do things you do not understand. You do not have to be an expert in cryptography, memory allocation or b-trees, etc, but if this is what your app requires, then you should take the time to read up on it and carefully research what is out there if you suspect it is beyond your abilities to implement. If you take the time to do your research, the choice between rolling your own, copying or adding a dependency will become clear. If it's not becoming clear, then you haven't finished your homework. Learning is a good thing, yes it takes time, but it's time well spent, and it's fun above all. You may discover that this thing that you thought was hard and needed a dependency is really a few lines of code (a good example is a graph implementation). It might even change your career path. At least that's been my experience in the nearly two decades of writing software.
surfmike 11y ago
if you're running multiple rails processes on a server like this, couldn't you somehow do the initialization in one process, then fork off the new processes? wouldn't that prevent the base libraries from being copied in memory?
- lotyrin 11y ago
  Yes. Many app servers do this. (Not being super familiar with the Ruby ecosystem, I am not sure specifically which ones.)
- paulannesley 11y ago
  Yes, http://unicorn.bogomips.org/ http://unicorn.bogomips.org/ popularized this for ruby / rack / rails with its forking model and preload_app option. http://puma.io/ http://puma.io/ does the same thing, but additionally runs multiple threads in each process. The garbage collector in Ruby 1.8 / 1.9 negated the benefits of copy-on-write forking, but that's fixed since Ruby 2.0
nickpsecurity 11y ago
Obligatory essay from PHK on the effect the author describes: http://queue.acm.org/detail.cfm?id=2349257 http://queue.acm.org/detail.cfm?id=2349257 History continues to repeat itself. Fake reuse and proliferation of unnecessary bloat are two of those recurring themes. Fight it whenever you can. The old TCL, LISP, Delphi, REBOL, etc clients and servers were tiny by modern standards. They still got the job done. Strip out bloat wherever you can. Also, standardize on one tool for each given job plus hide it behind a good interface to enable swapping it out if its own interface isn't great.
PaulHoule 11y ago
They should sell sonatype for this.
Rafert 11y ago
Excellent article. I tried to develop on GitLab once but the sheer amount of gems it pulls in (~100 directly declared, 350+ including dependencies if I remember correctly) with a bunch of installation problems made me decide it was not worth the hassle.
- sytse 11y ago
  I'm sorry to hear you had installation problems trying to develop for GitLab. Have you tried the GitLab development kit https://gitlab.com/gitlab-org/gitlab-development-kit https://gitlab.com/gitlab-org/gitlab-development-kit? If still are interested and experience problems please email support@gitlab.com and reference this comment for help. I agree with the article, the less dependencies the better. GitLab's gemfile.lock https://gitlab.com/gitlab-org/gitlab-ce/blob/master/Gemfile.lock https://gitlab.com/gitlab-org/gitlab-ce/blob/master/Gemfile.... has over 1000 lines and GitLab uses a lot of memory. We try to be careful what we pull in but if anyone has suggestions which can be removed please let us know. Recently we found out that we still had to remove Redcloth as a dependency, it will be gone in GitLab 8.5.
agentgt 11y ago
I find this argument sort related to the framework vs library and opinionated vs agnostic. Being an old fart Java developer I generally prefer things where you can plugin your own implementation (ie agnostic). That is there is an extreme for killing your dependencies of either extreme copy'npaste OR which every library offers a plugin SPI (ie inversion of control) (or a combo of both). The problem with the dependency injection above approach (aka Spring prior to Boot) is that you have developers doing lots of custom crap, bloated/overly engineered libraries, increased ramp up time, and configuration hell. But I still think this is probably better than ole copy'n paste.. most of the time. I do hate dependencies though.
- mwcampbell 11y ago
  You're probably more experienced in java development than me. But maybe a good rule of thumb is that most Java libraries shouldn't use reflection or dynamically loaded classes (i.e. using Class.forName or ClassLoader). That rules out most dependency injection frameworks.
  - agentgt 11y ago
    That is correct that the libraries should not have DI but I should be able to wire up the library on my own and not let the library do its own static initialization. What is far worse than Class.forName and other crap is libraries self imposed singletons. Take for example Hystrix. I'm just now fixing that the thing loads up its own configuration framework (Archaius) which uses static initialization. Archaius needed like 10 other dependencies. This is all really because Hystrix uses static singleton (HystrixPlugins) and many frameworks need this or else is incredibly difficult to get an implementation up (ie using pseudo singleton to avoid excessive passing of a context). https://github.com/Netflix/Hystrix/pull/1083 https://github.com/Netflix/Hystrix/pull/1083
  - lmm 11y ago
    Java makes plugin-like scenarios very hard. The kind of thing you'd do with a typeclass in languages that have them. There's OSGi (the horror, the horror) which maybe-kinda-sorta-works, or there's the SPI where you put the class name in a .service file (which will then be instantiated though... reflection). When those are the alternatives, DI frameworks aren't so bad.
giancarlostoro 11y ago
One issue I had with Ruby on Rails was getting MySQL drivers to even cooperate on both Windows and Linux. At the end of the day I wound up sticking to Python and other languages instead. I don't mind using any language, but if the language is fighting me due to native dependency hell then I can't really do much. Even to just use SQLite was a bit of a painful experience, yet on Python SQLite works out of the box without any effort on my part (on Windows). Oddly enough. I'm looking to getting back into Ruby at some point later this year, but I might ignore Rails altogether so I don't miss out on learning a new fun language.
3minus1 11y ago
I remember one nasty bug where someone has included one function from bootstrap.js library and someone else had included the entire library. So both functions were running causing an issue.
adenadel 11y ago
This reminds of of this article http://www.joelonsoftware.com/articles/fog0000000007.html http://www.joelonsoftware.com/articles/fog0000000007.html Apparently Microsoft's Excel team had even written their own C compiler.
ocdtrekkie 11y ago
So, I've been writing a home automation system using the .NET Framework (with Visual Basic, I'll wait until you finish laughing).......... Okay. I've made a point not to add any third parties references and packages I can avoid. I went ahead and got a third party scheduling engine, and the SQLite provider, but beyond that, I'm writing everything else myself so far. First of all, I'm learning a lot in having to write stuff myself. At the very least, it's a great educational experience. I've worked with a lot of code samples, so I'm not going totally from scratch, but they're all at the very least tailored to my needs. But for me, the big thing is keeping everything thin. The program loads in milliseconds. Almost all of the reference data for what it's built on is in one place (the .NET Framework Reference). And key, is that the features my program supports are the features I want and need, not the features some dependency has told me to have. The biggest dependency I have, Quartz.NET, is actually the most confusing part. It's not structured like the rest of my program is, it's documentation leaves some things to be desired, and it does a lot more than I need it to. There's a lot of bloat I could cut out if I wrote my own scheduler, and maybe someday I will.
jcoffland 11y ago
This is not only an issue for running software but also a huge issue for compiling/building software. Each dependency adds the potential to break your builds in new ways. As the software your program depends on evolves the risk increases that it will change the way your program executes or cause it to fail to build. Many devs will insist you do not reinvent the wheel by writing things like JSON parsers but you always have to weigh the cost of adding a dependency. It's not free.
cdnsteve 11y ago
Double edge sword. Deps are great! Functionality added quickly. Deps are terrible! They broke my app. If your app has a long shelf time, the less deps you rely on, the easier to manage from what I've seen. For some reason Golang feels like it makes sense here. Pretty much everything you need is in core. *Disclaimer, I don't have any Golang apps in prod but I'd love to hear from those that do.
beat 11y ago
I have one thing to say about all of this... Nokogiri.
twic 11y ago
Does any of this sound familiar: - Test gems loading in production. That does not sound familiar! Is this a thing which happens with Rails?
- nona 11y ago
  No, but sometimes people are a bit careless in their Gemfile I guess.
vinceguidry 11y ago
Gems I use fall into three categories. A lot of my projects are just wrappers around one main gem. Rails, Nokogiri, Roo, API wrapper gems. These are 'project gems'. If they give me problems, I'll re-evaluate the scope of the project and perhaps pick another gem to orient the project around. Once the project reaches maturity, I'll default to fixing the problem rather than re-engineering it unless the problems run deep. Sometimes I'll use gems like Phoner to handle datatypes that are too tricky to do with regular Ruby. I'll call these 'utility gems'. When I include a utility gem, generally it has one job and one job only, it's invoked in exactly one place in the code and gets included in that file. I can generally replace a utility gem with stdlib Ruby code if I really need to. I also have what I call 'infrastructure gems'. These are gems like pry, capistrano, and thor that I tend to include in every project where it seems they would be useful. These are gems that are worth getting to know very well because they solve really hard problems that you don't want to use stdlib for. If these give me problems I will do whatever I need to to resolve them and understand why the problem exists, because the costs of migrating off of them would be steep. The decision to use a gem should not be taken too lightly, but nor should it weigh large on the mind. Be quick to try it out, but also quick to take it out.
edejong 11y ago
The problem with dependencies is that developers approach it from a top-down approach. The question they answer is: I need an HTTP client, JSON API, monitoring tool, logging framework. Never do they ask the opposite: what kind of foundations do I need? What elementary blocks do I need to have or learn in order to make a JSON parser in 5 lines of code? Is it possible to do logging without all the cruft? Can I write the library in the same amount of time as I can read the docs? Could the code I write be the docs? Similar line of reasoning: can I leverage my OS to do scheduling/IPC/monitoring/security? If it can't, should we lobby for better OSes (that might scale over multiple machines?) Does Linux/Docker offer the right fundamentals? Dijkstra was truly right: the art of programming is the art of managing complexity.
LukeB_UK 11y ago
I disagree, and this quote I've seen floating around the internet sort of sums the idea up to me (albeit with a music analogy): > I thought using loops was cheating, so I programmed my own using samples. I then thought using samples was cheating, so I recorded real drums. I then thought that programming it was cheating, so I learned to play drums for real. I then thought using bought drums was cheating, so I learned to make my own. I then thought using premade skins was cheating, so I killed a goat and skinned it. I then thought that that was cheating too, so I grew my own goat from a baby goat. I also think that is cheating, but I’m not sure where to go from here. I haven’t made any music lately, what with the goat farming and all.
- gglitch 11y ago
  Charming quote. The same argument could be made about making your own food "from scratch." But it's not a solid refutation of the essay. Let's say that if one end of the spectrum is unlimited dependencies and complete indifference to the complexity and size of the project, and the other end of the spectrum is raising your own goats, there must be an ideal somewhere in the middle.
  - LukeB_UK 11y ago
    I completely agree, but the extremism in the essay is ridiculous. If I'm writing some software, I don't want to have to roll my own version of every little thing when there are battle tested libraries out there that already do it and benefit from many people using it. If my use case is vastly different, sure, but if it's the same use case as everyone else then I see little benefit.
    - hackbinary 11y ago
      Have you read a modest proposal by J Swift. Satire at its best! https://en.wikipedia.org/wiki/A_Modest_Proposal https://en.wikipedia.org/wiki/A_Modest_Proposal
    - justaaron 11y ago
      if they are so battle-tested then they probably are not like the email gem that the author spoke of that was using 10mb more memory than necessary for dependencies... node-land is also getting a bit silly this way... to say nothing of java etc... since we are into silly quotes i'll riff and probably get this wrong but: "you wanted a banana but you got the gorilla holding it and the whole jungle too" joe armstrong (erlang) on Class-based OO inheritance etc...
Walkman 11y ago
There is a counter example of his reasoning in the Python world. There is a HTTP client library "urllib" in the standard library, but nowadays everyone rather pulls in the external dependency "requests" because the urllib API is terrible. It is mature, well tested, good documented code though.
- _ZeD_ 11y ago
  just as a data point: I don't use request, I onestly prefer the urrlib (was urllib2) API offered by the standard library
fixermark 11y ago
Incidentally, though the author says this can apply to any ecosystem, finding it applies to Ruby too often is what pushed me out of developing Rails apps. At least at the time I was using it heavily, the Rails space just wasn't stable enough to trust that I wouldn't have to learn an entirely new wheel to get my work done every time I went in to fix a relatively small problem. "Can I implement the required minimial functionality myself? Own it" is advice one gives if one can't trust the libraries one depends upon to stay healthy, performant, and applicable to your use-case. Nobody'd recommend re-implementing readline or printf; if you have some heavy-lifting mathematics to do in Python, use numpy.
technoir 11y ago
If there is an analogue in the standard library you should have a compelling reason to use an alternative. Wish this could be filed under "common" sense. Thanks for articulating and presenting this principle, among others. Great writeup.
joesmo 11y ago
So to summarize, to get rid of an extra 10 (or even 100) megs in Ruby (or 0 megs in some other languages) of memory usage (and disk usage don't forget!) spend weeks rewriting, testing, and integrating your own code instead of using already written, tested, and integrated code. Now, that I've clarified the article's point, how can anyone not follow this "best-practices" advice? </sarcasm>
EternalFury 11y ago
How long did it take anyone to realize this nightmare? Since we are on the path to major discoveries, let's talk about runtime, runtime-dependencies and all that. Every Ruby app ever created is stuck somewhere on the time axis, before its origin.
justaaron 11y ago
amen
aidenn0 11y ago
Slightly related, I was packaging up a webapp in docker, and one thing the application did was form-fill PDFs. I had been using pdftk to do this, but it turns out pdftk is written in gcj, and gcj pulls in a lot for its runtime libraries. I wrote a small program using the mupdf libraries and cut the size of my docker image by over 400MB.
- EvanPlaice 11y ago
  Congrats, you just violated the GPL. You were already violating the proprietary license if you're using his for commercial purposes without paying. Not all free software is free.
  - aidenn0 11y ago
    Actually mupdf uses AGPL, and so the program I wrote and linked with mupdf is also under AGPL.
  - aidenn0 11y ago
    I just saw the commercial redistribution clause for pdfTK if that's what you're talking about. It does not affect me since this was not a commercial application, but it would seem to me that that clause itself is a violation of the GPL, since pdfTK links to GPL software and it is also a contradiction of a separate place on the site that claims the same software is licensed under the GPLv2. Preventing commercial redistribution is not compatible with the GPL.
justinator 11y ago
Perl takes a pragmatic take on this (as well as other takes...) with the collection of ::Tiny CPAN modules that just do one thing pretty OK. Things like Try::Tiny that help immensely with exception handling - something you don't want to really roll you own. It itself does not have any dependencies that aren't in core: http://deps.cpantesters.org/?module=Try%3A%3ATiny;perl=latest http://deps.cpantesters.org/?module=Try%3A%3ATiny;perl=lates...
cjhveal 11y ago
Interesting to note that the Stripe gem removed one of its dependencies seemingly in reaction to being called out at the end of this article.

409 points by twampss 11y ago

As everything, I think a bit of balance is needed. You're doing a quick MVP to demonstrate that your idea is working? Fuck it, just throw in dependencies for everything, just care about solving the problem you're trying to solve and proving/disproving your point. Once you verified it, then go and kill your dependencies. But don't do it just because you want to do it. If in the end the users doesn't benefit from you optimizing your dependencies, why do it? (Speaking from a product side rather than a OSS project used by other projects) Not sure KILL ALL DEPENDENCIES is helpful, but I'm not sure that MAKE EVERYTHING A DEPENDENCY is helpful either so...

badloginagain 11y ago
Ruby is great for prototyping, because its so easy to get things up and running. The big thing is transitioning to any kind of final production code. The rules for clean code apply as much for Ruby as it does for any other language. But its a good post, it can easily be something you overlook, due to gems being so damn convenient.
st3v3r 11y ago
That'd be good advice if those MVPs didn't so often become the actual product themselves. If industry and management understood that these things were proof of concepts, and realized that the actual product is going to have to be rewritten, then I'd agree with you.
fixermark 11y ago
s/kill/understand, which is useful advice for software engineering in general. As time approaches infinity, the number of magic "I use this package and it does something in my code, and then it all just works" dependencies you pull in should approach 0.
rhapsodyv 11y ago
Perfect. Get things done fast. Prove your product. If it succeed, you will have time to tune every aspect and invent your own wheel that fit your needs. But until that, RAM is a lot cheaper than your own time writing from scratch your version of things that are very stable and largely used. But, the advice is really important for gem writters. As a gem author, I think you really need think a little more about our dependencies as you do with our public interface.

peterwwillis 11y ago

Perl apps have thousands upon thousands of dependencies. It's intentional - reused code in CPAN means less downloading, more efficient code, and less bugs as the codebase gets refined. An app that relies mostly on dependencies is essentially an app with free support by hundreds of developers. That's the case with CPAN anyway; I don't know how Ruby people do things. Bugs happen, though. If you see a bug in a dependency, it is your job to report it at the very least, if not make an attempt to fix it. Without this community of people helping to improve a common codebase, we'd all be writing everything from scratch, and progress would move a lot slower.

rileymat2 11y ago

"The mime-types gem recently optimized its memory usage and saved megabytes of RAM. Literally every Rails app in existence can benefit from this optimization because Rails depends on the mime-types gem transitively: rails -> actionmailer -> mail -> mime-types." It seems like this could also be cast as a major success for "semi" standard dependencies.

EGreg 11y ago

I was just thinking about this today. But from the point of view of growinga community around a platform! Would you want to have one namespace for "official" modules and heavily influence everyone to use them? That's centralization (of governance). But, it's not centralization of a process that requires high availability. So the "drawback" is only that you centralize control and can make certain guarantees to developers on your platform. When you're starting an ecosystem, you can choose a "main namespace" as yum, npm etc. does or you can choose the more "egalitarian" convention of "Vendor/product" as github and Composer do. I think, in the end, the latter leads to a lot more proliferation of crap, and as the articls said, multiple versions of everything existing side-by-side. I have to deal with these issues when designing our company's platform (http://qbix.com/platform http://qbix.com/platform) and I think that having a central namespace is good. The platform installer etc. will make it super easy to download and install the "official" plugins. You can distribute your own "custom" plugins but the preferred way to contribute to the community would be to check what's already there first and respect that. If you REALLY want to make an alternative to something, make it good enough that the community's admins protecting the namespace will allow it into the namespace. Otherwise, promote it yourself, or fork the whole platform.

simonw 11y ago

Another benefit to minimizing your dependencies is security. The less external packages you are using (especially packages without active, security-conscious maintainers) the less likely you are to suffer a surprise vulnerability due to something deep down in your dependency hierarchy. This goes for client-side JavaScript too. XSS holes are one of the worst web app vulnerabilities out there and could easily be introduced accidentally by a simple mistake in a library. And this stuff is incredibly hard to audit these days thanks to the JavaScript community's cultural trend towards deeply nested dependencies.

riffraff 11y ago
but otoh, if you try to reinvent something instead of using a tried & true library, you might as well just add new bugs. I.e. I'd 100% use libxml to sanitize xml rather than trying and reimplementing xml parsing myself. As always, trade offs.
- fixermark 11y ago
  Yep. OpenSSL has major security issues encountered on a relatively regular basis. Do not do your users the disservice of rolling your own SSL implementation. ;)

yoz-y 11y ago

> No code runs faster than no code. > No code has fewer bugs than no code. > No code uses less memory than no code. > No code is easier to understand than no code. The dependencies you decide to implement yourself in a minimal fashion are code though. I generally agree with the article, but in the end It Depends™

bpicolo 11y ago
And are generally worse tested, worse supported, and you have to maintain it on your own

ninjakeyboard 11y ago

I'm not 100% sure I agree with this as stated. Sure if the functionality is in core lib, use it but... it depends... Consider these three statements: - No code runs faster than no code. - No code has fewer bugs than no code. - No code is easier to understand than no code. For a language like scala where there is no json processing in the standard lib, if there is a json library that is battle tested, then by removing my own json code and leaning on that well tried and tested code for serialization/de-serialization, I've removed a whole bunch of code from my own library. The whole point of having modules as abstractions is to keep concerns neatly tucked in their own places to to increase re-use. By subscribing to the idea that my module should implement all of the functionality it needs, we're loosing the benefits of modularization. I just went through this exercise myself in a library I maintain - I removed my own json code and put a library in. I removed a bunch of code and made the whole thing simpler by leaning on that abstraction.

skewart 11y ago
I think the point of the article is: use external libraries thoughtfully. I don't think he was suggesting not using them at all. Your example sounds like a situation where a dependency certainly makes sense.
falcolas 11y ago
> ... I've removed a whole bunch of code from my own library You removed a bunch of code you understood, and added a bunch more code you don't understand, along with whatever technical debt, edge cases, and performance issues which are lingering in that library. Adding a library is never removing code from your project, it's adding code you don't yet understand to your project. It can still be a net win, but it's not less code for you to maintain.
- cyphar 11y ago
  > > ... I've removed a whole bunch of code from my own library > You removed a bunch of code you understood, and added a bunch more code you don't understand, along with whatever technical debt, edge cases, and performance issues which are lingering in that library. Not all code you've written is good code. Hell, not all code you've written you actually understand. Libraries and dependencies make sense in many cases. Don't write yet another JSON parsing library unless you really need to. > Adding a library is never removing code from your project, it's adding code you don't yet understand to your project. It can still be a net win, but it's not less code for you to maintain. It's referencing code that you don't maintain. If the maintainer is bad, use a different library.

Eric_WVGG 11y ago

“Oh, I thought you said ‘dependents’” — Abraham

mschuster91 11y ago

This is not just true for Ruby but also for the entire npm ecosystem. I wonder how much traffic could be saved by optimizing npm packages... probably on terabyte scale at github alone, methinks.

allendoerfer 11y ago

To me, this seems more like an argument for optimizing beyond your own stack. Don't kill your own dependencies. Your app uses to much memory? Improve a dependency, you have now improved other peoples apps, too. Your app uses to much dependencies in total? Try to get all your first-level dependencies to standardize on the best http-client. (Which he is partially doing with his post.) Dependencies may have problems, but shared problems are better than problems only you have.

drunkenazi 11y ago
Couldn't agree more. I think the title is unreasonably one sided, and saying "be part of the solution" is equally one sided. Dependencies are great for the reasons you specified, and I saw nothing in that article suggesting otherwise. The part that feels the worst to read is: > Can I implement the required minimal functionality myself? Own it. This is largely a judgement call; "can I" and "minimal functionality" are subject to change based on many external circumstances. "Own it" also seems to imply owning it not as a dependency, based on the context, but rather as a part of a monolithic whole. It is also interesting that the sidekiq product makes use of gem dependencies. At top level 5 without platform dependencies, which (mostly due to rails) expands out to many more. The message should not be to "kill your dependencies", because that mindset is outdated and slow. So tired of hearing about how bad dependencies or scripting languages are. Would be much more excited to hear about how to contribute to open source dependencies, and how to write efficient scripts.
sargas 11y ago
I agree 100% with this. I used to bring in dependencies with the "don't reinvent the wheel" mentality. Then I realized how much trust I'm giving to the authors of all dependencies I pull in. Now I tend to do my best to understand the dependencies I bring so I can improve them if I can. The only problem I find with this decision is when I make an improvement/fix a bug on a dependency, and the project is either inactive or the authors don't give a crap about your work.
- allendoerfer 11y ago
  The only problem I find with this decision is when I make an improvement/fix a bug on a dependency, and the project is either inactive or the authors don't give a crap about your work. True, but I think a temporary fork, which will eventually be merged back in, is still better than your own code with its own bugs.

dec0dedab0de 11y ago

It sounds like this is advocating NIH syndrome. If a library is going to make my job easier I'm going to use it, unless there is a very specific benefit of doing it myself.

pcwalton 11y ago

A lot of apps (old-timey Windows apps, for example) have this philosophy, leading them to reinvent things like crypto and image decoding. Naturally, this leads to tons of bugs, including security bugs. I would revise this to: Don't bring in more code than you need. But if the choice is between writing something yourself and using someone else's well-tested, heavily-used library, always go for the latter.

edgyswingset 11y ago
> Don't bring in more code than you need. I see it as a sliding scale. If I'm parsing 1 string with the same date format into 1 object, I'm not going to pull in some general purpose time parsing library - I'll write the 10 lines of code myself, a few unit tests, and be happy. If in the future I start having to deal with different date strings and some need to do more than just throw up a single date on a page somewhere, I'll get a date/time library.
- amelius 11y ago
  You could also use the third party library for your unit tests.
  - deleted 11y ago
    [deleted]
jvehent 11y ago
This is pretty much what Rob Pike advocates in Go: "A little copying is better than a little dependency." http://go-proverbs.github.io/ http://go-proverbs.github.io/
- pcwalton 11y ago
  I pretty squarely disagree with Rob Pike on that one. Copying is how you introduce bugs and insulate yourself from upstream bugfixes. I'm suggesting that you should try to remove code first, add a dependency on well-trusted code if that doesn't work, and only copy/reinvent as a last resort.

tarr11 11y ago

Looks like the mime-types upgrade has some sort of hard dependency on Rails 5? I seem to be stuck on 2.99

Rafert 11y ago
Its the other way around actually: ActionMailer 4.2 depends on mail ~> 2.5, >= 2.5.4, so you'll get 2.6.3 now. That version depends on mime-types <3, >= 1.16, so you'll get 2.99. The big change in mime-types 3 is using the columnar store by default, which is where the memory savings come from. It's opt-in from mime-types 2.6 onwards because it's a breaking change. Mail and afaik most other gems have opted in already.

brightball 11y ago

This should be one of the perks of Go since the compiler won't let you include anything that you aren't using.

jerf 11y ago
That's an orthogonal issue. You won't accidentally bring in something completely unrelated because you forgot to remove it from the "include"s, but nothing technically stops you from having a deep dependency chain. Culturally the Go community is aware of the issue, though. Still, it isn't hard to bring in a chain accidentally. I have a program than needs to do a query against the local LDAP system to extract members of a specified group. The LDAP library brings in five more libraries for parsing all the various bits of LDAP. Since this isn't C, I'm a bit less nervous about pulling in, say, a BER decoding library, because at least Go is generally memory-safe, but, still, that's a somewhat large stack for such a simple query. (Traditionally in C, you might as well just expect any library that decodes anything remotely binary-esque will have buffer overflows. C is a DSL for writing buffer overflows.) And yet, I'd be insane to try to implement some sort of just-barely-minimal LDAP client to do it myself. Looking at my local godoc instance's full set of packages that have gotten pulled in one way or another is still sort of intimidating. Some of them are cases where I'm just pulling in a subdir and got an entire large repo (the golang experimental repos do that a lot), but, still, I've got a lot of stuff in there. If you're a go programmer and you haven't run godoc locally and had a look at the packages page, have a look. You may be surprised.

gravypod 11y ago

Yes, this is the sort of thing that scares me away from Ruby. I'm worried this sort of "screw it just add a library" is going to spread further in my language of choice: Java. In my time doing open source programming on the side, I've found that it has become more common with the advent of things like mvn and gradle to just slather on layers to your stack even for the simple tasks. Need a function to turn a byte buffer into a string? Download these 3 Apache commons libraries and their dependencies. I understand if you are relying on a large portion of a library and you need to use it, but why bring an entire library in for one function.

mbrock 11y ago
There are ideas floating around that make it appealing to do just that. For example, the commons library might be considered "battle-tested," and who really knows what could happen with your own custom byte-buffer-to-string function? Maybe you missed something? Maybe there is some "best practice" that you didn't follow? Maybe the commons library is optimized? And writing your own thing doesn't add business value. Developer time is more expensive than dependencies. And so on. Me, I very often prefer to write things myself, in a way that can get labelled as NIH. My inclination is based on bad experiences with trying to debug external libraries. Sometimes I look at open source library code and find staggering complexity that I have no need for. Yes, maybe the library is great, but if its combinatorial size is 10,000 times the functionality that we need, then depending on its correctness becomes scary to me. And when I need to customize it, due to some requirements alteration, I will find it difficult and tedious. Black-box type libraries for isolated complicated tasks like codecs and crypto I will happily use. Otherwise, I'm a fan of the "design patterns" approach to reuse, which is all about learning from others, but without creating reusable formal abstractions in library form. So if you teach me how to write an URL router, I can then use your insights without depending on your code base, and I can adapt the idea so it fits my application perfectly.
- gravypod 11y ago
  I agree completely. I'm not the one who will sit down and attempt to implement my own RSA, or hashing algorithms. It's like pornography. I don't know how to define it, but I definitely know when I see it. There are correct times to use libraries. But pulling a 15meg for some simple functionality is not a good practice in my opinion.

Animats 11y ago

Avoid shims. There are lots of libraries that just put one interface on top of another interface. They don't do much actual work. Pulling in shims, especially if they pull in lots of other stuff you're not using, should be avoided. If the dependency does real work you'd otherwise have to code, then use it.

dawnerd 11y ago
cough Mongoose. Been moving away from it on my projects. While it does provide a nice interface, it just creates more work down the road.

jeffdavis 11y ago

Your app/library inherits the technical debt of all its dependencies. There's a natural tension between code reuse and avoiding dependencies. If you can avoid a big dependency by writing a couple hundred lines of low-maintenance code, its probably worth it.

BinaryIdiot 11y ago

This is a great read that can be applied to node.js very much. I've seen apps that include 10, maybe 20 dependencies but when you flatten out the full dependency tree? Thousands. It's incredible and if one of those dependencies screws up semantic versioning or just screws up in general it can be a nightmare to debug and fix. This is why every 1.0 product I work on I include every dependency that speeds up my development. In 2.0 the first things to do is prune all unnecessary dependencies and start minor rewrites when a dependency can be done in house (yeah yeah reinventing the wheel is a problem but most npm dependencies are small and many can be recreated internally without issue). This is even more important if you're creating a library / module. My msngr.js library uses zero dependencies and yet can make http calls in node and the browser because it was easy to implement the minimal solution I needed without bringing in dependencies to support a single way of calling http.

nextos 11y ago
This sadly also happens in some Linux repositories that add too many dependencies to a few key packages. On NixOS, last time I tried, installing mutt ended up bringing python as well.
- davexunit 11y ago
  In Nix, it's very easy to make a minimal mutt variant by removing the python input.
  - nextos 11y ago
    Yes, but these things shouldn't be happening. Default builds of packages should be kept more minimal. I understand it's hard with Nix philosophy, and things are improving with different package outputs. For the record, python was pulled indirectly via the gnupg dependency I think.
    - davexunit 11y ago
      It's a difference in philosophy. I think packages by default should be full-featured, and minimal variants can be created for those that want it.

sksixk 11y ago

what's the point? none of them are realistic. "no code" - well, it's there for a reason. "own it" - do i really want to write my own minimal implementation? i understand that dependency is a pita but this post doesn't provide anything worthwhile.

jowiar 11y ago

I find dependencies to be a very good indicator for how my code should be modularized. That is, rather than pulling a boatload of dependencies into "the application", pull a couple dependencies into a module, and then depend on the module. It makes it very easy for dependencies to be a "well, it gets the job done for now, and I can reimplement that myself if that changes" sort of thing.

edvinbesic 11y ago
I found this approach to work well for me as well. It has payed off many times. I try to wrap most of my dependencies so that if I later feel like I need to pick some low-hanging fruit I can implement some of the functionality internally while maintaining the original api.
- jowiar 11y ago
  Also, this is probably the single biggest difference for me when working with a static vs. dynamically typed language. With Static Typing, there's a "translate the dependency's types into the application's types" step that pretty much screams "put a seam here!". With Dynamic Typing, it's a bit less loud.
  - Aeolos 11y ago
    That's pretty much a C/C++ problem though. I don't think I've ever had to translate types in C# or F#.
    - jowiar 11y ago
      I was referring to that, with Scala, I like to avoid leaking the innards of a JSON serialization library in the code for an API client, instead returning domain objects. Whereas with, JS or Python, the initial approach is to sling around the blob of JSON.

laumars 11y ago

This article would be more accurately written as "prefer the standard library over 3rd party solutions" since all the examples given still required dependencies, but ones that are shipped as part of the language runtime (Ruby in this case). However when discussing languages with no specific standard library or languages who's standard library is missing feature y, then it's quite understandable to use a 3rd party battle tested dependency. In fact I'd go further and say it would be advisable to use a respected 3rd party library when dealing with code which handles security or other complex concepts with high failure rates.

50CNT 11y ago
Matter of fact, sometimes it's better to be using a respectable 3rd party library. Requests vs. urllib2 in Python springs to mind, and I'm sure there's more examples.

spullara 11y ago

This is a huge mistake if applied without care. Building things from scratch necessarily will introduce more bugs, more maintenance costs and leave you with a codebase that suffers from a lack of maturity.

edvinbesic 11y ago
I don't think the sentiment is 'implement everything from scratch', but rather that if something exists natively try to use that instead of pulling in other dependencies, like in the http client example.
Chris_Newton 11y ago
Building things from scratch necessarily will introduce more bugs, more maintenance costs and leave you with a codebase that suffers from a lack of maturity. Unfortunately, in some programming language ecosystems where having many small and transitive dependencies on modules from an non-curated repository is common, none of those three things is necessarily true. Code reuse is not a trivial problem, and you always have to weigh the benefits against the costs and risks to decide whether it’s worth it. If we’re depending on GitHub repositories with a dozen files and three subdirectories just to provide some simple functionality that any junior programmer could implement directly in five lines of code, we’ve probably lost the plot. On the other hand, if we have a full in-house implementation of encryption algorithms we use to throw sensitive customer data around between the browser and our servers, we’ve also probably lost the plot.

liveoneggs 11y ago

mojolicious does a great job with this, supporting optional dependencies as progressive enhancement (installing EV will speed you up, but you don't necessarily need it) http://mojolicious.org/ http://mojolicious.org/

greggman 11y ago

This reminds me of an example I ran into yesterday. I haven't used webpack yet but I saw a question on SO of someone wanting to use some package called glslify. I thought I'd take a look and maybe learn webpage in the process. From the description all glslify does is look for files with the extensions .glsl, .frag, and .vert and lets you get their contents with `content = require(filename)`. Sounds like it would be at most 10-30 lines of code. Nope npm install --save glslify-loader webpack-glsl-test@1.0.0 /Users/gregg/temp/webpack-glsl-test └─┬ glslify-loader@1.0.2 └─┬ glslify@2.3.1 ├─┬ bl@0.9.5 │ └─┬ readable-stream@1.0.33 │ ├── core-util-is@1.0.2 │ ├── isarray@0.0.1 │ └── string_decoder@0.10.31 ├─┬ glsl-resolve@0.0.1 │ ├── resolve@0.6.3 │ └── xtend@2.2.0 ├─┬ glslify-bundle@2.0.4 │ ├─┬ glsl-inject-defines@1.0.3 │ │ └── glsl-token-inject-block@1.0.0 │ ├── glsl-token-defines@1.0.0 │ ├── glsl-token-depth@1.1.2 │ ├─┬ glsl-token-descope@1.0.2 │ │ ├── glsl-token-assignments@2.0.1 │ │ └── glsl-token-properties@1.0.1 │ ├── glsl-token-scope@1.1.2 │ ├── glsl-token-string@1.0.1 │ └── glsl-tokenizer@2.0.2 ├─┬ glslify-deps@1.2.5 │ ├── events@1.1.0 │ ├─┬ findup@0.1.5 │ │ ├── colors@0.6.2 │ │ └── commander@2.1.0 │ ├── graceful-fs@4.1.3 │ ├── inherits@2.0.1 │ └─┬ map-limit@0.0.1 │ └─┬ once@1.3.3 │ └── wrappy@1.0.1 ├── minimist@1.2.0 ├── resolve@1.1.7 ├─┬ static-module@1.3.0 │ ├─┬ concat-stream@1.4.10 │ │ ├── readable-stream@1.1.13 │ │ └── typedarray@0.0.6 │ ├─┬ duplexer2@0.0.2 │ │ └── readable-stream@1.1.13 │ ├─┬ escodegen@1.3.3 │ │ ├── esprima@1.1.1 │ │ ├── estraverse@1.5.1 │ │ ├── esutils@1.0.0 │ │ └─┬ source-map@0.1.43 │ │ └── amdefine@1.0.0 │ ├─┬ falafel@1.2.0 │ │ ├── acorn@1.2.2 │ │ ├── foreach@2.0.5 │ │ └── object-keys@1.0.9 │ ├─┬ has@1.0.1 │ │ └── function-bind@1.0.2 │ ├── object-inspect@0.4.0 │ ├─┬ quote-stream@0.0.0 │ │ ├── minimist@0.0.8 │ │ └─┬ through2@0.4.2 │ │ └─┬ xtend@2.1.2 │ │ └── object-keys@0.4.0 │ ├── shallow-copy@0.0.1 │ ├─┬ static-eval@0.2.4 │ │ └─┬ escodegen@0.0.28 │ │ ├── esprima@1.0.4 │ │ └── estraverse@1.3.2 │ └─┬ through2@0.4.2 │ └─┬ xtend@2.1.2 │ └── object-keys@0.4.0 ├── through2@0.6.5 └── xtend@4.0.1 > 4 meg of source files --- update: I think maybe I misunderstood the description. glslify actually parses GLSL and re-writes it in various ways so maybe this is a bad example. I've seen other though. Like 40k+ lines of deps for an ANSI color library or 200k+ lines deps and native node plugins for launching a browser from node.

mwcampbell 11y ago

A large number of dependencies is only a problem in environments that aren't amenable to per-function static linking or tree-shaking. These include dynamically typed languages like Python, Ruby, and JavaScript (except when using the Google Closure Compiler in advanced mode), but also platforms like the JVM and .NET when reflection is allowed. Where static linking or tree-shaking is feasible, the run-time impact of bringing in a large library but only using a small part of it is no more than the impact of rewriting the small part you actually use. Edit: Dart is an interesting case. It has dynamic typing, but it's static enough that tree-shaking is feasible. Seth Ladd's blog post about why tree-shaking is necessary [1] makes the same point that I'm making here. [1]: http://blog.sethladd.com/2013/01/minification-is-not-enough-you-need.html http://blog.sethladd.com/2013/01/minification-is-not-enough-...

shoover 11y ago
Yeah, but it seems like you listed most of the platforms most people actually use, all in the X column. How do we get from 100MB Electron deployment and fat, partially used jars and dlls to this magical tree-shaking world?
- mwcampbell 11y ago
  You're right; most of the languages and platforms used for developing applications don't have reliable support for tree-shaking or fine-grained static linking. But there's hope for some of them. When building Android applications, it's common to process the JVM bytecode with ProGuard before converting it to Dex bytecode. ProGuard includes a tree-shaking step. Sometimes it's necessary to tell ProGuard about specific classes or class members that it should leave alone, if they're accessed dynamically (e.g. using java.lang.reflect or Class.forName). But it's still better than nothing. Likewise, .NET applications for the Windows Store are compiled to native code using .NET Native, and that compilation includes a tree-shaking step. This introduces some limitations on the use of reflection. I'm guessing similar limitations will apply to the native compilation option of .NET Core. As for JavaScript, Google's Closure Compiler can do tree-shaking. But I don't know if the Closure Compiler's advanced mode works with any of the popular JavaScript libraries or frameworks, or just Google's Closure Library.

gtrubetskoy 11y ago

Your programs shouldn't do things you do not understand. You do not have to be an expert in cryptography, memory allocation or b-trees, etc, but if this is what your app requires, then you should take the time to read up on it and carefully research what is out there if you suspect it is beyond your abilities to implement. If you take the time to do your research, the choice between rolling your own, copying or adding a dependency will become clear. If it's not becoming clear, then you haven't finished your homework. Learning is a good thing, yes it takes time, but it's time well spent, and it's fun above all. You may discover that this thing that you thought was hard and needed a dependency is really a few lines of code (a good example is a graph implementation). It might even change your career path. At least that's been my experience in the nearly two decades of writing software.

surfmike 11y ago

if you're running multiple rails processes on a server like this, couldn't you somehow do the initialization in one process, then fork off the new processes? wouldn't that prevent the base libraries from being copied in memory?

lotyrin 11y ago
Yes. Many app servers do this. (Not being super familiar with the Ruby ecosystem, I am not sure specifically which ones.)
paulannesley 11y ago
Yes, http://unicorn.bogomips.org/ http://unicorn.bogomips.org/ popularized this for ruby / rack / rails with its forking model and preload_app option. http://puma.io/ http://puma.io/ does the same thing, but additionally runs multiple threads in each process. The garbage collector in Ruby 1.8 / 1.9 negated the benefits of copy-on-write forking, but that's fixed since Ruby 2.0

nickpsecurity 11y ago

Obligatory essay from PHK on the effect the author describes: http://queue.acm.org/detail.cfm?id=2349257 http://queue.acm.org/detail.cfm?id=2349257 History continues to repeat itself. Fake reuse and proliferation of unnecessary bloat are two of those recurring themes. Fight it whenever you can. The old TCL, LISP, Delphi, REBOL, etc clients and servers were tiny by modern standards. They still got the job done. Strip out bloat wherever you can. Also, standardize on one tool for each given job plus hide it behind a good interface to enable swapping it out if its own interface isn't great.

PaulHoule 11y ago

They should sell sonatype for this.

Rafert 11y ago

Excellent article. I tried to develop on GitLab once but the sheer amount of gems it pulls in (~100 directly declared, 350+ including dependencies if I remember correctly) with a bunch of installation problems made me decide it was not worth the hassle.

sytse 11y ago
I'm sorry to hear you had installation problems trying to develop for GitLab. Have you tried the GitLab development kit https://gitlab.com/gitlab-org/gitlab-development-kit https://gitlab.com/gitlab-org/gitlab-development-kit? If still are interested and experience problems please email support@gitlab.com and reference this comment for help. I agree with the article, the less dependencies the better. GitLab's gemfile.lock https://gitlab.com/gitlab-org/gitlab-ce/blob/master/Gemfile.lock https://gitlab.com/gitlab-org/gitlab-ce/blob/master/Gemfile.... has over 1000 lines and GitLab uses a lot of memory. We try to be careful what we pull in but if anyone has suggestions which can be removed please let us know. Recently we found out that we still had to remove Redcloth as a dependency, it will be gone in GitLab 8.5.

agentgt 11y ago

I find this argument sort related to the framework vs library and opinionated vs agnostic. Being an old fart Java developer I generally prefer things where you can plugin your own implementation (ie agnostic). That is there is an extreme for killing your dependencies of either extreme copy'npaste OR which every library offers a plugin SPI (ie inversion of control) (or a combo of both). The problem with the dependency injection above approach (aka Spring prior to Boot) is that you have developers doing lots of custom crap, bloated/overly engineered libraries, increased ramp up time, and configuration hell. But I still think this is probably better than ole copy'n paste.. most of the time. I do hate dependencies though.

mwcampbell 11y ago
You're probably more experienced in java development than me. But maybe a good rule of thumb is that most Java libraries shouldn't use reflection or dynamically loaded classes (i.e. using Class.forName or ClassLoader). That rules out most dependency injection frameworks.
- agentgt 11y ago
  That is correct that the libraries should not have DI but I should be able to wire up the library on my own and not let the library do its own static initialization. What is far worse than Class.forName and other crap is libraries self imposed singletons. Take for example Hystrix. I'm just now fixing that the thing loads up its own configuration framework (Archaius) which uses static initialization. Archaius needed like 10 other dependencies. This is all really because Hystrix uses static singleton (HystrixPlugins) and many frameworks need this or else is incredibly difficult to get an implementation up (ie using pseudo singleton to avoid excessive passing of a context). https://github.com/Netflix/Hystrix/pull/1083 https://github.com/Netflix/Hystrix/pull/1083
- lmm 11y ago
  Java makes plugin-like scenarios very hard. The kind of thing you'd do with a typeclass in languages that have them. There's OSGi (the horror, the horror) which maybe-kinda-sorta-works, or there's the SPI where you put the class name in a .service file (which will then be instantiated though... reflection). When those are the alternatives, DI frameworks aren't so bad.

giancarlostoro 11y ago

One issue I had with Ruby on Rails was getting MySQL drivers to even cooperate on both Windows and Linux. At the end of the day I wound up sticking to Python and other languages instead. I don't mind using any language, but if the language is fighting me due to native dependency hell then I can't really do much. Even to just use SQLite was a bit of a painful experience, yet on Python SQLite works out of the box without any effort on my part (on Windows). Oddly enough. I'm looking to getting back into Ruby at some point later this year, but I might ignore Rails altogether so I don't miss out on learning a new fun language.

3minus1 11y ago

I remember one nasty bug where someone has included one function from bootstrap.js library and someone else had included the entire library. So both functions were running causing an issue.

adenadel 11y ago

This reminds of of this article http://www.joelonsoftware.com/articles/fog0000000007.html http://www.joelonsoftware.com/articles/fog0000000007.html Apparently Microsoft's Excel team had even written their own C compiler.

ocdtrekkie 11y ago

So, I've been writing a home automation system using the .NET Framework (with Visual Basic, I'll wait until you finish laughing).......... Okay. I've made a point not to add any third parties references and packages I can avoid. I went ahead and got a third party scheduling engine, and the SQLite provider, but beyond that, I'm writing everything else myself so far. First of all, I'm learning a lot in having to write stuff myself. At the very least, it's a great educational experience. I've worked with a lot of code samples, so I'm not going totally from scratch, but they're all at the very least tailored to my needs. But for me, the big thing is keeping everything thin. The program loads in milliseconds. Almost all of the reference data for what it's built on is in one place (the .NET Framework Reference). And key, is that the features my program supports are the features I want and need, not the features some dependency has told me to have. The biggest dependency I have, Quartz.NET, is actually the most confusing part. It's not structured like the rest of my program is, it's documentation leaves some things to be desired, and it does a lot more than I need it to. There's a lot of bloat I could cut out if I wrote my own scheduler, and maybe someday I will.

jcoffland 11y ago

This is not only an issue for running software but also a huge issue for compiling/building software. Each dependency adds the potential to break your builds in new ways. As the software your program depends on evolves the risk increases that it will change the way your program executes or cause it to fail to build. Many devs will insist you do not reinvent the wheel by writing things like JSON parsers but you always have to weigh the cost of adding a dependency. It's not free.

cdnsteve 11y ago

Double edge sword. Deps are great! Functionality added quickly. Deps are terrible! They broke my app. If your app has a long shelf time, the less deps you rely on, the easier to manage from what I've seen. For some reason Golang feels like it makes sense here. Pretty much everything you need is in core. *Disclaimer, I don't have any Golang apps in prod but I'd love to hear from those that do.

beat 11y ago

I have one thing to say about all of this... Nokogiri.

twic 11y ago

Does any of this sound familiar: - Test gems loading in production. That does not sound familiar! Is this a thing which happens with Rails?

nona 11y ago
No, but sometimes people are a bit careless in their Gemfile I guess.

vinceguidry 11y ago

Gems I use fall into three categories. A lot of my projects are just wrappers around one main gem. Rails, Nokogiri, Roo, API wrapper gems. These are 'project gems'. If they give me problems, I'll re-evaluate the scope of the project and perhaps pick another gem to orient the project around. Once the project reaches maturity, I'll default to fixing the problem rather than re-engineering it unless the problems run deep. Sometimes I'll use gems like Phoner to handle datatypes that are too tricky to do with regular Ruby. I'll call these 'utility gems'. When I include a utility gem, generally it has one job and one job only, it's invoked in exactly one place in the code and gets included in that file. I can generally replace a utility gem with stdlib Ruby code if I really need to. I also have what I call 'infrastructure gems'. These are gems like pry, capistrano, and thor that I tend to include in every project where it seems they would be useful. These are gems that are worth getting to know very well because they solve really hard problems that you don't want to use stdlib for. If these give me problems I will do whatever I need to to resolve them and understand why the problem exists, because the costs of migrating off of them would be steep. The decision to use a gem should not be taken too lightly, but nor should it weigh large on the mind. Be quick to try it out, but also quick to take it out.

edejong 11y ago

The problem with dependencies is that developers approach it from a top-down approach. The question they answer is: I need an HTTP client, JSON API, monitoring tool, logging framework. Never do they ask the opposite: what kind of foundations do I need? What elementary blocks do I need to have or learn in order to make a JSON parser in 5 lines of code? Is it possible to do logging without all the cruft? Can I write the library in the same amount of time as I can read the docs? Could the code I write be the docs? Similar line of reasoning: can I leverage my OS to do scheduling/IPC/monitoring/security? If it can't, should we lobby for better OSes (that might scale over multiple machines?) Does Linux/Docker offer the right fundamentals? Dijkstra was truly right: the art of programming is the art of managing complexity.

LukeB_UK 11y ago

I disagree, and this quote I've seen floating around the internet sort of sums the idea up to me (albeit with a music analogy): > I thought using loops was cheating, so I programmed my own using samples. I then thought using samples was cheating, so I recorded real drums. I then thought that programming it was cheating, so I learned to play drums for real. I then thought using bought drums was cheating, so I learned to make my own. I then thought using premade skins was cheating, so I killed a goat and skinned it. I then thought that that was cheating too, so I grew my own goat from a baby goat. I also think that is cheating, but I’m not sure where to go from here. I haven’t made any music lately, what with the goat farming and all.

gglitch 11y ago
Charming quote. The same argument could be made about making your own food "from scratch." But it's not a solid refutation of the essay. Let's say that if one end of the spectrum is unlimited dependencies and complete indifference to the complexity and size of the project, and the other end of the spectrum is raising your own goats, there must be an ideal somewhere in the middle.
- LukeB_UK 11y ago
  I completely agree, but the extremism in the essay is ridiculous. If I'm writing some software, I don't want to have to roll my own version of every little thing when there are battle tested libraries out there that already do it and benefit from many people using it. If my use case is vastly different, sure, but if it's the same use case as everyone else then I see little benefit.
  - hackbinary 11y ago
    Have you read a modest proposal by J Swift. Satire at its best! https://en.wikipedia.org/wiki/A_Modest_Proposal https://en.wikipedia.org/wiki/A_Modest_Proposal
  - justaaron 11y ago
    if they are so battle-tested then they probably are not like the email gem that the author spoke of that was using 10mb more memory than necessary for dependencies... node-land is also getting a bit silly this way... to say nothing of java etc... since we are into silly quotes i'll riff and probably get this wrong but: "you wanted a banana but you got the gorilla holding it and the whole jungle too" joe armstrong (erlang) on Class-based OO inheritance etc...

Walkman 11y ago

There is a counter example of his reasoning in the Python world. There is a HTTP client library "urllib" in the standard library, but nowadays everyone rather pulls in the external dependency "requests" because the urllib API is terrible. It is mature, well tested, good documented code though.

_ZeD_ 11y ago
just as a data point: I don't use request, I onestly prefer the urrlib (was urllib2) API offered by the standard library

fixermark 11y ago

Incidentally, though the author says this can apply to any ecosystem, finding it applies to Ruby too often is what pushed me out of developing Rails apps. At least at the time I was using it heavily, the Rails space just wasn't stable enough to trust that I wouldn't have to learn an entirely new wheel to get my work done every time I went in to fix a relatively small problem. "Can I implement the required minimial functionality myself? Own it" is advice one gives if one can't trust the libraries one depends upon to stay healthy, performant, and applicable to your use-case. Nobody'd recommend re-implementing readline or printf; if you have some heavy-lifting mathematics to do in Python, use numpy.

technoir 11y ago

If there is an analogue in the standard library you should have a compelling reason to use an alternative. Wish this could be filed under "common" sense. Thanks for articulating and presenting this principle, among others. Great writeup.

joesmo 11y ago

So to summarize, to get rid of an extra 10 (or even 100) megs in Ruby (or 0 megs in some other languages) of memory usage (and disk usage don't forget!) spend weeks rewriting, testing, and integrating your own code instead of using already written, tested, and integrated code. Now, that I've clarified the article's point, how can anyone not follow this "best-practices" advice? </sarcasm>

EternalFury 11y ago

How long did it take anyone to realize this nightmare? Since we are on the path to major discoveries, let's talk about runtime, runtime-dependencies and all that. Every Ruby app ever created is stuck somewhere on the time axis, before its origin.

justaaron 11y ago

amen

aidenn0 11y ago

Slightly related, I was packaging up a webapp in docker, and one thing the application did was form-fill PDFs. I had been using pdftk to do this, but it turns out pdftk is written in gcj, and gcj pulls in a lot for its runtime libraries. I wrote a small program using the mupdf libraries and cut the size of my docker image by over 400MB.

EvanPlaice 11y ago
Congrats, you just violated the GPL. You were already violating the proprietary license if you're using his for commercial purposes without paying. Not all free software is free.
- aidenn0 11y ago
  Actually mupdf uses AGPL, and so the program I wrote and linked with mupdf is also under AGPL.
- aidenn0 11y ago
  I just saw the commercial redistribution clause for pdfTK if that's what you're talking about. It does not affect me since this was not a commercial application, but it would seem to me that that clause itself is a violation of the GPL, since pdfTK links to GPL software and it is also a contradiction of a separate place on the site that claims the same software is licensed under the GPLv2. Preventing commercial redistribution is not compatible with the GPL.

justinator 11y ago

Perl takes a pragmatic take on this (as well as other takes...) with the collection of ::Tiny CPAN modules that just do one thing pretty OK. Things like Try::Tiny that help immensely with exception handling - something you don't want to really roll you own. It itself does not have any dependencies that aren't in core: http://deps.cpantesters.org/?module=Try%3A%3ATiny;perl=latest http://deps.cpantesters.org/?module=Try%3A%3ATiny;perl=lates...

cjhveal 11y ago

Interesting to note that the Stripe gem removed one of its dependencies seemingly in reaction to being called out at the end of this article.