7 ms·
The Broken Promises of MRI/REE/YARV
- deleted 15y ago[deleted]
- lusis 15y agoThe article covers it but those are different C-based Ruby VM implementations: - 1.8 is MRI - 1.9 is YARV - 1.8 plus performance patches is REE
- deleted 15y ago[deleted]
- epochwolf 15y agoMRI = Matz's Ruby Interpreter = Ruby 1.8 YARV = Yet Another Ruby VM = Ruby 1.9 REE = Ruby Enterprise Edition = Ruby 1.8 with a modified garbage collection system to make it friendly to vforking. These are all versions of ruby written in C.
- CPlatypus 15y agoDamn, your reply and the other four pointing out that they're Ruby VMs reflects very poorly on the intelligence of HN commenters. Did you guys even RTFA? I know these are Ruby VMs, but the article is about the C code that's used within them. That's the only place where "volatile" has any meaning at all. Maybe "volatile" really is unknown among Ruby programmers, but among the people who implement interpreters for Ruby or any other language I can assure you it's pretty common knowledge. I confess, I just don't know how to deal with such epic stupidity more gracefully than this. Sheesh.
- grifaton 15y agoThey're Ruby interpreters -- Matz's Ruby Interpreter, Ruby Enterprise Edition, and Yet Another Ruby VM respectively.
- deleted 15y ago[deleted]
- nkassis 15y agoRuby virtual machines. MRI is the VM for ruby 1.8 (Matz ruby implementation and Matz is the creator of ruby), YARV is the newer incanation that was developed for ruby 1.9 these are all written in C. There plenty of other ruby VMs like rubinius or jruby etc.
- judofyr 15y agoMRI - Matz' Ruby Interpreter (the default Ruby) REE - Ruby Enterprise Edition (unofficial branch of Ruby 1.8) YARV - Yet Another Ruby VM (the VM used in Ruby 1.8) (Although in this case I think he means MRI = 1.8 and YARV = 1.9)
- ssmoot 15y agoMinor correction, YARV, written by ko1, has never been a part of the Ruby 1.8.x line to my knowledge. Some language features, (mostly stdlib I think) were back-ported in 1.8.7, but 1.8.x is and always has been an interpreter where 1.9.x has always been YARV (a VM). Useless trivia: Once upon a time Ruby2 was going to be called "Rete" IIRC. Or maybe "Rite"? I doubt it's in any shape to be called a "formal" plan at this point, and who knows if it'll ever actually see the light of day. It was supposed to drop optional parens IIRC, it's even in the original Pickaxe I think, but I doubt that's still on the board. Don't remember what else.
- deleted 15y ago[deleted]
- deleted 15y ago[deleted]
- deleted 15y ago[deleted]
- thibaut_barrere 15y agoI do appreciate the technicality of the article, but I'm not sure to agree with the first point of conclusion: how does it makes MRI (and related) 'fatally flawed' though? (real question). What makes it 1/ irreversible and 2/ bad for today's users? EDIT: as well, I wouldn't stop using Ruby because of that; I would use JRuby or Rubinius or IronRuby (if I understand well, these ones are not affected?)
- phillmv 15y agoIt's a bit hysterical. Shit! MRI/YARV/REE are inherently fatally flawed! All that code I have running in production must be a FIGMENT OF MY IMAGINATION! SAVE YOURSELVES
- benblack 15y agoI am running this code in production, hence it cannot have bugs. QED. Yours in perpetual bogglement, Lil' B
- koudelka 15y agoThe point was clearly not that it has no bugs, but that if something is working to spec, it's working.
- msbarnett 15y agoThat is an interesting strawman you've constructed, as accepting it requires the reader to conflate the idea of bugs in general and "fatal flaws". Obviously all non-trivial code working in production not only can have bugs, but will have bugs. Just as obviously, no reasonable person would consider those "fatal flaws" for any reasonable definition of the word fatal. MRI/YARV's Conservative GC opens up some bedevilling classes of bugs for gem writers, obviously. Calling that a "fatal flaw" when millions of lines of production code continue to function despite its presence is nothing but over-the-top hyperbole.
- pshc 15y ago
- wingo 15y agoCute. The Boehm-Demers-Weiser collector has GC_reachable_here for this reason. Guile has scm_remember_upto_here since before it switched to libgc. I'm sure other systems have their things too. That said, I like Handle, the RAII thing that V8 uses. It also allows for compacting collection. Too bad C doesn't do RAII.
- thibaut_barrere 15y ago.Net has GCHandle [1] and I believe the JVM calls to JNI have a similar mechanism (GetXXCritical [2]) [1] http://www.shafqatahmed.com/2008/05/memory-control.html http://www.shafqatahmed.com/2008/05/memory-control.html [2] http://publib.boulder.ibm.com/infocenter/javasdk/v5r0/index.jsp?topic=/com.ibm.java.doc.diagnostics.50/diag/understanding/jni_copypin.html http://publib.boulder.ibm.com/infocenter/javasdk/v5r0/index....
- onedognight 15y agoWhile C doesn't support RAII, gcc does: https://secure.wikimedia.org/wikipedia/en/wiki/Resource_Acquisition_Is_Initialization#C_Example_using_GCC_extensions https://secure.wikimedia.org/wikipedia/en/wiki/Resource_Acqu...
- softbuilder 15y agoWell plus one for a blog post with a theme song, anyway.
- davesims 15y agoThis post is a weird mix of careful technical analysis and douchey, Zed Shaw-style hysterical overstatement. However, I would like to see Matz' response to the recommended steps for a fix at the end. Sounds like a reasonable goal to add for Ruby 2.0. Note to self: Listening to Papoose while writing a technical blog post turns your otherwise important observations into a Chicken Littleish, end-of-the world rant.
- Nelson69 15y agoI kind of branded it a bit "douchey" at first too but then as I thought about it, it seemed remarkably restrained considering he debugged this issue. It's not like this happened all the time, had to get kind of lucky and build and calibrate a system just right to capture it. I don't intend this to be an inflammatory question, I'm sort of a perpetual ruby novice, it's never been my day job and I've never managed to sort of catch up with the community, as soon as I feel pretty good with something I find it's been obsoleted a couple times. I like it but how does the community at large deal with stuff like this? This guy found a real bug and invested some time in it, do other rubyists just deal with crashes and restart their stuff? Do they just consider it part of "being on the cutting edge?" Or do they not even notice?
- msbarnett 15y agoIn practice crashes due to this issue simply do not occur very often. I think I've had the VM segfault twice in the last two or three years. That's what makes the hyperbolic tone of this article so douchey; he wrote up an interesting dissection of an edge case issue as though it were an ongoing catastrophe, mostly just to inject a bunch of chest-thumping rock-star bravado that added nothing of value to the discussion.
- Nelson69 15y agoI get that. It's probably related to how many libraries you use and a lot of other things? There might be pathological ways to make it happen more frequently. It all depends on how and when it happens though.
- wonnage 15y agoCan someone dissect this a little more? My understanding is the pointer to str never gets written to the stack, and so str on the heap might get freed before zstream_append_input makes use of it. But how could the GC see this/what is the faulty assumption?
- eonwe 15y agoMy understanding is that Ruby GC just runs through its heap of Ruby objects and sees which of them are reachable based on other objects in the Ruby heap and C-stack/registers. Faulty assumption seems to be that counting references only to RVALUEs (Ruby objects in heap) is enough to determine if a part of memory can be freed. This breaks down in C-extensions where macros extract some part of the object or something pointed by it for use. In this case RSTRING_PTR extracts the C char-array used by str for zstream_append_input to use (lets call it arr). If zstream_append_input or any calls underneath it tries to allocate a new Ruby object, GC may get called and str (and thus arr) may get freed because there are no references left to it anymore (no heap/stack/register because the register value was overwritten). And this seems to require all Ruby C-extension writers to lock the objects they're using through macros with RB_GC_GUARD. Edit: note that there are no references left to str
- fhars 15y agoThe point is that the GC cannot see that and so assumes that the object is no longer referenced and can be freed. A conservative collector works by scanning the live memory of the process for things that look like pointers into the same live memory and then assumes that all objects that are not the target of any of these pointers are garbage. Tough luck if the only reference to a live object lives in a register.
- ice799 15y agoregisters are scanned, too. the bug is not that the ref is in a register. the bug is that there are no refs anywhere. not on the stack and not in any register.
- 15y ago
- xpaulbettsx 15y agoSo, what this really seems to boil down to, is: The Ruby C API is returning objects that are not correctly reference-counted for a short period of time and are incorrectly subject to GC. This doesn't seem fatal to me, just not reasonably fixable from the GC side. It might be true, that a new API is needed to hold refs in the C side.
- benblack 15y agoI am apparently in that foolish minority that believes language runtimes should not segfault/corrupt themselves while running correct code. That this problem requires significant effort just to hack around, while actually fixing it would take a major architectural change, is what elevates this from mere "lolwut?" to fatally flawed. There are good alternative runtimes for Ruby, such as the JVM and the CLR, that do not suffer from this problem. Y'all should use them. Funktacularly yours, Lil' B
- davesims 15y agoIf edge case segfaults were fatal flaws Windows should never have shipped. I say 'edge case' because obviously there are millions of lines of Ruby code running for years on MRI/YARV/REE that have not encountered this error often enough to cause the kind of breathless panic you seem to think is appropriate. BTW the CLR is not a good alternative runtime for Ruby, might not ever be: http://www.zdnet.com/blog/microsoft/whats-next-for-microsofts-ironruby/7034 http://www.zdnet.com/blog/microsoft/whats-next-for-microsoft... You did good work here -- don't hurt your credibility with overstatement.
- jjore 15y agoWell, the problem here is that C using gems are going to often be memory corruptingly buggy until and unless either the gem source is updated to declare the proper parts volatile or Ruby's own C API is reworked to evolve this bug out of existence and then gems would have to be updated to use the API anyway. Both problems are hard and the current state of affairs is apparently some random amount of the time we'll get memory corruption bugs.
- kingkilr 15y agoI think this goes to a pretty simple point: anything you have to do by hand you will eventually get wrong. Thus, to a first approximation anything that can be automated, probably ought to. To show off this principle I'm going to show off some of the PyPy source code: https://bitbucket.org/pypy/pypy/src/default/pypy/module/select/interp_epoll.py https://bitbucket.org/pypy/pypy/src/default/pypy/module/sele... This is the implementation of `select.epoll`. Somethings you'll notice there's no GC details (allocations outside the GC of C level structs are handled nicely with a context manager), and we have a declarative (rather than imperative) mechanism for specifying argument parsing to Python level methods, this ensures consistency in readability as well error handling, etc.
- deleted 15y ago[deleted]
- kingkilr 15y agoNope, wrapped values are interpreter level objects, they're the kind of things that exist at the Python level, in PyPy they're called things like W_IntObject, on CPython they're PyIntObject, I'm sure Ruby has the same. Then there are unwrapped ints which are machine level integers.
- deleted 15y ago[deleted]
- iam 15y agoI think this is a problem that exists across any VM that implements a GC, not just Ruby. .NET CLR has the exact same problem (perhaps a harder one, since CLR has a moving GC), so anytime they touch GC references (pointers to objects that are collectible) it's always wrapped in an explicit GC stack frame (think GC struct that lives on the stack). Furthermore, all reads/writes are carefully done with macros (which of course expands to volatile + some other stuff) to make sure the compiler doesn't optimize it away. On the one hand, this is nice because they don't need to scan the C-stack (it scans the VM stack and the fake GC frame stacks -- well it's one stack but you skip the native C frames), on the other hand this means that any time a GC object is used in C code (ok, actually it's C++) they have to be real careful to guard it. Of course bugs crop up all the time where an object gets collected where it shouldn't have, it happens so often that there is a name for it -- "GC Hole". Astute readers and users of p/invoke may remark that they don't have to set up any "GC frames" -- that is because this complicated scheme is not exposed outside of the CLR source. Regular users of .NET who want to marshal pointers between native/managed can simply request that a GC reference gets pinned, at which point I'm mostly sure it won't get collected until it's unpinned. The bad news is I'm almost positive there is nothing you can do with just C here to make this problem go away. You'd want stuff to magically just happen under the hood, and C++ is the right way to go for that. It's probably possible to create an RAII style C++ GC smart pointer that would be 99% foolproof at the expense of some performance. It gets a little bit trickier if we are doing a moving collector. I am thinking it could ref/unref at creation/destruction, and disallow any direct raw pointer usage not to shoot yourself in the foot. Of course the people writing the GC still need to worry about this..
- tptacek 15y agoA very similar pattern bit me in the ass with the ObjC GC and libevent.
- tsuyoshi 15y agoAnyone who has written an extension to a garbage-collected language in C will have run into this issue. Personally I've written extensions for Guile, OCaml, Ruby, MLton, and Java, and all of them have tricky rules for making your C code safe for garbage collection. Using volatile is the wrong way to do this though... this tells me that the people figuring this stuff out for Ruby don't really know C that well.
- CPlatypus 15y ago"Very few people out there know that the volatile type qualifier exists"? Only if there are "very few" kernel programmers, embedded programmers, and others who have used C for anything low-level and/or multi-threaded. Otherwise, no. Sorry, but knowing about it doesn't make you special. "Volatile" is the wrong fix, by the way. That's just depending on yet another non-required behavior. There is in fact no further reference to "str" between the function call and the reassignment at the start of the next iteration, so there's nothing for "volatile" to chew on. This particular version of this particular compiler just happens to add an extra pair of stack operations in this case, but it's not truly required to. A real fix would not only mark the variable as volatile but also add a reference after the function call. The same "(void)str;" type of statement that's often used to suppress "unused argument/variable" warnings should count as a reference to force correct behavior here.