9 ms·
Linus Torvalds: “I'm happily hacking on a new save format using ‘libgit2’”
- joelhaasnoot 13y agoWorked on a project a few years ago where we needed distributed sync capability. Using git (or bazaar or mercurial) was one of the options - store everything in it versus a database. Interesting to see the same thought "coming back".
- fit2rule 13y agoI've also used libgit as a means to a similar end - providing versioned data across a local filesystem. Its an idea whose time has come ..
- jmnicolas 13y agoFrom the comments (Tristan Colgate) : "XML is what you do to a sysadmin if waterboarding him would get you fired." Made my day :-)
- Ygg2 13y agoThat's just mean. Waterboarding isn't that bad...
- jmnicolas 13y agoBut it gets you fired ... on the other end, nobody has ever been fired for using XML.
- nzp 13y agoWith my occasional sysadmin hat on, until a few weeks ago I had the luck to never have had to deal with XML configuration files. Then came Solr and now I know what horror is. (To be clear, Solr itself is great, but those god damn config files...)
- oneeyedpigeon 13y agoI don't quite get Linus' problem with XML for document markup (for anything else - config files, build scripts - sure, XML is horrible). Does anyone know any more details about what his specific gripe is? For me, asciidoc (which looks very similar, conceptually, to markdown) suffers from one huge problem: it's incomplete. Substituting symbols for words results in a more limited vocabulary, if that vocabulary is to remain at all memorable. Sure, XML can be nasty, but thats very much a function of the care taken to a) format the file sensibly b) use appropriate structure (i.e. be as specific as necessary, and no more).
- adobriyan 13y agoThis link needs to be posted again and again and again. I'm sure quite a lot of people will easily recognize it. :^) Subject: Re: S-exp vs XML, HTML, LaTeX (was: Why lisp is growing) https://groups.google.com/forum/message/raw?msg=comp.lang.lisp/1KrHMfo1AiE/7icWIAjFMJoJ https://groups.google.com/forum/message/raw?msg=comp.lang.li...
- oneeyedpigeon 13y agoThat's a wonderful rant - I particularly appreciate the digression into anti-bush rhetoric - but: 1. There's very little detail here; it's a nicely worded, emotionally charged piece that leaves a lot of detail unaddressed, e.g. "'I would like to hear why you think it is so bad, can you be more specific please?' If you really need more information, search the Net, please." That's not very helpful. 2. It argues for 'simpler' markup via the removal of attributes. Where possible, I totally agree, as at least hinted at in my original post. Sometimes, though, this would be impossible or unwieldy (e.g. HREF attribute on an A element). 3. Character entities vs. unicode - totally agree. Wherever possible, I use proper unicode characters rather than ugly character entities in my markup. 4. "But the one thing I would change the most from a markup language ... is to go for a binary representation." Linus would vehemently disagree on this point.
- adobriyan 13y agoThere are whole lot more supplementary rants: http://www.xach.com/naggum/articles/search?q=xml http://www.xach.com/naggum/articles/search?q=xml In particular: http://www.xach.com/naggum/articles/3224334063725870@naggum.net.html http://www.xach.com/naggum/articles/3224334063725870@naggum.... with key words being "Whather what you are really after is foo, bar, or zot, depends on your application.". His articles on SGML are mandatory reading too. Several years ago someone posted these links and it opened wonderful world of Lisp to me. Not the language per se (there are many languages) but whole another Universe of how things could be done. I swear I jumped on the chair reading every page of CL standard, how brilliant it is on every level to C. Eventually it led me to rethink attitude to C and Unix in general, core parts of which I despise now. So here am I returning favor, maybe someone will follow these links too. Thank you, Erik. Rest in peace.
- tzury 13y agoI just realized that Linus' posts are the only reason I ever go to Google Plus.
- unsigner 13y agoLinus : G+ :: notch : Java
- xentronium 13y agoThat's unfair. Lots of infrastructural projects are done in java. E.g. my personal favorite: lucene (+ solr, elasticsearch).
- unsigner 13y agoYeah, and I hear some communities (photographers?) have taken G+ as their home. It was tongue-in-cheek and purely from my (PC, desktop, Windows/console game developer) perspective.
- jan_g 13y agoFor me it's not just G+, but also Facebook and Twitter. Only reason I ever visit those sites is indirectly through HN posts and similar.
- cbsmith 13y agoThe question nobody is asking, but actually should is: I wonder what other good G+ content you are missing? G+ is largely misunderstood. It is a lousy tool for interaction with people connected to you purely socially. It's a very good way to find and interact with people connected to you by interest.
- ChikkaChiChi 13y agoThis is exactly how I explain Google+ to folks. It's built for communities, not cliques.
- icefox 13y ago
- splitbrain 13y agohe talks about a save file format, not a file system. or do we have different concepts of "file system"?
- sp332 13y agoI agree it's confusing, I think the submitter just meant "system for files" or something.
- senthilnayagam 13y agowhy do you need to view filesystem and make it readable for humans, you would interact it via commands "ls" or some gui git as the basis of filesystem is interesting, hope we don't need to manually make branches and commits to use it
- oneeyedpigeon 13y agoDid you read the article? It's not really about the filesystem. 1 part your fault for seemingly not reading the article you're commenting about, 1 part the submitter's fault for choosing such a misleading title.
- twic 13y agoTitle is entirely misleading. Tech support! TECH SUPPORT!!
- anon4 13y agoHave you tried turning it off and on again?
- theandrewbailey 13y agoIs your title plugged in?
- k2enemy 13y agoI don't really understand what he's talking about here (my ignorance, not his fault.) Is it something like https://camlistore.org/ https://camlistore.org/ that is a content addressable (the git part) datastore?
- saljam 13y agoYep, I thought it sounded like Camlistore, but as a library.
- josephlord 13y agohttps://github.com/torvalds/subsurface https://github.com/torvalds/subsurface I didn't really know what he was talking about but I think this is it. The title does need changing though as it is definitely file formats under discussion not file systems.
- bananas 13y agoI think this title is wrong. Firstly some clarification - this appears to just be about the persistence format for his dive log. It was XML, now it's git based with plain text. As someone who had to manage a system which worked with plain text files structured in a filesystem for a number of years in the 1990s, this is done to death already. You now end up with the following problems: locking, synchronising filesystem state with the program, inode usage, file handles to manage galore and concurrency. All sorts. Basically this is a "look I've discovered maildir and stuffed it in a git repo". Not saying there is a better solution but this isn't a magic bullet. It's just a different set of pain.
- deleted 13y ago[deleted]
- xsace 13y agoMaybe you want to wait till he release something. Cause you know, if he took months to get the big picture in mind, I doubt you grasp what he envision just by reading his comment.
- bananas 13y agoIf it's not that, I'll eat my hat, and my pyjamas. There's not much more to infer from the comment. Unless he's invented a new ASN.1 encoding which plugs into libgit or something or a new text serialisation format (both unlikely).
- bsder 13y agoYes, because his design of git was so well-formed. Git is so well-designed that expert users manage to trash their repositories and propagate the damage. Maybe that's not a problem of libgit. But tools are both the infrastructure and the UI.
- taeric 13y agoNot sure what you are referring to. What are some common ways "expert users" manage to "trash their repositories?"
- bhaak 13y agoWhat's with all the XML hate? Of course, doing everything in XML is a stupid idea (e.g. XSLT and Ant) and thanks heaven that hype is over. But if I want something that is able to express data structures customized by myself, usually with hierarchical data that can be verified for validity and syntax (XML Schemas or old-school DTD), what other options are there? Doing hierarchical data in SQL is a bitch and if you want to transfer it, well good luck with a SQL dump. JSON and other lightweight markup languages fail the verification requirement.
- Sharlin 13y agoThe issue is probably that 99.999% of all XML use cases don't use (or need) the verification aspect. For all of those, XML is overkill. Besides, surely it would be possible to design a verification layer on top of JSON, for instance - the fact that one does not currently exist does not mean that XML (and abuse of XML!) should not be criticized.
- _delirium 13y agoPeople seem to prefer JSON, but I don't find it any better to hand-write/hand-edit than XML. If anything it's slightly worse, because it has more syntax edge cases.
- bananas 13y agoAnd it doesn't support the multitude of accurate numeric types that XML does implicitly. XML data is not just "strings", it's a sequence of characters. The deserializer determines what sort of type it is based on either the structure or the language's capabilities. With XML, you can define these policies. With JSON you're stuck with JavaScript being the semantic standard and type definitions which ties you to floats or numbers inside strings. The latter is criminal. Edit: clarification as HN won't let me reply any more.
- icebraining 13y agoHow so? XML by itself only supports strings; any other data types have to be derived from a schema. But you can do the same with any other format that supports strings, including JSON.
- fuzzix 13y ago> "I actually want to have a good mental picture of what I'm doing before I start prototyping. And while I had a high-level notion of what I wanted, I didn't have enough of a idea of the details to really start coding." This I like. The race away from the waterfall straw man has also stripped us of the advantages of BDUF. While rigid phase-driven project management helps nobody, I think there's still room for speccing as much as we can upfront within iterative processes. Or you could run to the IDE and start ramming design pattern boilerplate down its throat the second you're out of the first meeting ;)
- hvidgaard 13y agoYou should be speccing what you want to achieve: the goals, the why, the impact, the external limitations, measures of success and so forth. This also allows you to describe and plan testing up front. The "how" is best handled in an iterative manner. A lot of people use AGILE to avoid planning at all, which is a particular destructive anti-pattern, and the exact opposite of what you need.
- fuzzix 13y ago> "A lot of people use AGILE to avoid planning at all" Yup, I've seen this a lot. In one instance "Agile" meant I could finish a major task using an unfamiliar language, framework and code base in short order. Genuinely, the customer was told "Of course, fuzzix here is familiar with Agile processes so you should have this in 3 weeks". edit of course this also meant there was no formal spec for the task, though I did have a photo of the whiteboard.
- pessimizer 13y ago>The "how" is best handled in an iterative manner. I think that the first "how" should be planned as much as anything else. I understand how you refactor from v0.0.1 to v5.34.2 iteratively, but I think that getting from vNothing to v0.0.1 is qualitatively different. If I don't have a complete idea of how my minimally functional thing will work that is small enough that I can completely hold it in my head, and instead just architect by agglutination and test writing, 1) my results are going to be hacky garbage, 2) my first 50 iterations are going to be devoted to replacing it all haphazardly to fix bugs, and 3) the code and interface will become increasingly more complex, harder to work with, and strewn with special cases. When v0.0.1 is well planned, v2.5.2 may not look anything like the plan anymore, but in my experience it becomes shorter, cleaner, and more correct rather than a giant ball of band-aids propped up with tests.
- aashishkoirala 13y agoThis is what Linus does. He has strong opinions and he throws them around. You can't let that get to you. Both XML and JSON are just fine if used properly.
- theandrewbailey 13y agoThis is the first profanity-free Linus rant that I've read in a long time.
- vacri 13y agoAlmost all of Torvalds' "profanity rants" that get passed around are the result of frustration at an existing conversation, and you can find profanity-free comments by him simply by checking out a slightly earlier one.
- aashishkoirala 13y agoHaha, right. I play in the .NET space, so it's never going to happen, but God help us both if I ever have to end up working for this guy.
- icebraining 13y agohttp://harmful.cat-v.org/software/xml/ http://harmful.cat-v.org/software/xml/
- lifeisstillgood 13y agoWhat I like is the "I dont start prototyping till I have a good mental picture" I am currently stuck on a project I want to start becasue I cannot get it to fit right in my (future) head. And I am glad I am not an idiot for not being able to knock out my next great project in between lattes. (Ok, in direct comparison terms I am an idiot, but at least its not compounded)
- tim333 13y agoYeah, I noted that too, also that it took him months to to get his good mental picture. It makes me feel not so bad about spending months trying to get clear on some of my stuff.
- specialist 13y ago"A change in perspective is worth 80 IQ points." -- Alan Kay My biggest hurdle solving new problems is divining a unifying, simplifying metaphor. Once you have the right notion, that Eureka! moment, everything falls into place, like magic. Like how Kepler was able to fully explain Bache's astronomical data once he realized the planets orbits the sun. Personal example: I used to write print production software. Placing pages onto much larger sheets of paper that get folded and bound into a book. A task called image positioning aka imposition. It took me years to figure out how to model the problem. Key insight was simulating the work backwards, from binding back to the press. Then when I showed the new solution to my coworkers, the response was "Well, duh."
- signa11 13y agoerik-naggum's most excellent xml rant: http://www.schnada.de/grapt/eriknaggum-xmlrant.html http://www.schnada.de/grapt/eriknaggum-xmlrant.html
- meapix 13y agoxml haters!!! using other formats how can I define DTDs?
- 1ris 13y agohttps://news.ycombinator.com/item?id=7333354 https://news.ycombinator.com/item?id=7333354
- mrcharles 13y agoOn the game I'm currently working on, it's built very heavily around Lua. So for the save system, we simply fill a large Lua table, and then write that to disk, as Lua code. The 'save' file then simply becomes a Lua file that can be read directly into Lua. This is absolutely amazing for debugging purposes. Also you never have to worry about corrupt save files or anything of it's ilk. Development is easier, diagnosing problems is easier, and using a programmatic data structure on the backend means that you can pretty much keep things clean and forward compatible with ease. (Oh, also being able to debug by altering the save file in any way you want is a godsend).
- frik 13y agoSo, it's similar to JSON (JavaScript), but valid Lua syntax. local t = {} t = {["foo"] = "bar", [123] = 456} t.foo2 = "bar2"
- hootener 13y agoI've had to write output save file formats for various projects on several occasions, and it never occurred to me to take this approach. Thanks for sharing this, it's one of those ideas that (to me) seems so brilliant in its simplicity that I probably would've never thought of it. Any hiccups in the day-to-day work using this approach? I'm just trying to get a better idea of the workflow since I'm very seriously considering applying it to my next project.
- mrcharles 13y agoThe biggest hiccup is almost a literal one; serializing large lua structures and then writing them to disk can take a lot of time. But this can largely be mitigated by just saving compiled lua instead of text lua.
- Touche 13y agoThat's how people are going to cheat at your game.
- saucetenuto 13y ago
- Gonzih 13y agoCurrent title that I see "Linus Torvalds on implementation of human-readable file system" is off. It's about file formats, not file systems.
- tedchs 13y agoWhy reinvent on-disk data formats when you can just make a file of protocol buffers? https://code.google.com/p/protobuf/ https://code.google.com/p/protobuf/
- sparkie 13y agoWhy reinvent binary serialization when you could use ASN.1, or any of the thousand binary serialization formats that pre-date protobufs?
- lern_too_spel 13y agoFor that specific example, you can find a good discussion here: https://groups.google.com/forum/m/#!topic/protobuf/eNAZlnPKVW4 https://groups.google.com/forum/m/#!topic/protobuf/eNAZlnPKV...
- McP 13y agoIronically that has already been reinvented in the form of Cap'n Proto: http://kentonv.github.io/capnproto/ http://kentonv.github.io/capnproto/ (other than that I agree it's a good solution)
- WalterBright 13y agoBack in the bad old DOS days, instead of creating a file format for saving/loading the configuration of the text editor, I simply wrote out the image in memory of the executable to the executable file. (The configuration was written to static global variables.) Running the new executable then loaded the new configuration. This worked like a champ, up until the Age of Antivirus Software, which always had much grief over writing to executable files. It's a trick I learned from the original Fortran version of ADVENT.
- strictfp 13y agoThank you for that anecdote, it made my day. Simply awesome.
- WalterBright 13y agoI learned a heckuva lot from reading the ADVENT Fortran source code. I was floored when I figured out how it was saving its configuration - such a brilliant idea. And in DOS it could be implemented in about 5 lines of simple C code. (Code size was critical in the old 64Kb days.) The other huge thing I learned from ADVENT was polymorphism. The comment in the source code "the troll is a modified dwarf" was an epiphany for me.
- WalterBright 13y agoI forgot to mention, on a floppy disk system, saving the configuration in the exe file made for fast loading of the program, since it didn't need to do extra floppy file operations to load the config.
- chongli 13y agoReminds me of the approach taken by Xmonad where the configuration is compiled into a new executable and then run.
- picomancer 13y agoReaders may be familiar with the TI-83 programmable graphing calculator's assembly language functionality (especially those who took high school math classes in the mid-to-late 1990's). The TI-83's only user-writable storage was 32K of RAM (there was a small lithium battery to keep it powered when you changed the AAA's; also some of the RAM was used for system stuff so somewhat less than 32K was actually available for user purposes). You could write hex values in the program text editor, then you could tell the calculator to execute the hex codes as machine code. I understand the previous models, TI-82 and TI-85, were hacked / backdoored to run user-supplied assembly language, so TI responded by including an official entry point and developer documentation for the TI-83. People later wrote loaders which allowed programs to be stored as binary instead of text (using half the space). Some loaders also had the capability to run binary programs by swapping them into the target address rather than copying them (theoretically a third option would be possible, running programs in-place if they weren't written to depend on their load address, but this wasn't a direction the community went in. gcc users may be familiar with -fPIC which produces code which can run from any address, and this flag is necessary when compiling code for shared libraries.) This allowed people to create massive 20K+ applications (an RPG called Joltima comes to mind), that used most of the available RAM. The fact that this loading scheme made static variables permanent was also quite convenient. (And most variables were static; stack-based addressing would be tough because the Z80 only has two index registers, one of which is used -- or perhaps I should say "utterly wasted" -- by the TI-83 OS.) The next generation, the TI-83+, included I think 256K of flash ROM, and a special silver edition was released which contained 2 MB.
- sam_bwut 13y agoAt work we have a git backed document store that just saves as json - versioning makes keeping track of audit points nice and easy.
- beagle3 13y agoAnd the actual description is here: http://lists.hohndel.org/pipermail/subsurface/2014-March/010592.html http://lists.hohndel.org/pipermail/subsurface/2014-March/010...
- vfclists 13y agoWhat is it with HN commenters and their demented ability to send topics completely of track? I would have thought someone might have examined the code or what Linus is trying to implement and comment about it. But here we have threads about Lua, why people hate XML and love JSON and all kinds if irrelevant issues which have been well hashed elsewhere ad nauseam. Why not restrict to an analysis of whatever it is Linus developing? HN is getting truly annoying and sucky, if it isn't so already.
- hardwaresofton 13y agoWhy not sqlite or sexpressions? Linus states that databases can't hold previous state but that's not really true... I'm not sure why git is the best tool for the job in this case, even after reading the post & some of the contents.
- tmzt 13y agoThey can, if you recreate the primary feature of Git on top of them.
- pcj 13y ago>>So I've been thinking about this for basically months, but the way I work, I actually want to have a good mental picture of what I'm doing before I start prototyping. And while I had a high-level notion of what I wanted, I didn't have enough of a idea of the details to really start coding. This might be a tangential discussion. Earlier, I used to have a similar approach. Can't code until I have the complete picture. But, it's tough to do in a commercial world and you have deliverables. So, nowadays, I start with what I know and scramble my way until I get a better picture. There are times when that approach works. But, there have been days where I was like - "wish I had spent some more time thinking about this". I am curious how folks on HN handle this "coding block".
- tonyarkles 13y agoI've got a few strategies that might help, depending on the circumstances. A notebook: I'll write down some notes and just kind of free write whatever thoughts come to mind. If there's something that I think is important to come back to, I'll draw an empty box in the left margin (to be filled with a check mark later) Readme: start writing the Readme for the project, even if you're not entirely sure of the details. Include code examples. If you don't like how the API is coming together, change it. It's way less work to modify the API now than it will be later. Write a test: I don't always unit test, but when I do I test first :). This works well on projects that already have a decent test suite. It's kind of an executable version of the Readme. Branch and Hack: branches are cheap. Make one and start playing. Don't like how it's turning out? Make a new branch and try again! Ctrl-Z: maybe the answer won't come to you right away. Let it sit and run in the background for a while and come back to it. If I'm worried about forgetting details, I'll write it down in a notebook first.