5 ms·
SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem
- ctoth 11mo agoOnce builds are "fast enough," there's no business case for the painful work of making the codebase comprehensible. We're going to 1 billion LoC codebases and there's nothing stopping us!
- theossuary 11mo agoWhy tf does an electric vehicle need 500m+ lines of code
- jeffbee 11mo agoSome people actually write tests.
- serbancon 11mo agoWe actually picked a fairly conservative number - there are even larger automotive codebases today. For example, Mercedes’ MB.OS: “is powered by more than 650 million lines of code” - see: https://www.linkedin.com/pulse/behind-scenes-mbos-development-magnus-%C3%B6stberg-56rse https://www.linkedin.com/pulse/behind-scenes-mbos-developmen...
- menaerus 11mo ago650M LoC is certainly not a single codebase that you can "checkout" and "build". Also, the figure is a little bit hard to believe.
- api 11mo agoCould you just do the build in /dev/shm?
- ongy 11mo agoNo. `/dev/shm` would just be a build in `tmpfs`. Though from what I gather form the story, part of the spedup comes from how android composes their build stages. I.e. speeding up by not downloading everything only helps if you don't need everything you download. And adds up when you download multiple times. I'm not sure they can actually provide a speedup in a tight developer cycle with a local git checkout and a good build system.
- ongy 11mo agoWhile it looks like at least some of the team are ex-googlers, this isn't the srcfs we know from piper (Google internal tools). Looks like it's similar in some ways. But they also don't tell too much and even the self-hosting variant is "Talk to us" pricing :/
- jonnrb 11mo agoWDYM this seems very familiar. At commit deadbeef I don't need to materialize the full tree to build some subcomponent of the monorepo. Did I miss something? And as for pricing... are there really that many people working on O(billion) lines of code that can't afford $TalkToUs? I'd reckon that Linux is the biggest source of hobbyist commits and that checks out on my laptop OK (though I'll admit I don't really do much beyond ./configure && make there...)
- ongy 11mo agoOh yea, this is "srcfs the idea" but not "srcfs the project". I.e. this isn't something battel tested for hundreds of thousands of developers 24/7 over the last years. But a simple commercial product sold by people that liked what they used. Well, since android is their flagship example, anyone that wants to build custom android releases for some reason. With the way things are, you don't need billions of code of your own code to maybe benefit from tools that handle billions of lines of code.
- 7e 11mo agoGoogle or Meta needs to open source their magic VFSes. Maybe Meta is closest with EdenFS.
- mattnewton 11mo agoI have thought about this, but also wondered if it would be as magic without the highly paid team of fantastic SRE and maintainers, and the ridiculous amount of disk and compute available to them.
- 11mo ago
- Ericson2314 11mo agoThe headline times are a bit ridiculous. Are they trying to turn https://github.com/facebook/sapling/blob/main/eden/fs/docs/Overview.md https://github.com/facebook/sapling/blob/main/eden/fs/docs/O... or some git fuse thing into a product?
- jonnrb 11mo agoIt seems like that plus some build output caching?
- zokier 11mo agoWell they also claim to be able to cache build steps somehow build-system independently. > As the build runs, any step that exactly matches a prior record is skipped and the results are automatically reused > SourceFS delivers the performance gains of modern build systems like Bazel or Buck2 – while also accelerating checkouts – all without requiring any migration. Which sounds way too good to be true.
- fukka42 11mo agoSeems viable if you can wrap each build stap with a start/stop signal. At the start snapshot the filesystem. Record all files read & written during the step. Then when this step runs again with the same inputs you can apply the diff from last time. Some magic to automatically hook into processes and doing this automatically seems possible.
- bananaquant 11mo agoI think I got the magic part. You can store all build system binaries in the VFS itself. When any binary gets executed, VFS can return a small sham binary instead that just checks command line arguments, if they match, checks the inputs, and if they match, applies the previous output. If there is any mismatch, it can execute the original binary as usual and make the new output. Easy and no process hacking necessary.
- vlovich123 11mo ago
- DuckConference 11mo agoTheir performance claims are quite a bit ahead of the distributed android build systems that I've used, I'm curious what the secret sauce is.
- cogman10 11mo agoIs it going to be anything more than just a fancier ccache?
- vlovich123 11mo agoIt’s definitely not ccache as they cover that under compiler wrapper. This works for Android because a good chunk of the tree is probably dead code for a single build (device drivers and whatnot). It’s unclear how they benchmark - they probably include checkout time of the codebase which artificially inflates the cost of the build (you only checkout once). It’s a virtual filesystem like what Facebook has open sourced although they claim to also do build caching without needing a dedicated build system that is aware of this and that part feels very novel
- refulgentis 11mo agoRe: including checkout, it’s extremely unlikely. source: worked on Android for 7 years, 2 hr build time tracks to build time after checkout on 128 core AMD machine; checkout was O(hour), leaving only an hour for build if that was the case.
- serbancon 11mo agoObviously this is the best-case, hyper-optimized scenario and we were careful not to inflate the numbers. The machine running SourceFS was a c4d-standard-16, and if I remember correctly, the results were very similar on an equivalent 8-vCPU setup. As mentioned in the blog post, the results were 51 seconds for a full Android 16 checkout (repo init + repo sync) and ~15 minutes for a clean build (make) of the same codebase. Note that this run was mostly replay - over 99 % of the build steps were served from cache.
- bityard 11mo agoMeh, content marketing for a commercial biz. There are no interesting technical details here. I was a build engineer in a previous life. Not for Android apps, but some of the low-effort, high-value tricks I used involved: * Do your building in a tmpfs if you have the spare RAM and your build (or parts of it) can fit there. * Don't copy around large files if you can use symlinks, hardlinks, or reflinks instead. * If you don't care about crash resiliency during the build phase (and you normally should not, each build should be done in a brand-new pristine reproducible environment that can be thrown away), save useless I/O via libeatmydata and similar tools. * Cross-compilers are much faster than emulation for a native compiler, but there is a greater chance of missing some crucial piece of configuration and silently ending up with a broken artifact. Choose wisely. The high-value high-effort parts are ruthlessly optimizing your build system and caching intermediate build artifacts that rarely change.
- 7e 11mo agoThat’s all basic stuff, and none of it solves what this product claims to.
- serbancon 11mo agoWe hear you on the “we want more technical blogs” part - they’ll be coming once we get a breather. We kept this first post high-level to reach a broader audience. Thanks for reading!
- bityard 11mo agoNot as basic as you seem to think. I was brought into a team of 70-ish engineers working across 4-5 products. Big enterprise products written by very bright programmers. But build systems and infrastructure were not their core competency. Their flagship application took 6 hours to build when I was hired. I got it down to 30-45 minutes using a combination of the techniques above and a revamped build infrastructure. When I finally left that position, the build was much more modular, so you could rebuild a small part of it and glue it into a bunch of existing artifacts and have a final product in just a few minutes.
- vzaliva 11mo agoIt sounds from the page that it is Android-source-code specific. Why? Could this work with any source code base?
- everlier 11mo agoIf my understanding is correct, this only makes sense for codebases that do not fit in memory of a largest build box an organisation can run
- rs186 11mo agoI think the page itself answers your question pretty well.
- serbancon 11mo agoI posted a longer answer to a similar question above, if you're interested. Thanks!
- forrestthewoods 11mo agoThe world desperately needs a good open source VFS that supports Windows, macOS, and Linux. Waaaaay too many companies have independently reinvented this wheel. Someone just needs to do it once, open source it, and then we can all move on.
- 7e 11mo agoThis. Such a product also solves some AI problems by matting you version very large amounts of training data in a VCS like git, which can then be farmed out for distributed unit testing.
- forrestthewoods 11mo agoHuggingFace bought XetHub which is really cool. It’s built for massive blobs of weight data. So it’s not a general purpose VCS VFS. The world still needs the latter. I’d be pretty happy if Git died and it was replacing with a full Sapling implementation. Git is awful so that’d be great. Sigh.
- _1tan 11mo agoI want this but self hosted/integrated into our CI (Gitlab in our case).
- serbancon 11mo agoPlease fill in this form: https://www.source.dev/demo https://www.source.dev/demo . We’re prioritizing cloud deployments but are keen to hear about your use case and see what we can do.
- jeffrallen 11mo agoTldr: your build system is so f'd that you have gigs of unused source and hundreds of repeated executions of the same build step. They can fix that. Or, you could, I dunno, fix your build?
- jayd16 11mo agoYou could just have a mono-repo with a large amount of assets that aren't always relevant to pull. Incremental builds and diff only pulls are not enough in a modern workflow. You either need to keep a fleet of warm builders or you need to store and sync the previous build state to fresh machines. Games and I'm sure many other types of apps fall into this category of long builds, large assets, and lots of intermediate build files. You don't even need multiple apps in a repo to hit this problem. There's no simple off the shelf solution.
- Dylan16807 11mo agoFor a large amount of sometimes-relevant assets, is mapping them over NFS a bad solution? SourceFS also gets them across the network on demand, right? And a fleet of warm builders seems pretty reasonable at that scale. SourceFS sounds useful for extra smart caching but some of these problems do sound like they're just bad fixable configuration.
- jayd16 11mo ago> And a fleet of warm builders seems pretty reasonable at that scale. It's actually pretty hard. The more builders you have the older the workspace gets and scaling up or cycling machines causes the next builds to be super slow. Game engines end up making central intermediate asset caches like Unreal's UBA or Unity's Cache Server.
- blaz0 11mo agoYes, games are a common case where your repo can be very large but building your code only requires access to a small subset of it. For example, the League of Legends source repo is millions of files and hundreds of GB in size, because it includes things like game assets, vendored compiler toolchains for all of our target platforms, etc. But to compile the game code, only about 15,000 files and 600MB of data are needed from the repo. That means 99% of the repo is not needed at all for building the code, and that is why we are seeing a lot of success using VFS-based tech like the one described in this blog. In this case, we built our own virtual filesystem for source code based on our existing content-defined patching tech (which we wrote about a while ago [1]). It's similar to Meta's EdenFS in that we built it on top of the ProjFS API on Windows and NFSv3 on macOS and Linux. We can mount a view into the multimillion-file repo in 3 seconds, and file data (which is compressed and deduplicated and served through a CDN) is downloaded transparently when a process requests it. We use a normal caching build system to actually run the build, in our case FASTBuild. I recently timed it, and I can go from having nothing at all on disk to having locally built versions of the League of Legends game client and server in 20 seconds on a 32-core machine. This is with 100% cache hits, similar to the build timings mentioned in the article. [1] https://technology.riotgames.com/news/supercharging-data-delivery-new-league-patcher https://technology.riotgames.com/news/supercharging-data-del...
- serbancon 11mo agoHey everyone. I’m Serban, co-founder of Source.dev. Thanks for the upvotes and thoughtful discussion. I’ll reply to as many comments as I can. Nothing means more to an early-stage team than seeing we’re building something people truly value - thanks from all of us at Source.dev!
- CJefferson 11mo agoWhile I’m sure it’s much more advanced, out of interest is this similar to the Python tool ‘fabricate’, which would use strace to track all files a program read, and wrote?
- MarsIronPI 11mo ago> Fast builds are what truly makes a difference to developer productivity. With SourceFS builds complete over 9x faster on a regular developer machine. This sets a new standard as it enables developers to get their sword fighting time back and speeds-up the lengthy feedback loop on CI pipelines. Objection! Long build times are better for sword-fighting time. The longer it takes, the more sword-fighting we have time for!
- zar22 11mo ago[dead]
- deleted 11mo ago[deleted]
- sudahtigabulan 11mo agoThis reminds me of ClearCase and its MVFS. Builds were audited by somehow intercepting things like open(2) and getenv(3) invoked by a compiler or similar tool, and each produced object had an associated record listing the full path to the tool that produced it, its accurate dependencies (exact versions), and environment variables that were actually used. Anything that could affect the reproducibility was captured. If an object was about to be built with the exact same circumstances as those in an existing record, the old object was reused, or "winked-in", as they called it. It also provided versioning at filesystem level, so one could write something like file.c@@/trunk/branch/subbranch/3 and use it with any program without having to run a VCS client. The version part of the "filename" was seen as regular subdirectories, so you could autocomplete it even with ancient shells (I used it on Solaris).
- yencabulator 11mo agoVagueposts from the marketing department are not appreciated.