charles_irl

6 ms·

1.
Routing for serverless servers with Pingora, Envoy, and Spanner(modal.com)
11 points by charles_irl 3mo ago|0 comments
2.
Speculation Is All You Need(modal.com)
12 points by charles_irl 3mo ago|0 comments
3.
by charles_irl 4mo ago
Very cool! > Unless you've actually watched tokens stream at those rates, the numbers are hard to internalize. This is the rendering. I built something similar recently, for the same reason: https://modal.com/llm-alm
4.
by charles_irl 4mo ago
To clarify: we do content-based hashing, and when we say "shared bytes aren’t guaranteed to be in the exact same container image layer", what we mean is that FROM some/image RUN pip install torch==2.7.1 and FROM another/
5.
by charles_irl 4mo ago
You're absolutely right!
6.
by charles_irl 4mo ago
Yep! That should start in ten seconds or so -- about a second per gigabyte of weights, plus a second to start the container and a few seconds to load the memory snapshot. There are a few limitations with snapshotting, e.g. it generally fail
7.
by charles_irl 4mo ago
Cutting latencies by 40x! Unfortunately couldn't fit the whole title in the character limit :<
8.
Cutting inference cold starts by 40x with LP, FUSE, C/R, and CUDA-checkpoint(modal.com)
91 points by charles_irl 4mo ago|18 comments
9.
How to Achieve Serverless GPUs(modal.com)
8 points by charles_irl 4mo ago|0 comments
10.
by charles_irl 8mo ago
> Ed Zitron also called out the business model of GPU-as-a-service middleman companies like modal deeply unsustainable, and I also don't see how they can make a profit if they are only reselling public clouds. You got a link for tha
11.
by charles_irl 8mo ago
Sorry to lead with a bunch of jargon! Wanted to make it obvious that we'd give concrete recommendations instead of palaver. The technical terms there are later explained and diagrammed, and the recommendations derived from something cl
12.
by charles_irl 8mo ago
oof ty, willfix
13.
Three types of LLM workloads and how to serve them(modal.com)
75 points by charles_irl 8mo ago|5 comments
14.
Host overhead is killing your inference efficiency(modal.com)
3 points by charles_irl 10mo ago|0 comments
15.
by charles_irl 11mo ago
Inspired by https://float.exposed , which was on the front page recently, I put together this visualizer for lower precision/quantized floating point numbers -- specifically, all of the formats in the Open Compute Project&#x
16.
Quantized Float Exposed(quant.exposed)
2 points by charles_irl 11mo ago|1 comments
17.
Against SQL (2021)(scattered-thoughts.net)
82 points by charles_irl 11mo ago|77 comments
18.
Length-extension attacks are still a thing(00f.net)
2 points by charles_irl 11mo ago|1 comments
19.
The future of Python web services looks GIL-free(blog.baro.dev)
3 points by charles_irl 11mo ago|0 comments
20.
Lexical differential highlighting instead of syntax highlighting(wordsandbuttons.online)
2 points by charles_irl 11mo ago|0 comments
21.
CReact – JSX for the Cloud(github.com)
1 points by charles_irl 11mo ago|0 comments
22.
QUIC and the end of TCP sockets(codemia.io)
62 points by charles_irl 11mo ago|82 comments
23.
In C++ modules globally unique module names seem to be unavoidable(nibblestew.blogspot.com)
2 points by charles_irl 1y ago|0 comments
24.
Stupid jj Tricks(andre.arko.net)
3 points by charles_irl 1y ago|0 comments
25.
by charles_irl 1y ago
Thanks! I think computers are fun and I want reading about them to be fun too. I was also reminded of HazyResearch's MegaKernels. Didn't want to distract from the main thrust of the post, but definitely think that's a promisi
26.
by charles_irl 1y ago
Hey, one of the authors here! Reductively, software engineering means taking an idea and mapping it into code. So one form of "reverse" engineering would be taking the code and extracting the ideas. That's what we did here. B
27.
We reverse-engineered Flash Attention 4(modal.com)
5 points by charles_irl 1y ago|0 comments
28.
A Tour of eBPF in the Linux Kernel: Observability, Security and Networking(lucavall.in)
2 points by charles_irl 1y ago|0 comments
29.
Categorical Foundations for Cute Layouts(research.colfax-intl.com)
39 points by charles_irl 1y ago|6 comments
30.
Pocket Casts, You Altered the Deal, So I Will Alter Your App(blog.matthewbrunelle.com)
12 points by charles_irl 1y ago|3 comments

More ›