1a1a11a

6 ms·

1.
A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing(arxiv.org)
4 points by 1a1a11a 26d ago|0 comments
2.
Learning Augmented Heuristics(systems.seas.harvard.edu)
2 points by 1a1a11a 1mo ago|0 comments
3.
Burstiness is all you need for LLM serving(systems.seas.harvard.edu)
1 points by 1a1a11a 2mo ago|0 comments
4.
Why is the storage cluster used to train Llama 3 so slow?(scontent.fagc1-2.fna.fbcdn.net)
2 points by 1a1a11a 2y ago|1 comments
5.
by 1a1a11a 2y ago
"Tectonic (Pan et al., 2021), Meta’s general-purpose distributed file system, is used to build a storage fabric (Battey and Gupta, 2024) for Llama 3 pre-training. It offers 240 PB of storage out of 7,500 servers equipped with SSDs, and
6.
by 1a1a11a 3y ago
I had a try on a few traces; the random two-choice algorithm is only better when there is a big scan, and the LRU miss ratio curve shows a cliff. In all other cases, it is worse than LRU eviction. Implementation can be found https:/&
7.
by 1a1a11a 3y ago
Got it. Make sense, I like the reason why you want to make it circular. :)
8.
by 1a1a11a 3y ago
it looks like the ring is implemented using a linked list?
9.
by 1a1a11a 3y ago
Great point! We tried the oracle (Belady MIN) and found that the oracle often does not admit new objects (<20% chance admitting new objects) in most traces. That's why we need to evict new objects quickly
10.
by 1a1a11a 3y ago
This is an awesome reply that I would like to see here. Do you mind sharing more details on what type of cache or use case needs the performance you mentioned? Database bufferpool? > a lot of these practical details have never been writ
11.
by 1a1a11a 3y ago
I love your comments!
12.
by 1a1a11a 3y ago
This is fantastic work!
13.
by 1a1a11a 3y ago
It was made with keynote with the magic move between slides. Maybe @yazhuo can share the file.
14.
by 1a1a11a 3y ago
I agree with you; any non-random algorithm has the problem
15.
by 1a1a11a 3y ago
Yes if you add a ghost queue. :)
16.
by 1a1a11a 3y ago
I totally agree with you, this is not designed for hardware... and as others mentioned RRIP might be better for set-associative caches
17.
by 1a1a11a 3y ago
Do you have an example of a non-random algorithm that does not have adversarial workloads?
18.
by 1a1a11a 3y ago
You are probably thinking of block I/O or CPU cache workloads where scans cause the miss ratio to increase. This algorithm targets web workloads where scans are not common, and the miss ratio does not change abruptly. Yes, SIEVE does n
19.
by 1a1a11a 3y ago
Thank you! Yes, this addresses the scan-resistant issue, but I am not sure how different it would be on workloads without scan. But the workloads we used in evaluations have very few scans, and the power of SIEVE comes from quickly evictin
20.
by 1a1a11a 3y ago
Yup, SIEVE is good enough for web workloads :)
21.
by 1a1a11a 3y ago
Good points. The first point is probably true for most research in computer systems. Different systems have different constraints that make a simple design very complex or not possible. Can you elaborate on the second point more? It feels t
22.
by 1a1a11a 3y ago
Share a bit more information on how we got here. SIEVE is the more general form of the FIFO-Merge algorithm in Segcache. When I designed the FIFO-merge algorithm, I thought it was worse than LRU, but it turned out to be better. At first, I
23.
by 1a1a11a 3y ago
> How it should work is that new items should be added at a fixed distance from the Hand pointer. This would help with the implementation with a circular buffer. But for now, let's decouple the algorithm from the implementation. &gt
24.
by 1a1a11a 3y ago
Sorry, I may miss something. How would "random two choices" differ from LRU?
25.
by 1a1a11a 3y ago
In case someone is interested, there is an independent review and implementation at https://github.com/scalalang2/golang-fifo
26.
by 1a1a11a 3y ago
And Segcache is available in Rust as part of Pelikan project https://github.com/pelikan-io/pelikan
27.
by 1a1a11a 3y ago
Disclaimer: this is the co-author. Sorry for the hyped language. If my guess is correct, the blog was "polished" by ChatGPT, and the intention was to "polish" the English. The paper has more content and should be read mo
28.
by 1a1a11a 3y ago
Do you work with Peter? If not, this will probably be the third paper...
29.
by 1a1a11a 3y ago
I bet this is ChatGPT style
30.
by 1a1a11a 3y ago
1. I wrote a bit on how to implement lock-free FIFO queues here https://blog.jasony.me/system/cache/2023/12/28/fifo , let me know if any part is not clear or not correct. Moving an object from the sm

More ›