marcelroed

5 ms·

1.
by marcelroed 2mo ago
Yes! I will publish a Rust crate soon. If you have thoughts about how to structure the API I would love to hear them.
2.
by marcelroed 2mo ago
The output tokens are identical in either case, but there are quite a few additional settings and formats that huggingface compat mode can generate. In general it also requires inputting Python lists of Python strings. The Gigatoken API ins
3.
by marcelroed 2mo ago
Thanks for the kind words, Craig! I'm planning to do a technical writeup+paper and a presentation video on the project in the near future. Will make sure to share it with the Discord!
4.
by marcelroed 2mo ago
Added numbers here: https://news.ycombinator.com/item?id=49015014
5.
by marcelroed 2mo ago
Author here: Actually, depending on the nature of the inference you're doing it can be quite significant. Here are some numbers for time-to-first-token (time to process the entire input and produce the first token of output) for an 8B
6.
by marcelroed 2mo ago
Running the numbers now
7.
by marcelroed 2mo ago
It's usually not as binary as "hit" or "miss" with a prefix cache, and you need to know the token boundaries to know where the cache hit ends. The current structures used for KV-caching in vLLM and SGLang work by ch
8.
by marcelroed 2mo ago
Time to first token refers to the time until the model outputs one token, which includes the time to process the entire prompt (doing prefill). The GPU time per token is much lower when doing prefill, so the significance of tokenization is
9.
by marcelroed 2mo ago
I can add some benchmarks for compatibility mode in the future. I have a little more juice to squeeze out of the Python interop though, so not quite ready for it yet.
10.
by marcelroed 2mo ago
Numbers are for the Gigatoken API, but compatibility mode just means eating a bunch of Python overhead (creating lists, reading strings to bytes). You can expect a modest ~200-300x speedup with compatibility mode depending on how you use it
11.
by marcelroed 2mo ago
You can, but this usually results in sequences with padding/truncation, since you won't know how many tokens your inputs map to before you actually tokenize them. This also makes shuffling difficult. In practice every training pro
12.
by marcelroed 2mo ago
Author here! In my case it's mostly pretraining experiments, where you might want to change your data mixture/filtering/processing of training data, and splits are usually done at a token-level instead of a text level. In thi
13.
by marcelroed 4mo ago
We have autograding for code through tests written by hand, and additionally do manual code audits if we see suspicious behavior. We also do grading the old-fashioned way for writeups. We do indeed catch students who don't follow the h
14.
by marcelroed 4mo ago
TA here. Noted! I now have more resources to test more environments, and will do so whenever possible. I think freezing due to memory overuse is going to be a problem with anything you code yourself, but I do think we could be more rigorous
15.
by marcelroed 4mo ago
We do provide resources for enrolled students. The online suggestions are for external students or Stanford students who we weren't able to admit.
16.
by marcelroed 4mo ago
TA here. Biggest changes are in the second assignment (distributed) where we added a bunch of memory, profiling and distributed tasks, as well as in the fifth assignment (alignment), where most of the RL tasks are fresh this year. Assignmen
17.
by marcelroed 4mo ago
TA here. Definitely not! In fact we explicitly added sections in the first assignment to allow for scaling down to even local compute (M-series GPUs). For assignment 2 there are a few regions that require Triton support for your GPU, but ev
18.
by marcelroed 1y ago
I am only familiar with MLIR for accelerator-specific compilation, but my understanding is that by describing operations at a higher level, you don’t need the frontend to know what LLVM IR will lead to the best final performance. For instan
19.
by marcelroed 1y ago
After trying this out on my iphone, I can say your conclusion is exactly correct. The icons look subtly out of focus in a way that's quite unsettling.
20.
by marcelroed 2y ago
It’s worth noting that Shreyas (the first author) was a student with Tenenbaum at MIT before he went to Berkeley