jwan584

6 ms·

1.
Mistral Flash Answers Run on Cerebras(cerebras.ai)
5 points by jwan584 2y ago|1 comments
2.
by jwan584 2y ago
The point about using FP32 for training is wrong. Mixed precision (FP16 multiplies, FP32 accumulates) has been use for years – the original paper came out in 2017.
3.
by jwan584 2y ago
A good talk on how Cerebras does power & cooling (8min) https://www.youtube.com/watch?v=wSptSOcO6Vw&ab_channel=Appli...
4.
100x defect tolerance: How we solved the yield problem(cerebras.ai)
331 points by jwan584 2y ago|179 comments
5.
by jwan584 2y ago
batch size by Q4 will be solid double digits (cerebras employee)
6.
Cerebras CS-3: the fastest and most scalable AI accelerator(cerebras.net)
2 points by jwan584 3y ago|0 comments
7.
by jwan584 3y ago
when you go from 1B to 175B, the model no longer fits in memory. so in practice you have to re-factor the model using tensor/pipeline parallelism. that's why it goes from 600 to 20K LOC.
8.
by jwan584 3y ago
Everyone knows Cerebras by their wafer scale chips. The less understood part is the 12TB of external memory. That's the real reason why large models fit by default and you don't have to chop it up in software ala megatron/dee
9.
BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model(arxiv.org)
3 points by jwan584 3y ago|2 comments
10.
by jwan584 3y ago
A helpful paper with the full recipe Cerebras uses to train LLMs and their process including: - Extensively deduplicated dataset (SlimPajama) - Hyperparameter search using muP - Variable sequence length training + ALiBi - Aggressive LR deca
11.
BTLM-3B-8K: 7B Performance in a 3B Parameter Model(cerebras.net)
3 points by jwan584 3y ago|0 comments
12.
by jwan584 3y ago
Meta announced a partnership with Qualcomm to bring LLMs to mobile. But 3B is a lot more compact than LLaMA's 7B.
13.
NFTs Are Signatures That Come with Artworks(draecomino.substack.com)
4 points by jwan584 6y ago|1 comments