ml_hardware

5 ms·

1.
by ml_hardware 2y ago
Looks like someone has got DBRX running on an M2 Ultra already: https://x.com/awnihannun/status/1773024954667184196?s=20
2.
by ml_hardware 3y ago
Mosaic's MPT models are already supported in GGML: https://github.com/ggerganov/ggml Here's MPT-30B running in 4-bit precision on CPU :) https://twitter.com/abacaj/status/16731334433
3.
by ml_hardware 3y ago
The repo for training and finetuning this model is open source here: https://github.com/mosaicml/llm-foundry
4.
MosaicML MPT-7B: A Commercially-Usable LLaMa-Quality Model(mosaicml.com)
119 points by ml_hardware 3y ago|11 comments
5.
by ml_hardware 4y ago
Did you actually read the blog? The very first sentence is: > Try out our Stable Diffusion code here! > https://github.com/mosaicml/diffusion-benchmark
6.
GPT-3 quality models for $450k(mosaicml.com)
1 points by ml_hardware 4y ago|0 comments
7.
Training GPT-3 Quality Models For $450k(mosaicml.com)
2 points by ml_hardware 4y ago|1 comments
8.
Training billion-parameter GPTs with MosaicML(mosaicml.com)
2 points by ml_hardware 4y ago|0 comments
9.
by ml_hardware 4y ago
I think it depends what downstream task you're trying to do... DeepMind tried distilling big language models into smaller ones (think 7B -> 1B) but it didn't work too well... it definitely lost a lot of quality (for general lan
10.
by ml_hardware 4y ago
I agree! The models will definitely keep getting bigger, and MoEs are a part of that trend, sorry if that wasn’t clear. A pod of gen2-H100s might have 256 GPUs with 40 TB of total memory, and could easily run a 10T param model. So I think w
11.
by ml_hardware 4y ago
The 9x speedup is a bit inflated... it's measured at a reference point of ~8k GPUs, on a workload that the A100 cluster is particularly bad at. When measured at smaller #s of GPUs which are more realistic, the speedup is somewhere betw
12.
by ml_hardware 4y ago
Couple points: 1) NVIDIA will likely release a variant of H100 with 2x memory, so we may not even have to wait a generation. They did this for V100-16GB/32GB and A100-40GB/80GB. 2) In a generation or two, the SOTA model architectu
13.
by ml_hardware 4y ago
At inference time it will be possible to do 4000 TFLOPS using sparse FP8 :) But keep in mind the model won't fit on a single H100 (80GB) because it's 175B params, and ~90GB even with sparse FP8 model weights, and then more needed
14.
by ml_hardware 5y ago
Unfortunately it will be hard to investigate properties of large, powerful neural networks without access to their trained weights. And industrial labs that spend millions of dollars training them will not be keen to share. If academics wan
15.
by ml_hardware 5y ago
You may find this blog post useful for thinking about AI scaling: https://www.alignmentforum.org/posts/k2SNji3jXaLGhBeYP/extra... For general tasks like language modeling, we are still seeing predictable improveme
16.
by ml_hardware 5y ago
> "Though the spike seen in the data generates more questions than answers, one thing is clear: A single (albeit large and busy) store’s decision to report a majority of its shoplifting incidents doubled the entire city’s monthly sh
17.
by ml_hardware 5y ago
Wow! How are you able to achieve the cost reductions? Is it different hardware, software optimizations, or both? Also does this suggest that OpenAI is charging 6x markups on their models... >:(
18.
by ml_hardware 5y ago
I was providing costs for the exact instance types that NeuralMagic used in their blog post, if we’re allowed to change that then I can also find cheaper GPU providers. I can agree with you that on super, super small inference deployments,
19.
by ml_hardware 5y ago
That's true.. in fact, seeing V100 FP16 < T4 FP16 makes me believe you're right, the V100 should be much faster if the tensor cores were being used.
20.
by ml_hardware 5y ago
T4 is a gpu :) NVIDIA Tesla T4: https://www.nvidia.com/en-us/data-center/tesla-t4/
21.
by ml_hardware 5y ago
My guess is they are using tensor cores as they report FP16 throughput, but they seem to be measuring at batch size 1, which is hugely unfair to the GPUs. For inference workloads you usually batch incoming requests together and run once o
22.
by ml_hardware 5y ago
I think your math is backwards. The training workload W is the same, and the time to complete it is: W / (256 * speed_v4) = 1.82 W / (4096 * speed_v3) = 0.39 speed_v4 / speed_v3 = (4096 * 0.39) / (256 * 1.82) = 3.43 Note
23.
by ml_hardware 6y ago
Haven't seen this mentioned yet, but ResNet-50 is an old model. I would probably expect multiple-percentage-point gains from using a better (or honestly just larger) architecture and better training methodology. Throw an ML engineer at
24.
by ml_hardware 6y ago
GPT3 can write correct computer programs. There are many demos online. Stepping back it is just a translation problem e.g. English idea -> Python code, which an ideal language model would be able to do. I think what I’m picking a gripe o
25.
by ml_hardware 6y ago
If you had a language model whose responses were indistinguishable from a human, would you call that understanding? I agree GPT-3 is * just * a language model, but then again, so are you.
26.
by ml_hardware 6y ago
I've done a lot of work in ML numerics, and I think TF32 is a completely safe drop-in for FP32 for ML workloads. NVIDIA seems to think so too, which is why on A100 it won't even be an option, it will be the default mode for any FP
27.
by ml_hardware 6y ago
If I may ask, why are the Inception style workloads still popular, rather than architectures like EfficientNet? Also, why FP32? CNNs are some of the most robust models to train in FP16 (much easier than language models) so you could get you
28.
by ml_hardware 6y ago
Totally agree! I think 3090 could be a lot more cost effective for researchers to dabble with NLP. But it really grinds my gears when people post these misleading benchmarks... the 3090 is handicapped at half-rate tensor core performance wh
29.
by ml_hardware 6y ago
This is a poor comparison of performance. All of these networks are CNNs, and very old architectures at that. They are all probably memory bottlenecked which is why you see the consistent 50% improvement in FP32 perf. It is also not clear w
30.
Cerebras partners with Glaxo for AI drug development(zdnet.com)
1 points by ml_hardware 6y ago|0 comments

More ›