paul_mk1

8 ms·

1.
by paul_mk1 2y ago
Sub 1-bit has been done at least as far back as 2016 for VGG style networks (my work). I was able to get 0.68 "effective" bits. The idea is that in each forward pass you add noise to each weight independently drawn from normal d
2.
by paul_mk1 3y ago
Goes back before then. This got popularized by BinaryConnect in 2015, and groups were training binary networks as early as 2011. You are probably referring to XNOR net, and the novel piece there was also using binary activations (which bitn
3.
by paul_mk1 3y ago
It's clear that QLoRA has opened up finetuning to a wider audience with limited compute, which is a good thing. One thing I've wondered about: what are the drawbacks to using QLoRA? For example if compute is not a limit, I'm
4.
by paul_mk1 3y ago
I don't think there is anything conceptually new in this work, other than it is applied to LLMs. But in fairness, getting these techniques to work at scale is no small feat. In my experience quantization aware training at these low bit
5.
by paul_mk1 3y ago
Nice to know there is a trail to relevant citations. I missed the BitNet paper and need to catch up. Btw TrueNorth project evolved into "NorthPole" chip by the same group, and was recently in the press. From afar NorthPole looks l
6.
by paul_mk1 3y ago
Fun to see ternary weights making a comeback. This was hot back in 2016 with BinaryConnect and TrueNorth chip from IBM research (disclosure, I was one of the lead chip architects there). Authors seemed to have missed the history. They shoul
7.
Show HN: Watch Life – A compact "analog" watch in Conway's Game of Life(github.com)
23 points by paul_mk1 3y ago|2 comments
8.
by paul_mk1 3y ago
Comparisons between different chip architectures are imperfect. In our opinion the most fair thing to do is 1) match the TFLOPs (since these workloads are compute bound), and 2) find a similar card that can run the same size models. Since M
9.
by paul_mk1 3y ago
You can try it yourself on SageMaker using NVIDIA. There's a free trial. https://aws.amazon.com/marketplace/seller-profile?id=seller-... For AMD, you'll have to wait until these cards become available on the
10.
by paul_mk1 3y ago
This is a good observation, the cards do have different memory bandwidth with the MI210 having more than double the bandwidth via HBM2e. Note that the comparisons between the two cards (MI210 and A6000) are being made for high throughput wo
11.
by paul_mk1 3y ago
>The "-ngl 32" means that only 32 out of 35 layers are being run on the GPU, and this results in a huge slow down as the GPU syncs with the CPU, and then computes the last 3 layers on the CPU. Thanks for the updated run configu
12.
by paul_mk1 3y ago
Appreciate your response. We compared MKML mk600 (5.2GB) against llama.cpp Q5_1 (4.7GB) and Q6_k (5.1GB) on a 4090 for llama-7B. The test is the same in all cases: we generate 128 tokens from a single token prompt (batch=1) and measure perf
13.
by paul_mk1 3y ago
Hi, one of the founders here. Attempting to address some of the comments in a single message. To help understand why we decided not to compare to existing methods: I think it would be difficult to do so fairly, since there are many tradeoff