miven

9 ms·

1.
by miven 8mo ago
I'm really glad that these HNet-inspired approaches are getting traction, I'm a big fan of that paper. Though I wonder how much of the gains in this case are actually due to 75% extra parameters compared to the baseline, even if t
2.
by miven 1y ago
Correct me if I'm misinterpreting something in your argument but as I see it Matryoshka embeddings just sort the vector bases of the output space roughly by order of their importance for the task, PCA-style, so when you truncate your 4
3.
by miven 1y ago
The ARC Prize Foundation ran extensive ablations on HRM for their slew of reasoning tasks and noted that the "hierarchical" part of their architecture is not much more impactful than a vanilla transformer of the same size with no
4.
by miven 1y ago
As far as I understand the "chunking" of input bytes is learned completely end to end, so it's basically up to the model to figure out how to most efficiently delineate and aggregate the information from the inputs according
5.
by miven 1y ago
According to [0] it's partly due to a key change they introduced in interleaving layers that use standard RoPE positional encodings and layers using what's called NoPE [1], not encoding positions at all and letting the model to fi
6.
by miven 1y ago
That's absolutely correct, KV cache is just an optimization trick, you could run the model without it, that's how encoder-only transformers do it. I guess what I'm trying to convey is that the latent representations within a
7.
by miven 1y ago
The key and value vectors are cached, that's kind of the whole point of autoregressive transformer models, the "state" not only survives within the KV cache but, in some sense, grows continuously with each token added, and is
8.
by miven 1y ago
I'm not sure I understand what you're trying to say here, information between tokens is propagated through self-attention, and there's an attention block inside each transformer block within the model, that's a whole lot
9.
by miven 2y ago
Chain of thought isn't exactly transparent either, you shouldn't fall into the pitfall of believing that the final sequence of tokens thinking about the task is the only processing the model actually performs during CoT. There mig
10.
by miven 2y ago
AFAIK retrieving documents that look like the query is more commonly avoided by using a bi-encoder explicitly trained for retrieval, those generally are conditioned to align embeddings of queries to those of relevant documents, with each ha
11.
by miven 2y ago
In section 2 they briefly mention studies such as [1] that point out that the token outputs of a chain of thought aren't always entirely faithful to the responses of the models I'm not sure whether it wouldn't be more reliabl
12.
by miven 2y ago
I mean, it's no free lunch, you still need to expend significantly more compute for the QLoRA training compared to any usual PTQ method, be it SpinQuant or any other more conventional quantization approaches.
13.
by miven 2y ago
Is there an intuitive reason why this ends up working this well compared to, say, applying some kind of thresholding to attention activations that are below average for a given head to filter that same attention noise out?
14.
by miven 2y ago
Residual connections are also worth mentioning as an extremely ubiquitous adaptation, one will be hard-pressed to find a modern architecture that doesn't use those at least to some extent, to the point where the original Resnet paper s
15.
by miven 2y ago
Yeah, it just got updated, here's the new link, they added sections on block-wise quantization for both the rounding-based and LUT-based approach: https://apple.github.io/coremltools/docs-guides/source/op
16.
by miven 2y ago
Huh, generally whenever I saw the lookup table approach in literature it was also referred to as quantization, guess they wanted to disambiguate the two methods Though I'm not sure how warranted it really is, in both cases it's st
17.
by miven 2y ago
> For on-device inference, we use low-bit palletization, a critical optimization technique that achieves the necessary memory, power, and performance requirements. Did they go over the entire text with a thesaurus? I've never seen &
18.
by miven 2y ago
I don't get your point, how is what you're suggesting here different from a few papers we already have on KV cache pruning methods like [1]? [1] https://arxiv.org/abs/2305.15805
19.
by miven 2y ago
The authors mention that Jacobi decoding is equivalent to greedy autoregressive decoding, but in practice don't we often want the sampling temperature to be above zero to avoid repetitions and excessively generic responses? I'm co
20.
by miven 2y ago
Any guesses as to why they bumped the parameter count up from 7B to 8B?
21.
by miven 2y ago
What you describe here sounds a little like the line of work centered around Universal Transformers, which basically process the input embeddings through a single transformer block multiple times with a separate module deciding when the emb
22.
by miven 2y ago
I think the author of this method said it's coming in a month or so
23.
by miven 3y ago
Correct me if I'm wrong but usually when you do normal token by token inference in a transformer you store calculations made in the previous step in a KV cache so you can reuse it instead of calculating it all over again. But here sinc
24.
by miven 3y ago
Now that I think about it, doesn't this "technique" triple the amount of compute and memory per generated token since each model needs to also compute and store the KV values for the two previous tokens it didn't generat
25.
by miven 3y ago
It was mentioned to be a 20B in a comparison table in a paper co-written by Microsoft, but they've since claimed that it's just an error, and I mean, they'd need to be sitting on some really impressive distilling techniques t
26.
by miven 3y ago
What do they consider to be an "LLM of this size"? While this technique of scaling up an existing pre-trained model via fine-tuning is really impressive, it feels a bit unfair to compare what's essentially now an 8.3B model t
27.
by miven 3y ago
>a major feature of transformers being wildly faster inference than with LSTM Wasn't the main issue with RNNs the fact that inference during training can't be efficiently parallelized? The inference itself normally should be fa
28.
by miven 3y ago
I wonder what would be the most efficient tactic for offloading select layers of such a model to a GPU within a memory-constrained system As far as I understand usually layer offloading in something like llama.cpp loads the first few consec
29.
by miven 3y ago
There seems to be an experimental Mistral Medium model listed among other available model endpoints on [1], the comparison table they give shows that it outmatches 8x7B by a few percent on every benchmark listed [1] https://mistr
30.
by miven 3y ago
> You won't be able to run this on your home GPU. As far as I understand in a MOE model only one/few experts are actually used at the same time, shouldn't the inference speed for this new MOE model be roughly the same as f

More ›