xcodevn

9 ms·

31.
by xcodevn 2y ago
Look from a different point of view: this is a feature, not a bug. With this, every example has equal weight, while with the fix , every token has equal weight.
32.
by xcodevn 2y ago
Looking forward to Hinton receiving a Fields Medal for inventing backpropagation.
33.
Scaling up self-attention inference(neuralblog.github.io)
1 points by xcodevn 2y ago|0 comments
34.
Scaling up self-attention inference(neuralblog.github.io)
1 points by xcodevn 2y ago|0 comments
35.
Letter from Professors Bengio, Hinton, Lessig, & Russell(safesecureai.org)
2 points by xcodevn 2y ago|0 comments
36.
by xcodevn 2y ago
If you're reading this, you may be interested in my other work on "Exploring Llama-3 MLP Neurons" as well. https://neuralblog.github.io/llama3-neurons/
37.
Logit Prisms: Decomposing Transformer Outputs for Mechanistic Interpretability(neuralblog.github.io)
49 points by xcodevn 2y ago|8 comments
38.
by xcodevn 2y ago
Yes, you're correct. The tokens are highlighted based on the neuron activation value, which is scaled to a range of 0 to 10.
39.
by xcodevn 2y ago
On a related note: recently, I released a visualization of all MLP neurons inside the llama3 8B model. Here is an example "derivative" neuron which is triggered when talking about the derivative concept. https://neural
40.
by xcodevn 2y ago
There is a simple web page where you can explore the neurons and discover interesting features yourself! With more than 400k neurons in the Llama3-8B model, there are plenty of fascinating neurons for you to uncover. Check it out and explor
41.
Exploring MLP neurons inside Llama3 model(neuralblog.github.io)
3 points by xcodevn 2y ago|1 comments
42.
by xcodevn 2y ago
tldr: mamba is not as good as transformer.
43.
by xcodevn 2y ago
please help yourself and do a quick Google search about "zero shot" and "few shot" learning.
44.
by xcodevn 2y ago
Of course, it will require exponential data for zero shot. The keyword here is zero shot . If you think about it for a second, this applies to humans too. We also need exponential training data to do things without examples.
45.
by xcodevn 3y ago
> Probably not. Lmao.
46.
by xcodevn 3y ago
I must say, understanding how transformers work is arguably the most important research problem in history, assuming that AGI can be achieved by just scaling up current LLM models on text, video, audio, etc.
47.
by xcodevn 3y ago
I am not trying to downplay the contribution of the paper, but isn't it obvious that this is the case?
48.
by xcodevn 3y ago
Even if OpenAI falls apart, this is still a good move.
49.
by xcodevn 3y ago
> 5. It has deep implications for the trajectory of a technology that many see as heralding a revolution at least as significant as — if not more than — agriculture or industry, with truly existential implications for humanity. Yes, this
50.
by xcodevn 3y ago
So this is what it looks like inside a car's dreams.
51.
by xcodevn 3y ago
Can anyone conduct a proper measurement of resistivity versus temperature and determine whether it's a superconductor or not?
52.
Self-reference is so cool(old.reddit.com)
2 points by xcodevn 4y ago|0 comments
53.
Pax: a PyTorch-like pytree module on top of Jax(github.com)
3 points by xcodevn 5y ago|0 comments
54.
Google Colab Pro+ ($50/month)(colab.research.google.com)
5 points by xcodevn 5y ago|0 comments
55.
by xcodevn 5y ago
"Analysing the agent’s internal representations, we can say that by taking this approach to reinforcement learning in a vast task space, our agents are aware of the basics of their bodies and the passage of time and that they understan
56.
Prompting GTP-3 is like saying in your head
1 points by xcodevn 5y ago|0 comments
57.
by xcodevn 5y ago
It is a single matrix multiplication to predict probabilities of all possible outputs. For example, with a hidden state of 1024 dimensions, and 8 bits output, it is 1024x256 parameters. 10 bits will need 1024x1024 params.
58.
by xcodevn 5y ago
This is a well known problem. The noise is due to mu-law compression. The 16 bit audio samples are compressed to 8, 9, or 10 bits before feeding to the neutral net. The reason is because predicting a categorical distribution of 2^16 values
59.
by xcodevn 5y ago
The most amazing thing in this video is NOT how we can read signals from the monkey brain, but how a monkey brain can learn to generate signals to control the cursor. I think it is remarkable that a brain can learn to have such fine contr
60.
by xcodevn 5y ago
My immediate thought: train a Transformer (or Tacotron2) that transforms text to the encoded Lyra codes... And, we will finally have a good real-time open-source text-to-speech system running on mobile devices.

More ›