lappa

8 ms·

1.
by lappa 3mo ago
ChatGPT, request minimal necessary diff to make a specific change, review, ctrl+c, ctrl+v
2.
by lappa 7mo ago
Good post. I think you mixed torch.eye with torch.full though
3.
Lucidrains (Phil Wang) GitHub account has been suspended(github.com)
5 points by lappa 7mo ago|5 comments
4.
Show HN: Train Block Diffusion Models on Consumer Hardware (RTX 4090) in Hours(github.com)
5 points by lappa 1y ago|0 comments
5.
by lappa 1y ago
This isn't suggesting no one understands how these models are architected, nor is anyone saying that SDPA / matrix multiplication isn't understood by those who create these systems. What's being said is that the result o
6.
by lappa 1y ago
I use the SingleFile extension to archive every page I visit. It's easy to set up, but be warned, it takes up a lot of disk space. $ du -h ~/archive/webpages 1.1T /home/andrew/archive/webpages
7.
by lappa 2y ago
OpenAI clearly states that they train on your data https://help.openai.com/en/articles/5722486-how-your-data-is...
8.
by lappa 2y ago
It's easy to argue that Llama-3.3 8B performs better than GPT-3.5. Compare their benchmarks, and try the two side-by-side. Phi-4 is yet another step towards a small, open, GPT-4 level model. I think we're getting quite close. Ch
9.
Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning(techcommunity.microsoft.com)
439 points by lappa 2y ago|143 comments
10.
by lappa 2y ago
Great project, looking forward to seeing more as this develops. Also FYI, your mail server seems to be down.
11.
by lappa 2y ago
Provided a constant temperature of 1.0, you can train the model on prompts with probablistic requests, with loss determined by KL divergence. Expectation: 80% left, 20% right Model sampling probability: 99% left, 1% right >>> 0.80
12.
by lappa 2y ago
Look at it from an algorithmic perspective. In computer science many algorithms take a non-constant number of steps to execute. However, in transformers models, there are a limited number of decoder blocks, and a limited number of FFN layer
13.
by lappa 3y ago
Very interested in the expansion of RL for transformers, but I can't quite tell what this project is. Could you please add links to the documentation to the readme where it states "It includes detailed documentation". Also ma
14.
by lappa 3y ago
A few days ago I saw a post using NeuralFlow to help explain the repetition problem. https://old.reddit.com/r/LocalLLaMA/comments/1ap8mxh/what_ca... > I’ve done some investigation into this. In a wel
15.
Open Inference Engines – Feature Comparison of Language Model Inference Engines(github.com)
3 points by lappa 3y ago|0 comments
16.
by lappa 3y ago
Not a material science expert, however per their paper, they use DFT to verify the stability, then use the verification status to improve the model. >candidate structures filtered using GNoME are evaluated using DFT calculations with sta
17.
by lappa 3y ago
Paper: https://www.nature.com/articles/s41586-023-06735-9
18.
by lappa 3y ago
Excellent, appears Amazon has introduced two important things here: - Rope theta of 100,000, likely from the Llama 2 Long paper which found that a large theta helped regulate attention between distant tokens[0] - A 16k (effective 32k) conte
19.
by lappa 3y ago
More data, more parameters, more compute all result in a better model per "Scaling Laws for Neural Language Models" https://browse.arxiv.org/pdf/2001.08361v1.pdf Largeness is a valid goal.
20.
by lappa 3y ago
From the paper https://github.com/EleutherAI/math-lm https://huggingface.co/EleutherAI/llemma_34b https://huggingface.co/EleutherAI/llemma_7b
21.
by lappa 3y ago
Great work, lots of useful information here. The only thing I wish you did different was explored alpha > 2 * r. In this blog post, the author found that alpha of 4 * r (where r=64) outperformed all smaller alphas in terms of loss when f
22.
by lappa 3y ago
"For example, HyperAttention makes the inference time of ChatGLM2 50% faster on 32k context length while perplexity increases from 5.6 to 6.3." "when half of all attention layers are patched (i.e., 14 layers), we verify that
23.
by lappa 3y ago
This seems half-baked and there are numerous faulty assumptions in this article. For example, Bitcoin miners cannot computer gradients. Their ASICs can only calculate double-sha256. Additionally, the premise of sending gradients of models t
24.
by lappa 3y ago
Interesting how this method quantizes different layers / modules in a manner that minimizes perplexity as it adjusts parameters. I'd be interested to see how 2.5 bit quantization compares to an unadjusted 4-bit baseline. Additiona
25.
by lappa 3y ago
This is great! If I could suggest changes that would make this especially useful for me and likely others it would be 1) The ability to substitute in your own LLM, specifically I'd like to be able to use Code Llama which was released a
26.
by lappa 3y ago
They're based on the original technical report. "Refuel" has run a different set of benchmarks on GPT-3.5 and GPT-4 and found a decline in quality. https://www.refuel.ai/blog-posts/gpt-3-5-turbo-model-com
27.
by lappa 3y ago
Here are some benchmarks, excellent to see that an open model is approaching (and in some areas surpassing) GPT-3.5! AI2 Reasoning Challenge (25-shot) - a set of grade-school science questions. - Llama 1 (llama-65b): 57.6 - LLama 2 (llama-2
28.
by lappa 4y ago
This is a really interesting project with a lot of potential. If I were a sponsor looking for a podcast I would want my search process to look something like this: - Search for a term relevant to my line of business - See a list of podcasts
29.
by lappa 4y ago
>And in small cities/towns, getting a taxi was nearly impossible a lot of the times. My experience has been the opposite. In small towns you cannot get use rideshare apps, but you can dial a taxi service. This has been the case in e
30.
by lappa 4y ago
Or inform the company of the recruiter who is wasting their staffs time interviewing incompatible candidates. This doesn't seem to be the companies fault.

More ›