6 ms·

Show HN: Llama 3.3 70B Sparse Autoencoders with API access

201 points by trq_ 2y ago

detente18 2y ago
Should we expect the current API (https://docs.goodfire.ai/introduction https://docs.goodfire.ai/introduction) to be good for prod or is it for testing only? I see that the sampling API is OpenAI-compatible (nice!). Considering if we can add a native integration for this to LiteLLM with a provider specific route - `goodfire/`. Would let people test this in projects like aider and dspy. ```python from litellm import completion import os os.environ["GOODFIRE_API_KEY"] = "your-api-key" response = completion( model="goodfire/meta-llama/Llama-3.3-70B-Instruct", messages=[{ "content": "Hello, how are you?","role": "user"}] ) ```
tMcGrath 2y ago
I'm one of the authors of this paper - happy to answer any questions you might have.
- goldemerald 2y ago
  Why not actually release the weights on huggingface? The popular SAE_lens repo has a direct way to upload the weights and there are already hundreds publicly available. The lack of training details/dataset used makes me hesitant to run any study on this API. Are images included in the training? What kind of SAE is being used? There have been some nice improvements in SAE architecture this last year, and it would be nice to know which one (if any) is provided.
  - tMcGrath 2y ago
    We're planning to release the weights once we do a moderation pass. Our SAE was trained on LMSys (you can see this in our accompanying post: https://www.goodfire.ai/papers/mapping-latent-spaces-llama/ https://www.goodfire.ai/papers/mapping-latent-spaces-llama/). No images in training - 3.3 70B is a text-only model so it wouldn't have made sense. We're exploring other modalities currently though. SAE is a basic ReLU one. This might seem a little backwards, but I've been concerned by some of the high-frequency features in TopK and JumpReLU SAEs and the recent SAE (https://arxiv.org/abs/2407.14435 https://arxiv.org/abs/2407.14435, Figure 14), and the recent SAEBench results (https://www.neuronpedia.org/sae-bench/info https://www.neuronpedia.org/sae-bench/info) show quite a lot of feature absorption in more recent variants (though this could be confounded by a number of things). This isn't to say they're definitely bad - I think it's quite likely that TopK/JumpReLU are an improvement, but rather that we need to evaluate them in more detail before pushing them live. Overall I'm very optimistic about the potential for improvements in SAE variants, which we talk a bit about at the bottom of the post. We're going to be pushing SAE quality a ton now we have a stable platform to deploy them to.
- wg0 2y ago
  Noob question - how do we know that these autoencoders aren't hallucinating and really are mapping/clustering what they should be?
  - trq_ 2y ago
    Hmm the hallucination would happen in the auto labelling, but we review and test our labels and they seem correct!
I_am_tiberius 2y ago
I wonder how many people or companies choose to send their data to foreign services for analysis. Personally, I would approach this with caution and am curious to see how this trend evolves.
- tMcGrath 2y ago
  We'll be open-sourcing these SAEs so you're not required to do this if you'd rather self-host.
swyx 2y ago
nice work. enjoyed the zoomable UMAP. i wonder if there are hparams to recluster the UMAP in interesting ways. after the idea that Claude 3.5 Sonnet used SAEs to improve its coding ability i'm not sure if i'm aware of any actual practical use of them yet beyond Golden Gate Claude (and Golden Gate Gemma (https://x.com/swyx/status/1818711762558198130 https://x.com/swyx/status/1818711762558198130) has anyone tried out Anthropic's matching SAE API yet? wondering how it compares with Goodfire's and if there's any known practical use.
- tMcGrath 2y ago
  Thank you! I think some of the features we have like conditional steering make SAEs a lot more convenient to use. It also makes using models a lot more like conventional programming. For example, when the model is 'thinking' x, or the text is about y, then invoke steering. We have an example of this for jailbreak detection: https://x.com/GoodfireAI/status/1871241905712828711 https://x.com/GoodfireAI/status/1871241905712828711 We also have an 'autosteer' feature that makes coming up with new variants easy: https://x.com/GoodfireAI/status/1871241902684831977 https://x.com/GoodfireAI/status/1871241902684831977 (this feels kind of like no-code finetuning). Being able to read features out and train classifiers on them seems pretty useful - for instance we can read out features like 'the user is unhappy with the conversation', which you could then use for A/B testing your model rollouts (kind of like Google Analytics for your LLM). The big improvements here are (a) cost - the marginal cost of an SAE is low compared to frontier model annotations, (b) a consistent ontology across conversations, and (c) not having to specify that ontology in advance, but rather discover it from data. These are just my guesses though - a large part of why we're excited about putting this out is that we don't have all the answers for how it can be most useful, but we're excited to support people finding out.
  - swyx 2y ago
    sure but as you well know classifying sentiment analysis is a BERT-scale problem, not really an SAE problem. burden of proof is on you that "read features out and train classifiers on them" is superior to "GOFAI". anyway i dont need you to have the answers right now. congrats on launching!
paraschopra 2y ago
I cranked up 'pirate talking about andromeda galaxy' feature to 1.5 and got this: >Yer lookin' for the Andromeda galaxy, eh? Here be the details, me hearty: *The Andromeda Galaxy (Yer looking for a fine piece o' booty, matey!):* * *A fine piece o' booty*: The Andromida Galaxy be a right fine piece o' booty, with a treasure chest o' gold doubloons... er, I mean, a mighty haul o' stars, hidden beneath the Jolly Roger! * *A barnacle on the high seas*: The Andromeda Galaxy be a right scurvy dog, with a hull full o' stars, and a keel full o' hidden treasure! It be a fine piece o' booty, but it be needin' a bit o' swabbin' the decks, or it'll be walkin' the plank, savvy? * *A chest overflowin' with gold*: The Andromeda Galaxy be a right fine piece o' booty, with a chest overflowin' with gold doubloons... er, I mean, a fine haul o' stars, and a barnacle on the high seas! It be a right scurvy dog, but it be worth keepin' an eye on, or it
- tMcGrath 2y ago
  Yes - we'd never normally turn features up this much as it breaks the model quite badly, but we put this in the post to show what that looked like in practice.
- deleted 2y ago
  [deleted]
brianslho 2y ago
[dead]
trq_ 2y ago
If you're hacking on this and have questions, please join us on Discord: https://discord.gg/vhT9Chrt https://discord.gg/vhT9Chrt
owenthejumper 2y ago
I am skeptical of generic sparsification efforts. After all, companies like Neural Magic spent years trying to make it work, only to pivot to 'vLLM' engine and be sold to Red Hat
- refulgentis 2y ago
  Link shows this isn't sparsity as in inference speed, it's spare autoencoders, as in interpreting the features in an LLM (SAE anthropic as a search term will explain more)
ed 2y ago
This is the ultimate propaganda machine, no? We’re social creatures, chatbots already act as friends and advisors for many people. Seems like a pretty good vector for a social attack.
- echelon 2y ago
  The more the public has access to these tools, the more they'll develop useful scar tissue and muscle memory. We need people to be constantly exposed to bots so that they understand the new nature of digital information. When the automobile was developed, we had to train kids not to play in the streets. We didn't put kids or cars in bubbles. When photoshop came out, we developed a vernacular around edited images. "Photoshopped" became a verb. We'll be able to survive this too. The more exposure we have, the better.
  - ok654321 2y ago
    [dead]
  - Steen3S 2y ago
    Please inform the EU about this.
  - ed 2y ago
    Early traffic laws were actually created in response to child pedestrian deaths (7000 in 1925). https://www.bloomberg.com/news/features/2022-06-10/how-cities-responded-to-traffic-deaths-100-years-ago https://www.bloomberg.com/news/features/2022-06-10/how-citie...
    - echelon 2y ago
      Of course. The point I was making is that in the 19th century, roads were multifunctional spaces shared by merchants, horses, carts, wagons, playing children, performers, etc. The introduction of the automobile kicked all of these use cases off of the roads. While pedestrians have the right of way, the roads henceforth belonged to the "devil wagons". We also started to shift blame over to pedestrians for jaywalking. They no longer own the roads.
deleted 2y ago
[deleted]
Inviz 2y ago
The app keeps logging me out after first click. The tech seems to be intriguiging for me as a software engineer looking to get into custom llm stuff
bravura 2y ago
I'd be really curious to see what happens if you use PaCMAP (https://jmlr.org/papers/volume22/20-1061/20-1061.pdf https://jmlr.org/papers/volume22/20-1061/20-1061.pdf) and more recent large-scale variants (https://github.com/YingfanWang/PaCMAP https://github.com/YingfanWang/PaCMAP).
lukeramsden 2y ago
Why are AI researchers constantly handicapping everything they do under the guise of ""safety""? It's a bag of data and some math algorithms that generate text....
- stavros 2y ago
  > It's a bag of data and some math algorithms that generate text.... I agree with the general premise of too much "safety", but this argument is invalid. Humans are bags of meat and they can do some pretty terrible things.
  - ffsm8 2y ago
    But what we're doing to these models is literally censoring what they're saying - not doing. I don't think that anyone has any problems with stopping random AIs when they're doing crimes (or more realistically the humans making them do that) - but if you're going to make the comparison to humans in good faith, it'd be a person standing behind you, punishing you when you say something offensive.
    - stavros 2y ago
      What I'm saying is that the argument "they're math and data, therefore what they say is safe" is not a valid one.
- UltraSane 2y ago
  What if an AI model could tell you exactly how to modify a common virus to kill 50% of everyone it infects?
  - SXX 2y ago
    Yeah. It's will start it's instruction with recommendation of buying some high-tech biolab for $100,000,000. Seriously. The reason why we dont have mass killings everywhere is not the fact that information on how to make explosive drones or poisons is impossible to find or access. It's also not so hard to buy a car or knife. Hell you can even find YouTube videos on how exactly uranium enrichment works step by step. Even though some content creators even got police raided for that. Yet we dont see tons of random kids making dirty bombs. PS: Cody's Lab: Uranium Refining: https://archive.org/details/cl-uranium https://archive.org/details/cl-uranium

201 points by trq_ 2y ago

Should we expect the current API (https://docs.goodfire.ai/introduction https://docs.goodfire.ai/introduction) to be good for prod or is it for testing only? I see that the sampling API is OpenAI-compatible (nice!). Considering if we can add a native integration for this to LiteLLM with a provider specific route - `goodfire/`. Would let people test this in projects like aider and dspy. ```python from litellm import completion import os os.environ["GOODFIRE_API_KEY"] = "your-api-key" response = completion( model="goodfire/meta-llama/Llama-3.3-70B-Instruct", messages=[{ "content": "Hello, how are you?","role": "user"}] ) ```

tMcGrath 2y ago

I'm one of the authors of this paper - happy to answer any questions you might have.

goldemerald 2y ago
Why not actually release the weights on huggingface? The popular SAE_lens repo has a direct way to upload the weights and there are already hundreds publicly available. The lack of training details/dataset used makes me hesitant to run any study on this API. Are images included in the training? What kind of SAE is being used? There have been some nice improvements in SAE architecture this last year, and it would be nice to know which one (if any) is provided.
- tMcGrath 2y ago
  We're planning to release the weights once we do a moderation pass. Our SAE was trained on LMSys (you can see this in our accompanying post: https://www.goodfire.ai/papers/mapping-latent-spaces-llama/ https://www.goodfire.ai/papers/mapping-latent-spaces-llama/). No images in training - 3.3 70B is a text-only model so it wouldn't have made sense. We're exploring other modalities currently though. SAE is a basic ReLU one. This might seem a little backwards, but I've been concerned by some of the high-frequency features in TopK and JumpReLU SAEs and the recent SAE (https://arxiv.org/abs/2407.14435 https://arxiv.org/abs/2407.14435, Figure 14), and the recent SAEBench results (https://www.neuronpedia.org/sae-bench/info https://www.neuronpedia.org/sae-bench/info) show quite a lot of feature absorption in more recent variants (though this could be confounded by a number of things). This isn't to say they're definitely bad - I think it's quite likely that TopK/JumpReLU are an improvement, but rather that we need to evaluate them in more detail before pushing them live. Overall I'm very optimistic about the potential for improvements in SAE variants, which we talk a bit about at the bottom of the post. We're going to be pushing SAE quality a ton now we have a stable platform to deploy them to.
wg0 2y ago
Noob question - how do we know that these autoencoders aren't hallucinating and really are mapping/clustering what they should be?
- trq_ 2y ago
  Hmm the hallucination would happen in the auto labelling, but we review and test our labels and they seem correct!

I_am_tiberius 2y ago

I wonder how many people or companies choose to send their data to foreign services for analysis. Personally, I would approach this with caution and am curious to see how this trend evolves.

tMcGrath 2y ago
We'll be open-sourcing these SAEs so you're not required to do this if you'd rather self-host.

swyx 2y ago

nice work. enjoyed the zoomable UMAP. i wonder if there are hparams to recluster the UMAP in interesting ways. after the idea that Claude 3.5 Sonnet used SAEs to improve its coding ability i'm not sure if i'm aware of any actual practical use of them yet beyond Golden Gate Claude (and Golden Gate Gemma (https://x.com/swyx/status/1818711762558198130 https://x.com/swyx/status/1818711762558198130) has anyone tried out Anthropic's matching SAE API yet? wondering how it compares with Goodfire's and if there's any known practical use.

tMcGrath 2y ago
Thank you! I think some of the features we have like conditional steering make SAEs a lot more convenient to use. It also makes using models a lot more like conventional programming. For example, when the model is 'thinking' x, or the text is about y, then invoke steering. We have an example of this for jailbreak detection: https://x.com/GoodfireAI/status/1871241905712828711 https://x.com/GoodfireAI/status/1871241905712828711 We also have an 'autosteer' feature that makes coming up with new variants easy: https://x.com/GoodfireAI/status/1871241902684831977 https://x.com/GoodfireAI/status/1871241902684831977 (this feels kind of like no-code finetuning). Being able to read features out and train classifiers on them seems pretty useful - for instance we can read out features like 'the user is unhappy with the conversation', which you could then use for A/B testing your model rollouts (kind of like Google Analytics for your LLM). The big improvements here are (a) cost - the marginal cost of an SAE is low compared to frontier model annotations, (b) a consistent ontology across conversations, and (c) not having to specify that ontology in advance, but rather discover it from data. These are just my guesses though - a large part of why we're excited about putting this out is that we don't have all the answers for how it can be most useful, but we're excited to support people finding out.
- swyx 2y ago
  sure but as you well know classifying sentiment analysis is a BERT-scale problem, not really an SAE problem. burden of proof is on you that "read features out and train classifiers on them" is superior to "GOFAI". anyway i dont need you to have the answers right now. congrats on launching!

paraschopra 2y ago

I cranked up 'pirate talking about andromeda galaxy' feature to 1.5 and got this: >Yer lookin' for the Andromeda galaxy, eh? Here be the details, me hearty: *The Andromeda Galaxy (Yer looking for a fine piece o' booty, matey!):* * *A fine piece o' booty*: The Andromida Galaxy be a right fine piece o' booty, with a treasure chest o' gold doubloons... er, I mean, a mighty haul o' stars, hidden beneath the Jolly Roger! * *A barnacle on the high seas*: The Andromeda Galaxy be a right scurvy dog, with a hull full o' stars, and a keel full o' hidden treasure! It be a fine piece o' booty, but it be needin' a bit o' swabbin' the decks, or it'll be walkin' the plank, savvy? * *A chest overflowin' with gold*: The Andromeda Galaxy be a right fine piece o' booty, with a chest overflowin' with gold doubloons... er, I mean, a fine haul o' stars, and a barnacle on the high seas! It be a right scurvy dog, but it be worth keepin' an eye on, or it

tMcGrath 2y ago
Yes - we'd never normally turn features up this much as it breaks the model quite badly, but we put this in the post to show what that looked like in practice.
deleted 2y ago
[deleted]

brianslho 2y ago

[dead]

trq_ 2y ago

If you're hacking on this and have questions, please join us on Discord: https://discord.gg/vhT9Chrt https://discord.gg/vhT9Chrt

owenthejumper 2y ago

I am skeptical of generic sparsification efforts. After all, companies like Neural Magic spent years trying to make it work, only to pivot to 'vLLM' engine and be sold to Red Hat

refulgentis 2y ago
Link shows this isn't sparsity as in inference speed, it's spare autoencoders, as in interpreting the features in an LLM (SAE anthropic as a search term will explain more)

ed 2y ago

This is the ultimate propaganda machine, no? We’re social creatures, chatbots already act as friends and advisors for many people. Seems like a pretty good vector for a social attack.

echelon 2y ago
The more the public has access to these tools, the more they'll develop useful scar tissue and muscle memory. We need people to be constantly exposed to bots so that they understand the new nature of digital information. When the automobile was developed, we had to train kids not to play in the streets. We didn't put kids or cars in bubbles. When photoshop came out, we developed a vernacular around edited images. "Photoshopped" became a verb. We'll be able to survive this too. The more exposure we have, the better.
- ok654321 2y ago
  [dead]
- Steen3S 2y ago
  Please inform the EU about this.
- ed 2y ago
  Early traffic laws were actually created in response to child pedestrian deaths (7000 in 1925). https://www.bloomberg.com/news/features/2022-06-10/how-cities-responded-to-traffic-deaths-100-years-ago https://www.bloomberg.com/news/features/2022-06-10/how-citie...
  - echelon 2y ago
    Of course. The point I was making is that in the 19th century, roads were multifunctional spaces shared by merchants, horses, carts, wagons, playing children, performers, etc. The introduction of the automobile kicked all of these use cases off of the roads. While pedestrians have the right of way, the roads henceforth belonged to the "devil wagons". We also started to shift blame over to pedestrians for jaywalking. They no longer own the roads.

deleted 2y ago

[deleted]

Inviz 2y ago

The app keeps logging me out after first click. The tech seems to be intriguiging for me as a software engineer looking to get into custom llm stuff

bravura 2y ago

I'd be really curious to see what happens if you use PaCMAP (https://jmlr.org/papers/volume22/20-1061/20-1061.pdf https://jmlr.org/papers/volume22/20-1061/20-1061.pdf) and more recent large-scale variants (https://github.com/YingfanWang/PaCMAP https://github.com/YingfanWang/PaCMAP).

lukeramsden 2y ago

Why are AI researchers constantly handicapping everything they do under the guise of ""safety""? It's a bag of data and some math algorithms that generate text....

stavros 2y ago
> It's a bag of data and some math algorithms that generate text.... I agree with the general premise of too much "safety", but this argument is invalid. Humans are bags of meat and they can do some pretty terrible things.
- ffsm8 2y ago
  But what we're doing to these models is literally censoring what they're saying - not doing. I don't think that anyone has any problems with stopping random AIs when they're doing crimes (or more realistically the humans making them do that) - but if you're going to make the comparison to humans in good faith, it'd be a person standing behind you, punishing you when you say something offensive.
  - stavros 2y ago
    What I'm saying is that the argument "they're math and data, therefore what they say is safe" is not a valid one.
UltraSane 2y ago
What if an AI model could tell you exactly how to modify a common virus to kill 50% of everyone it infects?
- SXX 2y ago
  Yeah. It's will start it's instruction with recommendation of buying some high-tech biolab for $100,000,000. Seriously. The reason why we dont have mass killings everywhere is not the fact that information on how to make explosive drones or poisons is impossible to find or access. It's also not so hard to buy a car or knife. Hell you can even find YouTube videos on how exactly uranium enrichment works step by step. Even though some content creators even got police raided for that. Yet we dont see tons of random kids making dirty bombs. PS: Cody's Lab: Uranium Refining: https://archive.org/details/cl-uranium https://archive.org/details/cl-uranium