5 ms·

Google announces Sec-Gemini v1 a new experimental cybersecurity model

161 points by ebursztein 1y ago

esafak 1y ago
It's interesting how we're seeing the emergence of specialized models, much like trained humans.
- jgalt212 1y ago
  What's old is new again. Pretty much all ML and statistical models were specialized for a single task / domain.
deleted 1y ago
[deleted]
jruohonen 1y ago
> Next, in response to a question about the vulnerabilities in the Salt Typhoon description, Sec-Gemini v1 outputs not only vulnerability details (thanks to its integration with OSV data, the open-source vulnerabilities database operated by Google), but also contextualizes the vulnerabilities with respect to threat actors (using Mandiant data). I remain still skeptical about LLMs in this space, although I might be proven wrong, as often happens. Nevertheless, OSV has already been a big advance, so it is great that it gets a further commitment.
ziddoap 1y ago
Could be great for augmenting a cybersec professional's tasks; I'm certainly interested in trying it. However, I fear it will not be used as just one of the tools in the toolbox, and rather it will be used as something to defer (and consequently shed liability) to.
- booi 1y ago
  Has anybody been able to shed liability to AI yet?
  - ziddoap 1y ago
    In the legal sense? I'm not sure. In the corporate day-to-day? Absolutely.
  - deleted 1y ago
    [deleted]
  - walleeee 1y ago
    We have practiced the art of liability displacement from living, breathing human beings to artificial constructions for a lot longer than we've had a digital substrate for such
andy99 1y ago
Is this a "model" as in a set of transformer weights that inherently does security work or is it a system that has data lookup and or other tools along with an LLM to do the question interpretation, synthesis, and output presentation? From the description re data integrations it sounds like the latter, unless the data mentioned is in fact used for training. The distinction is important because a security-tuned model will have different limitations and uses than an actual pre-build security LLM app. Being an app also makes benchmarking against other "models" less straightforward.
qwertox 1y ago
There is generally something about the Gemini models which feels a bit different than Claude, ChatGPT or Mistral. I always have the feeling that I'm chatting with a model oriented towards engineering tasks. The seriousness, lack of interest of being humorous or cool. I don't know if this is because I interact with Gemini only through AI Studio, and it may have different system instructions (apart from those one can add oneself, which I never do) than the one at gemini.google.com. I never use gemini.google.com because of the lack of a simple export feature. And it's not even possible to save one chat to disk (well, neither do the others), I just wish it did. AI Studio saving to Google Drive is really useful. I lets you download the chat, strip it of verbose things like the thinking process, and reuse it in a new chat. I wish gemini.google.com had a "Save as Markdown" per answer and for the complete chat (with a toggle to include/exclude the thinking process). Then it would be a no brainer for me. It's the same as if Google Docs would not have an "Download.." menu entry but you could only "save" the documents via Takeout.
- Y_Y 1y ago
  > The seriousness, lack of interest of being humorous or cool. I love this. When ChatGPT compliments me on my great question or tries to banter it causes me great despair.
  - neodypsis 1y ago
    I've noticed 4o uses a lot of emojis, and, in general, is very enthusiastic. I find it funny. If I want a more formal bot, I switch to one of the o3 family.
    - deleted 1y ago
      [deleted]
    - jonplackett 1y ago
      Every now and then 4o seems to get a bit drunk and use tonnes of emojis or start swearing when I haven’t sworn myself in the chat. The other day I asked a fairly innocuous question and it LOLed and said it’d give me the ‘no Bullshit answer’
notepad0x90 1y ago
I'm always torn apart when it comes to LLMs and analytical tasks. When you perform an analytical task, whether it is something simple like assessing the potential risk and impact of a vulnerability or complex like analyzing an obfuscated malware sample to determine its capabilities, you have to thoroughly go over the data points available to you, and corroborate the data points or evidence you are using to come up with conclusions. LLMs can help with a lot of this, but you still have to go over their reasoning (black-box mostly) or backtrack their work before you can accept their conclusions. In other words, even with humans, their skills and experience are never enough. they have to show the reasoning behind their conclusions and then show that reasoning is backed up by an independent source of fact. Short of that, you can still perform analysis, but then you must clearly state that your analysis is weak and requires more follow-up work and validation. So with LLMs, I'm torn up because they kind of make your life a lot easier, but does it just feel that way or are they adding more work and uncertainty where that is intolerable?
ZYbCRq22HbJ2y7 1y ago
Does it seem like a bad idea to trust something that is probablistically correct with security?
- amitport 1y ago
  Like with any automatic procedure: Are humans better? Specifically, in their own example they are just citing Mandiant, which may itself be wrong... https://news.ycombinator.com/item?id=43595294 https://news.ycombinator.com/item?id=43595294
mmooss 1y ago
Using AI systems for high-speed security actions, proactive and reactive, seems necessary but not sufficient: I expect attackers will also use AI systems, trained on the latest in effective attacks. What about defense would make defenders' AI systems more effective than attackers'? I think it's necessary because, if the attackers use AI systems then the defenders need to keep up. Also, we need to be creating far more secure systems to start with. Now it is, to a degree, security through obscurity - something is secure when attackers can't find the bugs fast enough. Security through obsurity wouldn't seem to work well when the attacker uses AI software.
majestik 1y ago
I read the article, and while it’s great the model can generate relevant output- so what? The article doesn’t discuss any action being taken using that output. So what’s the big breakthrough here?
infoSecer 1y ago
It always blows my mind that nobody at Google thought it would be a good idea to very carefully review the answer of the AI. In the second screenshot, the prompt asks about CVE-2024-3400, and at first glance this appears ok. But in the affected systems section it states: > Also Hitachi Energy RTU500 firmware and Siemens Ruggedcom APE1808 firmware. I cannot find any reference that this Hitachi device is vulnerable to that CVE. Hitachi has a nice interface to list all vulnerabilities of their devices, this CVE is not part of it. In the Mitigation section any mention of Hitachi is also missing. Almost as if this device is not vulnerable. There is some more weirdness, like it doesn't mention the "portal" feature is also vulnerable.
- ebursztein 1y ago
  Thanks for looking in-depth in our post. The Hitachi RTU500 mention is not an hallucination, we did check for those. It is mentioned in the Mandiant threat intelligence data.
  - infoSecer 1y ago
    Have you considered that Mandiant is wrong? I cannot find any evidence that it would be vulnerable. Hitachi doesn't even appear to be a technology partner of Palo Alto (https://technologypartners.paloaltonetworks.com/English/directory?q=Hitachi https://technologypartners.paloaltonetworks.com/English/dire...). As far as I can tell, the only connection between those is, that CISA released this alert which mentions multiple unrelated advisories in one post. Which happens to be the Siemens Palo Alto and another unrelated Hitachi advisory in RTU500: https://www.cisa.gov/news-events/alerts/2024/04/25/cisa-releases-eight-industrial-control-systems-advisories https://www.cisa.gov/news-events/alerts/2024/04/25/cisa-rele...
    - fc417fc802 1y ago
      Isn't the tool doing its job in that case? I wouldn't generally expect it to independently determine that an otherwise reliable source made a mistake. In fact I feel like that would be a really bad idea. Imagine if a relatively clueless intern left something out of a report because the textbook "seemed wrong".
arresin 1y ago
Maybe this has something to do with the wiz acquisition.

161 points by ebursztein 1y ago

esafak 1y ago

It's interesting how we're seeing the emergence of specialized models, much like trained humans.

jgalt212 1y ago
What's old is new again. Pretty much all ML and statistical models were specialized for a single task / domain.

deleted 1y ago

[deleted]

jruohonen 1y ago

> Next, in response to a question about the vulnerabilities in the Salt Typhoon description, Sec-Gemini v1 outputs not only vulnerability details (thanks to its integration with OSV data, the open-source vulnerabilities database operated by Google), but also contextualizes the vulnerabilities with respect to threat actors (using Mandiant data). I remain still skeptical about LLMs in this space, although I might be proven wrong, as often happens. Nevertheless, OSV has already been a big advance, so it is great that it gets a further commitment.

ziddoap 1y ago

Could be great for augmenting a cybersec professional's tasks; I'm certainly interested in trying it. However, I fear it will not be used as just one of the tools in the toolbox, and rather it will be used as something to defer (and consequently shed liability) to.

booi 1y ago
Has anybody been able to shed liability to AI yet?
- ziddoap 1y ago
  In the legal sense? I'm not sure. In the corporate day-to-day? Absolutely.
- deleted 1y ago
  [deleted]
- walleeee 1y ago
  We have practiced the art of liability displacement from living, breathing human beings to artificial constructions for a lot longer than we've had a digital substrate for such

andy99 1y ago

Is this a "model" as in a set of transformer weights that inherently does security work or is it a system that has data lookup and or other tools along with an LLM to do the question interpretation, synthesis, and output presentation? From the description re data integrations it sounds like the latter, unless the data mentioned is in fact used for training. The distinction is important because a security-tuned model will have different limitations and uses than an actual pre-build security LLM app. Being an app also makes benchmarking against other "models" less straightforward.

qwertox 1y ago

There is generally something about the Gemini models which feels a bit different than Claude, ChatGPT or Mistral. I always have the feeling that I'm chatting with a model oriented towards engineering tasks. The seriousness, lack of interest of being humorous or cool. I don't know if this is because I interact with Gemini only through AI Studio, and it may have different system instructions (apart from those one can add oneself, which I never do) than the one at gemini.google.com. I never use gemini.google.com because of the lack of a simple export feature. And it's not even possible to save one chat to disk (well, neither do the others), I just wish it did. AI Studio saving to Google Drive is really useful. I lets you download the chat, strip it of verbose things like the thinking process, and reuse it in a new chat. I wish gemini.google.com had a "Save as Markdown" per answer and for the complete chat (with a toggle to include/exclude the thinking process). Then it would be a no brainer for me. It's the same as if Google Docs would not have an "Download.." menu entry but you could only "save" the documents via Takeout.

Y_Y 1y ago
> The seriousness, lack of interest of being humorous or cool. I love this. When ChatGPT compliments me on my great question or tries to banter it causes me great despair.
- neodypsis 1y ago
  I've noticed 4o uses a lot of emojis, and, in general, is very enthusiastic. I find it funny. If I want a more formal bot, I switch to one of the o3 family.
  - deleted 1y ago
    [deleted]
  - jonplackett 1y ago
    Every now and then 4o seems to get a bit drunk and use tonnes of emojis or start swearing when I haven’t sworn myself in the chat. The other day I asked a fairly innocuous question and it LOLed and said it’d give me the ‘no Bullshit answer’

notepad0x90 1y ago

I'm always torn apart when it comes to LLMs and analytical tasks. When you perform an analytical task, whether it is something simple like assessing the potential risk and impact of a vulnerability or complex like analyzing an obfuscated malware sample to determine its capabilities, you have to thoroughly go over the data points available to you, and corroborate the data points or evidence you are using to come up with conclusions. LLMs can help with a lot of this, but you still have to go over their reasoning (black-box mostly) or backtrack their work before you can accept their conclusions. In other words, even with humans, their skills and experience are never enough. they have to show the reasoning behind their conclusions and then show that reasoning is backed up by an independent source of fact. Short of that, you can still perform analysis, but then you must clearly state that your analysis is weak and requires more follow-up work and validation. So with LLMs, I'm torn up because they kind of make your life a lot easier, but does it just feel that way or are they adding more work and uncertainty where that is intolerable?

ZYbCRq22HbJ2y7 1y ago

Does it seem like a bad idea to trust something that is probablistically correct with security?

amitport 1y ago
Like with any automatic procedure: Are humans better? Specifically, in their own example they are just citing Mandiant, which may itself be wrong... https://news.ycombinator.com/item?id=43595294 https://news.ycombinator.com/item?id=43595294

mmooss 1y ago

Using AI systems for high-speed security actions, proactive and reactive, seems necessary but not sufficient: I expect attackers will also use AI systems, trained on the latest in effective attacks. What about defense would make defenders' AI systems more effective than attackers'? I think it's necessary because, if the attackers use AI systems then the defenders need to keep up. Also, we need to be creating far more secure systems to start with. Now it is, to a degree, security through obscurity - something is secure when attackers can't find the bugs fast enough. Security through obsurity wouldn't seem to work well when the attacker uses AI software.

majestik 1y ago

I read the article, and while it’s great the model can generate relevant output- so what? The article doesn’t discuss any action being taken using that output. So what’s the big breakthrough here?

infoSecer 1y ago

It always blows my mind that nobody at Google thought it would be a good idea to very carefully review the answer of the AI. In the second screenshot, the prompt asks about CVE-2024-3400, and at first glance this appears ok. But in the affected systems section it states: > Also Hitachi Energy RTU500 firmware and Siemens Ruggedcom APE1808 firmware. I cannot find any reference that this Hitachi device is vulnerable to that CVE. Hitachi has a nice interface to list all vulnerabilities of their devices, this CVE is not part of it. In the Mitigation section any mention of Hitachi is also missing. Almost as if this device is not vulnerable. There is some more weirdness, like it doesn't mention the "portal" feature is also vulnerable.

ebursztein 1y ago
Thanks for looking in-depth in our post. The Hitachi RTU500 mention is not an hallucination, we did check for those. It is mentioned in the Mandiant threat intelligence data.
- infoSecer 1y ago
  Have you considered that Mandiant is wrong? I cannot find any evidence that it would be vulnerable. Hitachi doesn't even appear to be a technology partner of Palo Alto (https://technologypartners.paloaltonetworks.com/English/directory?q=Hitachi https://technologypartners.paloaltonetworks.com/English/dire...). As far as I can tell, the only connection between those is, that CISA released this alert which mentions multiple unrelated advisories in one post. Which happens to be the Siemens Palo Alto and another unrelated Hitachi advisory in RTU500: https://www.cisa.gov/news-events/alerts/2024/04/25/cisa-releases-eight-industrial-control-systems-advisories https://www.cisa.gov/news-events/alerts/2024/04/25/cisa-rele...
  - fc417fc802 1y ago
    Isn't the tool doing its job in that case? I wouldn't generally expect it to independently determine that an otherwise reliable source made a mistake. In fact I feel like that would be a really bad idea. Imagine if a relatively clueless intern left something out of a report because the textbook "seemed wrong".

arresin 1y ago

Maybe this has something to do with the wiz acquisition.