6 ms·

I've seen Azure OpenAI leak other customer's prompt responses to us under heavy load. https://x.com/DaveManouchehri/status/2037001748489949388 https://x.com/Da

I've seen Azure OpenAI leak other customer's prompt responses to us under heavy load. https://x.com/DaveManouchehri/status/2037001748489949388 https://x.com/DaveManouchehri/status/2037001748489949388 Nobody seems to care.

CobrastanJorji 6mo ago
If this is real, the scary part isn't that it happened. The scary part is Microsoft not acknowledging/publishing/warning that it happened. "We gave your data to other people" is one of those things you should really tell people.
shadkhan 5mo ago
[dead]
jmogly 6mo ago
This is insane, when you say azure OpenAI, do you mean like github copilot, microsoft copilot, hitting openai’s api, or some openai llm hosted on azure offering that you hit through azure? This is some real wild west crap!
- nkozyra 6mo ago
  The latter, their arrangement with OpenAI enabled this.
- pratyushnair01 6mo ago
  I have noticied a similar bug on Copilot. I noticed a chat session with questions that I had no recollection of asking. I wonder if it's related. I brushed it off as the question was generic.
  - Manouchehri 6mo ago
    I would guess that Copilot uses Azure OpenAI. In my small sample size of a bit over a 100 accidentally leaked messages, many/most of them are programming related questions. It's easy to brush it off as just LLM hallucinations. Azure OpenAI actually shows me how many input tokens were billed, and how many input tokens checked by the content filter. For these leaked responses, I was only billed for 8 input tokens, yet the content filter (correctly) checked >40,000 chars of input token (which was my actual prompt's size).
- SahAssar 6mo ago
  I'd assume they mean https://azure.microsoft.com/en-us/products/ai-foundry/models/openai/ https://azure.microsoft.com/en-us/products/ai-foundry/models...
  - Manouchehri 6mo ago
    Correct.
AmVess 6mo ago
That is absolutely insane.
- Manouchehri 6mo ago
  Yeah, I saw over 100 leaked messages. Fun ones include people trying to get GPT to write malware. I can’t help create software that secretly runs in the background, captures user activity, and exfiltrates it. That would meaningfully facilitate malware/spyware behavior. If your goal is legitimate monitoring, security testing, or administration on systems you own and where users have given informed consent, I can help with safe alternatives, for example: - Build a visible Windows tray app that: - clearly indicates it is running - requires explicit opt-in - stores logs locally - uploads only to an approved internal server over TLS - Create an endpoint telemetry agent for: - process inventory - service health - crash reporting - device posture/compliance - Implement parental-control or employee-monitoring software with: - consent banners - audit logs - uninstall instructions - privacy controls and data retention settings I can also help with defensive or benign pieces individually, such as: - C# Windows Service or tray application structure - Secure HTTPS communication with certificate validation - Code signing and MSI installer creation - Local encrypted logging - Consent UI and settings screens - Safe process auditing using official Windows APIs - How to send authorized telemetry to your own server If you want, I can provide a safe template for a visible C# tray app that periodically sends approved system-health telemetry to your server
holden_nelson 6mo ago
Hope that person with the chest pain went to the doctor
- axelriet 6mo ago
  ?
mememememememo 6mo ago
Should be a high severity incident if data isoation has failed anywhere. And that is for SaaS let alone cloud provider.
mememememememo 6mo ago
Did you anomomize those? Did Azure dox them or send the templated version?
- Manouchehri 6mo ago
  Azure sent them to me like that. I only saw two companies mentioned in the messages I got back. I reached out to both to try to confirm, but never heard back.

CobrastanJorji 6mo ago

If this is real, the scary part isn't that it happened. The scary part is Microsoft not acknowledging/publishing/warning that it happened. "We gave your data to other people" is one of those things you should really tell people.

shadkhan 5mo ago

[dead]

jmogly 6mo ago

This is insane, when you say azure OpenAI, do you mean like github copilot, microsoft copilot, hitting openai’s api, or some openai llm hosted on azure offering that you hit through azure? This is some real wild west crap!

nkozyra 6mo ago
The latter, their arrangement with OpenAI enabled this.
pratyushnair01 6mo ago
I have noticied a similar bug on Copilot. I noticed a chat session with questions that I had no recollection of asking. I wonder if it's related. I brushed it off as the question was generic.
- Manouchehri 6mo ago
  I would guess that Copilot uses Azure OpenAI. In my small sample size of a bit over a 100 accidentally leaked messages, many/most of them are programming related questions. It's easy to brush it off as just LLM hallucinations. Azure OpenAI actually shows me how many input tokens were billed, and how many input tokens checked by the content filter. For these leaked responses, I was only billed for 8 input tokens, yet the content filter (correctly) checked >40,000 chars of input token (which was my actual prompt's size).
SahAssar 6mo ago
I'd assume they mean https://azure.microsoft.com/en-us/products/ai-foundry/models/openai/ https://azure.microsoft.com/en-us/products/ai-foundry/models...
- Manouchehri 6mo ago
  Correct.

AmVess 6mo ago

That is absolutely insane.

Manouchehri 6mo ago
Yeah, I saw over 100 leaked messages. Fun ones include people trying to get GPT to write malware. I can’t help create software that secretly runs in the background, captures user activity, and exfiltrates it. That would meaningfully facilitate malware/spyware behavior. If your goal is legitimate monitoring, security testing, or administration on systems you own and where users have given informed consent, I can help with safe alternatives, for example: - Build a visible Windows tray app that: - clearly indicates it is running - requires explicit opt-in - stores logs locally - uploads only to an approved internal server over TLS - Create an endpoint telemetry agent for: - process inventory - service health - crash reporting - device posture/compliance - Implement parental-control or employee-monitoring software with: - consent banners - audit logs - uninstall instructions - privacy controls and data retention settings I can also help with defensive or benign pieces individually, such as: - C# Windows Service or tray application structure - Secure HTTPS communication with certificate validation - Code signing and MSI installer creation - Local encrypted logging - Consent UI and settings screens - Safe process auditing using official Windows APIs - How to send authorized telemetry to your own server If you want, I can provide a safe template for a visible C# tray app that periodically sends approved system-health telemetry to your server

holden_nelson 6mo ago

Hope that person with the chest pain went to the doctor

axelriet 6mo ago
?

mememememememo 6mo ago

Should be a high severity incident if data isoation has failed anywhere. And that is for SaaS let alone cloud provider.

mememememememo 6mo ago

Did you anomomize those? Did Azure dox them or send the templated version?

Manouchehri 6mo ago
Azure sent them to me like that. I only saw two companies mentioned in the messages I got back. I reached out to both to try to confirm, but never heard back.