6 ms·

I picked up a shitty NUC from ewaste and it had a label on it for an AI company

204 points by _ugfj 2y ago

avidphantasm 2y ago
“Move fast and break things” strikes again.
daniel-s 2y ago
It's hard to explain to my non-technical friends/relatives, i.e., everyone I know outside work, why I'm careful about digital security and privacy.
- deleted 2y ago
  [deleted]
teddyh 2y ago
I used to be a little bit sad that dumpster diving would be a phenomenon lost to time. Not so, it turns out; I guess everything old is new again, just in slightly altered forms.
- massysett 2y ago
  Just Google “dumpster diving for food” and how alive and well that is might surprise you.
  - teddyh 2y ago
    I was was obviously referring to <http://www.catb.org/jargon/html/D/dumpster-diving.html http://www.catb.org/jargon/html/D/dumpster-diving.html>.
2muchcoffeeman 2y ago
What the hell is happening that developers still have the hubris to roll their own secrets manager?
- Rinzler89 2y ago
  Resümee driven engineering?
  - exitb 2y ago
    Can't imagine a bigger red flag on a resume.
    - Rinzler89 2y ago
      >see resume of dude claiming to roll out his own OS instead of using one of the established ones >view it as red flag and throw his resume away >dude is Linus Torvalds
  - bayindirh 2y ago
    Sometimes you feel like implementing one will take shorter than configuring one. It's a phase we all go through, but some of us are stuck there forever.
  - fragmede 2y ago
    This is the opposite of that. If they had practiced resumé driven development, that NUC would have been an k8s node that had secrets encrypted at-rest on top of FDE.
- tajd 2y ago
  “Ah I can just roll my own in half an hour!” Says the person with a learning curve ahead of them!
- theshrike79 2y ago
  "We've got $10M VC money and I've always wanted to try writing a secrets manager"
- xyzzy123 2y ago
  Lack of senior infra engineer-itis. It's fairly normal not to realise there are entire ecosystems of tools and conventions to solve certain problems if you haven't come across them many times before.
deleted 2y ago
[deleted]
atemerev 2y ago
Security by irrelevance. On paper, that’s a catastrophe. In practice, however, even criminals won’t extract much value from these random logs and video records.
- viraptor 2y ago
  I can come up with some ideas for the voice assistant logs to be useful if the S3 bucket contains a lot of them: - getting private numbers of known people (from "call Some Name") - spam targeting from calendar event creation - various private info available from dictated notes
  - atemerev 2y ago
    Just get any recent database leak for that on any of the numerous sites trading breaches / leaks. Voice samples can be useful for scammers of course, but the victim needs to be rich enough to justify the attack (and it is easy enough to get a voice sample anyway if you have the phone number). Etc.
    - viraptor 2y ago
      It's a new source. Of course there's a different way to get that info too. It's like saying "we had one leak, there's no value in any new data leaks ever".
keepamovin 2y ago
Is there a danger this is from a cloud provider, not the startup, and the provider just dumped old rack parts without sanitizing them?
- BoredPositron 2y ago
  Your data, your responsibility. No matter where you host.
  - perbu 2y ago
    Sure. A cloud provider can copy your data without you knowing, they might migrate a host from one cluster to another and copy the data along with it and I would find it unreasonable to be held accountable if they forgot to scrub the source. It would still be my problem at the end of the day.
- philipwhiuk 2y ago
  No - because the startup had labelled their kit.
- rcarmo 2y ago
  A cloud provider wouldn't use NUCs. Seriously, they have better things to do with their time than rack and stack multiple tiny boxes with PSUs (and, apparently, external USB fans as well). Also, serious cloud providers (besides using server-grade hardware) have to follow proper equipment destruction and recycling procedures--Azure datacenters, for instance, used to have an on-site industrial shredder for disks (which were nevertheless hardware encrypted, but any failing storage was destroyed anyway).
- fragmede 2y ago
  encrypted at rest means no, but you're welcome to encrypt it another time if you don't believe them.
- acdha 2y ago
  It’d be a huge screwup for a major provider. They promise at-rest encryption and secure disposal of media to their enterprise customers so that’d be a very expensive omission. In this case, there’s no indication of that and it’s so poorly handled that it radiates a startup winging it where they “didn’t have time” to hire anyone with a clue since the AI gold rush was right there. Given the reported healthcare data this seems like an especially bad choice.
teekert 2y ago
Every time I hook my dev NUC up to my HDMI cable and plug in a keyboard, just to decrypt the ssd en get back to working after a reboot, I will now be thinking of this story and feel better about this little nuisance. (Yeah I know there are better ways, I could ssh into the boot env with systemd, I could just encrypt the home dir, I could somehow use the TPM (but that is still pita) etc, I'll look at that the next time I set up the a dev machine...)
- inductive_magic 2y ago
  YubiKey
  - microtonal 2y ago
    Or remote unlock: https://github.com/gsauthof/dracut-sshd https://github.com/gsauthof/dracut-sshd
    - inductive_magic 2y ago
      Very cool, ty.
- adriancr 2y ago
  You can use "keyscript=" in "/etc/crypttab" to run any script you want. Using that you could set up networking and download the key from somewhere (say a remote KMS that would need user approval before continuing). So you would reduce your problem to being prompted on phone to unlock your desktop. You could even use some sort of hardware key to prove request came from the physical server. (simplest might be a RP2350 with some key burnt in) Or you could use something like SGX if it's still available anywhere to prove the same.
  - michaelt 2y ago
    > You can use "keyscript=" in "/etc/crypttab" to run any script you want. ...unless your distro uses systemd, which removes keyscript support [1] because the systemd guy "really dislikes generic callouts" [1] https://github.com/systemd/systemd/pull/3007 https://github.com/systemd/systemd/pull/3007
beAbU 2y ago
Where do I need to hang out so that I can also score stacks of NUCs from a dumpster?
- diggan 2y ago
  Start bringing good coffee and treats to your local recycling center for the person/people who work there when you recycle stuff. After doing that a couple of times, ask nicely if you can take some hardware people are throwing. Alternative approach, hang out outside your target location to figure out when/where they throw stuff, and when trash collectors come. Arrive somewhere in between and dumpster dive :)
  - pjmlp 2y ago
    Beware that both approaches might be considering stealing in some jurisditions. How often that is taken into consideration is another matter.
    - codetrotter 2y ago
      It’s actually so sad. Last time I was at the recycling center a couple of years ago there was a Nintendo 64 in the electronics waste. I bet that it probably still worked. And even if it didn’t it would have been fixable. But I wasn’t allowed to pick it up. Instead this console just has to get destroyed. And for what? It’s not even like a Nintendo 64 is going to have any personal data on it that poses any danger to the previous owner. And on the flip side if the argument is that the electronics could be dangerous because they are broken. I probably run the same risk when buying electronics second hand anyway. So I don’t think that should prevent them from letting people pick up things either.
    - diggan 2y ago
      > Beware that both approaches might be considering stealing in some jurisditions. Yeah, most definitely, which is why you need to befriend them before asking, otherwise it's a guaranteed "No, we cannot do that".
- anorangecat 2y ago
dncornholio 2y ago
[flagged]
rcarmo 2y ago
Another classic Foone story. I heartily recommend the follow on Mastodon, there's at least a gem a week, if not more. Also, worth noting that they are currently in need of some help, so consider supporting them: https://digipres.club/@foone/112929955279707608 https://digipres.club/@foone/112929955279707608
crispyambulance 2y ago
Yeah, those boxes in the loading docks of data centers... Lots of interesting stuff. Picked up a SFF Dell desktop from a huge pile of identical ones in a large cardboard box last spring. They had the good sense, however, to harvest the SSD and memory. Got replacements for a song and now I run home assistant on that thing. There's also sometimes "old iron" in the dock. Sun servers from back in the day. Beautiful hardware but not something one would ever want to take home.
- pbhjpbhj 2y ago
  Harvesting the memory, that's a bit (ha!) extreme. If the box is powered down then it's there any evidence residual memory can be imaged?
  - vetinari 2y ago
    More like RAM can be sold separately. The computer itself would sell for the same amount whether with ram or not.
deleted 2y ago
[deleted]
flumpcakes 2y ago
Charity subsidising the broken? US healthcare industry.
- rcarmo 2y ago
  So you'd rather we not help someone in need? Because your comment is _very_ insensitive in a personal context.
  - flumpcakes 2y ago
    I don't think it was insensitive, just a fact. Why do I have to read begging messages on a tech forum for a citizen of the world's biggest and richest economy? Maybe that's insensitive to the 4bn other people on the planet living in poverty with no chance of access to healthcare, begging or not.
Ekaros 2y ago
I feel justified on hammering the decommissioning process now. Just pull the disk and destroy it...
ajb 2y ago
I guess insolvency law needs to be changed to place some duty of care towards data subjects on insolvency practitioners, rather than allowing everything to be flogged off to the highest bidder. Since winding up user data storage safely is not free, this probably means data processors[1] should be required to get insolvency insurance to cover it. [1] "data processor" in the sense of the various user data protection laws, not just any data processor
raxxorraxor 2y ago
My work NUC just shuts down if I start any AI process, be that running an LLM or stable diffusion. Pooof, system off. Temperature doesn't matter. It also doesn't boot when I activate all processor cores. I have to leave one of those unused. Although that may be due some "valuable" feature like secure boot or bitlocker. I heard NUC now goes to ASUS, perhaps the devices can improve. There are quite a few problems here. I mean I expected abysmal performance on any pure CPU AI task, but some of those could have run in the background. I wonder what Intel did with the time they were the dominant player on a lot of markets... The security here is just that the device is so bad, that no party could extract something useful. Seriously, these things...
susanhea 2y ago
[flagged]

204 points by _ugfj 2y ago

“Move fast and break things” strikes again.

It's hard to explain to my non-technical friends/relatives, i.e., everyone I know outside work, why I'm careful about digital security and privacy.

deleted 2y ago
[deleted]

teddyh 2y ago

I used to be a little bit sad that dumpster diving would be a phenomenon lost to time. Not so, it turns out; I guess everything old is new again, just in slightly altered forms.

massysett 2y ago
Just Google “dumpster diving for food” and how alive and well that is might surprise you.
- teddyh 2y ago
  I was was obviously referring to <http://www.catb.org/jargon/html/D/dumpster-diving.html http://www.catb.org/jargon/html/D/dumpster-diving.html>.

2muchcoffeeman 2y ago

What the hell is happening that developers still have the hubris to roll their own secrets manager?

Rinzler89 2y ago
Resümee driven engineering?
- exitb 2y ago
  Can't imagine a bigger red flag on a resume.
  - Rinzler89 2y ago
    >see resume of dude claiming to roll out his own OS instead of using one of the established ones >view it as red flag and throw his resume away >dude is Linus Torvalds
- bayindirh 2y ago
  Sometimes you feel like implementing one will take shorter than configuring one. It's a phase we all go through, but some of us are stuck there forever.
- fragmede 2y ago
  This is the opposite of that. If they had practiced resumé driven development, that NUC would have been an k8s node that had secrets encrypted at-rest on top of FDE.
tajd 2y ago
“Ah I can just roll my own in half an hour!” Says the person with a learning curve ahead of them!
theshrike79 2y ago
"We've got $10M VC money and I've always wanted to try writing a secrets manager"
xyzzy123 2y ago
Lack of senior infra engineer-itis. It's fairly normal not to realise there are entire ecosystems of tools and conventions to solve certain problems if you haven't come across them many times before.

deleted 2y ago

[deleted]

atemerev 2y ago

Security by irrelevance. On paper, that’s a catastrophe. In practice, however, even criminals won’t extract much value from these random logs and video records.

viraptor 2y ago
I can come up with some ideas for the voice assistant logs to be useful if the S3 bucket contains a lot of them: - getting private numbers of known people (from "call Some Name") - spam targeting from calendar event creation - various private info available from dictated notes
- atemerev 2y ago
  Just get any recent database leak for that on any of the numerous sites trading breaches / leaks. Voice samples can be useful for scammers of course, but the victim needs to be rich enough to justify the attack (and it is easy enough to get a voice sample anyway if you have the phone number). Etc.
  - viraptor 2y ago
    It's a new source. Of course there's a different way to get that info too. It's like saying "we had one leak, there's no value in any new data leaks ever".

keepamovin 2y ago

Is there a danger this is from a cloud provider, not the startup, and the provider just dumped old rack parts without sanitizing them?

BoredPositron 2y ago
Your data, your responsibility. No matter where you host.
- perbu 2y ago
  Sure. A cloud provider can copy your data without you knowing, they might migrate a host from one cluster to another and copy the data along with it and I would find it unreasonable to be held accountable if they forgot to scrub the source. It would still be my problem at the end of the day.
philipwhiuk 2y ago
No - because the startup had labelled their kit.
rcarmo 2y ago
A cloud provider wouldn't use NUCs. Seriously, they have better things to do with their time than rack and stack multiple tiny boxes with PSUs (and, apparently, external USB fans as well). Also, serious cloud providers (besides using server-grade hardware) have to follow proper equipment destruction and recycling procedures--Azure datacenters, for instance, used to have an on-site industrial shredder for disks (which were nevertheless hardware encrypted, but any failing storage was destroyed anyway).
fragmede 2y ago
encrypted at rest means no, but you're welcome to encrypt it another time if you don't believe them.
acdha 2y ago
It’d be a huge screwup for a major provider. They promise at-rest encryption and secure disposal of media to their enterprise customers so that’d be a very expensive omission. In this case, there’s no indication of that and it’s so poorly handled that it radiates a startup winging it where they “didn’t have time” to hire anyone with a clue since the AI gold rush was right there. Given the reported healthcare data this seems like an especially bad choice.

teekert 2y ago

Every time I hook my dev NUC up to my HDMI cable and plug in a keyboard, just to decrypt the ssd en get back to working after a reboot, I will now be thinking of this story and feel better about this little nuisance. (Yeah I know there are better ways, I could ssh into the boot env with systemd, I could just encrypt the home dir, I could somehow use the TPM (but that is still pita) etc, I'll look at that the next time I set up the a dev machine...)

inductive_magic 2y ago
YubiKey
- microtonal 2y ago
  Or remote unlock: https://github.com/gsauthof/dracut-sshd https://github.com/gsauthof/dracut-sshd
  - inductive_magic 2y ago
    Very cool, ty.
adriancr 2y ago
You can use "keyscript=" in "/etc/crypttab" to run any script you want. Using that you could set up networking and download the key from somewhere (say a remote KMS that would need user approval before continuing). So you would reduce your problem to being prompted on phone to unlock your desktop. You could even use some sort of hardware key to prove request came from the physical server. (simplest might be a RP2350 with some key burnt in) Or you could use something like SGX if it's still available anywhere to prove the same.
- michaelt 2y ago
  > You can use "keyscript=" in "/etc/crypttab" to run any script you want. ...unless your distro uses systemd, which removes keyscript support [1] because the systemd guy "really dislikes generic callouts" [1] https://github.com/systemd/systemd/pull/3007 https://github.com/systemd/systemd/pull/3007

beAbU 2y ago

Where do I need to hang out so that I can also score stacks of NUCs from a dumpster?

diggan 2y ago
Start bringing good coffee and treats to your local recycling center for the person/people who work there when you recycle stuff. After doing that a couple of times, ask nicely if you can take some hardware people are throwing. Alternative approach, hang out outside your target location to figure out when/where they throw stuff, and when trash collectors come. Arrive somewhere in between and dumpster dive :)
- pjmlp 2y ago
  Beware that both approaches might be considering stealing in some jurisditions. How often that is taken into consideration is another matter.
  - codetrotter 2y ago
    It’s actually so sad. Last time I was at the recycling center a couple of years ago there was a Nintendo 64 in the electronics waste. I bet that it probably still worked. And even if it didn’t it would have been fixable. But I wasn’t allowed to pick it up. Instead this console just has to get destroyed. And for what? It’s not even like a Nintendo 64 is going to have any personal data on it that poses any danger to the previous owner. And on the flip side if the argument is that the electronics could be dangerous because they are broken. I probably run the same risk when buying electronics second hand anyway. So I don’t think that should prevent them from letting people pick up things either.
  - diggan 2y ago
    > Beware that both approaches might be considering stealing in some jurisditions. Yeah, most definitely, which is why you need to befriend them before asking, otherwise it's a guaranteed "No, we cannot do that".
anorangecat 2y ago

dncornholio 2y ago

[flagged]

rcarmo 2y ago

Another classic Foone story. I heartily recommend the follow on Mastodon, there's at least a gem a week, if not more. Also, worth noting that they are currently in need of some help, so consider supporting them: https://digipres.club/@foone/112929955279707608 https://digipres.club/@foone/112929955279707608

crispyambulance 2y ago

Yeah, those boxes in the loading docks of data centers... Lots of interesting stuff. Picked up a SFF Dell desktop from a huge pile of identical ones in a large cardboard box last spring. They had the good sense, however, to harvest the SSD and memory. Got replacements for a song and now I run home assistant on that thing. There's also sometimes "old iron" in the dock. Sun servers from back in the day. Beautiful hardware but not something one would ever want to take home.

pbhjpbhj 2y ago
Harvesting the memory, that's a bit (ha!) extreme. If the box is powered down then it's there any evidence residual memory can be imaged?
- vetinari 2y ago
  More like RAM can be sold separately. The computer itself would sell for the same amount whether with ram or not.

deleted 2y ago

[deleted]

flumpcakes 2y ago

Charity subsidising the broken? US healthcare industry.

rcarmo 2y ago
So you'd rather we not help someone in need? Because your comment is _very_ insensitive in a personal context.
- flumpcakes 2y ago
  I don't think it was insensitive, just a fact. Why do I have to read begging messages on a tech forum for a citizen of the world's biggest and richest economy? Maybe that's insensitive to the 4bn other people on the planet living in poverty with no chance of access to healthcare, begging or not.

Ekaros 2y ago

I feel justified on hammering the decommissioning process now. Just pull the disk and destroy it...

ajb 2y ago

I guess insolvency law needs to be changed to place some duty of care towards data subjects on insolvency practitioners, rather than allowing everything to be flogged off to the highest bidder. Since winding up user data storage safely is not free, this probably means data processors[1] should be required to get insolvency insurance to cover it. [1] "data processor" in the sense of the various user data protection laws, not just any data processor

raxxorraxor 2y ago

My work NUC just shuts down if I start any AI process, be that running an LLM or stable diffusion. Pooof, system off. Temperature doesn't matter. It also doesn't boot when I activate all processor cores. I have to leave one of those unused. Although that may be due some "valuable" feature like secure boot or bitlocker. I heard NUC now goes to ASUS, perhaps the devices can improve. There are quite a few problems here. I mean I expected abysmal performance on any pure CPU AI task, but some of those could have run in the background. I wonder what Intel did with the time they were the dominant player on a lot of markets... The security here is just that the device is so bad, that no party could extract something useful. Seriously, these things...

susanhea 2y ago

[flagged]