5 ms·

I also wouldn't be surprised if labs were starting to mix in a few pelican SVGs into their training data.

quantumHazer 1y ago
SVG benchmarking is a thing since GPT-4, so probably all major labs are overfitting on some dataset ov svg images for sure
diggan 1y ago
Even "accidentally" it makes sense that "SVGs of pelicans riding bikes" are now included into datasets used for training as it has spread as a wildfire on the internet, making it less useful as a simple benchmark. This is why I keep all my benchmarks private and don't share anything about them publicly, as soon as you write about them anywhere publicly they'll stop being useful in some months.
- toyg 1y ago
  > This is why I keep all my benchmarks private This is also why, if I were an artist or anyone commercially relying on creative output of any kind, I wouldn't be posting anything on the internet anymore, ever. The minute you make anything public, the engines will clone it to death and turn it into a commodity.
  - __mharrison__ 1y ago
    Somewhat defeats the purpose of being an artist, doesn't it?
    - toyg 1y ago
      Defeating the purpose of creating almost anything, really. AI is definitely breaking the whole "labor for money" architecture of our world.
    - diggan 1y ago
      Worth noting, in case people weren't aware, but bunch of people have different motivations for being artists, not everyone wants everything they do to be as widely shared as possible, some are happy playing/drawing/whatever for a small group of people, and not even making money off it.
    - zhengyi13 1y ago
      Eeeehhhh. Maybe the thing to do is provide public, physical exhibits of your art in search of patronage.
  - debugnik 1y ago
    That makes it so much harder to show art to people and market yourself though. I considered experimenting with web DRM for art sites/portfolios, on the assumption that scrappers won't bother with the analog loophole (and dedicated art-style cloners would hopefully be disappointed by the quality), but gave up because of limited compatible devices for the strongest DRM levels, and HDCP being broken on those levels anyway. If the DRM technique caught on it would take attackers, at most, a few bucks and hours once to bypass it, and I don't think users would truly understand that upfront.
simonw 1y ago
I'll believe they are doing that when one of the models draws me an SVG that actually looks like a pelican.
- __mharrison__ 1y ago
  Someone needs to craft a beautifully bike donned by a pelican, throw in some seo, and see how long it takes a model to replicate it. Simon probably wouldn't be happy about killing his multi-year evaluation metric though...
  - simonw 1y ago
    I would be delighted. My pelican on a bicycle benchmark is a long con. The goal is to finally get a good SVG of a pelican riding a bicycle, and if I can trick AI labs into investing significant effort in cheating on my benchmark then fine, that gets me my pelican!

SVG benchmarking is a thing since GPT-4, so probably all major labs are overfitting on some dataset ov svg images for sure

diggan 1y ago

Even "accidentally" it makes sense that "SVGs of pelicans riding bikes" are now included into datasets used for training as it has spread as a wildfire on the internet, making it less useful as a simple benchmark. This is why I keep all my benchmarks private and don't share anything about them publicly, as soon as you write about them anywhere publicly they'll stop being useful in some months.

toyg 1y ago
> This is why I keep all my benchmarks private This is also why, if I were an artist or anyone commercially relying on creative output of any kind, I wouldn't be posting anything on the internet anymore, ever. The minute you make anything public, the engines will clone it to death and turn it into a commodity.
- __mharrison__ 1y ago
  Somewhat defeats the purpose of being an artist, doesn't it?
  - toyg 1y ago
    Defeating the purpose of creating almost anything, really. AI is definitely breaking the whole "labor for money" architecture of our world.
  - diggan 1y ago
    Worth noting, in case people weren't aware, but bunch of people have different motivations for being artists, not everyone wants everything they do to be as widely shared as possible, some are happy playing/drawing/whatever for a small group of people, and not even making money off it.
  - zhengyi13 1y ago
    Eeeehhhh. Maybe the thing to do is provide public, physical exhibits of your art in search of patronage.
- debugnik 1y ago
  That makes it so much harder to show art to people and market yourself though. I considered experimenting with web DRM for art sites/portfolios, on the assumption that scrappers won't bother with the analog loophole (and dedicated art-style cloners would hopefully be disappointed by the quality), but gave up because of limited compatible devices for the strongest DRM levels, and HDCP being broken on those levels anyway. If the DRM technique caught on it would take attackers, at most, a few bucks and hours once to bypass it, and I don't think users would truly understand that upfront.

simonw 1y ago

I'll believe they are doing that when one of the models draws me an SVG that actually looks like a pelican.

__mharrison__ 1y ago
Someone needs to craft a beautifully bike donned by a pelican, throw in some seo, and see how long it takes a model to replicate it. Simon probably wouldn't be happy about killing his multi-year evaluation metric though...
- simonw 1y ago
  I would be delighted. My pelican on a bicycle benchmark is a long con. The goal is to finally get a good SVG of a pelican riding a bicycle, and if I can trick AI labs into investing significant effort in cheating on my benchmark then fine, that gets me my pelican!