6 ms·

Laion Big Video Dataset

91 points by ks2048 21d ago

topwalktown 21d ago
"Overall, we attempted to download 130M videos and achieved a link success rate of approximately 60%, resulting in 80M successfully retrieved videos with a total duration of 10M hours." I am astonished that the success rate is so high. How Youtube didn't block them, I don't know. But I think that this URL list won't age well because youtube will very quickly block any researcher trying to download these videos themselves.
- ks2048 21d ago
  I believe they have the actual videos to download, "exclusively for academic and non-commercial research." and you have to request access, https://projects.laion.ai/bvd/bvd-raw-access.html https://projects.laion.ai/bvd/bvd-raw-access.html They say they used yt-dlp and "employ a residential proxy network". But, yeah, youtube seems to now more aggressively block yt-dlp.
  - kennyadam 21d ago
    Ha, is it just me or does a “residential proxy network” sound like a fancy way of saying “botnet”?
    - m00dy 21d ago
      Nope, they’re different things. Proxy networks like Proxybase [0] use open-source clients and ask for the user’s consent before allowing them to join the network. [0] https://proxybase.xyz https://proxybase.xyz
    - MallocVoidstar 21d ago
      Basically all of them are botnets, yes. They generally claim to have consent but I'm pretty sure 99% of it is "some app the user uses has it buried in a 200 page ToS"-style consent.
      - m00dy 21d ago
        No, they are not. Just ask any AI.
        MallocVoidstar 21d ago
        The companies say they have consent because saying "yeah we'll give you access to a botnet" is illegal. https://cloud.google.com/blog/topics/threat-intelligence/disrupting-largest-residential-proxy-network https://cloud.google.com/blog/topics/threat-intelligence/dis... >These SDKs, which are offered to developers across multiple mobile and desktop platforms, surreptitiously enroll user devices into the IPIDEA network. If I install an app with a big ToS and buried in there is "we use third-party monetization SDKs" and buried in there is "welcome to our botnet" they do not actually have my consent.
    - stymaar 21d ago
      It's the "legal version" of botnets, yes.
  - kevmo314 21d ago
    That’s a pretty penny on the bandwidth bill alone!
  - maybewhenthesun 21d ago
    > youtube seems to now more aggressively block yt-dlp. Another thing to thank AI for! /s
- ks2048 21d ago
  I was wondering if youtube blocked them a lot and they got a lot of videos from other sources - but no, you can see on this HF page that 93.1% of URls are youtube.com, https://huggingface.co/datasets/laion/BVD-URLs https://huggingface.co/datasets/laion/BVD-URLs
deleted 21d ago
[deleted]
vivzkestrel 21d ago
- can someone with expertise give us an overview of the architecture involved doing this - let us say you ran yt-dlp inside python aiohttp - surely your ll run a limit soon as your ip address will be flagged - what solutions do we have to auto rotate proxies in python - are there better, faster and more reliable ways to go about downloading a 100 million videos without getting your ip address blocked?
- faangguyindia 21d ago
  Google may supply cache boxes to ISPs which can be manipulated to let download any video you want
- earthnail 21d ago
  Use Tor to proxy. Every time you get a connection error, send a SIGHUP to the Tor daemon, which will tell the Tor daemon to establish a new connection, thereby giving you a new IP. You may not get as good results as a residential proxy, but these proxies are prohibitively expensive if you're doing this for your own personal or academic use, and the Tor method works surprisingly well.
  - vivzkestrel 21d ago
    - am not familiar with tor except for the fact that it uses something called onion with multiple hops and is very hard to trace making it privacy friendly - are we talking here about using tor programmatically? - what I am referring to on my end is how to go about programmatically downloading a 100 million videos one after the other on s3 - what kind of bot detection / evasion / ip address / proxy rotation mechanisms are needed - are you familiar with libraries in the pypi ecosystem capable of tackling some of these concerns? - is aiohttp / asyncio python enough for this paired with multiprocessing or do we have better options?
    - earthnail 21d ago
      - Use the python Library stem to control the Tor daemon and send sighup commands when necessary - yt dlp has builtin support for tor - write the retry logic using your favourite coding AI - spin up a cheap cloud instance and let it run
voidUpdate 21d ago
Damn, that's a lot of videos for them to contact the creators and ask for permission to use their content as machine learning training content. Unless of course, they didn't, and just went ahead with it anyway...
- lucrbvi 21d ago
  Under the EU's AI Act non-profits and universities can just bypass these constents to make non-commercial datasets btw
  - voidUpdate 21d ago
    Does that apply to everyone who doesn't live in the EU as well? And what happens when a major AI company uses this dataset as training data, ignoring consent as usual?
  - stymaar 21d ago
    But I've been told that the AI act was a terrible anti-innovation legislation made by clueless and corrupt bureaucrats. How is it possible that they made a sensible decision?
  - Someone 21d ago
    I don’t see the AI act giving those permissions. In fact, I don’t think the AI act applies at all here. https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=OJ:L_202401689#art_2 https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=OJ:...: “This Regulation does not apply to AI systems or AI models, including their output, specifically developed and put into service for the sole purpose of scientific research and development.” https://github.com/LAION-AI/BVD https://github.com/LAION-AI/BVD: “LAION-BVD is released exclusively for research purposes and not for commercial use”
techsage 20d ago
Curious how useful very large video datasets are compared with smaller, better curated ones. At some point, adding more videos probably helps less than improving the quality of the data.
deleted 20d ago
[deleted]

91 points by ks2048 21d ago

"Overall, we attempted to download 130M videos and achieved a link success rate of approximately 60%, resulting in 80M successfully retrieved videos with a total duration of 10M hours." I am astonished that the success rate is so high. How Youtube didn't block them, I don't know. But I think that this URL list won't age well because youtube will very quickly block any researcher trying to download these videos themselves.

ks2048 21d ago
I believe they have the actual videos to download, "exclusively for academic and non-commercial research." and you have to request access, https://projects.laion.ai/bvd/bvd-raw-access.html https://projects.laion.ai/bvd/bvd-raw-access.html They say they used yt-dlp and "employ a residential proxy network". But, yeah, youtube seems to now more aggressively block yt-dlp.
- kennyadam 21d ago
  Ha, is it just me or does a “residential proxy network” sound like a fancy way of saying “botnet”?
  - m00dy 21d ago
    Nope, they’re different things. Proxy networks like Proxybase [0] use open-source clients and ask for the user’s consent before allowing them to join the network. [0] https://proxybase.xyz https://proxybase.xyz
  - MallocVoidstar 21d ago
    Basically all of them are botnets, yes. They generally claim to have consent but I'm pretty sure 99% of it is "some app the user uses has it buried in a 200 page ToS"-style consent.
    - m00dy 21d ago
      No, they are not. Just ask any AI.
      - MallocVoidstar 21d ago
        The companies say they have consent because saying "yeah we'll give you access to a botnet" is illegal. https://cloud.google.com/blog/topics/threat-intelligence/disrupting-largest-residential-proxy-network https://cloud.google.com/blog/topics/threat-intelligence/dis... >These SDKs, which are offered to developers across multiple mobile and desktop platforms, surreptitiously enroll user devices into the IPIDEA network. If I install an app with a big ToS and buried in there is "we use third-party monetization SDKs" and buried in there is "welcome to our botnet" they do not actually have my consent.
  - stymaar 21d ago
    It's the "legal version" of botnets, yes.
- kevmo314 21d ago
  That’s a pretty penny on the bandwidth bill alone!
- maybewhenthesun 21d ago
  > youtube seems to now more aggressively block yt-dlp. Another thing to thank AI for! /s
ks2048 21d ago
I was wondering if youtube blocked them a lot and they got a lot of videos from other sources - but no, you can see on this HF page that 93.1% of URls are youtube.com, https://huggingface.co/datasets/laion/BVD-URLs https://huggingface.co/datasets/laion/BVD-URLs

deleted 21d ago

[deleted]

vivzkestrel 21d ago

- can someone with expertise give us an overview of the architecture involved doing this - let us say you ran yt-dlp inside python aiohttp - surely your ll run a limit soon as your ip address will be flagged - what solutions do we have to auto rotate proxies in python - are there better, faster and more reliable ways to go about downloading a 100 million videos without getting your ip address blocked?

faangguyindia 21d ago
Google may supply cache boxes to ISPs which can be manipulated to let download any video you want
earthnail 21d ago
Use Tor to proxy. Every time you get a connection error, send a SIGHUP to the Tor daemon, which will tell the Tor daemon to establish a new connection, thereby giving you a new IP. You may not get as good results as a residential proxy, but these proxies are prohibitively expensive if you're doing this for your own personal or academic use, and the Tor method works surprisingly well.
- vivzkestrel 21d ago
  - am not familiar with tor except for the fact that it uses something called onion with multiple hops and is very hard to trace making it privacy friendly - are we talking here about using tor programmatically? - what I am referring to on my end is how to go about programmatically downloading a 100 million videos one after the other on s3 - what kind of bot detection / evasion / ip address / proxy rotation mechanisms are needed - are you familiar with libraries in the pypi ecosystem capable of tackling some of these concerns? - is aiohttp / asyncio python enough for this paired with multiprocessing or do we have better options?
  - earthnail 21d ago
    - Use the python Library stem to control the Tor daemon and send sighup commands when necessary - yt dlp has builtin support for tor - write the retry logic using your favourite coding AI - spin up a cheap cloud instance and let it run

voidUpdate 21d ago

Damn, that's a lot of videos for them to contact the creators and ask for permission to use their content as machine learning training content. Unless of course, they didn't, and just went ahead with it anyway...

lucrbvi 21d ago
Under the EU's AI Act non-profits and universities can just bypass these constents to make non-commercial datasets btw
- voidUpdate 21d ago
  Does that apply to everyone who doesn't live in the EU as well? And what happens when a major AI company uses this dataset as training data, ignoring consent as usual?
- stymaar 21d ago
  But I've been told that the AI act was a terrible anti-innovation legislation made by clueless and corrupt bureaucrats. How is it possible that they made a sensible decision?
- Someone 21d ago
  I don’t see the AI act giving those permissions. In fact, I don’t think the AI act applies at all here. https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=OJ:L_202401689#art_2 https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=OJ:...: “This Regulation does not apply to AI systems or AI models, including their output, specifically developed and put into service for the sole purpose of scientific research and development.” https://github.com/LAION-AI/BVD https://github.com/LAION-AI/BVD: “LAION-BVD is released exclusively for research purposes and not for commercial use”

techsage 20d ago

Curious how useful very large video datasets are compared with smaller, better curated ones. At some point, adding more videos probably helps less than improving the quality of the data.

deleted 20d ago

[deleted]