9 ms·
ByteDance’s Bytespider is scraping at much higher rates than other platforms
- MaKey 2y agoSomehow the headline made me think of a parent with a TikTok account.
- skrebbel 2y agoMaybe the people running the crawler are also parenting tiktokers
- bilekas 2y agoLeave no bot behind!
- beAbU 2y agoAll of a sudden the title makes so much more sense. Thanks. Now I might read the article actually.
- smittywerben 2y agoTitle edited out the apostrophe from the headline, plus a few small differences shown in brackets. TikTok[’s] parent launched a [web] scraper [that’s] gobbling up [the] world’s [online] data 25x[-times] faster than OpenAI
- bilekas 2y ago> does not respect robots.txt research shows. It would be nice then for the investigators to help people with the identifying markers for such crawlers. Apart from a mention of darkvisitors, which it seems is a paid service to "Block agents who try to ignore your robots.txt" I'm not sure how much that could be trusted given their business model also.
- koolba 2y ago> The Bytespider bot, much like those of OpenAI and Anthropic, does not respect robots.txt, the research shows. Robots.txt is a line of code that publishers can put into a website that, while not legally binding in any way, is supposed to signal to scraper bots that they cannot take that website’s data. Does any of these scrapers uniquely and unambiguously identify themselves as a bot? Or are those days long over?
- spiderfarmer 2y agoThis one does and I blocked them categorically from all my domains.
- diggan 2y ago> Does any of these scrapers uniquely and unambiguously identify themselves as a bot? It seems like all of them do, yeah: https://github.com/eob/isai/blob/b9060db7dc1a7789b322b8c2838514fc4e596779/src/patterns.json https://github.com/eob/isai/blob/b9060db7dc1a7789b322b8c2838... Not sure if they're really "scrapers" though, if they're initiated by a user for a single webpage/website, more like "user-agents" in that case, unless it automatically fans out from there to get more content.
- deleted 2y ago[deleted]
- netdevnet 2y agoSome of the scrapers used by big companies do identify themselves as bots by using unique user agents. Of course, it does not mean that they don't have other bots running around without the bot user agent name. Whether those days are over or not will greatly depend on the outcome of the ongoing New York Times vs OpenAI lawsuit. If OpenAI wins, then it pretty much green lights all the other scrappers to feast upon the web
- jeroenhd 2y agoMost of the good ones will tag themselves in the user agent and follow robots.txt. The ones that don't are the ones people are trying to block the most. Sometimes Google or Bing go crazy and start scraping the same resource over and over again, but most scraping tools causing load peaks are the badly written/badly configured/malicious ones.
- benreesman 2y agoIndiscriminate scraping is a dick move. But if you’re going to do it, do it properly. I would have hung it off the Like button with an ungodly ZooKeeper ensemble and trained a GBDT on which parts of which URLs I could just obliterate with Proxygen. We’d have it all in about 4 days. Don’t ask me how I know. The second worse thing about the AI megacorps after being evil is being staffed by people who use Cursor. Edit: on the back of the valued feedback of a valued commenter I’d like to acknowledge that I made a sloppy mistake and have corrected in haste, making no excuses. It would be super great if the largest private institutions in history of the world took the care with give or take everything that I do with trolling on a forum.
- t_6_t 2y ago[flagged]
- trunch 2y agoOnly on HN could Lex Fridman's endorsement mean anything when it comes to an IDE
- benreesman 2y agoI remember when people said things like “X is endorsed by John Carmack”. But Engine John is just a guest on the real show. I didn’t hang onto the first fortune I made in this game, which in general is a real nuisance, but it has the silver lining that I’m still working and likely will be when Zuckerberg realizes that he hasn’t graduated a class of legitimate E5s since 2017.
- t_6_t 2y ago[flagged]
- benreesman 2y ago^ it’s considered polite to use an “Edit:” annotation when dramatically changing what someone replied to. Cursor is a great product, done by brilliant people. I very much doubt that they set out to dramatically amplify the byte-denominated output and performance-cycle clout of the exact group of people who want to generate their code but find emacs macros or protobuf or any of the more sophisticated zero-temperature codegen mechanisms too high a complexity bar to clear before blasting metric tonnage of generated slop into our collective lives. The target audience of Cursor is the last group of people you want jizzing on your codebase at scale while simultaneously capturing mindshare proportional to NVIDIA’s market capitalization. It’s somewhere between dorky and cringe when Jensen signs some girl’s breasts like he’s Mick Jagger or something. That vibe in your repository is my job security in ten years.
- wtk 2y agohttps://archive.ph/https://fortune.com/2024/10/03/bytedance-tiktok-bytespider-scraper-bot/ https://archive.ph/https://fortune.com/2024/10/03/bytedance-...
- neonate 2y agohttps://archive.md/btyIo https://archive.md/btyIo
- jl6 2y agoI have observed this bot requesting URLs that haven’t been live for over a decade, and to which no reference can now be found in search engines. I imagine there must be a private trade in URL lists.
- is_true 2y agomaybe they are using commoncrawl, webarchive, yandex as indexes?
- jefozabuss 2y agoIn addition to those it's also possible they just found a website that published a scraped list back then and got de-indexed for obvious spammy content. I would not be surprised if there are still some auto generated link directories left from the "golden ages" of blackhat.
- aaron695 2y ago[dead]
- Havoc 2y agoGoing to be hard to enforce anything against this if it’s happening across jurisdiction like this. I don’t see how copyright survives long term in this sort of context
- sct202 2y agoBytedance has physical presences in most major markets now for ad sales/support so there are measures that can be taken and money is flowing that could be halted if needed.
- jfoster 2y agoCopyright isn't required once any work can be created faster than you can snap your fingers. It was originally a way to motivate creation of artistic works, since they used to involve a lot of effort.
- diggan 2y ago> It was originally a way to motivate creation of artistic works, since they used to involve a lot of effort. So true, then abstract expressionism appeared and suddenly copyright wasn't a thing anymore.
- neilv 2y ago> Copyright isn't required once any work can be created faster than you can snap your fingers. Copyright isn't required if you use a tool built upon violating copyright? Breathing isn't required if someone strangles everyone to death. (Now we can all transcend breathing, in the new post-living higher plane of existence. Which surely is viable and great, and totally won't be abused to enrich the worst people, to the detriment of everyone else.)
- jfoster 2y agoThat's a very valid point if courts around the world are about to rule against every AI company in hundreds or thousands of court cases. Do you believe that is going to happen?
- deleted 2y ago
- deleted 2y ago[deleted]
- shellac 2y agoI'm pretty sure this bot has been operating for much longer than the article suggests (April this year), and truly is a pain. I work in academia and see a lot of ill considered web scraping by ML / AI researchers, but Bytespider is in a league of its own.
- sflefties 2y ago[dead]
- welder 2y agoSo what, who cares? Is this newsworthy? It's definitely not something to get upset about, web scraping is a normal part of the internet.
- OutOfHere 2y agoJust how would a scraper catch up with the internet if not by accelerating the rate? It is to be expected if the scraping is to succeed.
- buro9 2y agoAlso the Facebook hit scraper. Which does not respect robots.txt and definitely is just scraping. AS blocks are the only really effective tool now, there are many scrapers that do not even respect user agent
- wiredfool 2y agoFacebook and someone who’s using a Firefox UA are the big hitters for me today, each at a sustained 6 req per second for the last 24 hours on one site. Today is actually pretty good, there’s some real looking UA traffic in the top 10.
- 486sx33 2y agoIt’s unfortunate and kind of dystopian. We have an opportunity to properly archive all of the worlds online data and catalog it for very very low cost (historically), so that the future of our planet will have a much better reference point for the past. Instead of that, companies are sucking up as much crap as possible, and tokenizing it and then scrubbing it, and adding “safety” to it. Reality is always much stranger than fiction.
- Spivak 2y agoWe have billions of people, we can accomplish two maybe three things at a time. This is a valid use as any of that archived data. The part that sucks isn't that people are doing unusual things with it like training AI, but that copyright & capitalism make it so that everyone has to go get their own data themselves to the annoyance of web admins. The biggest technical hurdle to sharing the work among interested parties is the web only authenticates the pipe, not the content.
- jrochkind1 2y agoCommonCrawl tries to archive the web and share it openly so everyone doesn't have to scrape it themselves. "Our goal is to democratize the data so that everyone, not just big companies, can do high-quality research and analysis." Because they share it openly including with those doing AI, they wind up on "AI crawler" lists, which are increasingly used by blocking tools that just "use the AI list", by people who don't like AI, or, quite ironically, people who are trying to prevent the excess traffic that poorly mannered AI crawlers cause. (Common Crawl's crawler is well mannered, uses good user-agent, respects robots.txt including crawl-delay, etc) https://commoncrawl.org/ https://commoncrawl.org/
- anileated 2y agoCopyright & capitalism is the crucial part of how we have the technical foundation that got us ML and most of the material used for training it. Big tech companies that want to monetize it at scale would like us to not think about that (or any long term consequences that do not affect shareholder value beyond current management), of course. If anything, the problem with intellectual property law is that they feel it’s safe for them to ignore it when it comes to ordinary people’s work (good luck suing ClosedAI).
- Ironlikebike 2y agoIn my last job, we observed ByteDance scraping TBs of OS testing data using the restful API that our OSS community front-end was using to serve it's CI result to the OSS community. The scraping was so relentless it was causing performance problems. We were also worried they were going to cause us large network egress fees as well. We specifically locked down the API after that, and anyone who wanted to use results had to ask explicit permission and be granted access.
- jgrahamc 2y agoStuff like this is why Cloudflare launched the AI Audit feature and the ability to block "AI bots". We're about to launch a feature that'll enforce your robots.txt.
- andrethegiant 2y agoI’m working on a platform[1] (built on Cloudflare!) that lets devs deploy well-behaved crawlers by default, respecting robots.txt, 429s, etc. The hope is that we can introduce a centralized caching layer to alleviate network congestion from bot traffic. [1] https://crawlspace.dev https://crawlspace.dev
- zebomon 2y agoI love the sentiment, but the real issue is one of incentives and not ability. The problem crawlers have more than enough technical ability to minimize their impact. They just don't have a reason to care right now.
- notachatbot123 2y agoIt would be nice to share this tooling free and open-source so that anyone can protect themselves.
- deleted 2y ago[deleted]
- kgen 2y agoTo be honest, it's probably not enough to just block these scrapers if they are acting maliciously, people should just start serving generated content back to it and see how long it takes for them to catch on and fix the problem
- OuterVale 2y agoThis pops to mind: https://youtube.com/watch?v=Hi5sd3WEh0c https://youtube.com/watch?v=Hi5sd3WEh0c
- nubinetwork 2y ago> The China-based parent company of video app TikTok released its own web crawler or scraper bot, dubbed Bytespider, sometime in April Uh, no... bytespider has been around for a long time...
- sieabahlpark 2y ago[dead]