6 ms·
Classifying all of the pdfs on the internet
- buildbot 2y agoI have 20-40TB (pre-dedup) of PDFs - 8TB is a lot but not even close to the total number of PDFs available.
- sporedro 2y agoJust wondering what do you collect? Is it mainly mirroring things like libgen? I have a decent collection of ebooks/pdfs/manga from reading. But I can’t imagine how large a 20TB library is.
- buildbot 2y agoNo torrents at all in this data, all publicly available/open access. Mostly scientific pdfs, and a good portion of those are scans not just text. So the actual text amount is probably pretty low compared to the total. But still, a lot more than 8TB of raw data out there. I bet the total number of PDFs is close to a petabyte if not more.
- tylerflick 2y ago> I bet the total number of PDFs is close to a petabyte if not more. That's a safe bet. I'v seen PDF's in the GBs from users treating it like a container format (which it is).
- Maxion 2y agoIt's probably tens of petabytes if not more, if you count PDFs that'd be private. Invoices, order confirmations, contracts. There's just so so much.
- reaperducer 2y agoJust wondering what do you collect? I can't speak for the OP, but you can buy optical media of old out-of-print magazines scanned as PDFs. I bought the entirety of Desert Magazine from 1937-1985. It arrived on something like 15 CD-ROMS. I drag-and-dropped the entire collection into iBooks, and read them when I'm on the train. (Yes, they're probably on archive.org for free, but this is far easier and more convenient, and I prefer to support publishers rather than undermine their efforts.)
- buildbot 2y agoYep, a good bit of them are from sources like this :)
- deleted 2y ago[deleted]
- mehulashah 2y agoCare to make it publicly available? Or is that not permitted on your dataset? Certainly, there’s a lot more PDFs out there than 8TB. I bet there’s a lot of redundancy in yours, but doesn’t dedup well because of all the images.
- buildbot 2y agoI think that would be legally iffy for the stuff like collections of old magazines that were purchased on CD/DVD and such :/
- qingcharles 2y agoI have >10TB of magazines I've collected so far, and I could probably source another 50TB if I had the time. I'm working on uploading them, but I've had too much on my plate lately: https://en.magazedia.wiki/ https://en.magazedia.wiki/ There is a significant issue with copyright, though. I'll remove anything with a valid DMCA, but 99.9% of the world's historical magazine issues are now in IP limbo as their ownership is probably unknown. Most of the other .1% aren't overly concerned as distribution is their goal and their main income is advertising, not sales.
- layer8 2y agoI would have expected categories for product brochures and product manuals.
- afh1 2y agoInteresting read, I did not know about Common Crawl. I feel like RTBF is kind of a lost battle these days with more and more crawlers for AI and whatnot. Once on the internet there is no way back, for better or for worse. This tangent aside, 8TB is really not a lot of data, it's just 8 consumer-grade 1TB hard drives. I find it hard to believe this is "the largest corpus of PDFs online", maybe the largest public one. Not sure how representative it is of "the whole internet".
- deweller 2y agoIs it possible that the 8 TB is just the extracted text?
- tokai 2y agoNo, the Safedocs dataset is unprocessed pdfs.
- Propelloni 2y agoDoesn't sound like a lot, but where I am now we routinely work on very large infrastructure projects and the plans, documents and stuff mostly come as PDF. We are talking of thousands of documents, often with thousands of pages, per project and even very big projects almost never break 20 GB. If you like, you could say, PDF are information dense, but data sparse. After all it is mostly white space ;)
- IggleSniggle 2y agoThey often aren't like you're describing, though. For example, pdfs with high res images embedded that are drafts of future book or pamphlets prints. These can be hundreds of Mbs for a single pdf with less than 100 pages, and are so common in marketing departments that it's hard to imagine that you could fit anywhere close to all the pdfs on 8TB.
- Propelloni 2y agoTrue, we get plenty of high-res pictures of film in PDF here and some of them are ridiculously large, easily approaching gigabyte sizes, like you said. But that's more a problem of the user creating the PDF than inherent to PDFs. A raw 36 megapixels (our fancy 4K displays are only 8.3 megapixels, for comparison) picture reproduction of an ISO 400 film takes only about 70 MB, which tells us that something went wrong in the transfer if a PDF containing 10 pages of them cracks 1 GB. So, yeah, there are these monsters that send even beefy computers thrashing. But in my experience something in the creation process went wrong and it is appallingly common for a trade where PDFs are the go-to transfer format (I'm looking at you AutoCAD users!) I'd guess that the archive is doing the same we do, reprocess them for sensible results and store them. I assume you think the archive does not and then I'd agree with you. One determined civil engineer with AutoCAD can fill 8 TB in a week ;)
- Thaxll 2y agoFirst you need a good PDF library :/
- llm_trw 2y agoBack in 2006 there were multiple 1tb collections of textbooks as torrents. I imagine the size and number has only grown since then.
- namrog84 2y agoThat was before hoarding and building questionable businesses around them became a thing. I remember it being really easy to find textbooks, solution manuals, and related pdf and other stuff as late as 2008 far easier than 6-8 years later. The main difference were sites like chegg and many other sites started slurping them up to resell in some way.
- loa_in_ 2y agoIt doesn't take away the torrents, no?
- CaptainFever 2y agoOne of my pet peeves is the way people use words like slurp, hoover, take, vaccuum, suck up, or steal, when in reality they mean copy. I mean if Chegg manages to sell something you can get for free, then all the more power to them lol. Though we could probably do more to educate the younger generation on the magic of torrents. Ignoring angry textbook publishers, of course.
- namrog84 2y agoIf they copy then do things to help take down the initial thing. They've done more than just copy. And that's what I believe many places did. You get your copy. Then send out fake dmca noticed or buy out places. Then sell your copy after everyone else's copies aren't available anymore. It's a very standard practice in all walks of life. You gain access to a device then improve security and fix issues so that others can't get in anymore. That's a bad person approach. The legal way is to pull up the proverbial ladder or legal loopholes behind you. Good, bad, or whatever. Tons of people and places do it. It's far more nefarious than just so innocently copying
- TuringNYC 2y agoIve been playing with https://www.aryn.ai/ https://www.aryn.ai/ for Partitioning. Curious if anyone has tried these tools for better data extraction from PDFs. Any other suggestions? (I'm a bit disappointed that most of the discussion is about estimating the size of PDFs on the internet, I'd love to hear more about different approaches to extracting better data from the PDFs.)
- dwynings 2y agohttps://www.sensible.so/ https://www.sensible.so/ Full disclosure: I'm an employee
- noleary 2y agoThis is a really cool idea, thanks for sharing. I don't have that much free time these days, but I was thinking of trying a similar-but-different project not too long ago. I wanted to make a bit of an open source tool to pull down useful time series data for the social sciences (e.g. time series of social media comments about grocery prices). Seems like LLMs have unlocked all kinds of new research angles that people aren't using yet. I may steal some of your good ideas if I ever get to work on that side project :)
- pxdm 2y agoMy first thought on seeing the PCA embeddings scatterplot was "I wonder what pdfs are at the centre of those two clusters?" The most typical pdfs on the internet.
- whistle650 2y agoInteresting read with lots of good detail, thank you. A comment: if you are balancing the classes when you do one vs all binary training, and then use the max probability for inference, your probabilities might not be calibrated well, which could be a problem. Do you correct the probabilities before taking the argmax?
- minimaxir 2y agoOne of the now-underdiscussed features of embeddings is that you can indeed use any existing statistical modeling techniques on them out of the box, and as a bonus avoid the common NLP preprocessing nuances and pitfalls (e.g. stemming) entirely. This post is a good example on why going straight to LLM embeddings for NLP is a pragmatic first step, especially for long documents.
- throw10920 2y agoYou can apply statistical techniques to the embeddings themselves? How does that work?
- mkl 2y agoYou can apply statistical techniques to anything you want. Embeddings are just vectors of numbers which capture some meaning, so statistical analysis of them will work fine.
- throw10920 2y agoDon't most statistical techniques rely on specific structure in the spaces containing the objects they operate on, in order to be useful?
- mkl 2y agoEmbeddings have structure, or they wouldn't be very useful. E.g. cosine similarity works because (many) embeddings are designed to support it.
- throw10920 2y agoOh, that should have been obvious. Thank you for explaining.
- guiomie 2y agoInteresting and fun article! I've been experimenting with various LLMs/GenAI solutions to extract tabular data from PDFs with underwhelming results. It seems like they are good at extracting strings of text and summarizing (e.g what was the total price? when was this printed?) but extracting reliably into a CSV has a decent margin of error.
- abhi_p 2y agoDisclosure: I'm an employee. Give the Aryn partitioning service a shot: https://www.aryn.ai/post/announcing-the-aryn-partitioning-service https://www.aryn.ai/post/announcing-the-aryn-partitioning-se... We recently released it and we've a few examples here: https://sycamore.readthedocs.io/en/stable/aryn_cloud/get_started.html https://sycamore.readthedocs.io/en/stable/aryn_cloud/get_sta... that show you how to turn the tabular data from the pdf into a pandas dataframe(which you can then turn into csv).
- snats 2y agoHi! Author here, I wasn't expecting this to be at the top of HN, AMA
- bprew 2y agoHi snats, great article. You mention the accuracy of the various techniques you used, could you explain more about how you calculated the accuracy? Were the pdfs already categorized? Thanks!
- snats 2y agohi! i used the average accuracy over the entire dataset made originally made by the llm
- dangoodmanUT 2y agoGreat post, I'm wondering if there are resources you'd suggest to learn this kind of analysis? I dug through the code and it seemed like a ton of things I'm not familiar with, probably a lot of techniques I don't know rather than python ofc.
- autokad 2y agowould be interesting to see if they tried LDA (latent direchelet allocation) topics
- ks2048 2y agoI don’t have 8TB laying around, but we can be a bit more clever.... In particular I cared about a specific column called url. I really care about the urls because they essentially tell us a lot more from a website than what meats the eye. I'm I correct that it is only only using the URL of the PDF to do classification? Maybe still useful, but that's quite a different story than "classifying all the pdfs".
- xattt 2y agoIt’s just classifying the URLs if that’s the case. The legwork to classify PDFs is already done, and the authorship of the article can go to anyone who can get a grant for a $400 NewEgg order for an 8TB drive.
- deleted 2y ago[deleted]
- gnewton77 2y agoDid some similar work with similar visualizations ~2009, on ~5.7M research articles (PDFs, private corpus) from scientific publishers Elsevier, Springer: Newton, G., A. Callahan & M. Dumontier. 2009. Semantic Journal Mapping for Search Visualization in a Large Scale Article Digital Library. Second Workshop on Very Large Digital Libraries at the European Conference on Digital Libraries (ECDL) 2009. https://lekythos.library.ucy.ac.cy/bitstream/handle/10797/14057/ECDL105.pdf?sequence=1&isAllowed=y https://lekythos.library.ucy.ac.cy/bitstream/handle/10797/14... I am the first author.
- Loughla 2y agoHow do you decide who is listed first? And does the ampersand symbolize something that the word and doesn't, or is that just citation style?
- j_bum 2y agoIn biomedical research or tangential fields, author order generally follows these guidelines: First author(s): the individual(s) who organized and conducted the study. Typically there is only a single first author, but nowadays there are often two first authors. This is because the amount of research required to generate “high impact” publications simply can’t be done by a single person. Typically, the first author is a Ph.D. student or lab scientist. Middle authors: Individuals that provide critical effort, help, feedback, or guidance for the study and publication of the research. Different fields/labs have varying stringencies for what is considered “middle authorship worthy”. In many labs, simply being present and helping with the research warrants authorship. In other labs, you need to contribute a lot of energy to the project to be included as an author. Senior author(s): The primary investigators (PI’s) or lead researchers that run the lab that conducted and published the study. The senior authors are typically the ones that acquire funding and oversee all aspects of the published research. PI’s have varying degrees of hands-on management. There is some variation in whether the central research question for a manuscript is developed by the first vs. the senior author, but usually it’s the senior author. Also, the first and senior authors typically write the manuscript and seek edits/feedback from middle authors. In other cases, there can be dedicated writers that write the manuscript, who sometimes do/don’t get middle authorship. A main takeaway is: the general outline I’ve provided above is not strictly adhered to. I’ll take some liberty to apply this outline to this article at hand: First Author: G. Newton (OP). The scientist who mostly likely conducted all of the data mining and analysis. He likely wrote the article as well. Middle Author: A. Callahan. It seems like this author was a grad student at the time the article was written. She likely performed essential work for the paper’s publication. This could’ve been: helping with the analysis, data mining, or ideation. Senior Author: M. Dumontier. A data science professor, now at Maastricht U. He’s a highly cited scientist! Lastly… if you check out the acknowledgements, you can see three additional names. These people likely helped with setting up compute access, editing, or general ideation. This is a cool manuscript! Hopefully this overview isn’t TMI and provides some insight into the biomedical/data science publication process.
- byteknight 2y agoThis seems like cool work but with a ton of "marketing hype speak" that immediately gets watered down by the first paragraph. Ordering of statements. 1. (Title) Classifying all of the pdfs on the internet 2. (First Paragraph) Well not all, but all the PDFs in Common Crawl 3. (First Image) Well not all of them, but 500k of them. I am not knocking the project, but while categorizing 500k PDFs is something we couldnt necessarily do well a few years ago, this is far from "The internet's PDFs".
- deleted 2y ago[deleted]
- 1-6 2y agoOverpromise with headline, underdeliver on details.
- schneehertz 2y agoMoreover, the classification was not done on 500,000 PDF files themselves, but rather on the metadata of those 500,000 PDFs.
- muratsu 2y agoI would have expected the finetuned model to perform much better. Would be curious to see the performance with other models
- mehulashah 2y agoClassification is just a start. Wondering if it's worth doing something more -- like turning all of the text into Markdown or HTML? Would anyone find that interesting?
- Treesrule14 2y agoThere are a lot of webcrawlers where the chief feature is turning the website into markdown, I don't quite understand what they are doing for me thats useful since I can just do something like `markdownify(my_html)` or whatever, all this to say is that I wouldn't find this useful, but also clearly people think this is a useful feature as part of an LLM pipeline.
- loa_in_ 2y agoYou don't want the footer or navigation in the output. Ideally you want the main content of the page, if it exists. How do you assign header level if they're only differentiated by CSS left-margin in a variety of units? How do you interpret documents that render properly but are hardly correct HTML?
- Treesrule14 2y agoThanks, I guess, none of that stuff seemed super useful to cut systematically, but I'm gonna run some tests.
- excalibur 2y ago> How would you classify all the pdfs in the internet? Definitely as 'hot dog' or 'not a hot dog'.
- josh-sematic 2y agoVery cool! At Airtrain we’ve also found embeddings can be very valuable for building classification models. If you’re looking to play around with a large amount of text and embeddings we actually recently deduped and embedded all of fineweb-edu (also mentioned in the article) and put the resulting dataset on Hugging Face: https://huggingface.co/datasets/airtrain-ai/fineweb-edu-fortified https://huggingface.co/datasets/airtrain-ai/fineweb-edu-fort...
- niels_bom 2y agoTypo: “meats the eye”
- Mindey 2y agoWhoever said there "internet," they fail to grasp how big the internet really is.
- sireat 2y agoNice work! You've taken multiple approaches similar to what I sometimes do at the national library, I've used all kind of embeddings -> classifiers / LDA. Curious on your prompt: https://github.com/snat-s/m/blob/main/classify_metadata/prompt.txt https://github.com/snat-s/m/blob/main/classify_metadata/prom... Wouldn't this be basically prompting to classify by the type of URL?