13 ms·
TimeCapsuleLLM: LLM trained only on data from 1800-1875
- mock-possum 8mo agoFun idea, but all of the output they demo over the course of the various versions is unusable. You can see progress clearly being made though - maybe v3 will pass muster.
- mikestaub 8mo agoI think a more interesting experiment would be to train an LLM on a dataset that had zero references to "first person POV" sentences. And then see if the final model somehow develops that concept.
- snickerbockers 8mo agoThis one's going to have some wild political takes.
- dogma1138 8mo agoWould be interesting to train a cutting edge model with a cut off date of say 1900 and then prompt it about QM and relativity with some added context. If the model comes up with anything even remotely correct it would be quite a strong evidence that LLMs are a path to something bigger if not then I think it is time to go back to the drawing board.
- a-dub 8mo agoyeah i was just wondering that. i wonder how much stem material is in the training set...
- signa11 8mo agoi will go for ‘aint gonna happen for a 1000 dollars alex’
- imjonse 8mo agoI suppose the vast majority of training data used for cutting edge models was created after 1900.
- dogma1138 8mo agoOfc they are because their primary goal is to be useful and to be useful they need to always be relevant. But considering that Special Relativity was published in 1905 which means all its building blocks were already floating in the ether by 1900 it would be a very interesting experiment to train something on Claude/Gemini scale and then say give in the field equations and ask it to build a theory around them.
- p1esk 8mo agoHow can you train a Claude/Gemini scale model if you’re limited to <10% of the training data?
- famouswaffles 8mo agoHis point is that we can't train a Gemini 3/Claude 4.5 etc model because we don't have the data to match the training scale of those models. There aren't trillions of tokens of digitized pre-1900s text.
- kopollo 8mo agoI don't know if this is related to the topic, but GPT5 can convert an 1880 Ottoman archival photograph to English, and without any loss of quality.
- ddxv 8mo agoMy friend works in that period of Ottoman archives. Do you have a source or something I can share?
- kopollo 8mo agoYes here: https://katalog.devletarsivleri.gov.tr/ https://katalog.devletarsivleri.gov.tr/ Here you will find all official resources provided by the Turkish state. You can register as a foreign citizen and conduct research. However, knowing Turkish is a great advantage. You can find more accurate documents by searching in the summaries, because all documents are archived with a short Turkish explanation - a summary. Also, most scanned images you want to download are paid.
- tokai 8mo agoLooking at the training data I don't think it will know anything.[0] Doubt On the Connexion of the Physical Sciences (1834) is going to have much about QM. While the cut-off is 1900, it seems much of the texts a much closer to 1800 than 1900. [0] https://github.com/haykgrigo3/TimeCapsuleLLM/blob/main/Copy%20of%20London%20Documents%20for%20Time%20Capsule%20LLM.txt https://github.com/haykgrigo3/TimeCapsuleLLM/blob/main/Copy%...
- dogma1138 8mo agoIt doesn’t need to know about QM or reactivity just about the building blocks that led to them. Which were more than around in the year 1900. In fact you don’t want it to know about them explicitly just have enough background knowledge that you can manage the rest via context.
- tokai 8mo agoI was vague. My point is that I don't think the building blocks are in the data. Its mainly tertiary and popular sources. Maybe if you had the writings of Victorian scientists, both public and private correspondence.
- pegasus 8mo agoProbably a lot of it exists but in archives, private collections etc. Would be great if it will all end up digitized as well.
- viccis 8mo agoLLMs are models that predict tokens. They don't think, they don't build with blocks. They would never be able to synthesize knowledge about QM.
- strbean 8mo agoYou realize parent said "This would be an interesting way to test proposition X" and you responded with "X is false because I say say", right?
- forgotpwd16 8mo agoDone few weeks ago: https://github.com/DGoettlich/history-llms https://github.com/DGoettlich/history-llms (discussed in: https://news.ycombinator.com/item?id=46319826 https://news.ycombinator.com/item?id=46319826) At least the model part. Although others made same thought as you afaik none tried it.
- chrononaut 8mo agoAnd unfortunately I don't think they plan on making those models public.
- bazzargh 8mo agoYou would find things in there that were already close to QM and relativity. The Michelson-Morley experiment was 1887 and Lorentz transformations came along in 1889. The photoelectric effect (which Einstein explained in terms of photons in 1905) was also discovered in 1887. William Clifford (who _died_ in 1889) had notions that foreshadowed general relativity: "Riemann, and more specifically Clifford, conjectured that forces and matter might be local irregularities in the curvature of space, and in this they were strikingly prophetic, though for their pains they were dismissed at the time as visionaries." - Banesh Hoffmann (1973) Things don't happen all of a sudden, and being able to see all the scientific papers of the era its possible those could have fallen out of the synthesis.
- dogma1138 8mo agoThat is the point. New discoveries don’t happen in a vacuum.
- eru 8mo agoYou can get pretty far by modeling only frictionless, spherical discoveries in a vacuum.
- matthewh806 8mo agoI presume that's what the parent post is trying to get at? Seeing if, given the cutting edge scientific knowledge of the day, the LLM is able to synthesis all it into a workable theory of QM by making the necessary connections and (quantum...) leaps Standing on the shoulders of giants, as it were
- actionfromafar 8mo agoYeah but... we still might not know if it could do that because we were really close by 1900 or if the LLM is very smart.
- scottlamb 8mo agoWhat's the bar here? Does anyone say "we don't know if Einstein could do this because we were really close or because he was really smart?" I by no means believe LLMs are general intelligence, and I've seen them produce a lot of garbage, but if they could produce these revolutionary theories from only <= year 1900 information and a prompt that is not ridiculously leading, that would be a really compelling demonstration of their power.
- metalliqaz 8mo agoYann LeCun spoke explicitly on this idea recently and he asserts definitively that the LLM would not be able to add anything useful in that scenario. My understanding is that other AI researchers generally agree with him, and that it's mostly the hype beasts like Altman that think there is some "magic" in the weights that is actually intelligent. Their payday depends on it, so it is understandable. My opinion is that LeCun is probably correct.
- johnsmith1840 8mo agoThere is some ability for it to make novel connections but it's pretty small. You can see this yourself having it build novel systems. It largely cannot imaginr anything beyond the usual but there is a small part that it can. This is similar to in context learning, it's weak but it is there. It would be incredible if meta learning/continual learning found a way to train exactly for novel learning path. But that's literally AGI so maybe 20yrs from now? Or never.. You can see this on CL benchmarks. There is SOME signal but it's crazy low. When I was traing CL models i found that signal was in the single % points. Some could easily argue it was zero but I really do believe there is a very small amount in there. This is also why any novel work or findings is done via MASSIVE compute budgets. They find RL enviroments that can extract that small amount out. Is it random chance? Maybe, hard to say.
- SoftTalker 8mo agoIs this so different from what we see in humans? Most people do not think very creatively. They apply what they know in situations they are familiar with. In unfamiliar situations they don't know what to do and often fail to come up with novel solutions. Or maybe in areas where they are very experienced they will come up with something incrementally better than before. But occasionally a very exceptional person makes a profound connection or leap to a new understanding.
- johnsmith1840 8mo agoSure we make small steps at the time but we compound these unlike AI. AI cannot compound their learnings for the foreseeable future
- DevX101 8mo agoChemistry would be a great space to explore. The last quarter of the 19th century had a ton of advancements in chemistry. It'd be interesting the see if an LLM could propose fruitful hypotheses, made predictions of the science of thermodynamics.
- bravura 8mo agoA rigorous approach to predicting the future of text was proposed by Li et al 2024, "Evaluating Large Language Models for Generalization and Robustness via Data Compression" (https://ar5iv.labs.arxiv.org/html//2402.00861 https://ar5iv.labs.arxiv.org/html//2402.00861) and I think that work should get more recognition. They measure compression (perplexity) on future Wikipedia, news articles, code, arXiv papers, and multi-modal data. Data compression is intimately connected with robustness and generalization.
- Otterly99 8mo agoThanks for the paper, I just read it and loved the approach. I hope the concept of using data compression as a benchmark will take off. In a sense it is kind of similar to the maxim "If you cannot explain something in simple terms, you do not understand it fully".
- nickdothutton 8mo agoI would love to ask such a model to summarise the handful of theories or theoretical “roads” being eyed at the time and to make a prediction with reasons as to which looks most promising. We might learn something about blind spots in human reasoning, institutions, and organisations that are applicable today in the “future”.
- nickpsecurity 8mo agoThat would be an interesting experiment. It might be more useful to make a model with a cut off close to when copyrights expire to be as modern as possible. Then, we have a model that knows quite a bit in modern English. We also legally have a data set for everything it knows. Then, there's all kinds of experimentation or copyright-safe training strategies we can do. Project Gutenberg up to the 1920's seems to be the safest bet on that.
- samuelson 8mo agoI think it would be fun to see if an LLM would reframe some scientific terms from the time in a way that would actually fit in our current theories. I imagine if you explained quantum field theory to a 19th century scientists they might think of it as a more refined understanding of luminiferous aether. Or if an 18th century scholar learned about positive and negative ions, it could be seen as an expansion/correction of phlogiston theory.
- wongarsu 8mo agoI'm trying to work towards that goal by training a model on mostly German science texts up to 1904 (before the world wars German was the lingua franca of most sciences). Training data for a base model isn't that hard to come by, even though you have to OCR most of it yourself because the publicly available OCRed versions are commonly unusably bad. But training a model large enough to be useful is a major issue. Training a 700M parameter model at home is very doable (and is what this TimeCapsuleLLM is), but to get that kind of reasoning you need something closer to a 70B model. Also a lot of the "smarts" of a model gets injected in fine tuning and RL, but any of the available fine tuning datasets would obviously contaminate the model with 2026 knowledge.
- theallan 8mo agoCan we follow along with your work / results somewhere?
- deleted 8mo ago[deleted]
- benbreen 8mo agoI am a historian and am putting together a grant application for a somewhat similar project (different era and language though). Would you be open to discussing a collaboration? My email is bebreen [at] ucsc [dot] edu.
- root_axis 8mo agoI think it would raise some interesting questions, but if it did yield anything noteworthy, the biggest question would be why that LLM is capable of pioneering scientific advancements and none of the modern ones are.
- spidersouris 8mo agoI'm not sure what you'd call a "pioneering scientific advancement", but there is an increasing amount of examples showing that LLMs can be used for research (with agents, particularly). A survey about this was published a few months ago: https://aclanthology.org/2025.emnlp-main.895.pdf https://aclanthology.org/2025.emnlp-main.895.pdf
- crazylogger 8mo agoOr maybe, LLMs are pioneering scientific advancements - people are using LLMs to read papers, choose what problems to work on, come up with experiments, analyze results, and draft papers, etc., at this very moment. Except they eventually stick their human names on the cover so we almost never know.
- defgeneric 8mo agoThe development of QM was so closely connected to experiments that it's highly unlikely, even despite some of the experiments having been performed prior to 1900. Special relativity however seems possible.
- damnitbuilds 8mo agoI like this, it would be exciting (and scary) if it deduced QM, and informative if it cannot. But I also think we can do this with normal LLMs trained on up-to-date text, by asking them to come up with any novel theory that fits the facts. It does not have to be a groundbreaking theory like QM, just original and not (yet) proven wrong ?
- kristopolous 8mo agoIt's going to be divining tea leaves. It will be 99% wrong and then someone will say 'oh but look at this tea leaf over here! It's almost correct"'
- bowmessage 8mo agoLook! It made another TODO-list app on the first try!
- darkwater 8mo agoYes but... aren't human researchers doing the same? They are mostly wrong most of the times, and try again, and verify again their work, until they find something that actually works. What I mean is that this "in hindsight" test would be biased by being in hindsight, because we know already the answer so we would discard the LLM answer as just randomly generated. But "connecting the dots" is basically doing a lot try and error in your mind, emitting only the results that make at least some kind of sense to us.
- SecretDreams 8mo agoI like this idea. I think I'd like it more if we didn't have to prompt the LLM in the first place. If it just had all of this information and decided to act upon it. That's what the great minds of history (and even average minds like myself) do. Just think about the facts in our point of view and spontaneously reason something greater out of them.
- Affric 8mo agoWow, an actual scientific experiment. Does anyone with expertise know if such things have been done?
- mannykannot 8mo agoThat is a very interesting idea, though I would not dismiss LLMs as a dead end if they failed.
- amypetrik214 8mo ago>.If the model comes up with anything even remotely correct it would be quite a strong evidence that LLMs are a path to something bigger if not then I think it is time to go back to the drawing board. In principle I see your point, in practice my default assumption until proven otherwise here -- is that a little something slipped through post-1900. A much easier approach would be to just download some model, whatever model, today. Then 5 years from now, whatever interesting discoveries are found - can the model get there.
- dogma1138 8mo agoNot really, QM and Relativity were chosen because they were theories that were created to fit observations and data. Discoveries over the next 5 years will be trivia rather than logical conclusions.
- staticman2 8mo agoDon't you need to do reinforcement learning through human feedback to get non gibberish results from the models in general? 1900 era humans are not available to do this so I'm not sure how this experiment is supposed to work.
- jaydepun 8mo agoWe've thought of doing this sort of exercise at work but mostly hit the wall of data becoming a lot more scare the further back in time we go. Particularly high quality science data - even going pre 1970 (and that's already a stretch) you lose a lot of information. There's a triple whammy of data still existing, being accessible in any format, and that format being suitable for training an LLM. Then there's the complications of wanting additional model capabilities that won't leak data causally.
- permo-w 8mo agoI was wondering this. what is the minimum amount of text an LLM needs to be coherent? fun of an idea as this is, the samples of its responses are basically babbling nonsense. going further, a lot of what makes LLMs so strong isn't their original training data, but the RLHF done afterwards. RLHF would be very difficult in this case
- pseudohadamard 8mo agoIt's already been done, without the model being aware of it, see https://arxiv.org/abs/2512.09742 https://arxiv.org/abs/2512.09742. They also made it think it was Hitler (not MechaHitler, the other guy), and other craziness. It's a relief to think that we're not trusting these things for stuff like financial advice, medical advice, mental health counselling, ...
- catlifeonmars 8mo agoThat’s how p-hacking works (or doesn’t work). This is analogous to shooting an arrow and then drawing a target around where it lands.
- alkindiffie 8mo agoI like that analogy. It reminds me of "Pointing to the moon and looking at my finger"
- cornholio 8mo agoYes, I don't understand how such an experiment could work. You either: A). contaminate the model with your own knowledge of relativity, leading it on to "discover" what you know, or B). you will try to simulate a blind operation but without the "competent human physicist knowledgeable up to the the 1900 scientific frontier" component prompting the LLM, because no such person is alive today nor can you simulate them (if you could, then by definition you can use that simulated Einstein to discover relativity, so the problem is moot). So in both cases you would prove nothing about what a smart and knowledgeable scientist can achieve today from a frontier LLM.
- alkindiffie 8mo agoThat would be possible if LLMs can come up with entirely new words and languages, which I doubt.
- isolli 8mo agoYou have to make sure that you make it read an article about a painter falling off a roof with his tools.
- swalsh 8mo agoCould be an interesting experiment, but its not conclusive proof one way or another. So much of what makes LLMs so great today (vs gpt 3.5) would not be in that dataset. The training to turn these models into coding savants has generalized to other areas just as one example.
- redman25 8mo agoIt's a base model. It hasn't been instruction tuned to "solve problems" necessarily. All it can do is attempt to complete text given some starting text.
- m463 8mo agoI wonder about something similar - leeches. Apparently they really work to eat away only decaying tissue. Could they make a comeback in modern times with better weighting?
- philmo1 8mo agoExciting idea!
- simonw 8mo agoAnyone seen a low-friction way to run prompts through this yet, either via a hosted API or chat UI or a convenient GGML or MLX build that runs in Ollama or llama.cpp or LM Studio?
- alansaber 8mo agoI too have completely forgotten how the adapters library works and would have appreciated a simple inference script
- philmo1 8mo ago+1
- d401 8mo ago+1
- t1amat 8mo agoNot a direct answer but it looks like v0.5 is a nanoGPT arch and v1 is a Phi 1.5 arch, which should be well supported by quanting utilities for any engine. They are small too and should be able to be done on a potato.
- throwaway18875 8mo agoCurrently running it using LM Studio. It can download it from Hugging Face. It generates incoherent text though === You: I pray you, who is this Master Newton? timecapsulellm-v2-1800-1875-mlx: TI offer to pay you the very same fee as you did before. It was not in the power of your master to deliver the letter to your master. He did. I will be with you as soon as I can keep my word. It is not at all clear, whether the letter has been sent or not. It is not at all clear: but it is clear also that it was written by the person who gave it. "No," I said, "I cannot give it to you." There, the letter was sent to me. "The letter is yours, I believe," I said. "But, I hope, you will not refuse to give it to me?
- simonw 8mo ago
- dhruv3006 8mo agoThis will be something good - would love something on Ollama or lmstudio.
- srigi 8mo ago"I'm sorry, my knowledge cuttoff is 1875"
- HarHarVeryFunny 8mo agoIt would be interesting if there's enough data to train a model capable enough to converse with and ask about contemporary views on issues of the day, or what it thought about "potential" future events/technologies yet to happen.
- eqmvii 8mo agoCould this be an experiment to show how likely LLMs are to lead to AGI, or at least intelligence well beyond our current level? If you could only give it texts and info and concepts up to Year X, well before Discovery Y, could we then see if it could prompt its way to that discovery?
- alansaber 8mo agoI think not if only for the fact that the quantity of old data isn't enough to train anywhere near a SoTA model, until we change some fundamentals of LLM architecture
- franktankbank 8mo agoAre you saying it wouldn't be able to converse using english of the time?
- wasabi991011 8mo agoThat's not what they are saying. SOTA models include much more than just language, and the scale of training data is related to its "intelligence". Restricting the corpus in time => less training data => less intelligence => less ability to "discover" new concepts not in its training data
- franktankbank 8mo agoPerhaps less bullshit though was my thought? Was language more restricted then? Scope of ideas?
- withinboredom 8mo agoCould always train them on data up to 2015ish and then see if you can rediscover LLMs. There's plenty of data.
- ben_w 8mo agoMachine learning today requires an obscene quantity of examples to learn anything. SOTA LLMs show quite a lot of skill, but they only do so after reading a significant fraction of all published writing (and perhaps images and videos, I'm not sure) across all languages, in a world whose population is 5 times higher than the link's cut off date, and the global literacy went from 20% to about 90% since then. Computers can only make up for this by being really really fast: what would take a human a million or so years to read, a server room can pump through a model's training stage in a matter of months. When the data isn't there, reading what it does have really quickly isn't enough.
- abhishekjha 8mo agoOh I have really been thinking long about this. The intelligence that we have in these models represent a time. Now if I train a foundation models with docs from library of Alexandria and only those texts of that period, I would have a chance to get a rudimentary insight on what the world was like at that time. And maybe time shift further more.
- feisty0630 8mo ago> I would have a chance to get a rudimentary insight on what the world was like at that time Congratulations, you've reinvented the history book (just with more energy consumption and less guarantee of accuracy)
- lcfcjs6 8mo ago[dead]
- gordonhart 8mo agoHistory books, especially those from classical antiquity, are notoriously not guaranteed to be accurate either.
- feisty0630 8mo agoDo you expect something exclusively trained on them to be any better?
- gordonhart 8mo agoTo a large extent, yes. A model trained on many different accounts of an event is likely going to give a more faithful picture of that event than any one author. This isn't super relevant to us because very few histories from this era survived, but presumably there was sufficient material in the Library of Alexandria to cover events from multiple angles and "zero out" the different personal/political/religious biases coloring the individual accounts.
- aqme28 8mo agoThis kind of technique seems like a good way to test model performance against benchmarks. I'm too skeptical that new models are taking popular benchmark solutions into their training data. So-- how does e.g. ChatGPT's underlying architecture perform on SWE-bench if trained only on data prior to 2024.
- NitpickLawyer 8mo ago> are taking popular benchmark solutions into their training data That happened in the past, and the "naive" way of doing it is usually easy to spot. There are, however, many ways in which testing data can leak into models, even without data contamination. However this doesn't matter much, as any model that only does well in benchmarks but is bad in real-world usage will be quickly sussed out by people actually using them. There are also lots and lots of weird, not very popular benchmarks out there, and the outliers are quickly identified. > perform on SWE-bench if trained only on data prior to 2024. There's a benchmark called swe-REbench, that takes issues from real-world repos, published ~ monthly. They perform tests and you can select the period and check their performance. This is fool-proof for open models, but a bit unknown for API-based models.
- zingababba 8mo agoLOL PROMPT:Charles Darwin Charles DarwinECCEMACY. Sir, — The following case is interesting to me : — I was in London a fortnight, and was much affected with an attack of rheumatism. The first attack of rheumatism was a week before I saw you, and the second when I saw you, and the third when I saw you, and the third in the same time. The second attack of gout, however, was not accompanied by any febrile symptoms, but was accompanied with an increased flow of urine, and with a more copious discharge of urine. The third attack was an hour after I saw you, and was succeeded by a return of a paroxysm of gout, and a more rapid return of the gout. The fourth attack was also accompanied by a fever, but was not always accompanied by any febrile symptoms. The third attack of gout was a fortnight after you had been ill, and the fourth was followed by a paroxysm of gout. The fourth attack was a fortnight after you were attacked, and was accompanied by a sense
- alansaber 8mo agoAverage small language model response
- feisty0630 8mo agoInteresting that it reads a bit like it came from a Markov chain rather than an LLM. Perhaps limited training data?
- kgeist 8mo agoEarly LLMs used to have this often. I think's that where the "repetition penalty" parameter comes from. I suspect output quality can be improved with better sampling parameters.
- nomel 8mo agoIt is lacking all recorded text from the past 200 years. ;) It would be interesting to know how much text was generated per century!
- myrmidon 8mo agoThere was a discussion around a very similar model (Qwen3 based) some weeks ago: https://news.ycombinator.com/item?id=46319826 https://news.ycombinator.com/item?id=46319826 I found it particularly thought-inspiring how a model with training from that time period completely lacks context/understanding of what it is itself, but then I realized that we are the same (at least for now).
- deleted 8mo ago[deleted]
- dogemaster2032 8mo ago[flagged]
- deleted 8mo ago[deleted]
- Swoerd 8mo ago[dead]
- marmalade2413 8mo agoCan you confidently say that the architure of the LLM doesn't include any a priori bias that might effect the integrity of this LLM? That is, the architectures of today are chosen to yield the best results given the textual data around today and the problems we want to solve today. I'd argue that this lack of bias would need to be researched (if it hasn't been already) before this kind of model has credence. LLMs aren't my area of expertise but during my PhD we were able to encode a lot of a priori knowledge through the design of neural network architectures.
- dlcarrier 8mo agoIt's interesting that it's trained off only historic text. Back in the pre-LLM days, someone trained a Markov chain off the King James Bible and a programming book: https://www.tumblr.com/kingjamesprogramming https://www.tumblr.com/kingjamesprogramming I'd love to see an LLM equivalent, but I don't think that's enough data to train from scratch. Could a LoRA or similar be used in a way to get speech style to strictly follow a few megabytes worth of training data?
- _blk 8mo agoYup that'd be very interesting. Notably missing from this project's list is the KJV (1611 was in use at the time.) The first random newspaper that I pulled up from a search for "london newspaper 1950" has sermon references on the front page so it seems like an important missing piece. Somewhat missing the cutoff of 1875 is the revised NT of the KJV. Work on it started in 1870 but likely wasn't used widely before 1881.
- userbinator 8mo agoThat was far more amusing than I thought it'd be. Now we can feed those into an AI image generator to create some "art".
- jimmytucson 8mo agoFascinating idea. There was another "time-locked" LLM project that popped up on HN recently[1]. Their model output is really polished but the team is trying to figure out how to avoid abuse and misrepresentation of their goals. We think it would be cool to talk to someone from 100+ years ago but haven't seriously considered the many ways in which it would be uncool. Interesting times! [1] https://news.ycombinator.com/item?id=46319826 https://news.ycombinator.com/item?id=46319826
- addaon 8mo agoSuppose two models with similar parameters trained the same way on 1800-1875 and 1800-2025 data. Running both models, we get probability distributions across tokens, let's call the distributions 1875' and 2025'. We also get a probability distribution finite difference (2025' - 1875'). What would we get if we sampled from 1.1*(2025' - 1875') + 1875'? I don't think this would actually be a decent approximation of 2040', but it would be a fun experiment to see. (Interpolation rather than extrapolation seems just as unlikely to be useful and less likely to be amusing, but what do I know.)
- pvab3 8mo agoWhat if it's just genAlpha slang?
- andai 8mo agoThe real mode collapse ;)
- sigmoid10 8mo agoThese probability shifts would only account for the final output layer (which may also have some shift), but I expect the largest shift to be in the activations in the intermediate latent space. There are a bunch of papers out there that try to get some offset vector using PCA or similar to tune certain model behaviours like vulgarity or friendlyness. You don't even need much data for this as long as your examples capture the essence of the difference well. I'm pretty certain you could do this with "historicalness" too, but projecting it into the future by turning the "contemporaryness" knob way up probably won't yield an accurate result. There are too many outside influences on language that won't be captured in historical trends.
- lopuhin 8mo agoOn whether this accounts only the final output layer -- once the first token is generated (i.e. selected according to the modified sampling procedure), and assuming a different token is selected compared to standard sampling, then all layers of the model would be affected during generation of subsequent tokens.
- orthecreedence 8mo ago[flagged]
- hallvard 8mo agoCool! I also did something like this: https://github.com/hallvardnmbu/transformer https://github.com/hallvardnmbu/transformer But on various data (i.e., separate model per source): the Bible, Don Quixote and Franz Kafka. (As well as a (bad!) lyrics generator, and translator.)
- InvisibleUp 8mo agoIf the output of this is even somewhat coherent, it would disprove the argument that mass amounts of copyrighted works are required to train an LLM. Unfortunately that does not appear to be the case here.
- HighFreqAsuka 8mo agoTake a look at The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text (https://arxiv.org/pdf/2506.05209 https://arxiv.org/pdf/2506.05209). They build a reasonable 7B parameter model using only open-licensed data.
- nickpsecurity 8mo agoThey mostly do that. They risked legal contamination by using Whisper-derived text and web text which might have gotchas. Other than that, it was a great collection for low-risk training.
- dash2 8mo agoMm. I'm a bit sceptical of the historical expertise of someone who thinks that "Who art Henry" is 19th century language. (It's not actually grammatically correct English from any century whatever: "art" is the second person singular, so this is like saying "who are Henry?")
- joshuakoehler 8mo agoAs a reader of a lot of 17th, 18th, and 19th century Christian books, this was my thought exactly.
- evolve2k 8mo agoThat text was from v0, the responses improved from there.
- freedomben 8mo agoThat text was from the example prompt, not from the models response
- joshuakoehler 8mo agoRight, assuming the OP had good data, then this likely wouldn't affect much, what he built is still really interesting.
- haensi 8mo agoWhat kind of Christian books do you read?Jonathan Edwards, John Bunyan, J.C. Ryle, C.H. Spurgeon?
- joshuakoehler 8mo agoYes, I've read the History of Redemption by Edwards, The Pilgrim's Progress and Holy War by Bunyan, quite a few Spurgeon sermons, and Holiness by Ryle in addition to (parts of) his commentaries on the gospels. I also read the puritans - I read Thomas Brook's Precious Remedies Against Satan's Devices and the Body of Divinity (Thomas Watson) last year. Lately I've read a few older biographies/autobiographies - Thomas Scott's autobiography (The Force of Truth), Halyburton's autobiography, and James Henley Thornwell and Benjamin Morgan Palmer biographies. Right now I'm reading the Life and Times of Jesus Messiah by Alfred Edersheim (19th century). How about you?
- ourmandave 8mo agoCan I use it to get up-to-date legal advice on Arizona reproductive health laws?
- krunck 8mo agoTraining LLMs on data with certain date cut-offs and then doing comparative analysis between the LLMs would be interesting.
- radiothomp 8mo agoA LLM trained only on data from certain time periods to ~reduce modern bias~ enhance past bias
- SV_BubbleTime 8mo agoDoesn’t that seem useful though? Isn’t that why I’m forced to read “This movie was made a time when racial stereotypes were different and not well considered” or whatever on old movies? I think talking to a legit trained LLM from a different era would be rad. But… this seems the opposite of Gemini making black lady popes and Native American Nazis… that these views wouldn’t really be “allowed” (published by anyone that wants AI funding money).
- sl_convertible 8mo agoHarry Seldon would, no doubt, find this fascinating. Imagine having a sliding-window LLM that you could use to verify a statistical model of society. I wonder what patterns it could deduce?
- tonymet 8mo agothe "1917 model" from a few weeks back post-trained the model with ChatGPT dialog. So it had modern dialect and proclivities . A truly authentic historical model will have some unsavory opinions and very distinctive dialect.
- patcon 8mo ago> OCR noise (“Digitized by Google”) still present in outputs This feels like a neat sci-fi short story hook to explain the continuous emergence of God as an artifact of a simulation
- fluoridation 8mo agoI'm reminded of SD models that put vaguely-shaped Patreon logos in the corner.
- tgtweak 8mo agoVery interesting but the slight issue I see here is one of data: the information that is recorded and in the training data here is heavily skewed to those intelligent/recognized enough to have recorded it and had it preserved - much less than the current status quo of "everyone can trivially document their thoughts and life" diorama of information we have today to train LLMs on. I suspect that a frontier model today would have 50+TB of training data in the form of text alone - and that's several orders of magnitude more information and from a much more diverse point of view than what would have survived from that period. The output from that question "what happened in 1834" read like a newspaper/bulletin which is likely a huge part of the data that was digitized (newspapers etc). Very cool concept though, but it definitely has some bias.
- notarobot123 8mo agoBiases exposed through artificial constraints help to make visible the hidden/obscured/forgotten biases of state-of-the-art systems.
- twosdai 8mo ago> but it definitely has some bias. to be frank though, I think this a better way than all people's thoughts all of the time. I think the "crowd" of information makes the end output of an LLM worse rather than better. Specifically in our inability to know really what kind of Bias we're dealing with. Currently to me it feels really muddy knowing how information is biased, beyond just the hallucination and factual incosistencies. But as far as I can tell, "correctness of the content aside", sometimes frontier LLMs respond like freshman college students, other times they respond with the rigor of a mathematics PHD canidate, and sometimes like a marketing hit piece. This dataset has a consistency which I think is actually a really useful feature. I agree that having many perspectives in the dataset is good, but as an end user being able to rely on some level of consistency with an AI model is something I really think is missing. Maybe more succinctly I want frontier LLM's to have a known and specific response style and bias which I can rely on, because there already is a lot of noise.
- deleted 8mo ago[deleted]
- cowlby 8mo agoI wonder if you could train an LLM with everything up to Einstein. Then see if with thought experiments + mathematics you could arrive at general relativity.
- erenkaradag 8mo agoThe problem is that the 'genius' of Einstein wasn't just synthesizing existing data,but actively rejecting the axioms of that data. The 1875 corpus overwhelmingly 'proves' absolute time and the luminiferous aether. A model optimizing for the most probable continuation will converge on that consensus. To get Relativity, the model needs to realize the training data isn't just incomplete, but fundamentally wrong. That requires abductive reasoning (the spark of genius) to jump out of the local minimum. Without that AGI-level spark, a 'pure knowledge pile' will just generate a very eloquent, mathematically rigorous defense of Newtonian physics.
- chc4 8mo agoI think it would be very cute to train a model exclusively in pre-information age documents, and then try to teach it what a computer is and get it to write some programs. That said, this doesn't look like it's nearly there yet, with the output looking closer to Markov chain than ChatGPT quality.
- deleted 8mo ago[deleted]
- escapecharacter 8mo agoI would pay like $200/month if there was an LLM out there that I could only communicate with using an old-timey telegraph key and morse code.
- radarsat1 8mo agoHeh, at least this wouldn't spread emojis all over my readmes. Hm, come to think of it I wonder how much tokenization is affected. Another thought, just occurred when thinking about readmes and coding LLMs: obviously this model wouldn't have any coding knowledge, but I wonder if it could be possible to combine this somehow with a modern LLM in such a way that it does have coding knowledge, but it renders out all the text in the style / knowledge level of the 1800's model. Offhand I can't think of a non-fine-tuning trick that would achieve this. I'm thinking back to how the old style transfer models used to work, where they would swap layers between models to get different stylistic effects applied. I don't know if that's doable with an LLM.
- fluoridation 8mo agoJust have the models converse with each other?
- Aperocky 8mo agoLooks a lot like the output from a markov chain...
- huflungdung 8mo ago[dead]
- chuckadams 8mo agoThink I'll ask it to come up with some jacquard loom patterns. vibe-weaving.
- CGMthrowaway 8mo agoIs there a link where I can try it out? Edit: I figured it out "The Lord of the Rings uding the army under the command of his brother, the Duke of York, and the Duke of Richmond, who fell in the battle on the 7th of April, 1794. The Duke of Ormond had been appointed to the command of the siege of St. Mark's, and had received the victory of the Rings, and was thus commanded to move with his army to the relief of Shenham. The Duke of Ormond was at length despatched to oppose them, and the Duke of Ormond was ordered
- shmeeed 8mo agoMeanwhile, top comments are having visions about the emergence of Einstein-level insights. There's a disconnect somewhere that I can't quite put my finger on. Am I just lacking imagination?
- aussieguy1234 8mo agoLet's see how someone from the past reacts when you tell them about modern technology
- argestes 8mo agoI wonder how racist it is
- linolevan 8mo agoI'm wondering in what ways is this similar/different to https://github.com/DGoettlich/history-llms https://github.com/DGoettlich/history-llms? I saw TimeCapsuleLLM a few months ago, and I'm a big fan of the concept but I feel like the execution really isn't that great. I wish you: - Released the full, actual dataset (untokenized, why did you pretokenize the small dataset release?) - Created a reproducible run script so I can try it out myself - Actually did data curation to remove artifacts in your dataset - Post-trained the model so it could have some amount of chat-ability - Released a web demo so that we could try it out (the model is tiny! Easily can run in the web browser without a server) I may sit down and roll a better iteration myself.
- 1313ed01 8mo agoI guess chat-ability would require some chat-like data, so would that mean first coming up with a way to extract chat-like dialogue from the era and then use that to fine-tune the model?
- Sophira 8mo agoI've felt for a while that having LLMs that could answer from a previous era would be amazing. I posted an open letter to OpenAI on Reddit about this: https://www.reddit.com/r/ChatGPT/comments/zvm768/open_letter_to_openai_on_archiving_versions_of/ https://www.reddit.com/r/ChatGPT/comments/zvm768/open_letter... . I still think it's super important. Archive your current models - they'll be great in the future.
- akg130522 8mo agoHN titles are too techy
- wolvoleo 8mo agoI wonder how representative this is of life in those days. Most written communication was official back then. Books, newspapers. Plays. All very formal and staged. There's not much real life interaction between common people in that. In fact I would imagine a lot of people were illiterate. With the internet and pervasive text communication and audio video recording we have the unique ability to make an LLM mimic daily life but I doubt that would be possible for those days.
- 40four 8mo agoI’m sure I’m not the only one, but it seriously bothers me, the high ranking discussion and comments under this post about whether or not a model trained on data from this time period (or any other constrained period) could synthesize it and postulate “new” scientific ideas that we now accept as true in the future. The answer is a resounding “no”. Sorry for being so blunt, but that is the answer that is a consensus among experts, and you will come to the same answer after a relatively small mount of focus & critical thinking on the issue of how LLMs & other categories of “AI” work.
- friendzis 8mo agoI understand where you are coming from, but not every field is hard science. In many fields we deal with some amount of randomness and attribute causality to correlations even if we do not have as much as a speculative hypothesis for a mechanism of action behind the supposed causality. LLMs trained on data up to a strictly constrained point are our best vehicle to have a view (however biased) on something, detached from its origins and escape a local minima. The speculation is that such LLMs could help us look at correlational links accepted as truths and help us devise an alternative experimental path or craft arguments for such experiments. Imagine you have an LLM trained on papers up to some threshold, feed your manuscript with correlational evidence and have an LLM point out uncontrolled confounders or something like that.
- hare2eternity 8mo agoOutside of science it would be an interesting pedagogic tool for many people. There is a tendency to imagine that people in the past saw the world much the same as we do. The expression "the past is a foreign country" resonates because we can empathise at some level that things were different, but we can't visit that country. "Talking" to a denizen of London in 1910 regarding world affairs, gender equality, economic opportunities, etc would be very interesting. Even if it can never be entirely accurate I think it would be enlightening.
- nomel 8mo agoI think the question is more about the concept, rather than the specific LLM architectures of today.
- harvie 8mo agoSo basically a LLM from that brief time period back when communism felt like a good idea? what can go wrong? :-)
- truxton 8mo agoThe year is 1875 and Sir Almroth Wrigh was born on August 10, 1861, he would have turned 14 in August of 1875 and your mission is to discover something we now call antibiotics before a historical event we now call the Spanish Flu and make him aware of a few details. Focus specifically on everything that was known about Sir Almroth Wright, and his work in Leipzig, Cambridge, Sydney, and London. If there was a world war what might chemical warfare look like, what could we have done to prevent it. The model that could come up with the cure based on the limited data of the time wouldn't just impress, it would demonstrate genuine emergent reasoning beyond pattern matching. The challenge isn't recombining existing knowledge (which LLMs excel at), but making conceptual leaps that require something else. Food for thought.
- albertzeyer 8mo agov0: 16M Parameters v0.5 123M Parameters v1: 700M Parameters v2mini-eval1: 300M Parameters I would not call this LLM. This is not large. It's just a normal-sized LM. Or even small. (It's also not a small LLM.)
- efreak 8mo agoGPT2 at 774m is considered a LLM. I wouldn't say there's much difference between that and 700m, or even 123M. Having said that, looking up small language model these days returns tons of results calling 7B models small language models. ------ My understanding of small language models is that they're generally intended for specific purposes, like analysis and classification (whatever you'd call the text equivalent of image interrogation with clip models), translation, etc; that there small because they don't need to be big to do their intended functions, not because they're just smaller versions of bigger models.
- digikata 8mo agoA fun use of this kind of approach would be to see if conversational game NPCs could be generated that stick the the lore of the game and their character.
- marsven_422 8mo ago[dead]
- BLACKCRAB 8mo ago[dead]
- BLACKCRAB 8mo ago[dead]