Dorialexander

7 ms·

1.
by Dorialexander 3mo ago
hi, so Pleias co-founder here. Common Corpus is commonly used now in pretraining, including by close labs, but rarely as the only source (which would be the actual ethical commitment). Like most people training efficient models we move towa
2.
by Dorialexander 7mo ago
Actually there is even a straight connection: Step-Fun DeepResearch trained on SYNTH (the open Baguettotron dataset).
3.
by Dorialexander 1y ago
Hi. Yes this is wholly correct. On the second points: * Well I'm very much involved in making open more models, pretrained the first model on free and open data without copyrigh issues, released the first version fo GRPO that can run o
4.
by Dorialexander 1y ago
Seeing LLM as a motor was a legitimate view until recently. But what we're start seeing with actual agentification is models taking the driver seat, making the call about search, tool use, API. Like DeepSearch, these models are likely
5.
by Dorialexander 1y ago
So to clarify: the important product that people will ultimately want is the model. Obviously you need to design an infra/UI around it but that's not the core product. The really important distinction is between workflow (what eve
6.
by Dorialexander 1y ago
Hi. So quickly: * RL is Reinforcement Learning. Already used for a while as part of RLHF but now we have started to find a very nice combo of reasoning+RL on verifiable tasks. Core idea is that models are not just good a predicting the next
7.
by Dorialexander 1y ago
Hi, author here. An important background is the imminent rise of actual LLM agents I discuss in the next post: https://vintagedata.org/blog/posts/designing-llm-agents So answering to a few comments: *The shift is
8.
by Dorialexander 3y ago
Ah it's completely volontary on my part: I want to keep the historical spelling as much a possible. That's why I used the google books OCR which does a better work at it than Gallica. That's still a bit erased in the current
9.
by Dorialexander 3y ago
Either that or appending archaic expressions in the prompts (a bit like the prompt extension of Midjourney)
10.
by Dorialexander 3y ago
Well you're not going to believe it but I do have a FoucaultGPT just being trained (indirectly: Foucault is just part of my extended French historical corpus). As a sample: Prompt : Écrit un livre de Michel Foucault sur les mesures de
11.
by Dorialexander 3y ago
I published the completely dataset here: https://huggingface.co/datasets/Pclanglais/MonadGPT While I don't think Saint-Simon is included, a French colleague did a few try with it that turned out better than C
12.
by Dorialexander 3y ago
In a way you could do so by prompting Monad with artificial intelligence stuff. I did a try lately on the latest OpenAI events and it went on like this: "In this sad and tragical storye, you shall heare how Sam Altman, a manne of great
13.
by Dorialexander 3y ago
Yes I needed that for the conversational/instructional capacities. I've made a lot of tests with base models and it would not listen to instruction very well…
14.
by Dorialexander 3y ago
Yes you're perfectly right. I've currently tried to maintain some kind of uneasy balance between good conversational capacities (so that it really is a "chatGPT") and cultural reset, which means it may revert from its 17
15.
by Dorialexander 3y ago
Not feasible to go with pretraining only. What is possible is to use a larger learning rate but this will be a hard trade-off with conversational capacities. Fine tuning is currently based on original texts with a synthetic prompt. The issu
16.
by Dorialexander 3y ago
Yes. I think we may have enough for "full finetuning" and erasing to a large extent the previous knowledge. But that's still very far off for pretraining. "RomeGPT" is next on my list of Monad successors and to give
17.
by Dorialexander 3y ago
Yes it happens once in a while. It's still a small model (7B) and I've done very weird things with it. If I were historically reconditioned in the 17th century mindset, I would also likely have strange lapses of insanity.
18.
by Dorialexander 3y ago
Already in the work. Just had a meeting today with two latinists about it.
19.
by Dorialexander 3y ago
Certainly. In fact I see we already follow each other on Twitter :D And yes totally. The other massive impact could be in source analysis. I have started using Mistral-Hermes for text annotation and it is both impressive and very fast.
20.
by Dorialexander 3y ago
The model can also be tried directly on this space: https://huggingface.co/spaces/Pclanglais/MonadGPT HuggingFace has generously provided free GPUs.
21.
by Dorialexander 3y ago
Hi! Model creator here. I happen to be a cultural historian and that's a main use case that I see. It's not complicated to learn about past events but having a general idea of the culture of the time (and its alieness from our per
22.
by Dorialexander 3y ago
Hi. TheBloke has quantized the model: https://huggingface.co/TheBloke/MonadGPT-GGUF You may be able to run the Q3 or Q4 variant. Although in my experience, the quality of quantization takes a hit on "weirder"