lllllm

6 ms·

1.
by lllllm 1y ago
Swiss AI Initiative | https://www.swiss-ai.org/ | Hybrid/ONSITE (in Europe) We are a young team, and the creators of the Apertus LLM, the currently leading open-data open-weights AI model. Join us to work on cutting ed
2.
by lllllm 1y ago
yes this seems a good way to go. for example you can already find many quantized versions under https://huggingface.co/models?search=apertus%20mlx and elsewhere
3.
by lllllm 1y ago
thank you!
4.
by lllllm 1y ago
We hear you, nevertheless this is one of the very few open-weights and open-data LLMs, and the license is still very permissive (compare for example to Llama). Personally of course I'd like to remove the additional click, but the unive
5.
by lllllm 1y ago
The pretraining (so 99% of training) is fully global, in over 1000 languages without special weighting. The posttraining (See section 4 of the paper) had also as many languages as we could get, and did upweight some languages. The posttrain
6.
by lllllm 1y ago
common crawl anyway respects the CCbot opt-out every time they do a crawl. we went a step further because back in old ages (2013 is our oldest training data) LLMs did not exist, so website owners opting out today of AI crawlers might like t
7.
by lllllm 1y ago
martin here from the apertus team, happy to answer any questions if i can. the full collection of models is here: https://huggingface.co/collections/swiss-ai/apertus-llm-68b6... PS: you can run this locally on you
8.
by lllllm 1y ago
we compared to GPT-OSS-20B, Llama 4, Qwen 3, among many others. Which models do you think are missing, among open weights and fully-open models? Note that we have a specific focus on multilinguality (over 1000 languages supported), not only
9.
by lllllm 1y ago
we didn't have time to write one yet, but there is the tech report which has a lot of details already
10.
by lllllm 1y ago
posttraining codebase is here: https://github.com/swiss-ai/posttraining
11.
by lllllm 1y ago
we released 81 intermediate checkpoints of the whole pretraining phase, and the code and data to reproduce. so full audit is surely possible - still it would depend on what you consider 'practical' here.
12.
by lllllm 1y ago
benchmarks: we provide plenty in the over 100 page tech report here https://github.com/swiss-ai/apertus-tech-report/blob/main/Ap... quantizations: available now in MLX https://github.com/
13.
by lllllm 1y ago
this is what this paper tries to answer: https://arxiv.org/abs/2504.06219 the quality gap is surprisingly small between compliant and not
14.
by lllllm 1y ago
absolutely! i've sent you a linkedin message last week. but here seems to work much better, thanks a lot!
15.
by lllllm 1y ago
we kept all 1800+ (script/language) pairs, not only the quality filtered ones. the question if a mix of quality filtered and not languages impacts the mixing is still an open question. preliminary research (Section 4.2.7 of https:&#x2
16.
by lllllm 1y ago
no. the main source is fineweb2, but with additional filtering for compliance, toxicity removal, and quality filters such as fineweb2-hq
17.
by lllllm 1y ago
Yes this is an interesting question. In our arxiv paper [1] we did study this for news articles, and also removed duplicates of articles (decontamination). We did not observe an impact on the downstream accuracy of the LLM, in the case of n
18.
by lllllm 1y ago
No, the model has nothing do to with Llama. We are using our own architecture, and training from scratch. Llama also does not have open training data, and is non-compliant, in contrast to this model. Source: I'm part of the training te
19.
by lllllm 1y ago
animation of it: https://youtu.be/ewg36czOOiI?si=moL9g9Xz2-vVClZX
20.
Distributed Collaborative ML (and LLMs) in the Browser(github.com)
2 points by lllllm 2y ago|0 comments
21.
by lllllm 4y ago
it takes quadratically more time the larger your context is.
22.
by lllllm 4y ago
The current systems like chatGPT actually have just such two parts. One is the raw LLM as you describe. The second one is another network acting as a filter on top of the first one. To be more precise, that second part is the process of fin