xfalcox

6 ms·

1.
by xfalcox 18d ago
Yeah, DeepSeek 4 Flash on vLLM has been an adventure indeed. It finally stabilized for me on 2 x H200 using a commit a few days before 0.28, so this release should be good for you.
2.
by xfalcox 1mo ago
Have you tried running it on a single 5090? Dual 5090 require https://github.com/aikitoria/open-gpu-kernel-modules for higher perf. Are you using TP?
3.
by xfalcox 2mo ago
I'm wondering the same! How that article has no links is beyond me.
4.
by xfalcox 2mo ago
Question to the OP, have you tested this on a machine where the entire model and context fit in RAM ?
5.
by xfalcox 2mo ago
README covers that https://github.com/JustVugg/colibri#ssd-wear-warning
6.
by xfalcox 4mo ago
Given my dev machine has 32GB of RAM and 32GB of VRAM that sits mostly idle when I'm not running AI models, this is not that bad of an idea.
7.
by xfalcox 6mo ago
Comparing a model you can downloads weights for with an API-only model doesn't make much sense.
8.
by xfalcox 6mo ago
Our CEO did that at our company and found 33 CVEs. Rails also did that and found 7 or 8.
9.
by xfalcox 8mo ago
I just made a new installer for Discourse on CharmRuby, now I gotta check this out and see if porting is feasible. Hopefully this reduces the app size, that is quite large with CharmRuby
10.
by xfalcox 8mo ago
That is a great fit for the GIF integration in Discourse. I was able to quickly add support for it at https://github.com/discourse/discourse-gifs/pull/107 Love to see WEBP support. Do you plan on adding suppo
11.
by xfalcox 8mo ago
First time I was in San Francisco and someone introduced themselves like that, going even beyond, was indeed a super weird experience being a brazilian.
12.
by xfalcox 9mo ago
We have vLLM for running text LLMs in production. What is the equivalent for this model?
13.
by xfalcox 10mo ago
I am partial to https://huggingface.co/Qwen/Qwen3-Embedding-0.6B nowadays. Open weights, multilingual, 32k context.
14.
by xfalcox 10mo ago
It's the Amazon own model. I'm baffled someone would pick it, even more that someone would test Llama 4 for a task in an age where Sonnet 4.5 is already out, so in the last 45 days. Looks like they were limited by AWS Bedrock opti
15.
by xfalcox 11mo ago
> what does the rag for uploaded files do in discourse? You can upload files that will act as RAG files for an AI bot. The bot can also have access to forum content, plus the ability to run tools in our sandboxed JS environment, making i
16.
by xfalcox 11mo ago
We host thousands of forums but each one has its own database, which means we get a sort of free sharding of the data where each instance has less than a million topics on average. I can totally see that at a trillion scale for a single sh
17.
by xfalcox 11mo ago
I was taken back when I saw what was basically zero recall loss in the real world task of finding related topics, by doing the same thing you described where we over capture with binary embeddings, and only use the full (or half) precision
18.
by xfalcox 11mo ago
In Discourse embeddings power: - Related Topics, a list of topics to read next, which uses embeddings of the current topic as the key to search for similar ones - Suggesting tags and categories when composing a new topic - Augmented search
19.
by xfalcox 11mo ago
Also worth mentioning that we use quantization extensively: - halfvec (16bit float) for storage - bit (binary vectors) for indexes Which makes the storage cost and on-going performance good enough that we could enable this in all our hostin
20.
by xfalcox 11mo ago
> Nobody’s actually run this in production We do at Discourse, in thousands of databases, and it's leveraged in most of the billions of page views we serve. > Pre- vs. Post-Filtering (or: why you need to become a query planner ex
21.
by xfalcox 1y ago
Depends on your needs. You surely don't want 32k long chunks for doing the standard RAG pipeline, that's for sure. My use case is basically a recommendation engine, where retrieve a list of similar forum topics based on the curren
22.
by xfalcox 1y ago
Just migrated all embeddings to this same model a few weeks ago in my company, and it's a game changer. Having 32k context is a 64x increase when compared with our previous used model. Plus being natively multilingual and producing ver
23.
by xfalcox 1y ago
Having a public tokenizer is quite useful, specially for embeddings. It allows you to do the chunking locally without going to the internet.
24.
by xfalcox 1y ago
Qwen 3 is not slow by any metrics. Which model, inference software and hardware are you running it on? The 30BA3B variant flies on any GPU.
25.
by xfalcox 1y ago
You'd be surprised how often people in enterprise can be left waiting months to get an API key approved for an LLM provider.
26.
by xfalcox 1y ago
This looks like a great fit for allowing people to monetize their Discourse forums, by having partners stores and plugging those instead of ads. Will build a quick poc integration. How can I contact you with feedback?
27.
by xfalcox 1y ago
This looks super cool, exactly what I've been wanting to create some useful widgets! Thanks for sharing!
28.
by xfalcox 1y ago
I guess one aspect missing here is weighting more popular songs on that analysis. I assume that the analysis is simply counting every song chords, so a unknown band you've never heard about has the same impact as The Ramones. I'd
29.
by xfalcox 1y ago
No downloadable open weights ? Looks like I'll stay on [bge-m3]( https://huggingface.co/BAAI/bge-m3 )
30.
by xfalcox 1y ago
My wife runs a small retail makeup shop on Shopify, which started before pix and those surprise false fraud claims almost killed the business. Pix was such a game changer. It is perfect.

More ›