mmoskal

5 ms·

1.
by mmoskal 2mo ago
Also 425 RMB is about $59 so $1 of tokens for $0.017 not $0.13 (the discount rate quoted also seems off).
2.
by mmoskal 2mo ago
https://github.com/openai/cdc-lean
3.
by mmoskal 4mo ago
I believe the App Store policy is you have to have a setting to disable ads. And Uber actually has it (though it has 8 different channels or so, apparently "Uber teen accounts" marketing was added recently). I used the setting and
4.
by mmoskal 4mo ago
If you want something with very small pcb footprint and DIY cheap cable try Hack connect https://arcade.makecode.com/hardware/dbg
5.
by mmoskal 10mo ago
Grammars work best when aligned with prompt. That is, if your prompt gives you the right format of answer 80% of the time, the grammar will take you to a 100%. If it gives you the right answer 1% of the time, the grammar will give you synta
6.
by mmoskal 10mo ago
OpenAI is using [0] LLGuidance [1]. You need to set strict:true in your request for schema validation to kick in though. [0] https://platform.openai.com/docs/guides/function-calling#lar... [1] https://g
7.
by mmoskal 11mo ago
I had good experience with carefully spaced holes in PCB and a 50 mil header, see https://jacdac.github.io/jacdac-docs/ddk/firmware/jac-connec...
8.
by mmoskal 1y ago
The previous article is in the same issue, in science and technology section. This is how they typically do it - leader article has a longer version in the paper. Leaders tend to be more opinionated.
9.
by mmoskal 1y ago
Consciousness (subjective experience) is possibly orthogonal to intelligence (ability to achieve complex goals). We definitely have a better handle on what intelligence is than consciousness.
10.
by mmoskal 1y ago
Counting to 2^61 probably is. To actually find a collision in 128b cryptographic hash function it would take closer to 2^65 hashes. Back of the envelope calculations suggest that with Pollard's rho it would cost a few million dollars o
11.
by mmoskal 1y ago
Airplanes are dirty, unsafe and unclean?
12.
by mmoskal 1y ago
I think this is like unsafe - most of your code won’t have it, so you get the benefits of borrow checker (memory safety and race freedom) elsewhere.
13.
by mmoskal 1y ago
This seems way too readable! I think you should remove the character literals in the name of purity. Also, this is likely way more compact than Brainfuck, as the lambda calculus is written essentially as usual. And seriously, very cool!
14.
by mmoskal 1y ago
https://github.com/verus-lang/verus is similar tool for Rust (developed by previous heavy users of Dafny).
15.
by mmoskal 1y ago
They are reducing forward pass time from say 1.5ms to 1ms. On bigger model you would likely reduce from 15ms to 14.2ms or something like that.
16.
by mmoskal 1y ago
The sglang and vllm numbers are with cuda graphs enabled. Having said that, 1B model is an extreme example - hence the 1.5x speedup. For regular models and batch sizes this would probably buy you a few percent.
17.
by mmoskal 1y ago
As mentioned in other comments - in TypeScript which follows this gradual typing there is a number of flags to disable it (gradually so to speak). No reason ty wouldn't do it.
18.
by mmoskal 1y ago
If you change one letter in the prompt, however insignificant you may think it is, it will change the results in unpredictable ways, even with temperature 0 etc. The same is not true of renaming a variable in a programming language, most re
19.
by mmoskal 1y ago
I don't know. I suspect most people rate data centers higher than almond milk...
20.
by mmoskal 1y ago
My understanding is that printing 300 page paperback costs like $2 while 50 pages cost $1.50. However you can clearly claim way more money for the 300 pages so publishers are not interested in short books, business or otherwise.
21.
by mmoskal 1y ago
Very typical for anything with CUDA (they tend to compile everything for 10 different architectures times hundreds of template kernel parameters). Not sure about ClickHouse though.
22.
by mmoskal 1y ago
To put this into perspective, driving for an hour in an electric car (15kW avg consumption) consumes about as much energy as 50,000 chatgpt queries [0] Running your laptop for an hour would be around 100 queries. [0] https://epoc
23.
by mmoskal 1y ago
Spec decoding only depends on the tokenizer used. It's transfering either the draft token sequence or at most draft logits to the main model.
24.
by mmoskal 1y ago
Just for some callibration: approx. no one runs 32 bit for LLMs on any sort of iron, big or otherwise. Some models (eg DeepSeek V3, and derivatives like R1) are native FP8. FP8 was also common for llama3 405b serving.
25.
by mmoskal 1y ago
Also ~noone runs h100 at home, ie at batch size 1. What matters is throughput. With 37b active parameters and a massive deployment throughout (per gpu) should be similar to Gemma.
26.
by mmoskal 1y ago
The way I understand it: if the instruction are at the top, the KV entries computed for "content" can be influenced by the instructions - the model can "focus" on what you're asking it to do and perform some computa
27.
by mmoskal 1y ago
According to Stack Overflow developer survey [0] Rust is at 12.5%, roughly a half of C# or Java and a quarter of Python. Also more than twice Ruby. So definitely not niche. [0] https://survey.stackoverflow.co/2024/techn
28.
by mmoskal 1y ago
Just to clarify: simple prefix KV cache doesn't require any special model training. It does require the inference framework to support it, but most do by now. You can see dramatic improvements in latency and throughput if there is a la
29.
by mmoskal 1y ago
Memory grows linearly, compute grows quadratically (but with small constant - until ~100k the inference will be still dominated by non-quadratic factors).
30.
by mmoskal 1y ago
The bitter lesson is about the balance of human ingenuity and compute being thrown at the problem. We've seen a few years of LLM compute being scaled up 10x every year, but this is hitting limits (fabs), and we will see more human effo

More ›