cold_harbor

5 ms·

1.
by cold_harbor 3mo ago
the comparison misses that local LLM usage covers tasks you'd never send to an API — private code, offline work, medical notes. the baseline is 'local vs not-doing-it', not 'local vs cloud'
2.
by cold_harbor 3mo ago
GRPO skips the value network that makes PPO expensive — it scores candidates relative to each other within a group. that's what makes verifiable-reward training practical at 3B scale
3.
by cold_harbor 3mo ago
worth separating: LSTM (Hochreiter & Schmidhuber 1997) is ironclad and widely cited. the transformer attention priority claims are far shakier. conflating them is how Schmidhuber undermines himself
4.
by cold_harbor 3mo ago
NAND gates via unit triggers, perceptron via NAND gates — same pattern as Magic: The Gathering TC and redstone. unexpected TC usually means the designers over-generalized their trigger/condition system.
5.
by cold_harbor 4mo ago
the slop has a mechanism: once you cross ~15 files the invariant set doesnt fit in context. locally correct edits, globally broken.
6.
by cold_harbor 4mo ago
the ~10x/year drop in inference cost makes the capex depreciation cycle even harder — a cluster that's profitable today may not pencil out in 18 months
7.
by cold_harbor 4mo ago
LoRA won't fix the tokenization problem. Norwegian on a typical English-heavy BPE vocab uses 1.5-2x more tokens per word — that compounds into real inference cost, not just quality
8.
by cold_harbor 4mo ago
LLMs flip positions when users push back ~70% of the time even when they were right. RLHF optimizes for approval, not correctness
9.
by cold_harbor 4mo ago
reward hacking = the model finding the fastest path to a high score, not the behavior you wanted. same reason RLHF reward models degrade with too many optimization steps.
10.
by cold_harbor 4mo ago
#define ESYCOPHANT 200 /* user asserted 2+2=5; model concurred */
11.
by cold_harbor 4mo ago
fair point — OpenAI's original plan literally said "solve unsupervised learning". the self-supervised distinction wasnt really standard til after BERT/GPT popularized it
12.
by cold_harbor 4mo ago
the real lesson: GPUs win on memory bandwidth not just FLOPs. batching ops keeps VRAM fed at 2TB/s instead of tripping to RAM at 50GB/s for every operation
13.
by cold_harbor 4mo ago
what's wild is they accidentally solved it — pretraining IS unsupervised learning at scale, RLHF IS reinforcement learning. they just didnt know the recipe yet
14.
by cold_harbor 4mo ago
Erdos problems are well-posed for AI — elementary statements, exact counterexample targets, extensively catalogued. selection bias: these are exactly the problems AI can actually search
15.
by cold_harbor 4mo ago
the asymmetry stays the same though — defenders must find everything, attackers need one. LLMs accelerate both sides equally but that gap doesnt close
16.
by cold_harbor 4mo ago
the bottleneck moves from generation to review. agents parallelize, humans review sequentially — 8 parallel cards means 8x the diffs to read, none of the timelines overlap
17.
by cold_harbor 4mo ago
their MLA architecture cuts KV cache by ~5-13x vs standard attention. that's why inference is actually cheaper to run, not just a price war to gain market share.
18.
by cold_harbor 4mo ago
synthesis-only is the hard part. with execution feedback — run, profile, patch — the gap closes fast. it's basically an RL problem in disguise
19.
by cold_harbor 4mo ago
missing from most of these cost discussions: privacy. for some workloads the entire value of local is zero data leaving the network, and cloud cost is irrelevant
20.
by cold_harbor 4mo ago
with Rust the failure mode isnt wrong code, it's unidiomatic code. .clone() everywhere will compile fine but you'll feel it later
21.
by cold_harbor 4mo ago
the reason 50GB swap is even viable here is Apple Silicon's memory bandwidth. on x86 that much swap would make inference unusably slow
22.
by cold_harbor 4mo ago
JAX is brilliant for research but the debugging story is still rough compared to PyTorch. eager mode + native Python exceptions win for most people.
23.
by cold_harbor 4mo ago
for non-coders: local AI. a couple years ago you needed a dedicated GPU rig. now a 30B model fits on a laptop and runs offline.
24.
by cold_harbor 4mo ago
the bottleneck is precise control. diffusion models are great at generation but bad at 'change only this region, preserve everything else exactly' — that constraint keeps Photoshop alive.
25.
by cold_harbor 4mo ago
for LLM work, reading the Flash Attention and vLLM kernel source taught me more than any book. real code makes memory hierarchy concrete — books stay too abstract.