lukebechtel

6 ms·

1.
by lukebechtel 5d ago
Fatalism is a toxic mindset. Is it hard? yes. Is it impossible? No.
2.
by lukebechtel 1mo ago
I relate to this; there are some things I remember very well, but in many areas I think my talent for improvisation stems from a ~mid long-term memory.
3.
by lukebechtel 2mo ago
> Chip Design > As an early proof of concept, Kimi K3 designed a chip to serve a nano model built on its own architecture. In a single 48-hour autonomous run, K3 built, optimized, and verified the chip using open-source EDA tools on t
4.
by lukebechtel 5mo ago
Some people seem to be suggesting these are coverups for quantization... Those who work on agent harnesses for a living realize how sensitive models can be to even minor changes in the prompt. I would not suspect quantization before I would
5.
by lukebechtel 6mo ago
really fun :) thanks!
6.
by lukebechtel 6mo ago
it sounds like you described it pretty well!
7.
by lukebechtel 6mo ago
~/.claude/projects is where the real fun is :)
8.
by lukebechtel 6mo ago
What is your domain?
9.
by lukebechtel 6mo ago
so we need to make some crazy llms...
10.
by lukebechtel 6mo ago
there used to be https://default-filename-tv.neocities.org/ but it got taken down :/
11.
by lukebechtel 6mo ago
The bitter lesson strikes again, I suppose!
12.
by lukebechtel 6mo ago
Good questions! It's clear I need to gather more metrics from our next generated inference library.
13.
by lukebechtel 6mo ago
Unfortunately it hasn't been open sourced. We're debating how / when to do this right now.
14.
by lukebechtel 6mo ago
This is a fair critique! We plan to use our system to generate many more inference libraries of this nature, and I'll make it a point to release better, broader correctness measures when we do so.
15.
by lukebechtel 6mo ago
Yes, great question! The system started without paged attention, and recreated its own paged attention implementation automatically once it realized it was a bottleneck. Pretty cool!
16.
by lukebechtel 6mo ago
Unfortunately, not at present; we went for FP8 because we believed it was generally the best tradeoff of quality and speed. Allowed faster iteration as well. We believe our improvements would hold on BF16, but let me check.
17.
by lukebechtel 6mo ago
Yes, speculative decoding will make both us and VLLM faster, but we believe it would be a relatively even bump on both sides, so we didn't include it in this comparison. Worth another test!
18.
by lukebechtel 6mo ago
We validate with MMLU and Hellaswag presently, and are getting this independently verified by a 3rd party. We have considered open-sourcing some of our optimized inference libraries in the future, but have not yet come to a decision on this
19.
Surpassing vLLM with a Generated Inference Stack(infinity.inc)
62 points by lukebechtel 6mo ago|22 comments
20.
by lukebechtel 7mo ago
I also do this!
21.
by lukebechtel 7mo ago
Thank you Georgi <3
22.
by lukebechtel 7mo ago
sonnet 4.6 is a third, and equivalent to opus 4.5, which is enough for me usually :) EDIT: Gemini does have 1m context for "free" though so that's great.
23.
by lukebechtel 7mo ago
Arc-AGI-2: 84.6% (vs 68.8% for Opus 4.6) Wow. https://blog.google/innovation-and-ai/models-and-research/ge...
24.
by lukebechtel 7mo ago
Oh, I always use opus 4.5 thinking mode. Maybe that's the diff.
25.
by lukebechtel 7mo ago
Why? I use it for all and love it. That doesn't mean you have to, but I'm curious why you think it's behind in the personal assistant game.
26.
by lukebechtel 7mo ago
> Context compaction (beta). > Long-running conversations and agentic tasks often hit the context window. Context compaction automatically summarizes and replaces older context when the conversation approaches a configurable threshold
27.
by lukebechtel 8mo ago
> We generally favor cultivating good values and judgment over strict rules and decision procedures, and to try to explain any rules we do want Claude to follow. By “good values,” we don’t mean a fixed set of “correct” values, but rather
28.
by lukebechtel 8mo ago
Interesting! Was kinda disappointed with Codex last time I tried it ~2m ago, but things change fast.
29.
by lukebechtel 8mo ago
1. Start with a plan. Get AI to help you make it, and edit. 2. Part of the plan should be automated tests. AI can make these for you too, but you should spot check for reasonable behavior. 3. Use Claude 4.5 Opus 4. Use Git, get the AI to ch
30.
by lukebechtel 8mo ago
Good question! 1. The post was written before this was common :) 2. If using Cursor (as I usually am), this isn't what it always does by default, though you can invoke something like it using "plan" mode. It's default is

More ›