hansonw

6 ms·

1.
by hansonw 7mo ago
The skill source is here: https://github.com/openai/skills/blob/main/skills/.curated/p... $skill-installer playwright-interactive in Codex! the model writes normal JS playwright code in a Node
2.
by hansonw 10mo ago
Rest assured that we are better at training models than naming them ;D - New benchmark SOTAs with 77.9% on SWE-Bench-Verified, 79.9% on SWE-Lancer, and 58.1% on TerminalBench 2.0 - Natively trained to work across many hours across multiple
3.
Building more with GPT-5.1-Codex-Max(openai.com)
483 points by hansonw 10mo ago|319 comments
4.
by hansonw 1y ago
More about that here! https://platform.openai.com/docs/codex#advanced-configuratio...
5.
by hansonw 2y ago
The ELI5 of the paper is that most "unlearning" methods can be regarded as adding some delta `w` to the parameters of the network, but most of `w` just gets "rounded away" during quantization (i.e. `quantize(X+w) ~= quan
6.
by hansonw 2y ago
It looks like they didn't want to make a public submission in order to avoid disclosing the model internals: https://cosine.sh/blog/genie-technical-report#:~:text=SWE%2D... .
7.
by hansonw 2y ago
It’s probably more. Pretty conservatively, if the KV embedding dimension for each token is ~10K x 100 attention layers (this is roughly the scale of Llama3.1 405B) that’s already 1M 16-bit floats per token = 2MB. They have likely needed to
8.
by hansonw 2y ago
https://news.ycombinator.com/item?id=40675577
9.
by hansonw 2y ago
This is also a good paper on the subject: What Algorithms can Transformers Learn? A Study in Length Generalization https://arxiv.org/abs/2310.16028
10.
by hansonw 2y ago
https://predibase.com
11.
by hansonw 2y ago
Yes. But also note that the new function calling is actually “tool calling” where the model is also fine-tuned to expect and react to the output of the function (and there are various other nuances like being able to call multiple functio
12.
by hansonw 2y ago
Not the author, but really nice that they shared some real data points: > Once our Text-to-SQL solution was in production, we were also able to observe how users interacted with the system. As our implementation improved and as users bec
13.
How we built Text-to-SQL at Pinterest(medium.com)
3 points by hansonw 2y ago|1 comments
14.
by hansonw 3y ago
If you think about it, RAG is a relatively primitive “first pass attention layer” that is binary and semi-heuristic based. I think it’s fairly safe to say that in the long term RAG will be integrated into the model architecture somehow, jus
15.
by hansonw 3y ago
If sub-quadratic architectures (eg Mamba) become a thing, it will become feasible to precompute most of the work for a fixed prefix (i.e. system prompt) and the latency can be pretty minimal. Even with current transformers, if you have a fi
16.
by hansonw 3y ago
Indeed: https://arxiv.org/pdf/2402.01032.pdf Perhaps future iterations of SSMs will accommodate dynamically sized (but still non-linearly-growing) hidden states / memories!
17.
by hansonw 3y ago
“RNN-mode inference” is also extremely exciting because you can precompute the hidden state of any prompt prefix (i.e. a long system prompt, or statically retrieved context) and continued generations pay the same cost irrespective of the pr
18.
by hansonw 3y ago
Our startup is building https://arcwise.app , which allows you to embed full-fledged SQL tables inside Google Sheets! We’re in the process of building out support for joins & subqueries, would be curious what people think.
19.
by hansonw 3y ago
I’m building a solution that works like this - we directly connect spreadsheet models to company databases (even converting pivots/formulas to SQL). Would love to chat with anyone that might find this valuable: https://arcwi
20.
by hansonw 3y ago
This is the best comparison I've found that benchmarks the current OSS inference solutions: https://hamel.dev/notes/llm/inference/03_inference.html IME the streaming API in text-generation-inference work
21.
by hansonw 3y ago
gcsfuse has controllable built-in caching of _metadata_ but not contents: https://cloud.google.com/storage/docs/gcsfuse-performance-an... You'd have to use your own cache otherwise. IME the OS-level page cach
22.
by hansonw 3y ago
What about datasets like https://huggingface.co/datasets/b-mc2/sql-create-context ? An OSS model based on StarCoder was also recently published which is roughly between GPT-3.5 and GPT-4: https://github.
23.
MPT-30B: Raising the bar for open-source foundation models(mosaicml.com)
34 points by hansonw 3y ago|2 comments
24.
by hansonw 4y ago
Arcwise | https://arcwise.app | Engineering | Full-time | SF or Remote (US) At Arcwise, we’re on a mission to make analytics and data science accessible to anybody - starting with spreadsheet users. We're big believers that
25.
by hansonw 4y ago
It appears the indexing for the model parts is deliberately not contiguous; the 03-82 range represents the main 80 transformer layers. https://github.com/yandex/YaLM-100B/blob/main/megatron_lm/me...
26.
by hansonw 4y ago
(Note: I worked on https://ide.atom.io and Facebook's Nuclide team). On the topic of performance - one of the architecture decisions that VSCode nailed was the 'extension host' ( https://code.visualstudi
27.
by hansonw 4y ago
a bunch of Parquet files in S3 ;)
28.
by hansonw 4y ago
plug in https://www.etebase.com/ and ship it :D
29.
by hansonw 4y ago
PyScript is built on top of Pyodide! But it's probably the first well-supported way to build frontend apps with it. I imagine the fact that Anaconda is backing PyScript means that there will be increased investment into Pyodide as well
30.
by hansonw 4y ago
Here's a ~5 minute talk with 5 such examples (where relatively simple ML models made a 1M+ impact at a FAANG) :) https://youtu.be/zyOEOd1HkSY?t=946 Happy to talk about more details if you message me through my profile!

More ›