angusturner

19 ms·

1.
by angusturner 5d ago
This seems highly optimistic to me. What's to say current AI won't be highly power-concentrating by default? The best models are owned by a few companies, and displacement of knowledge-workers mainly seems to benefit the capital c
2.
by angusturner 13d ago
Define LLMs. Current models look pretty different to the vanilla transformer from 2017. And its not even known what architecture and training innovations the frontier is using, right?
3.
by angusturner 13d ago
Its turtles all the way down
4.
by angusturner 13d ago
People want human art. Probably they will want it more than ever in post AGI world.
5.
by angusturner 13d ago
What does this even mean. There is strong evidence of LLMs doing in context learning. Some of the linear RNN layers in recent models are provably doing SGD in hidden space during inference
6.
by angusturner 19d ago
What unique differentiator could musk possibly offer though? Is there really a high demand for racist chatbots with no safety controls? And even if there is, will governments even allow it to keep operating?
7.
by angusturner 2mo ago
I find the presentation of this as "factoring training" very confusing. Factoring is something we do to distributions, which we then parameterize with our neural net. An autoregressive model pertains to a certain factorisation of
8.
by angusturner 2mo ago
I immediately thought of IWAE as well.
9.
by angusturner 3mo ago
In 2017 I worked tirelessly with my colleagues to implement and replicate the first transformer paper. Yesterday I left Opus 4.8 to go do some architecture research, with GPU access. It replicated and trained a credible baseline. It impleme
10.
by angusturner 4mo ago
For small to medium projects, an LLM can write functional (if not well crafted) Rust. Considering how easy this is now, why choose a heavier, slower and less typesafe language?
11.
by angusturner 5mo ago
why read that, vs an actually well-written compiler though?
12.
by angusturner 10mo ago
I thought exceptions tended to be made when its highly relevant to the technical topic at hand and also non controversial. Outside a few weird online bubbles and pockets of the US, hardly anyone disputes the claim you are objecting to.
13.
by angusturner 1y ago
I feel this. I've had a few tasks now where in honest retrospect I find myself asking "did that really speed me up". Its a bit demoralising cause not only do you waste time, you have a worse mental model of the resulting code
14.
by angusturner 1y ago
I think most SWEs do have a good idea where I work. They know that its a significant, but not revolutionary improvement. If you supervise and manage your agents closely on well scoped (small) tasks they are pretty handy. If you need a proto
15.
by angusturner 1y ago
Interesting... Maybe I need to investigate PayPal as an option here. Best case would be my bank eventually adds tap to pay natively
16.
by angusturner 1y ago
I recently made the shift to graphene from iOS and am mostly enjoying it. The user profiles was slow to set up and not having shared filesystem between the user profiles creates friction. But I love that I can effectively sandbox my work ap
17.
by angusturner 1y ago
Yeah, worlds slowest and most in-efficient write-only database. And as soon as you need to interact with goods or services in the real world, then you still need trust anyway. All these people harping on about: "Bro I just need to move
18.
by angusturner 1y ago
Fuck google for this. Awful decision. Guaranteed to be abused when Google or government despots decide that certain apps (or developers) aren't aligned with their interests. Feeling very frustrated with the way the internet is going la
19.
by angusturner 1y ago
You mean the guy that bans people from twitter for disagreeing with him? And has made a chatbot that spouts right-wing conspiracies in the name of being "anti-woke"?
20.
by angusturner 1y ago
As an Aussie, I was feeling somewhat consoled about the state of the US by the fact that the EU and UK still seem to have their heads screwed on. OSA and chat control have made me seriously rethink that… Has everyone lost their mind?
21.
by angusturner 1y ago
I wish people would stop parroting the view that LLMs are lossy compression. There is kind of a vague sense in which this metaphor holds, but there is a much more interesting and rigorous fact about LLMs which is that they are also _lossles
22.
by angusturner 1y ago
There is an excellent talk by Jack Rae called “compression for AGI”, where he shows (what I believe to be) a little known connection between transformers and compression; In one view, you can view LLMs as SOTA lossless compression algorithm
23.
by angusturner 1y ago
By the definition you have provided though, someone that has access to stable, safe or functional housing but then chooses to not to use it (eg opting to camp instead), is not homeless. Edit: the word “lack” really is the key word. This imp
24.
by angusturner 1y ago
Agree the media is having a field day with this and a lot of people will draw bad conclusions about it being sentient etc. But I think the thing that needs to be communicated effectively is that these these “agentic” systems could cause ser
25.
by angusturner 1y ago
One under appreciated / misunderstood aspect of these models is they use more compute than an equivalent sized autoregressive model. It’s just that for N tokens, autoregressive model has to make N sequential steps. Where diffusion does
26.
by angusturner 1y ago
You assume that for small steps (I.e taking some noisy code and slightly denoising) you can make an independence assumption. (All tokens conditionally independent, given the current state). Once you chain many steps you get a very flexible
27.
by angusturner 1y ago
Hm suppose for argument sake that feeding a batch of data through some moderately large FF architectures takes on the order of 100ms (I realise this depends on a lot parameters - but this seems reasonable for many tasks / networks). No
28.
by angusturner 1y ago
I’m really glad to see people converging on this view because I feel a bit insane for not understanding all the hype. Like, yeah, we need a standard way to connect LLMs with tools etc, but MCP in its current state is not a solution.
29.
by angusturner 1y ago
I think this is article is too generous about the use of stdio - I have found this extremely buggy so far, especially in the python sdk. Also if you want to wrap any existing code that logs or prints to stdout then it causes heaps of ugly m
30.
by angusturner 1y ago
Yeah this is what I don’t really get about the stable coin hype. In Australia for example, we are almost cashless now and most bank transfers are instant and cheap - no blockchain required. Digital currency is a good idea - unclear what val

More ›