8 ms·
Apple Research unearthed forgotten AI technique and using it to generate images
- celias 1y agoPaper at https://machinelearning.apple.com/research/normalizing-flows https://machinelearning.apple.com/research/normalizing-flows
- rfv6723 1y agoApple AI team keeps going against the bitter lesson and focusing on small on-device models. Let's see how this would turn out in longterm.
- echelon 1y agoEdge compute would be clutch, but Apple feels a decade too early.
- 7speter 1y agoMaybe for a big llm, but if they add some gpu cores and added a magnitude or 2 more unified memory to their i devices, or shoehorned m socs into high tier iDevices (especially as their lithography process advances), image generation becomes more viable, no? Also, I thought I read somewhere that apple wanted to infer simpler queries locally and switch to datacenter inference when the request was more complicated. If they approach things this way, and transistor progress continues linearly (relative to the last few years) maybe they can make their first devices that can meet these goals in… 2-3 years?
- sipjca 1y agosomewhat hard to say how the cards fall when the cost of 'intelligence' is coming down 1000x year over year while at the same time compute continues to scale. the bet should be made on both sides probably
- furyofantares 1y ago10x year over year, not 1000x, right? The 1000x is from this 10x observation having held for 3 years.
- sipjca 1y agoI believe the 1000x number I pulled is from SemiAnalysis or similar, using MMLU as the baseline benchmark and the cost per token from a year ago to today at the same score. Model improvements, hardware improvements and software improvements all make a massive difference when combined to make much greater than 10x gains in terms to intelligence/$
- peepeepoopoo137 1y ago"""The bitter lesson""" is how you get the current swath of massively unprofitable AI companies that are competing with each other over who can lose money faster.
- furyofantares 1y agoI can't tell if you're perpetuating the myth that these companies are losing money on their paid offerings, or just overestimating how much money they lose on their free offerings.
- janalsncm 1y agoIf it costs you a billion dollars to train a GPT5 and I can distill your model for a million dollars and get 90% of the performance, that’s a terrible deal for you. Or more realistically, whoever you borrowed from.
- rfv6723 1y agoThen if you offer your distilled model for commercial services, you would get sued by OpenAI in court.
- janalsncm 1y agoThe bitter-er lesson is that distillation from bigger models works pretty damn well. It’s great news for the GPU poor, not great for the guys training the models we distill from.
- rfv6723 1y agoDistillation is great for researchers and hobbyists. But nearly all frontier models have anti-distillation ToS, so distillation is out of question for western commercial companies like Apple.
- janalsncm 1y agoEven if Apple needs to train an LLM from scratch, they can distill it and deploy on edge devices. From that point, inference is free to them.
- yorwba 1y agoThey took a simple technique (normalizing flows), instantiated its basic building blocks with the most general neural network architecture known to work well (transformer blocks), and trained models of different sizes on various datasets to see whether it scales. Looks very bitter-lesson-pilled to me. That they didn't scale beyond AFHQ (high-quality animal faces: cats, dogs and big cats) at 256×256 is probably not due to an explicit preference for small models at the expense of output resolution, but because this is basic research to test the viability of the approach. If this ever makes it into a product, it'll be a much bigger model trained on more data. EDIT: I missed the second paper https://arxiv.org/abs/2506.06276 https://arxiv.org/abs/2506.06276 where they scale up to 1024×1024 with a 3.8-billion-parameter model. It seems to do about as well as diffusion models of similar size.
- nextaccountic 1y agoThis subject is fascinating and the article is informative, but I wish that HN had a button like "flag", but specific for articles that seems written by AI (well at least the section "How STARFlow compares with OpenAI’s 4o image generator" sounds like it)
- CharlesW 1y agoFWIW, you can always report any HN quality concerns to hn@ycombinator.com and it'll be reviewed promptly and fairly (IMO).
- Veen 1y agoIt reads like the work of a professional writer who uses a handful of variant sentence structures and conventions to quickly write an article. That’s what professional writers are trained to do.
- janalsncm 1y agoI had the opposite reaction, it definitely reads like a tech journalist who doesn’t have a great understanding of the tech. AI would’ve written a less clunky (and possibly incorrect) explanation.
- lukan 1y agoIf you enjoyed the article, why would you want to flag or tag it? For what purpose?
- nextaccountic 1y agoWell maybe this article isn't AI written after all. But the intent was adding an (AI) besides the title.
- kelseyfrog 1y agoForgotten from like 2021? NVAE[1] was a great paper but maybe four years is long enough to be forgotten in the AI space? shrug 1. NVAE: A Deep Hierarchical Variational Autoencoder https://arxiv.org/pdf/2007.03898 https://arxiv.org/pdf/2007.03898
- bbminner 1y agoRight, it is bizzare to read that someone "unearthed a forgotten AI technique" that you happened to have worked with/on when it was still hot - when did I become a fossil? :D Also, if we're being nitpicky, diffusion model inference has been proven equivalent to (and is often used as) a particular NF so.. shrug
- nabla9 1y agoThey are both variational inference, but Normalizing Flow (NF) is not VAE.
- kelseyfrog 1y agoIf you read the paper, you'll find "More Expressive Approximate Posteriors with Normalizing Flows" is in the methods section. The authors are in fact using (inverse) normalizing flows within the context of VAEs. The appendix goes on to explain, "We apply simple volume-preserving normalizing flows of the form z′ = z + b(z) to the samples generated by the encoder at each level".
- bitpush 1y agoI find it fascinating that Apple-centric media sites are stretching so much to position the company in the AI race. The title is meant to say that Apple found something unique that other people missed, when the simplest explanation is they started working on this a while back (2021 paper, afterall) and just released it. A more accurate headline would be - Apple starting to create images using 4 year old techniques.
- danhau 1y agoThis „4 year old technique“ apparently could give Apple an edge for on-device workloads. > short: both Apple and OpenAI are moving beyond diffusion, but while OpenAI is building for its data centers, Apple is clearly building for our pockets.
- bitpush 1y agoThe same edge Apple had summarizing notifications so poorly that they had to turn it off? https://arstechnica.com/apple/2024/11/apple-intelligence-notification-summaries-are-honestly-pretty-bad/ https://arstechnica.com/apple/2024/11/apple-intelligence-not...
- janalsncm 1y agoThat was a bad and unnecessary feature but the privacy benefits of running a model on device rather than in the cloud are undeniable.
- bitpush 1y agoThe fact that they shipped it shows they don't know what they were doing, private or not.
- janalsncm 1y agoThat’s a little unfair imo. Statistical models make mistakes and have failure modes which are difficult to predict. When the bug popped up, turning the feature off was easier than retraining and redeploying.
- OhNoNotAgain_99 1y ago[dead]
- tomhow 1y agoComments moved to https://news.ycombinator.com/item?id=44400105 https://news.ycombinator.com/item?id=44400105.