13 ms·
Stretch iPhone to its limit: 2GiB Stable Diffusion model runs locally on device
- burk96 4y agoThis looks so cool! Unfortunately, my 16.1 11pro is crashing before an image can be generated, let me know if I can do anything to debug/test this.
- liuliu 4y agoWhich generation of iPad is that? Honestly I haven't tested iPad yet, still in designing phase.
- mbroncano 4y agoI got I working nicely on my M1 iPad
- MikePlacid 4y agoIPad Air 4 is ok too.
- burk96 4y agoMy bad I wasn't clear, iPhone 11 Pro on iOS 16.1
- gorbypark 4y agoIt’s working fine on my iPhone 11 Pro / iOS 16.1 with all default settings. Just generated an image of an astronaut skiing that looks pretty good!
- boppo1 4y agoAww man, no ipad version? Tsk what did I get this 16gb of ram for?
- sosodev 4y agoThis is amazing. I'm kind of surprised that it doesn't have an NSFW image blocker. I want to be able to generate NSFW images but it probably should have one enabled by default.
- vardump 4y agoNow I see why that coffee cup icon is on the "generate images" button... fingers burning after a few images. Haha, awesome app!
- drawingthesun 4y agoIs there any option to set so that every image is automatically saved, and not to camera roll, but to the local app folder (same folder that contains this apps data)
- secretsatan 4y agoReally cool, but for some reason, it fails to share the images using Airdrop for me. I have to save the images to photos then airdrop from there
- patentatt 4y agoCrashes every time... am I doing something wrong? iPhone 11 Pro.
- liuliu 4y agoDoes it crash upon downloading models, or generating? I haven't tested on all the devices, but 11 Pro seems have 4GiB RAM, and should run with 384x384 resolution (check if that is the selected resolution at top right). There are reports that iOS is not happy with how I computing SHA256 for downloaded model file by loading them all in memory for Xr (3GiB RAM). If this is happening for other devices, I may need to do streaming hash computing and put up a bugfix.
- olliej 4y agoIt _might_ be easier just to use mmio and save any futzing around rewriting the actual code.
- liuliu 4y agoYeah, I thought Data(contentsOf:) already do that, but it appears not (tested, indeed allocated all the memory to load the data). Adding `mapIfSafe` in the reading options solved this.
- lll-o-lll 4y agoI have an iPhone 11 and it works for me. Had to keep the app in focus until all the downloading completed.
- gorbypark 4y agoI can also confirm it works (for me) on iPhone 11 Pro.
- WantonQuantum 4y agoThis is cool and looks like a great way to drain my phone battery :) I just used the prompt "A person looking at their phone in amazement" and got a good picture. Beware that on startup the app downloads almost 2 gig of data.
- tibbon 4y agoIt does warn you on startup about the download if you're not on wifi.
- odysseus 4y agoGood time to try 5G ultra capacity if you have it on an unlimited plan - will be faster than most people's wifi.
- jcims 4y agoPivot alert I reached 3Gbps over Verizon 5G in San Antonio last year and this year i get about 4Mbps over Verizon 5G in Ohio. It’s so bad I disabled it. I did read an article that iPhone 12 (which is what I have) have some kind of radio issue with 5G. Can anyone in here confirm?
- badwolf 4y agoVerizon has been real wonky all over Austin. If there's more than a handful of people in the area, bandwidth just goes to the crapper. I'll go from a couple hundred mbps on a good day with no clouds/wind/holding my phone just perfectly right in the right space, but usually get less than 1mbps on their 5g UW.
- tacotime 4y agoHahah now I can use my phone as a hand warmer this winter. It's incredible that this is an app!
- desro 4y agoThis is incredible work and a tremendous achievement. Bravo and thanks for sharing.
- miohtama 4y agoCould in-memory compression used to bring down the RAM requirements? There are some performance compressors like Blosch tuned for this: https://www.blosc.org/pages/blosc-in-depth/ https://www.blosc.org/pages/blosc-in-depth/ “Faster than memcpy” is the slogan.
- addaon 4y agoMacOS has transparent memory compression. Unclear to me if that's made its way to iPhone, but if it hasn't yet it will sooner or later.
- comboy 4y agoMemory compression? I can't find any good resources to read about it, any hints? I'm having trouble imaging how could it possibly work without totally destroying performance.
- miohtama 4y agoOn Mac, you can find Compressed memory in Activity monitor. It’s something similar to swap - apps do not need to have built in support for it.
- flatiron 4y agoIt segments a certain amount of ram to “swap” to which means compress and store. Normal blue sky ram operations are not compressed on macOS
- comboy 4y agoOh, yes compressed swap makes much more sense, thanks.
- kergonath 4y agoIt is not compressed swap, the compressed data is still in RAM. The OS just compresses inactive memory, with a couple of criteria to define “inactive”.
- Retr0id 4y agoThis is absolutely incredible. It takes about 45 seconds to generate a whole image on my iPhone SE3 - which is about as fast as my M1 Pro macbook was doing it with the original version!
- liuliu 4y agoSE 3rd Gen has 4GiB RAM, therefore the app defaults to 384x384 size. This is about 1/2 computation of your normal run (512x512) and the original version uses PLMS sampler, which defaults to 50 steps, while this one uses the newer DPM++ 2M Karras sampler, that defaults to 30 steps. All in all, your M1 Pro MBP is still 4x of your SE 3rd Gen in raw performance (although my implementation should be faster than PyTorch at about 2x on M1 chips)
- 2Gkashmiri 4y agohey. 9th gen ipad 10.9, i think it has 3GB ram so it will never work on it?
- codetrotter 4y ago> although my implementation should be faster than PyTorch at about 2x on M1 chips Do you plan to make a macOS version of your app also? Hope you will :)
- ShamelessC 4y agoFor what it's worth, you can decrease resolution and use the sampler mentioned on the pytorch versions. The AUTOMATIC web UI supports this, for instance. I would also welcome the additional optimizations, however.
- Kye 4y agoI'm downloading it now. This should be fun.
- wwarner 4y agoworks really well thank you!
- xingped 4y agoFantastic! Can it be made available for iPad?
- spideymans 4y agoIt works on the iPad, but it uses the iPhone UI. But it works just as well
- ir77 4y agoMeh… 2gb download only to try “image of unicorn pooping” to get the lamest results ever. If I can’t amuse my 5 yr old with this AI it’s nonsense. It transcribed as “image of unicorn poo ping” in tags :(
- stephenitis 4y agosmall add. display multiple examples while it's downloading the model. Its a long time to stare at an astronaut on a horse in space
- pellias 4y agoHmmm, my ipad keeps crashing and redownloading the sd_v1.4_f16.ckpt file.
- deleted 4y ago[deleted]
- liuliu 4y agoYeah, seems iPad has bunch of issues (curiously most of it related to how I translate tensor back to CGImage ...) stay tuned.
- MarcusE1W 4y agoOn my iPad mini 5th generation with A12 the download is fast and fine. But with standard settings it first warns “Device capability warning” and then indeed crashes every time. Is there a way to solve this? A12 chip should work, no?
- ollin 4y agohere's a direct app store link, if anyone wants to try the iPhone app immediately: https://apps.apple.com/us/app/draw-things-ai-generation/id6444050820 https://apps.apple.com/us/app/draw-things-ai-generation/id64... congratulations to liuliu on the launch!
- ttyyzz 4y agoCrashes on my iPhone SE 2. Gen which was to be expected :-(
- deleted 4y ago[deleted]
- antihero 4y agoCrashes on my 13 Mini, which is unexpected! Anything we need to do to prep the phones?
- rsynnott 4y agoI think that probably is to be expected, unfortunately. The model is 2GB; per the article iOS will kill apps that use 2GB on 4GB RAM devices (and the 13 Mini is 4GB).
- harrisi 4y agoIt works just fine on my iPhone 12 Mini, which has (basically) the same RAM, so it seems like it's something else. They're basically the same phone in general though, so I would be surprised if it was a hardware issue.
- deleted 4y ago[deleted]
- pulvinar 4y agoIt's running on my 13 Mini (so far). I did have trouble closing the "adjustments" dialog (upper-right button) due to its close button being underneath the status bar, but found that I could just drag the dialog down to the bottom and it closed.
- globalvisualmem 4y agoI just tried it works beautifully! I suggest defaulting to lower resolution 384 X 384 since it will speed up.
- gok 4y agoDid you try using Core ML for inference?
- iseanstevens 4y agoThis is really impressive !
- ZeroCool2u 4y agoThis is super cool. I just tried the default prompt on my iPhone 13 with the image size set to 768x512 and using the 3D Model (Redshift v1) and it just crashed the whole phone and restarted. Just like when I get BSOD's at work on my Windows GPU desktop :)
- deleted 4y ago[deleted]
- liuliu 4y agoYou should see a warning when selecting that size? 4GiB model cannot run at that resolution until someone implemented FlashAttention on Metal :)
- GistNoesis 4y agoNice work :) Porting FlashAttention to Metal will be quite hard. Because for performance reasons, they did a lot of shenanigans to respect the memory hierarchy. Thankfully, you can probably do something slower but more adapted to your memory constraints. If you relax this need for performance and allow some re-computations, you can write a qkvatt function which takes q,k,v and a buffer to store the resulting attention, and compute without needing any extra memory. The algorithm is still quadratic in time with respect to the attention horizon (although with a bigger constant (2x or 3x) due to the re computation). But it doesn't need any extra memory allocation which makes it easy to parallelize. Alternatively you can use an O(attention horizon * number of thread in parallel) (like flash attention) extra memory buffer to avoid the re-computation. Concerning the backward pass, that's the same thing, you don't need extra memory if you are willing to do some re-computation, or linear in attention horizon to not do re-computation. One interesting thing to notice in the backward pass, is that it doesn't use the attn of the forward pass, so it doesn't need to be kept preserved (only need to preserve Q,K,V). One little caveat of the backward pass (which you only need for training) is that it needs atomic_add to be easy to parallelize. This mean, it will be hard on Metal (afaik they don't have atomics for floats though they do have atomics for integer so you can probably use fixed points numbers).
- skykooler 4y agoI wonder if any of these tricks would be applicable to make a version of Stable Diffusion which could run on the Steam Deck.
- asadlionpk 4y agocant you just run it the regular way? it's a pc.
- skykooler 4y agoIt doesn't have enough VRAM to load the regular model; it ends up triggering the OOM killer.
- IceWreck 4y agoI dont think the Steam Deck is powerful enough. It might be possible but it will take hours.
- ShamelessC 4y agoApple's mobile processors (especially the M1) are waaay faster than a steam deck. Even with the optimizations, it would take like half an hour to run I bet.
- sabalaba 4y agoliuliu has always been a fucking god
- deleted 4y ago[deleted]
- nl 4y agoThis is some impressive work. You might like to look at the work HuggingFace has been doing (on non-iOS versions). They can run it in under 1GB RAM: > If is also possible to chain it with attention slicing for minimal memory consumption, running it in as little as < 800mb of GPU vRAM https://huggingface.co/docs/diffusers/optimization/fp16#offloading-to-cpu-with-accelerate-for-memory-savings https://huggingface.co/docs/diffusers/optimization/fp16#offl...
- liuliu 4y agoCPU offloading doesn't work because Apple has shared memory arch already. The head slicing is similar to https://machinelearning.apple.com/research/neural-engine-transformers https://machinelearning.apple.com/research/neural-engine-tra... I think it is quite practical only if MPSGraph less mysterious about its allocation strategy. It is not the ideal way though. Ideally, FlashAttention / XFormer is the way to go.
- propogandist 4y agothe developer is about to have a MASSIVE hosting bill the download restarts from 0% if the app is sent to the background, as there does not seem to be a download manager. This is especially problematic for the large 1.5gb file.
- liuliu 4y agoI am using Cloudflare R2, which doesn't have egress fee and I am getting about 5k Class B operations right now. Unless Cloudflare changes their end of the deal, I think it is OK.
- propogandist 4y agogreat to hear! please consider introducing a mechanism to suspend the download vs restarting, this may be especially valuable for those with a slower connection. With the traffic growth you'll be seeing chances are Cloudflare's enterprise team will soon be in-touch ;)
- Gigachad 4y agoI wonder why the file can’t be distributed inside the app on the App Store. That way downloads would be much more convenient.
- asadlionpk 4y agoappstore has app size limits.
- lxgr 4y agoThere's also a way to host app resources in the App Store, usually used for levels in games and such: https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/On_Demand_Resources_Guide/index.html https://developer.apple.com/library/archive/documentation/Fi...
- 4y ago
- happyopossum 4y agoKinda ironic that this links to a site that is almost unreadable on an iPhone (14 pro max if you’re wondering).
- 4mitkumar 4y agoI frequently see such comments about sites not loading fine on iPhone 13/14, while they continue to load just fine on for me on a 4 year old Android device (not pixel / Samsung). I wonder if it's the hardware or just the blockers that i use. Might be worth trying using blockers to see if it makes the general browsing experience better on Apple devices
- Tagbert 4y agoThe site loads fine but the font size is a little small for mobile. It reads fine if you rotate to landscape, though.
- sneak 4y agoExtreme respect to the developer for not including the "industry standard" clientside tracking/analytics/phone-home in this app. The fact that this runs locally on-device and doesn't send any information to anyone anywhere about what you're doing on your local device is wonderful. All apps used to be like this, and now the ones that actually respect user privacy are a rare and glorious exception. Thank you!
- 2Gkashmiri 4y agoUh..... Android is there? I mean android version?
- ranguna 4y agoYeah was wondering the same, wanted to see if a snapdragon 8 gen 1 would beat these numbers out of the park or not.
- hustwindmaple1 4y agoauthor is an iOS dev ...
- tluyben2 4y agoBrilliant!
- lll-o-lll 4y agoAny tips on how you “ If the face is cropped, now I know how to use the inpainting model to fill it in. If the inpainting model doesn’t do its job, you can always use a paint brush to paint it over and do an image-to-image generation again focused in that area.” using the app?
- liuliu 4y agoI have a Twitter thread on this: https://twitter.com/liuliu/status/1587978815208407041?s=46&t=gw6yY0kN06WRlUFmnpRq5w https://twitter.com/liuliu/status/1587978815208407041?s=46&t... (Note that at that time, there is an implemention bug in inpainting model that caused the weirdness that I need to manually fix)
- ragazzina 4y agoliuliu, this is simply incredible. Were you focused on just making it work on the iPhone, or do you think you will keep adding functionalities to the app? Do you think it will ever be possible to train one's own model on an iPhone?
- liuliu 4y agoI think that fine-tuning the whole model (a.k.a. Dreambooth) on iPhone would require more RAM / processing power than it currently has. More viable path is to implement Hypernetwork + Textual Inversion, that is within possibility of today's hardware.
- annoyingnoob 4y agoMakes a nice hand warmer on a cool evening.
- 2Gkashmiri 4y agoi have an ipad. the regular non air/pro/m1 one and the app installed and when i run it, it says "could be device incompatibility" and subsequently crashes
- dt3ft 4y agoCongrats on the release! I gave this and other available applications a try and I don’t understand what people see in ai image generation. A simple prompt generated a person with 3 nostrils, 7 squished fingers, deformities everywhere I look, it just mashes a bunch of photographs together and generates abominations. Pay close attention to generated models and you will find details which are simply wrong. What is the use case that I’m missing?
- tomcam 4y ago> and generates abominations You make it sound like that’s a bad thing
- dt3ft 4y agoHorror movie material for sure :) Who knows, maybe this inspires better horror movie creatures :)
- frankzander 4y agoNothing. It's IMHO just a hype of the younger nerdy generation. The real world applications of NN-based (there is no I in A) image generation are limited. One hype comes the other hype goes. IMHO it did not come to stay ;-)
- gpderetta 4y agoAs a 40ish year old, the future shock from all these AI image generators is extreme. This stuff was literally science fiction just a couple of years ago. Now you run it in your phone.
- nelsondev 4y agoPlay around with other prompts. For example, “golden gate bridge in the style of Van Gogh futuristic”. Architecture is cool. It doesn’t do people well
- pdntspa 4y agotry some of the prompts listed on lexica.art. Stable Diffusion needs good prompt engineering to turn out well
- donlinglok 4y agoI have tested the app on my old iPhone XS Max, it takes less than 3 mins for an image. And also can choose a model, steps, scale, and sampler! Thank you for your great work!
- senthilnayagam 4y agoon my iPhone 13, generates 384x384 images in under a minute. discovered they have stable diffusion 1.4,1.5, waifu diffusion(Anime), redshhift(3d model) and other models. iPhone becomes warm after couple of runs and starts draining battery, so do it while connected to charger.
- smcleod 4y agoHas anyone found anything similar (self-hosted or local on device) - but for text generation?
- aarkay 4y agoThis is amazing! Finally a use case for using all that compute power on the phone
- sisama 4y agoUnstable trying with 10 iterations on 348x348 on iPad 9th Gen https://support.apple.com/kb/SP849?locale=en_US https://support.apple.com/kb/SP849?locale=en_US. Looks cool tho!
- hamilyon2 4y agoThis will sell Iphone to me.
- vletal 4y agoAwesome! Although I wish to see the intermediate denoised image instead of the progress bar. Just a suggestion.
- machina_ex_deus 4y agoThis isn't recommended, the decoding takes as much time as processing next step. I learned it the hard way when I tried displaying the intermediate steps for debugging.
- ollin 4y agoyeah, running the full decoder takes a while. though, since the "latent" is just 4 channels and pretty close to representing RGB, you can use a linear combination of latent channels and get a basic (grainy, low-res) preview image like this [0] without much trouble. I expect you could go further, and train a shallow conv-only decoder to get nicer preview results, but I'm not sure if anyone's bothered yet. [0] https://github.com/madebyollin/maple-diffusion https://github.com/madebyollin/maple-diffusion
- secretsatan 4y agoPreserve your privacy, send us your phone number!
- secretsatan 4y agoI see now there's a download link in small grey text under that
- haunter 4y agoDefault prompt gives me an arm, or feels like a crop of a full photo (iPhone 11, iOS 16.1) https://files.catbox.moe/ivy15m.PNG https://files.catbox.moe/ivy15m.PNG
- nier 4y agoSame here. Going to the maximum of 512 × 320 pixels on my device gives me the feeling that only more capable devices can see the whole picture. This is not a resolution setting but a crop setting.
- alex_suzuki 4y agoKudos. I hope liuliu has a good deal on egress traffic... :-)
- RBerenguel 4y agoIt works extremely fast on an iPad Pro M1 (kind of expected, but it's _impressive_) although the app is built as "iPhone only", and strangely enough the iPad is cropping the upscaled iPhone app so the lower bar of image controls don't show at all, which is a pity
- jamil7 4y agoReach out to the author, they can likely fix this easily.
- RBerenguel 4y agoYup, done. I thought the author would see it better here (also would make it visible for other people stumbling on the issue) but I have contacted separately explaining the issue.
- spideymans 4y agoThis is about to become this single most important app on my iPad Pro. It totally accelerates my workflow
- dirtyid 4y agoIs this something that might make it's way to tensor/pixel or unique to Apple silicon?
- donkeyd 4y agoThis is awesome. It has killed my battery over the last couple of hours, because I couldn't stop generating new images.
- cph123 4y agoWorks great on my iPhone 14 Pro Max
- holoduke 4y agoI cant wait till this is possible with 24fps and do live camera view modifications. That woud be insane during meetings :) I guess 5 more years.
- mistersquid 4y agoUpdate: Draw Things uses “One-time photo selection” which according to Settings > Privacy & Security > Photos “Even if your photos were recently shown to you to select from, the app did not have access to your photo library.” Still, I didn’t realize apps could save to Photos without explicitly asking permission. I don’t recall giving “Draw Things” permissions to access my photo library, yet the app is able to save to my photo library without prompting and able to read existing images. I may have misunderstood what permissions apps should ask for when saving to the photo library.
- liuliu 4y agoI use PHPickerViewController: https://developer.apple.com/documentation/photokit/phpickerviewcontroller https://developer.apple.com/documentation/photokit/phpickerv... which runs out of the process such that when you want to select a photo into the app, I have no access to any information about your other photos and the location info is erased from what PHPickerViewController passed to me. When save the photo, I only use UIImageWriteToSavedPhotosAlbum (https://developer.apple.com/documentation/uikit/1619125-uiimagewritetosavedphotosalbum https://developer.apple.com/documentation/uikit/1619125-uiim...) which asks you permission to write to the album, not read permission (they are separate). There are more things I can do if I have read permissions (like create a "Draw Things" collections and save to that, rather than save to generic Camera Roll). Ultimately decided to not do that because I don't want more permissions than I minimally absolutely need.
- mistersquid 4y agoThank you for the detailed explanation. I appreciate it.
- mrtranscendence 4y agoEven though I understood very little of that it was still wild fun reading it. I'm glad such wizards exist, because I and most people I know certainly don't qualify.
- simonh 4y agoThanks, Ive been looking forward to something like this coming out. Runs great on my iPad but a UI optimised for the form factor would be nice.
- TuringNYC 4y agoOn a related note, has anyone been able to utilize Apple silicon GPUs? Running CPU-only is incredibly slow (and sad, since i've got these Apple accelerators idle!)
- stoobs 4y agoAutomatic1111's stable diffusion web gui works with apple silicon: https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Installation-on-Apple-Silicon https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki...
- richrichardsson 4y agoThis repo is working well on my M1 Mini. https://github.com/invoke-ai/InvokeAI https://github.com/invoke-ai/InvokeAI
- ActionHank 4y agoI wonder how long before the appstore policies are updated to require models be embedded in apps.
- ajconway 4y agoApp Store has a CDN-like feature that allows uploading large resources separately and download them after running the app for the first time.
- ShamelessC 4y agoThe quality of comments here is absolutely abysmal given the deeply technical nature of the article.