9 ms·

Stable Diffusion with Core ML on Apple Silicon

723 points by 2bit 4y ago

neonate 4y ago
https://github.com/apple/ml-stable-diffusion https://github.com/apple/ml-stable-diffusion
- christiangenco 4y ago
  Oh gosh that's an intimidating installation process. I'll be much more interested when I can just `brew install` a binary.
  - MuffinFlavored 4y ago
    I could be wrong but I think part of the issue is this needs some large files for the trained dataset?
    - deleted 4y ago
      [deleted]
  - artimaeis 4y ago
    A bit different take is DiffusionBee, if you're curious to try it out in a GUI form. https://diffusionbee.com https://diffusionbee.com
    - aryamaan 4y ago
      does it use the optimised model for Apple chips?
      - belthesar 4y ago
        Not yet, likely, but the project is very active. I could see it coming quite soon.
      - Gigachad 4y ago
        I just tested that app and it was taking about 1s/it using the "Double quality, double time" version. Spat out quite nice images at 25 iterations. Way better than stuff I had tried before which looked worse after a minute than this generates in 25 seconds.
    - bredren 4y ago
      I’ve used this a fair amount but am not sure it’s much better place to begin than automatic1111, especially for the HN crowd.
      - Terretta 4y ago
        automatic1111 does have an M1 workaround in the wiki, but it is incorrect it's correct enough that if you know your way around a CLI, git, and package management you can figure it out
        serpix 4y ago
        It sucks to have to figure it out, any person who figures it out should submit a PR on the very outdated Apple Silicon readme.
        swyx 4y ago
        you cant send a PR on a wiki right? also wonder if anyone did a blogpost yet
    - Cyberdog 4y ago
      On the one hand, I appreciate the attempt to bring this stuff into the realm of "double click to run" boneheads like me, but on the other hand, I really despise Electron apps when they're multi-platform, where such use is somewhat understandable if still despicable. For a Mac-only app to use Electron… Why do they hate us so?
      - yboris 4y ago
        I'm baffled by continued hate on Electron. The option isn't between Electron and a lean OS-native application, but between Electron and nothing. I can build an Electron app in under a day with a pretty UI. It would take me several months to get anything sensible that is OS native. And I'm not going to sit down and learn the alternative. So please just say "thank you" to the developers that are sharing free things with you.
        Cyberdog 4y ago
        I would argue that shipping bad software is worse than shipping no software at all, yes. And it's impossible not to create bad software when you start with "it runs in a web browser, but it's not a web page." I say this as a web developer with over fifteen years of professional experience. Worst of all is the shamelessness, though. Don't Electron developers feel ashamed when they ship their products? Or have their brains been so muddled by this "JavaScript everywhere" mentality that they don't realize it's bad? Will future generations even know what a native application is anymore? This program suggests quitting other applications while it runs. Maybe that wouldn't be so necessary if it wasn't using a framework which needs like 2GB of memory before it can draw a window. I note that my OP hasn't been downvoted into oblivion as most of my critical HN posts are. I think there's at least a significant silent minority who agree with me on this one.
        throwaway675309 4y ago
        Developers just like any other inventive field have to balance time and work towards a good product. Just because the app is written using a chromium framework does not necessarily mean that it's written poorly, VS code is a great example of a fast performance application written in electron. I don't know where you're getting 2 GB of required memory but if you spin up an electron app it's rare that it requires more than 100 if it's not doing anything. If you knew anything about these types of stable diffusion interfaces you know that they basically have to load the entire model into memory so that's likely where the multiple gigabytes is coming from. A lot of us got into development work because we want to create new things, you sound more like the person who spends 99% of their time endlessly optimizing the game engine without actually remembering to build a compelling game experience. You're getting downvoted because your arrogant tone makes you sound like an insufferable bore.
  - thepasswordis 4y ago
    Where are you seeing the installation process?
  - gedy 4y ago
    > Oh gosh that's an intimidating installation process I'm not seeing any installation instructions on either link - what am I missing?
    - alexfromapex 4y ago
      All I had to do was: - create a virtual environment (Python 3.8.15 worked best) - upgrade pip - pip install wheel - pip install -r requirements.txt - and then, python setup.py install - Had to update my XCode to use the generated mlpackage files :/ - Expand drawer with instructions and follow them to download model and convert it to Core ML format - Run their CLI command as mentioned
      - philsnow 4y ago
        > Had to update my XCode to use the generated mlpackage files :/ I keep running into this, message is RuntimeError: Error compiling model: "Error reading protobuf spec. validator error: The model supplied is of version 7, intended for a newer version of Xcode. This version of Xcode supports model version 6 or earlier.". I upgraded XCode, tried re-installing the command line tools with various invocations of `sudo rm -rf /Library/Developer/CommandLineTools ; xcode-select --install` etc but still get the above message (thanks in advance, in case you see this and reply) edit: I see from https://github.com/apple/ml-stable-diffusion/issues/7 https://github.com/apple/ml-stable-diffusion/issues/7 that somebody upgraded to macos 13.0.1 and that fixed the issue for them. I've put off upgrading to Ventura so far and don't want to upgrade just to mess around with stable diffusion on m1, if it can be avoided.
        philsnow 4y ago
        I'm past the edit window, but: I'm a dope, I didn't see the quite clear "macos 13 or newer" requirement.
      - Cyberdog 4y ago
        Where did you get those instructions from? Is creating a virtual environment necessary if I'm fine with it running on my real system? I assume the environment part is what the "conda" commands on the GitHub repo readme are doing, but finding "conda" to install seems to be its own process. It's not on MacPorts, pip seems to only install a Python package instead of an executable, and getting a package from some other site feels sketchy. What is it with ML and Python, anyway? Why is this amazing new technology being shrouded in an ecosystem and language which… well, I guess if I can't say anything nice…
  - artdigital 4y ago
    Let's give it a few days and someone will have something semi-automatic ready
pkage 4y ago
How does this compare with using the Hugging Face `diffusers` package with MPS acceleration through PyTorch Nightly? I was under the impression that that used CoreML under the hood as well to convert the models so they ran on the Neural Engine.
- liuliu 4y ago
  It doesn't. MPS largely is on GPU. PyTorch's MPS implementation is incomplete a few weeks ago as well. This is about 3x faster.
  - wincy 4y ago
    Is it? I just ran it on my M1 MacBook Air and am getting 3 it/sec, same as I was using Stable Diffusion for M1. Maybe I'm doing something wrong?
    - liuliu 4y ago
      That's surprising to me, although I did the look about 3 weeks ago, and MPS support is a moving target. It is just M1 without Pro or Ultra right? Also, diffusers does support different backends other than PyTorch.
- deleted 4y ago
  [deleted]
cloogshicer 4y ago
I think it's sad that Apple doesn't even give attribution to any of the authors. If you copy the Bibtex from this site, the Author field is just empty. Their names are also not mentioned anywhere on this site. This site is purely a marketing effort.
- rvz 4y ago
  > I think it's sad that Apple doesn't even give attribution to any of the authors. Pretty much like Stable Diffusion and the grifters using it in general and they will never credit the artists and images that they stole to generate these images.
  - ClumsyPilot 4y ago
    Do your point is that Apple and those grifters are equally reputable? two wrongs don't make a right.
    - rvz 4y ago
      I'm neither defending Apple or the grifters using Stable Diffusion in my comment. Both are as bad as each other, giving no attribution or credit.
  - astrange 4y ago
    This is sort of like if you learned English from reading a book and the author said they owned all your English sentences after that. Of course you can see the original images (https://rom1504.github.io/clip-retrieval/ https://rom1504.github.io/clip-retrieval/), it was legal to collect them (they used robots.txt for consent just like Google Image Search) and it was legal to do this with them (but not using US legal principles since it's made in Germany). "Crediting the artist" isn't a legal principle - it's more like some kind of social media standard which is enforced by random amateur artists yelling at you if you don't do it. It's both impossible (there are no original artists for a given output) and wouldn't do anything to help the main social issue (future artists having their jobs taken by AIs).
noduerme 4y ago
Can anyone explain in relatively lay terms how Apple's neural cores differ from a GPU? If they can run stable diffusion so much faster, which normally runs on a GPU, why aren't they used to run shaders for AAA games?
- Synaesthesia 4y ago
  They're designed to run ML specific functions like matrix multiply and stuff. Nvidia has a similar idea in "tensor cores". I think because they're low but operations like 8 or 16 bit which is faster but too low res for GPU work.
siraben 4y ago
While running locally on an M1 Pro is nice, recently I've switched over to a Runpod[0] instance running Stable Diffusion instead. The main reasons being high workloads placed on the laptop degrade the battery faster and it takes ~40s to render a single image. On an A5000 it takes mere seconds to do 40 steps. The cost is around $0.2/hr. [0] https://runpod.io https://runpod.io
- Joe_Boogz 4y ago
  can't the battery problem be mitigated if you plug in your Macbook while running Stable Diffusion?
  - siraben 4y ago
    The laptop body still heats up and over long periods of time this can degrade the battery, I’ve measured a sharp drop in capacity from the device itself.
calrizien 4y ago
Where is the community for this project?
tosh 4y ago
Atila from Apple on the expected performance: > For distilled StableDiffusion 2 which requires 1 to 4 iterations instead of 50, the same M2 device should generate an image in <<1 second https://twitter.com/atiorh/status/1598399408160342039 https://twitter.com/atiorh/status/1598399408160342039
- cammikebrown 4y ago
  If you told me this was possible when I bought an M1 Pro less than a year ago, I wouldn’t believe you. This is insane.
  - ncr100 4y ago
    Agreed. And the posted benchmarks for the M2 Macbook Air make me consider 'upgrading' to an Air.
    - Terretta 4y ago
      That laptop feels like liquid power. It's uncanny. Macbook Airs (way back when) felt sluggish. The MBA M1 changed that, it was "fine". These M2s are unexpectedly responsive on an ongoing basis. The MacBook Pro M1 Max is great (would be fantastic except they lost a Thunderbolt port in favor of legacy HDMI and memory card jacks), but you expect that machine to be responsive, so it's less surprising. The Studio Ultra, though, never slows down for anything. Still, if the Air could drive two external screens instead of one, I'd "downgrade" from the Max.
      - jclardy 4y ago
        I'd give the M1 air more credit - I moved from a 2019 16" Pro to the Air and performance was nearly identical except for long running tasks (> 10 minutes.) So for mobile app builds, it was blazing fast. And in the meantime the intel machine was blaring fans after the first 30 seconds while the Air barely got warm.And then the real kicker was watching the battery on the intel machine visibly dropping a few percentage points, while the air sits at the same level the whole time. I've since moved to the M2 air, and it is noticeably faster than M1, but it isn't the huge leap from last gen intel that the M1 was. But the hardware itself feels way better.
mark_l_watson 4y ago
Great stuff. I like that they give directions for both Swift and Python This gets you text descriptions to images. I have seen models that given a picture, then generate similar pictures. I want this because while I have many pictures of my grandmothers, I only have a couple of pictures of my grandfathers and it would be nice to generate a few more. Core ML is so well done. A year ago I wrote a book on Swift AI and used Core ML in several examples.
- astrange 4y ago
  That’s DreamBooth. There are some services that will do it for you.
  - mark_l_watson 4y ago
    Thanks!
    - mromanuk 4y ago
      I’m making one of those services, if you are interested, please reach me at my email. I would like to know what you have in mind regarding your grandmothers
behnamoh 4y ago
This may sound naive, but what are some use cases of running SD models locally? If the free/cheap options exist (like running SD on powerful servers), then what's the advantage of this new method?
- tosh 4y ago
  Works offline, privacy, independent of SaaS (API stability, longevity, …). I'm sure there are more.
- gjsman-1000 4y ago
  Powerful servers with GPUs are expensive. Laptops you already own, aren't.
- sofaygo 4y ago
  > There are a number of reasons why on-device deployment of Stable Diffusion in an app is preferable to a server-based approach. First, the privacy of the end user is protected because any data the user provided as input to the model stays on the user's device. Second, after initial download, users don’t require an internet connection to use the model. Finally, locally deploying this model enables developers to reduce or eliminate their server-related costs.
  - huggingmouth 4y ago
    Stability! The main reason why I use it locally is because I don't want some random dev unilaterally deciding to change or "sunsetting" features I rely on. Centralized services small and large are guilty of this and I'm sick of it.
- yazaddaruvala 4y ago
  "Hey Siri, draw me a purple duck" and it all happens without an internet connection! If you mean monetary usecases: Roughly something like Photoshop/Blender/UnrealEngine with ML plugins that are low latency, private, and $0 server hosting costs.
- jwitthuhn 4y ago
deleted 4y ago
[deleted]
zimpenfish 4y ago
Man, this takes a ton of room to do the CoreML conversions - ran out of space doing the unet conversion even though I started with 25GB free. Going on a delete spree to get it up to 50GB free before trying again.
- pyinstallwoes 4y ago
  How much space do you have and how much do you try to keep free? I get freaked out if I have less than 400gb free.
  - zimpenfish 4y ago
    /dev/disk3s5 926Gi 857Gi 52Gi 95% 8067489 540828800 1% /System/Volumes/Data It normally hovers around 30-35Gi free.
- password4321 4y ago
  All hail Grand Perspective back in the day, not sure who is carrying the "what's wasting my disk space" torch for free these days. Edit: still alive! https://grandperspectiv.sourceforge.net/ https://grandperspectiv.sourceforge.net/
  - jtbayly 4y ago
    Just used this again on 3 different computers, including mine. Works fantastically still. Found a >100GB accidental “livestream” recording on one computer. Would have taken forever to find what was taking up all the room otherwise.
  - peddling-brink 4y ago
    ncdu is the best in my book. TUI, supports deletion of files and folders, and very simple to understand. GUI apps for this task like GP and the like are more visually complex than they need to be.
    - password4321 4y ago
sorenjan 4y ago
How come you always have to install some version of pytorch or tensor flow to run these ml models? When I'm only doing inference shouldn't there be easier ways of doing that, with automatic hardware selection etc. Why aren't models distributed in a standard format like onnx, and inference on different platforms solved once per platform?
- 0x008 4y ago
  In the professional context (apart of individual apps distributed by small creators / indiehackers) usually models are run using standardized runtimes in native code (C++ usually), using runtimes TensorRT (for Nvidia Devices), onnxruntime (agnostic), etc.
- GeekyBear 4y ago
  >How come you always have to install some version of pytorch or tensor flow to run these ml models? The repo is aimed at developers and has two parts. The first adapts the ML model to run on Apple Silicon (CPU, GPU, Neural Engine), and the second allows you to easily add Stable Diffusion functionality to your own app. If you just want an end user app, those already exist, but now it will be easier to make ones that take advantage of Apple's dedicated ML hardware as well as the CPU and GPU. >This repository comprises: python_coreml_stable_diffusion, a Python package for converting PyTorch models to Core ML format and performing image generation with Hugging Face diffusers in Python StableDiffusion, a Swift package that developers can add to their Xcode projects as a dependency to deploy image generation capabilities in their apps. The Swift package relies on the Core ML model files generated by python_coreml_stable_diffusion https://github.com/apple/ml-stable-diffusion https://github.com/apple/ml-stable-diffusion
- pmarreck 4y ago
  DiffusionBee is an app that is completely self-contained and lets you play with this stuff completely trivially, no installs required. https://diffusionbee.com/ https://diffusionbee.com/
  - janandonly 4y ago
darkteflon 4y ago
For the uninitiated, which MacOS GUI app is this library most likely to show up in first/best? DiffusionBee?
- pksebben 4y ago
  automatic111's webui typically gets the most frequent updates. Middling easy to install.
  - darkteflon 4y ago
    Great, thank you. Look like there’s already a GH issue: https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/5309 https://github.com/AUTOMATIC1111/stable-diffusion-webui/issu...
syspec 4y ago
There's also https://draw.nnc.ai/ https://draw.nnc.ai/ - which is an iOS / iPad app running Stable Diffusion. The author has a detailed blogpost outlining how he modified the model to use Metal on iOS devices. https://liuliu.me/eyes/stretch-iphone-to-its-limit-a-2gib-model-that-can-draw-everything-in-your-pocket/ https://liuliu.me/eyes/stretch-iphone-to-its-limit-a-2gib-mo...
- antal 4y ago
  Yeah, that's what immediately came to mind for me as well. I don't know how similar/different the two solutions are, but it made me smile a bit that what Apple is showing off here has been already done by a single independent developer :)
tamersalama 4y ago
I can't get fine-tune the model ron Apple Silicon due to PyTorch supportability issues. I don't have high-hopes it will be supported. https://github.com/pytorch/pytorch/issues/77794 https://github.com/pytorch/pytorch/issues/77794 https://github.com/pytorch/pytorch/issues/77764 https://github.com/pytorch/pytorch/issues/77764
tomr75 4y ago
anyone know how to link this to a GUI?
Viluskaran 4y ago
8 gb ram
- Synaesthesia 4y ago
  What about it?
wellthisisgreat 4y ago
Macbook Air M1 / 16GB RAM took 3.56 to generate an image, this is pretty wild
- zimpenfish 4y ago
  > 3.56 to generate an image 3.56 seconds?
  - wellthisisgreat 4y ago
    ah 3.56 minutes, my mistake
personjerry 4y ago
Can't wait to see this integrated into automatic1111 so I can use it as a normie
wilsongoode 4y ago
I’ve been using InvokeAI: https://github.com/invoke-ai/InvokeAI https://github.com/invoke-ai/InvokeAI Great support for M1, basically since the beginning. The install is painless. Release video for InvokeAI 2.2: https://www.youtube.com/watch?v=hIYBfDtKaus https://www.youtube.com/watch?v=hIYBfDtKaus
dustedcodes 4y ago
What are some good resources to get into working with this and learning the basics around ML to get some fundamental understanding of how this works?
- videlov 4y ago
  I found the blog posts by Jay Alammar to be particularly good. Here are my starting suggestions (in this order) — https://jalammar.github.io/illustrated-word2vec/ https://jalammar.github.io/illustrated-word2vec/ https://jalammar.github.io/illustrated-transformer/ https://jalammar.github.io/illustrated-transformer/ https://jalammar.github.io/illustrated-bert/ https://jalammar.github.io/illustrated-bert/ https://jalammar.github.io/illustrated-stable-diffusion/ https://jalammar.github.io/illustrated-stable-diffusion/
joss82 4y ago
Would it be possible to run 2 SD instances in parallel on a single M1/M2 chip? One on the GPU and another on the ML core?

723 points by 2bit 4y ago

neonate 4y ago

https://github.com/apple/ml-stable-diffusion https://github.com/apple/ml-stable-diffusion

christiangenco 4y ago
Oh gosh that's an intimidating installation process. I'll be much more interested when I can just `brew install` a binary.
- MuffinFlavored 4y ago
  I could be wrong but I think part of the issue is this needs some large files for the trained dataset?
  - deleted 4y ago
    [deleted]
- artimaeis 4y ago
  A bit different take is DiffusionBee, if you're curious to try it out in a GUI form. https://diffusionbee.com https://diffusionbee.com
  - aryamaan 4y ago
    does it use the optimised model for Apple chips?
    - belthesar 4y ago
      Not yet, likely, but the project is very active. I could see it coming quite soon.
    - Gigachad 4y ago
      I just tested that app and it was taking about 1s/it using the "Double quality, double time" version. Spat out quite nice images at 25 iterations. Way better than stuff I had tried before which looked worse after a minute than this generates in 25 seconds.
  - bredren 4y ago
    I’ve used this a fair amount but am not sure it’s much better place to begin than automatic1111, especially for the HN crowd.
    - Terretta 4y ago
      automatic1111 does have an M1 workaround in the wiki, but it is incorrect it's correct enough that if you know your way around a CLI, git, and package management you can figure it out
      - serpix 4y ago
        It sucks to have to figure it out, any person who figures it out should submit a PR on the very outdated Apple Silicon readme.
        swyx 4y ago
        you cant send a PR on a wiki right? also wonder if anyone did a blogpost yet
  - Cyberdog 4y ago
    On the one hand, I appreciate the attempt to bring this stuff into the realm of "double click to run" boneheads like me, but on the other hand, I really despise Electron apps when they're multi-platform, where such use is somewhat understandable if still despicable. For a Mac-only app to use Electron… Why do they hate us so?
    - yboris 4y ago
      I'm baffled by continued hate on Electron. The option isn't between Electron and a lean OS-native application, but between Electron and nothing. I can build an Electron app in under a day with a pretty UI. It would take me several months to get anything sensible that is OS native. And I'm not going to sit down and learn the alternative. So please just say "thank you" to the developers that are sharing free things with you.
      - Cyberdog 4y ago
        I would argue that shipping bad software is worse than shipping no software at all, yes. And it's impossible not to create bad software when you start with "it runs in a web browser, but it's not a web page." I say this as a web developer with over fifteen years of professional experience. Worst of all is the shamelessness, though. Don't Electron developers feel ashamed when they ship their products? Or have their brains been so muddled by this "JavaScript everywhere" mentality that they don't realize it's bad? Will future generations even know what a native application is anymore? This program suggests quitting other applications while it runs. Maybe that wouldn't be so necessary if it wasn't using a framework which needs like 2GB of memory before it can draw a window. I note that my OP hasn't been downvoted into oblivion as most of my critical HN posts are. I think there's at least a significant silent minority who agree with me on this one.
        throwaway675309 4y ago
        Developers just like any other inventive field have to balance time and work towards a good product. Just because the app is written using a chromium framework does not necessarily mean that it's written poorly, VS code is a great example of a fast performance application written in electron. I don't know where you're getting 2 GB of required memory but if you spin up an electron app it's rare that it requires more than 100 if it's not doing anything. If you knew anything about these types of stable diffusion interfaces you know that they basically have to load the entire model into memory so that's likely where the multiple gigabytes is coming from. A lot of us got into development work because we want to create new things, you sound more like the person who spends 99% of their time endlessly optimizing the game engine without actually remembering to build a compelling game experience. You're getting downvoted because your arrogant tone makes you sound like an insufferable bore.
- thepasswordis 4y ago
  Where are you seeing the installation process?
- gedy 4y ago
  > Oh gosh that's an intimidating installation process I'm not seeing any installation instructions on either link - what am I missing?
  - alexfromapex 4y ago
    All I had to do was: - create a virtual environment (Python 3.8.15 worked best) - upgrade pip - pip install wheel - pip install -r requirements.txt - and then, python setup.py install - Had to update my XCode to use the generated mlpackage files :/ - Expand drawer with instructions and follow them to download model and convert it to Core ML format - Run their CLI command as mentioned
    - philsnow 4y ago
      > Had to update my XCode to use the generated mlpackage files :/ I keep running into this, message is RuntimeError: Error compiling model: "Error reading protobuf spec. validator error: The model supplied is of version 7, intended for a newer version of Xcode. This version of Xcode supports model version 6 or earlier.". I upgraded XCode, tried re-installing the command line tools with various invocations of `sudo rm -rf /Library/Developer/CommandLineTools ; xcode-select --install` etc but still get the above message (thanks in advance, in case you see this and reply) edit: I see from https://github.com/apple/ml-stable-diffusion/issues/7 https://github.com/apple/ml-stable-diffusion/issues/7 that somebody upgraded to macos 13.0.1 and that fixed the issue for them. I've put off upgrading to Ventura so far and don't want to upgrade just to mess around with stable diffusion on m1, if it can be avoided.
      - philsnow 4y ago
        I'm past the edit window, but: I'm a dope, I didn't see the quite clear "macos 13 or newer" requirement.
    - Cyberdog 4y ago
      Where did you get those instructions from? Is creating a virtual environment necessary if I'm fine with it running on my real system? I assume the environment part is what the "conda" commands on the GitHub repo readme are doing, but finding "conda" to install seems to be its own process. It's not on MacPorts, pip seems to only install a Python package instead of an executable, and getting a package from some other site feels sketchy. What is it with ML and Python, anyway? Why is this amazing new technology being shrouded in an ecosystem and language which… well, I guess if I can't say anything nice…
- artdigital 4y ago
  Let's give it a few days and someone will have something semi-automatic ready

pkage 4y ago

How does this compare with using the Hugging Face `diffusers` package with MPS acceleration through PyTorch Nightly? I was under the impression that that used CoreML under the hood as well to convert the models so they ran on the Neural Engine.

liuliu 4y ago
It doesn't. MPS largely is on GPU. PyTorch's MPS implementation is incomplete a few weeks ago as well. This is about 3x faster.
- wincy 4y ago
  Is it? I just ran it on my M1 MacBook Air and am getting 3 it/sec, same as I was using Stable Diffusion for M1. Maybe I'm doing something wrong?
  - liuliu 4y ago
    That's surprising to me, although I did the look about 3 weeks ago, and MPS support is a moving target. It is just M1 without Pro or Ultra right? Also, diffusers does support different backends other than PyTorch.
deleted 4y ago
[deleted]

cloogshicer 4y ago

I think it's sad that Apple doesn't even give attribution to any of the authors. If you copy the Bibtex from this site, the Author field is just empty. Their names are also not mentioned anywhere on this site. This site is purely a marketing effort.

rvz 4y ago
> I think it's sad that Apple doesn't even give attribution to any of the authors. Pretty much like Stable Diffusion and the grifters using it in general and they will never credit the artists and images that they stole to generate these images.
- ClumsyPilot 4y ago
  Do your point is that Apple and those grifters are equally reputable? two wrongs don't make a right.
  - rvz 4y ago
    I'm neither defending Apple or the grifters using Stable Diffusion in my comment. Both are as bad as each other, giving no attribution or credit.
- astrange 4y ago
  This is sort of like if you learned English from reading a book and the author said they owned all your English sentences after that. Of course you can see the original images (https://rom1504.github.io/clip-retrieval/ https://rom1504.github.io/clip-retrieval/), it was legal to collect them (they used robots.txt for consent just like Google Image Search) and it was legal to do this with them (but not using US legal principles since it's made in Germany). "Crediting the artist" isn't a legal principle - it's more like some kind of social media standard which is enforced by random amateur artists yelling at you if you don't do it. It's both impossible (there are no original artists for a given output) and wouldn't do anything to help the main social issue (future artists having their jobs taken by AIs).

noduerme 4y ago

Can anyone explain in relatively lay terms how Apple's neural cores differ from a GPU? If they can run stable diffusion so much faster, which normally runs on a GPU, why aren't they used to run shaders for AAA games?

Synaesthesia 4y ago
They're designed to run ML specific functions like matrix multiply and stuff. Nvidia has a similar idea in "tensor cores". I think because they're low but operations like 8 or 16 bit which is faster but too low res for GPU work.

siraben 4y ago

While running locally on an M1 Pro is nice, recently I've switched over to a Runpod[0] instance running Stable Diffusion instead. The main reasons being high workloads placed on the laptop degrade the battery faster and it takes ~40s to render a single image. On an A5000 it takes mere seconds to do 40 steps. The cost is around $0.2/hr. [0] https://runpod.io https://runpod.io

Joe_Boogz 4y ago
can't the battery problem be mitigated if you plug in your Macbook while running Stable Diffusion?
- siraben 4y ago
  The laptop body still heats up and over long periods of time this can degrade the battery, I’ve measured a sharp drop in capacity from the device itself.

calrizien 4y ago

Where is the community for this project?

tosh 4y ago

Atila from Apple on the expected performance: > For distilled StableDiffusion 2 which requires 1 to 4 iterations instead of 50, the same M2 device should generate an image in <<1 second https://twitter.com/atiorh/status/1598399408160342039 https://twitter.com/atiorh/status/1598399408160342039

cammikebrown 4y ago
If you told me this was possible when I bought an M1 Pro less than a year ago, I wouldn’t believe you. This is insane.
- ncr100 4y ago
  Agreed. And the posted benchmarks for the M2 Macbook Air make me consider 'upgrading' to an Air.
  - Terretta 4y ago
    That laptop feels like liquid power. It's uncanny. Macbook Airs (way back when) felt sluggish. The MBA M1 changed that, it was "fine". These M2s are unexpectedly responsive on an ongoing basis. The MacBook Pro M1 Max is great (would be fantastic except they lost a Thunderbolt port in favor of legacy HDMI and memory card jacks), but you expect that machine to be responsive, so it's less surprising. The Studio Ultra, though, never slows down for anything. Still, if the Air could drive two external screens instead of one, I'd "downgrade" from the Max.
    - jclardy 4y ago
      I'd give the M1 air more credit - I moved from a 2019 16" Pro to the Air and performance was nearly identical except for long running tasks (> 10 minutes.) So for mobile app builds, it was blazing fast. And in the meantime the intel machine was blaring fans after the first 30 seconds while the Air barely got warm.And then the real kicker was watching the battery on the intel machine visibly dropping a few percentage points, while the air sits at the same level the whole time. I've since moved to the M2 air, and it is noticeably faster than M1, but it isn't the huge leap from last gen intel that the M1 was. But the hardware itself feels way better.

mark_l_watson 4y ago

Great stuff. I like that they give directions for both Swift and Python This gets you text descriptions to images. I have seen models that given a picture, then generate similar pictures. I want this because while I have many pictures of my grandmothers, I only have a couple of pictures of my grandfathers and it would be nice to generate a few more. Core ML is so well done. A year ago I wrote a book on Swift AI and used Core ML in several examples.

astrange 4y ago
That’s DreamBooth. There are some services that will do it for you.
- mark_l_watson 4y ago
  Thanks!
  - mromanuk 4y ago
    I’m making one of those services, if you are interested, please reach me at my email. I would like to know what you have in mind regarding your grandmothers

behnamoh 4y ago

This may sound naive, but what are some use cases of running SD models locally? If the free/cheap options exist (like running SD on powerful servers), then what's the advantage of this new method?

tosh 4y ago
Works offline, privacy, independent of SaaS (API stability, longevity, …). I'm sure there are more.
gjsman-1000 4y ago
Powerful servers with GPUs are expensive. Laptops you already own, aren't.
sofaygo 4y ago
> There are a number of reasons why on-device deployment of Stable Diffusion in an app is preferable to a server-based approach. First, the privacy of the end user is protected because any data the user provided as input to the model stays on the user's device. Second, after initial download, users don’t require an internet connection to use the model. Finally, locally deploying this model enables developers to reduce or eliminate their server-related costs.
- huggingmouth 4y ago
  Stability! The main reason why I use it locally is because I don't want some random dev unilaterally deciding to change or "sunsetting" features I rely on. Centralized services small and large are guilty of this and I'm sick of it.
yazaddaruvala 4y ago
"Hey Siri, draw me a purple duck" and it all happens without an internet connection! If you mean monetary usecases: Roughly something like Photoshop/Blender/UnrealEngine with ML plugins that are low latency, private, and $0 server hosting costs.
jwitthuhn 4y ago

deleted 4y ago

[deleted]

zimpenfish 4y ago

Man, this takes a ton of room to do the CoreML conversions - ran out of space doing the unet conversion even though I started with 25GB free. Going on a delete spree to get it up to 50GB free before trying again.

pyinstallwoes 4y ago
How much space do you have and how much do you try to keep free? I get freaked out if I have less than 400gb free.
- zimpenfish 4y ago
  /dev/disk3s5 926Gi 857Gi 52Gi 95% 8067489 540828800 1% /System/Volumes/Data It normally hovers around 30-35Gi free.
password4321 4y ago
All hail Grand Perspective back in the day, not sure who is carrying the "what's wasting my disk space" torch for free these days. Edit: still alive! https://grandperspectiv.sourceforge.net/ https://grandperspectiv.sourceforge.net/
- jtbayly 4y ago
  Just used this again on 3 different computers, including mine. Works fantastically still. Found a >100GB accidental “livestream” recording on one computer. Would have taken forever to find what was taking up all the room otherwise.
- peddling-brink 4y ago
  ncdu is the best in my book. TUI, supports deletion of files and folders, and very simple to understand. GUI apps for this task like GP and the like are more visually complex than they need to be.
  - password4321 4y ago

sorenjan 4y ago

How come you always have to install some version of pytorch or tensor flow to run these ml models? When I'm only doing inference shouldn't there be easier ways of doing that, with automatic hardware selection etc. Why aren't models distributed in a standard format like onnx, and inference on different platforms solved once per platform?

0x008 4y ago
In the professional context (apart of individual apps distributed by small creators / indiehackers) usually models are run using standardized runtimes in native code (C++ usually), using runtimes TensorRT (for Nvidia Devices), onnxruntime (agnostic), etc.
GeekyBear 4y ago
>How come you always have to install some version of pytorch or tensor flow to run these ml models? The repo is aimed at developers and has two parts. The first adapts the ML model to run on Apple Silicon (CPU, GPU, Neural Engine), and the second allows you to easily add Stable Diffusion functionality to your own app. If you just want an end user app, those already exist, but now it will be easier to make ones that take advantage of Apple's dedicated ML hardware as well as the CPU and GPU. >This repository comprises: python_coreml_stable_diffusion, a Python package for converting PyTorch models to Core ML format and performing image generation with Hugging Face diffusers in Python StableDiffusion, a Swift package that developers can add to their Xcode projects as a dependency to deploy image generation capabilities in their apps. The Swift package relies on the Core ML model files generated by python_coreml_stable_diffusion https://github.com/apple/ml-stable-diffusion https://github.com/apple/ml-stable-diffusion
pmarreck 4y ago
DiffusionBee is an app that is completely self-contained and lets you play with this stuff completely trivially, no installs required. https://diffusionbee.com/ https://diffusionbee.com/
- janandonly 4y ago

darkteflon 4y ago

For the uninitiated, which MacOS GUI app is this library most likely to show up in first/best? DiffusionBee?

pksebben 4y ago
automatic111's webui typically gets the most frequent updates. Middling easy to install.
- darkteflon 4y ago
  Great, thank you. Look like there’s already a GH issue: https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/5309 https://github.com/AUTOMATIC1111/stable-diffusion-webui/issu...

syspec 4y ago

There's also https://draw.nnc.ai/ https://draw.nnc.ai/ - which is an iOS / iPad app running Stable Diffusion. The author has a detailed blogpost outlining how he modified the model to use Metal on iOS devices. https://liuliu.me/eyes/stretch-iphone-to-its-limit-a-2gib-model-that-can-draw-everything-in-your-pocket/ https://liuliu.me/eyes/stretch-iphone-to-its-limit-a-2gib-mo...

antal 4y ago
Yeah, that's what immediately came to mind for me as well. I don't know how similar/different the two solutions are, but it made me smile a bit that what Apple is showing off here has been already done by a single independent developer :)

tamersalama 4y ago

I can't get fine-tune the model ron Apple Silicon due to PyTorch supportability issues. I don't have high-hopes it will be supported. https://github.com/pytorch/pytorch/issues/77794 https://github.com/pytorch/pytorch/issues/77794 https://github.com/pytorch/pytorch/issues/77764 https://github.com/pytorch/pytorch/issues/77764

tomr75 4y ago

anyone know how to link this to a GUI?

Viluskaran 4y ago

8 gb ram

Synaesthesia 4y ago
What about it?

wellthisisgreat 4y ago

Macbook Air M1 / 16GB RAM took 3.56 to generate an image, this is pretty wild

zimpenfish 4y ago
> 3.56 to generate an image 3.56 seconds?
- wellthisisgreat 4y ago
  ah 3.56 minutes, my mistake

personjerry 4y ago

Can't wait to see this integrated into automatic1111 so I can use it as a normie

wilsongoode 4y ago

I’ve been using InvokeAI: https://github.com/invoke-ai/InvokeAI https://github.com/invoke-ai/InvokeAI Great support for M1, basically since the beginning. The install is painless. Release video for InvokeAI 2.2: https://www.youtube.com/watch?v=hIYBfDtKaus https://www.youtube.com/watch?v=hIYBfDtKaus

dustedcodes 4y ago

What are some good resources to get into working with this and learning the basics around ML to get some fundamental understanding of how this works?

videlov 4y ago
I found the blog posts by Jay Alammar to be particularly good. Here are my starting suggestions (in this order) — https://jalammar.github.io/illustrated-word2vec/ https://jalammar.github.io/illustrated-word2vec/ https://jalammar.github.io/illustrated-transformer/ https://jalammar.github.io/illustrated-transformer/ https://jalammar.github.io/illustrated-bert/ https://jalammar.github.io/illustrated-bert/ https://jalammar.github.io/illustrated-stable-diffusion/ https://jalammar.github.io/illustrated-stable-diffusion/

joss82 4y ago

Would it be possible to run 2 SD instances in parallel on a single M1/M2 chip? One on the GPU and another on the ML core?