6 ms·

Show HN: Wall-mounted diffusion mirror that turns reflections into paintings

472 points by cataPhil 2y ago

amelius 2y ago
Prize for the most power-hungry mirror. I also notice that the paintings are not stable over small changes in the inputs.
- shermantanktop 2y ago
  To me, that’s the charm.
  - amelius 2y ago
    How about changing the style every time the viewer blinks?
jaredhallen 2y ago
It would be cool to intercept the input from a webcam, transform it like this, and then present it to the OS as another virtual camera so you could select it in Teams or Zoom or whatever.
- noman-land 2y ago
  You can do this with OBS.
broast 2y ago
Super cool! I've had similar ideas when I first saw StreamDiffusion. The possibilities are endless
shermantanktop 2y ago
Have you considered a high frame rate morph effect between images? That would increase the effective frame rate and probably would look pretty cool.
- JKCalhoun 2y ago
  Even a simple crossfade.
- doctorhandshake 2y ago
  That’s the technique I used on a piece that does inpainting across an image at a rate of about 1 image every 8 seconds - I ‘melt’ the results in for the duration until the next patch is ready: https://hardwork.party/aws-epoch-optimizer/ https://hardwork.party/aws-epoch-optimizer/
- lancesells 2y ago
  I actually think the high frame rate will look worse unless the styling changes less in between frames. Otherwise, it's going to look erratic and there will be a ton of "popping" if each frame changes as much as the frames in the current video. Or maybe op wants it to be a chaotic feeling.
  - flir 2y ago
    Many years ago I did a couple of hundred variations on the mona lisa in photoshop, and stacked them as a gif. As the eyes remained unchanged in each frame, they gave a fixed point you could focus on while the rest of the face strobed chaotically. It was a nice effect.
theendisney4 2y ago
Sounds cool! I would like pictures with people in the room doing weird things energetically like stand on their chair. Dancing. Confetti booze strange outfits. Bordering the offensive then deleted after 30 minutes.
autoexec 2y ago
It's pretty neat, but I'd never have a webcam steaming everything in view to someone else's server on the internet. It'd be cool if the server ran on my own hardware, and ideally in the frame itself. The privacy policy at runpod.io wouldn't even display in my browsers. (their ToS loaded without an issue).
- beeflet 2y ago
  I wonder if you could do something like this with hardware acceleration (like google coral)
  - refulgentis 2y ago
    Unfortunately no, they only have a TensorFlow target and none of the hobbyist / released big model image stuff works with it
imsaw 2y ago
I think the low framerate adds a unique ambience, at least to me, watching the recorded videos. It gives time to appreciate the art style and generated features. It also feels like a time lapse of gradual changes. I could imagine it being annoying to look at in real time though.
tsunamifury 2y ago
Micasso
shib71 2y ago
> Art is ... mostly about surfacing the inner world, and only in part about skill. I like the phrasing of the first part. But what art is "about" is very subjective. For me, part of what I look for in art is intentionality - the notion that the artist has crafted each element toward a purpose, consciously or not. The less an artist contributes to the final piece, the less meaning I assign to it. In this case: I would say that the individual pictures being displayed are not "art" - they have no meaning. But I think the device in whole is a piece of art. That is a creation that surfaces the creator's inner world, because they designed the device, wrote the code, crafted the prompts to achieve pieces that reflected their notion of beauty.
- disqard 2y ago
  I love this framing (pun unintended) of "art". Besides what you articulated as the "intention", I often think of the "story" behind the art -- whether an idea in the creator's head was expressed via the piece (or not) makes me go "yes, this is Art" (or not). By that token, when I see automated projects like this, I think of the "installation" as art, but the pixels or arrangements generated by the piece itself is less art-like IMO.
- d0gsg0w00f 2y ago
  Yes. I think we attribute the cost of the human sacrifice into the value of the art. It's a like owning 1/80th of a human soul and hanging it on the wall.
viraptor 2y ago
> It’s unfortunate that art selects so strongly for skill. Can we decorrelate the two? I really like this direction. I understand why some object to the genai approaches, but in practice sometimes I get an idea of something cool and don't have the skills to create it myself. I'm not going to invest months/years to create each of those ideas and they're not important enough to spend hundreds of dollars that a skilled artist would request. Now there's a way people can try generating the thing and may end up enjoying it - and that's great. (At least for personal use, it gets a bit complicated for commercial purposes)
- happymellon 2y ago
  But unfortunately they did not. Not everyone could build what is presented here, and inadvertently they have validated that it requires skill to produce a machine that distorts your reality. The pictures being produced isn't the art.
- d0gsg0w00f 2y ago
  This is assuming an unimplemented idea has value. I think the value we attribute to an idea is actually the dissemination of the idea to others. Often a sufficiently radical idea is difficult for others to grasp and therefore an example must be created by the originator of the idea. So when someone says "That's a great idea" what they mean is "That's great work". Time will tell if others see an idea backed by AI work as valuable. Can they even tell? Who knows.
deleted 2y ago
[deleted]
rnxrx 2y ago
This really does change the interaction with art. As a future expansion it might be neat to recognize images on camera that would make for interesting art (i.e. detection of people/animals or recognition of certain styles of composition) as well as being able to choose amongst different styles. It seems sort of akin to some modern art that incorporated TV screens and video to make dynamic installations, like Nam June Paik.
r00fus 2y ago
Is there any way to freeze some of the images? I thought some of them would be awesome to keep (e.g. pfp or lockscreen photo).
melony 2y ago
Some optimization suggestions: - cache the prompting somehow, unless you are doing dynamic stuff with the prompts, the language embeddings generated should be static (this depends on the architecture of the model that you are using, it's only possible with certain setups where the language processing is a separate part in the pipeline) - consider fine-tuning an img to img model with your current outputs instead of using a language-coupled model. My intuition is that this is currently significantly over-engineered on the ML side. - Play around with local hardware acceleration instead of sending everything to the cloud, you also probably don't need particularly high resolution for the images either.
- cataPhil 2y ago
  Love it thanks, will look into it!
caelinsutch 2y ago
how much does this end up costing to run?
deleted 2y ago
[deleted]
GradientSurfer 2y ago
Nice! I peeked at the code and thought I’d share a few tips for improving the low frame rate: Base64 encoding the JPEG bytes will increase payload size up to ~30% and burns CPU cycles on both client and server. This is unnecessary, as Websocket protocol can send binary payloads (doesn’t need to be text). Consider removing lossy jpg compression as well, ie just send the raw RGB bytes over the network. Then on the server side you can simply call Image.frombuffer(…). StreamDiffusion can achieve high frame rates because of extensive batching in the pipeline. You’re not benefiting from that here as the client is only sending one frame at a time and then waiting for a response. See this example for an idea of how to queue input frames and consume them in batches https://github.com/cumulo-autumn/StreamDiffusion/blob/main/examples/screen/main.py https://github.com/cumulo-autumn/StreamDiffusion/blob/main/e... . Alternatively you could take a look at the SDXL Turbo and Lightning models. They are very fast at img2img but have limited resolution of 512² or 1024² pixels respectively. Which appears a bit lower than what you’re aiming for here, but they can be run locally in real time on a high end consumer grade GPU. For reference I have some code demonstrating this here https://github.com/GradientSurfer/Draw2Img/tree/main https://github.com/GradientSurfer/Draw2Img/tree/main
- MrLeap 2y ago
  yeah yeah yeah, do all these things, and afterwards, look at 2d interpolation methods that don't require AI for your inbetweens. There's some real fast kernel math that can lerp from one blob to another at 8 billion fps.
  - enjeyw 2y ago
    I think you’re getting downvoted because “yeah yeah yeah” is normally a sign that someone is sarcastically dismissing an idea, but the rest of your comment suggests you’re not at all - linerp is a great idea!
    - cataPhil 2y ago
      These are great ideas thank you!
    - 2y ago
randmeerkat 2y ago
> It’s unfortunate that art selects so strongly for skill. Jackson Pollock begs to differ.
- aspenmayer 2y ago
  Artistic skill lies not merely in the hand, but also in the eye.
  - randmeerkat 2y ago
    > Artistic skill lies not merely in the hand, but also in the eye. Precisely. There is no skill in artistry, it’s rather just a developed sense of style, that doesn’t come from a medium or method, it comes from growing to know oneself.
    - aspenmayer 2y ago
      I agree. I think the distinction between arts and crafts is largely one of utility, rather than skill or technique.
      - randmeerkat 2y ago
        > I agree. I think the distinction between arts and crafts is largely one of utility, rather than skill or technique. Which is why I take offense to someone claiming that it selects for and requires skill. Especially because the ones making that argument are usually the ones arguing that they’re incapable of making art. I would tell them they just need to spend more time finding themselves.
        aspenmayer 2y ago
        > Which is why I take offense to someone claiming that it selects for and requires skill. Especially because the ones making that argument are usually the ones arguing that they’re incapable of making art. I would tell them they just need to spend more time finding themselves. And spend more time (and intentional effort) in making art! It's like people want a shortcut to end result, when "real" artists know that the process of self-discovery is the reason and means through which they make the art in the first place. The map is not the territory. The purpose of the journey is the journey itself; the destination is simply a guiding star. A whole lot of aspirational mapmakers think if they only had "this one weird trick," they'd be gods.
malux85 2y ago
Let’s hook it up to a sentiment and toxicity model, and then if you’re being too negative online, it will start to deform you into a monster, we could have a real Picture of Dorian Gray!
- A4ET8a8uTh0 2y ago
  I chuckled, but that is actually an interesting ( and very doable ) fun project idea.
- CaptainFever 2y ago
  This is actually such a cool idea.
bravura 2y ago
FYI, the link to the frame used to mount the display just blocks you: https://www.leroymerlin.fr/produits/decoration-eclairage/decoration-murale/cadre-photo/cadre-noir/cadre-milo-21-x-29-7-cm-noir-inspire-71670942.html https://www.leroymerlin.fr/produits/decoration-eclairage/dec... For those curious, it's a MILO 21 x 29.7 cm black frame. These links work for me: https://www.leroymerlin.pt/produtos/decoracao-e-tapetes/molduras/molduras/moldura-com-passe-partout/moldura-multifotos-inspire-milo-21-x-29-7-cm-preto-19412743.html https://www.leroymerlin.pt/produtos/decoracao-e-tapetes/mold... Also, the screen he uses (HMTECH Raspberry Pi Screen 10.1) is pretty hard to find. Do people have other good recommendations for screens with similar quality and specs? Any idea why this uses infrared light and an infrared camera?
- esperent 2y ago
  Every time I've looked into building something like this, when I price it out, just using an old Android tablet in kiosk mode and making it a web app ends up being much cheaper.
- piva00 2y ago
  > Any idea why this uses infrared light and an infrared camera? I'd guess it's to make it work in the dark as well.
  - cataPhil 2y ago
    That's correct. Plus it also adds a layer of interaction that's pretty fun, kind of like a wand.
- bambax 2y ago
  The screen is available on Amazon? But it's a touch screen, which seems overkill; any screen would probably work? One can always build a custom frame around it.
  - bravura 2y ago
    Touch screen I think is cool because then you could interact with the art. I did a little research and these Waveshare displays look really beautiful. For a particular size display, they have models in different resolutions (from lower to higher res). But they're only for rpis and windows, not like mac or ipad. The QLED ones are the best. https://www.waveshare.com/product/ai/displays/10.1hp-capqled.htm?___SID=U https://www.waveshare.com/product/ai/displays/10.1hp-capqled...
tiborsaas 2y ago
Really neat idea, I'd also love to have on my bookshelf. > The main issue with the current setup is the low frame rate. I'd call it a feature rather a limitation, it's not bad that I have second to process the image for a bit. I would even increase the update frequency to 5 - 15 minutes and let it capture and generate a new image whenever it detects something changing / moving.
8n4vidtmkvmk 2y ago
I think this would be cooler if the camera was somewhere different than the frame. Looking at an artistic mirror seems a little boring. Maybe make a 2nd one, and put it in someone else's house, and then feed the camera from one into the other. So you can look at the 'reflection' of someone else and have these little moments where you're both looking at the picture at the same time. Heck, make many so you never know who you're looking at. It'll be the Omegle of picture frames.
- stefs 2y ago
  that's a good idea but something completely different than OP wanted to accomplish. i remember that this has been done already as an art installation in public places so people could see - and interact - with others from around the world. edit: like this here - https://www.inavateonthenet.net/news/article/vc-art-installation-opens-portals-between-cities https://www.inavateonthenet.net/news/article/vc-art-installa...
  - bambax 2y ago
    > an art installation in public places so people could see - and interact - with others from around the world Sure, and it wasn't long before this happened: > It took less than a week for people eager to share their ‘assets’ with the world to shut down the visual portals set up between the cities of Dublin and New York on 8 May. Although the project was aimed at bringing people together and connecting cultures, a few visitors to the locations have decided to take the invitation of getting to know others to a whole other level. https://traveltomorrow.com/new-york-dublin-portals-shut-down-after-indecent-exposure/ https://traveltomorrow.com/new-york-dublin-portals-shut-down... (One wonders why flashing is such a big problem that the whole installation needs to be shut down; but it seemed quite obvious from the start that people would try to do this...)
    - 8n4vidtmkvmk 2y ago
      We're getting quite good at nudie filters. Can we not just black out the goods or the whole screen if they're detected?
swayvil 2y ago
>But art is mostly about surfacing the inner world Oh good lord.
- more_corn 2y ago
  What’s wrong with that? If art is expression shouldn’t everyone have the right to artistic expression? Even people who express themselves with nerdy projects and code and hardware instead of paint and canvas? What do you think art is?
king_magic 2y ago
This is very cool and I applaud you OP.
stonethrowaway 2y ago
> But art is mostly about surfacing the inner world, and only in part about skill. It’s unfortunate that art selects so strongly for skill. Can we decorrelate the two? I don’t think the poor lad knows what Art is. Put another way, we have hundreds of years of recorded philosophical texts and diatribes on what constitutes Art, and what art-making is. Often written by serious practitioners who dedicate their life to it rather than internet-dwelling dabblers and dilettantes. We have people who are deemed artists, not necessarily painters but people who are wired a certain way and are industrious with their abilities. Math geniuses attend certain schools and the other pupils may pick up a thing or two from them, but that doesn’t mean the other pupils are geniuses also. So too, do artists walk among us and may do what we do and we may imitate what they do. But that doesn’t put us on equal footing whatsoever. Art doesn’t select for skill. This is a red herring and a misunderstanding. Art doesn’t select for anything, because if it did it wouldn’t be Art. This is an old somewhat trite topic that, historically, boiled down to no more than a pithy phrase: “Art cannot be taught.” as expounded by many teachers of incredible talent in their own right who have attempted to distill it into teachable material and realized their talent is not transferable as easily as they had hoped. Most of what you read on this subject is nonsense sold to you by grifters who want your money. Now and today more than ever. I’m all for, say, “Art and Fear” and “The Art Spirit” and even a bit of “War of Art” to name some household items on the subject. These are all great recent texts. But let’s take these for what they are: self-help literature, and nothing more. The further back in literature you go the less of this patting-on-the-back attitude you get, and more serious the subject matter is treated (example: read the lectures on Art by the presidents of the Royal Academy, they are numerous, Archive has them all. One president basically tells students to choose a different profession, discussed as an aside topic in a book on portraiture from that time.) Elsewhere in the comments people saying how art is simply good taste seem to be oblivious to the creations of artists that led them to make such a blundering conclusion. Your taste wouldn’t exist had an artist not created a thing to begin with. We’ve used image generators for decades now. It gets the job done. The person using it may be an artist or just someone who wants a dynamic, changing generated image on the wall.
- A4ET8a8uTh0 2y ago
  << Archive has them all. Thank you. I added those to my reading list. I don't think I ever delved into the topic. << Often written by serious practitioners who dedicate their life to it rather than internet-dwelling dabblers and dilettantes. And yet, here we have someone not burdened by the serious business of art and gives his personal perspective on it. I am not saying a lot of everything is not mostly crap, because it mostly is, but I found this child-like honesty oddly endearing. << Art doesn’t select for skill. Artist without a skill is just a dreamer, who can't put his vision into place. Barrier of entry has been lowered now, but I am relatively certain that was not always the case.
arathis 2y ago
This is great.
igornadj 2y ago
> Making art is hard. But art is mostly about surfacing the inner world, and only in part about skill. It’s unfortunate that art selects so strongly for skill. Not to sound like a luddite, but I do question the idea that the skill gap is merely an inconvenience. I suspect learning how to paint or make music changes something in yourself which teaches you some deeper life lessons. I've heard the phrase (paraphrased): No great work of art was made by a genius, genius comes to you unexpectedly like a gust of wind. It seems that cultivating these opportunities is the most an artist can do, and removing the skill gap seems to be removing the cultivation, the thing that changes you, the essence. There seems to be a few of these inherent deep workings that we as a people keep coming back to, without knowing what they are or how to discuss them (personally at least!). Not to rain on your parade OP, the project looks fun and super useful to a lot people! Just something I ponder on at times.
- terhechte 2y ago
  This goes opposite to the saying "Experts say it cannot be done; amateurs accomplish it every day.". Sometimes it's good to have someone with fresh eyes looking and something and not be shaped by decades of prior history.
  - NavinF 2y ago
    > "Experts say it cannot be done; amateurs accomplish it every day." I love this because I seem to encounter situations like that every day. Who came up with this saying? Recent example: This guy asked a very simple question about something that's commonly done in industry (wiring two power supplies in parallel and balancing the current between them): https://www.reddit.com/r/AskElectronics/comments/1g84zd7/using_multiple_usb_psus_in_parallel_to_increase/ https://www.reddit.com/r/AskElectronics/comments/1g84zd7/usi... Literally 95% of the replies in that thread are irrelevant bullshit from "experts" that have no idea how redundant server PSUs work. I replied to some under the same username. Meanwhile another guy successfully wired two 100W USB-C ports in parallel to power an entire PC. He had no idea that the resistance of his crappy wires kept the two smps control loops stable and divided the current evenly between the two ports ensuring that neither one trips OCP: https://www.reddit.com/r/UsbCHardware/comments/1g8pser/let_me_trigger_you_guys/ https://www.reddit.com/r/UsbCHardware/comments/1g8pser/let_m...
xlii 2y ago
> But art is mostly about surfacing the inner world, and only in part about skill. I always thought that art is all about emotions, both preserving and creating them. That’s why I find a banana on the wall art, popular music is still art. (Might be school influence but I also tend to ask myself „why”) This invention is definitely work of art, but its output - in my eyes - not. It’s like a cloud. It can make different shapes, and some are funny, some might remind me of my close ones, but it’s still averaged randomness. However the idea to reflect reality to this digital randomness I find without doubt - art. And, well: while not visual or audible, there is software, hardware, code and design which are also art (and difficult to make so counterpoint to thesis in first paragraph ;)).
- khafra 2y ago
  > I always thought that art is all about emotions, both preserving and creating them. But this leaves the role of intention ambiguous. If I double-park a BMW across two handicap spaces because I'm angry and entitled, is that art? It certainly evokes emotion, and it's also produced by emotion.
  - happymellon 2y ago
    If you present it as art, sure. But the best part about art is that other people don't have to consider what you do as art. I would find it hard to believe you are presenting it as art though, as you'll want your BMW back after you've bought whatever crap at the store. I don't have to believe your AI generated slop is art because it doesn't actually convey any emotion. This generation of the picture itself is art, similar to other "experience" art pieces. Its referred to as immersive art.
    - Lerc 2y ago
      >If you present it as art, sure. >But the best part about art is that other people don't have to consider what you do as art. I often think of art in a way that is remarkably consistent with these statements. My view was "Art is an invitation to consider" Sticking a banana to a wall becomes art when you do it to make people to engage or think about it. Engagement is not compulsory, but I think the invitation might be. Sticking a banana to a wall to keep it away from ants is simple utility, not intended to be artistic expression. >I don't have to believe your AI generated slop is art because it doesn't actually convey any emotion. You don't have to engage, but choosing to denigrate is actively hostile engagement. There are plenty of people using AI for expression of ideas. There are also people doing a bunch of dumb things. Lowbrow art is still art in it's own way. Much of it may have very little to say, but I don't think there are very many people who have pretensions that low effort images are much more than a kind of doodling.
unwind 2y ago
I don't have anything to say about the artistic value, but as a part-time tool collector with dreams of being a handy person, this passage was the most interesting: I used a puncher to cut a hole in the frame’s cardboard for the camera (drills didn’t work). I would love some more detail, or just people's interpretations ... in what manner can a drill fail to "work" on cardboard? There can be issues with tearing, perhaps, that I think punching works around nicely.
- esperent 2y ago
  It's too soft and layered, the drill tends to chew it up. I've had some luck putting duct tape over both sides and drilling through that, but for very clean holes you'd still probably have to clean the edges with a craft knife.
- bambax 2y ago
  My guess is he tried with a wood drill bit, which will tear cardboard and make a big mess. For cardboard, or plastic, you need a metal drill bit.
  - cataPhil 2y ago
    That's right, sorry for not explaining further in the post! The cardboard was so thin than neither wood nor metal drill bits made a clean enough cut.
    - kadoban 2y ago
      I've had success by putting the material between two pieces of wood and drilling through it all. Ideally you clamp it, but you can also just kind of push down on it.
quercus 2y ago
reminds me of the classic 80's "Take on Me" music video
dbspin 2y ago
This is great and all, but just to address the 'art' issue. It's the creation of the 'mirror' that's artistic (to whatever extent running a copy of stable diffusion is a creative act), not looking in the mirror. The enormous number of affordances of 'AI art' will be like this - passively consumptive, creative only in the sense that chatting about a movie or playing a video game can be creative. All of us can already 'surface the inner world' (which I think is actually a pretty great definition of art, or part of one at least) - it's not skill based, it's practice based. Just start drawing, or writing or progaming with Processing or futzing with Ableton or what have you. Skill builds over time, expression needs no automation.
Tepix 2y ago
> The main issue with the current setup is the low frame rate. I think lowering the framerate to something between 0.3 to 1 fps might even be better!
Mumps 2y ago
On the low framerate: why get a pi5 when a Jetson Nano could be had for about the same price?
- jansan 2y ago
  What do you want a high frame rate for when looking at art? What is the frame rate of the Mona Lisa?
  - Mumps 2y ago
    What's the frame rate of a zoetrope, a 1920's silent film, Charlie Chaplin's greatest works, Eraserhead? Not all art is still. So, why the snark?
ogou 2y ago
The interesting thing to me about this, and most other demos, is what people are simulating as examples of art. In this case, it's a very specific Picasso time period. Others tend to center on fantasy or well-known science fiction painters. This makes sense if your only exposure to art is the internet. Maybe a few Wikipedia sessions and a Google search for "modern art". The actual art world is in a huge transition right now and the idea of this Western dominance of art history is being completely re-evaluated. I haven't seen many demos of these tools, or AI art in general, that makes use of any actual modern art from the past 20-30 years. I definitely don't see examples from non-European art history (except Ukiyo-e or anime). That says a lot about what most of you expect from tools like this. It's not just technical, it's culturally normative. Think of it, (without looking it up) can any of you name any living artists that have been exhibited in the Museum of Modern Art in the past 30 years? I see so many strong opinions here about what art is or isn't, but wonder if anybody actually goes to museums or galleries to see what is considered modern art.
xrd 2y ago
I would love to see you add a networking layer so that I could connect my frame to a random one somewhere in the universe. Then I could see another person through this style transfer. They could see me. It would be like an art chat roulette.
tivert 2y ago
You should get a couple of old-work low voltage brackets and brush wall plates (https://www.homedepot.com/p/Carlon-1-Gang-Non-Metallic-Low-Voltage-Old-Work-Bracket-SC100RR/100160916 https://www.homedepot.com/p/Carlon-1-Gang-Non-Metallic-Low-V..., https://www.homedepot.com/p/Commercial-Electric-1-Gang-Brush-Plastic-Wall-Plate-White-5038-WH/207161871 https://www.homedepot.com/p/Commercial-Electric-1-Gang-Brush...), and route the power cable through the wall. It would look much nicer.
pogue 2y ago
I'd love to know the cost of all the materials used in the project and how much time it took you to assemble everything (I didn't see it in the article unless I missed it) This would definitely be a very cool installation piece for a house or commercial property for people to gather around and talk about!

472 points by cataPhil 2y ago

Prize for the most power-hungry mirror. I also notice that the paintings are not stable over small changes in the inputs.

shermantanktop 2y ago
To me, that’s the charm.
- amelius 2y ago
  How about changing the style every time the viewer blinks?

jaredhallen 2y ago

It would be cool to intercept the input from a webcam, transform it like this, and then present it to the OS as another virtual camera so you could select it in Teams or Zoom or whatever.

noman-land 2y ago
You can do this with OBS.

broast 2y ago

Super cool! I've had similar ideas when I first saw StreamDiffusion. The possibilities are endless

shermantanktop 2y ago

Have you considered a high frame rate morph effect between images? That would increase the effective frame rate and probably would look pretty cool.

JKCalhoun 2y ago
Even a simple crossfade.
doctorhandshake 2y ago
That’s the technique I used on a piece that does inpainting across an image at a rate of about 1 image every 8 seconds - I ‘melt’ the results in for the duration until the next patch is ready: https://hardwork.party/aws-epoch-optimizer/ https://hardwork.party/aws-epoch-optimizer/
lancesells 2y ago
I actually think the high frame rate will look worse unless the styling changes less in between frames. Otherwise, it's going to look erratic and there will be a ton of "popping" if each frame changes as much as the frames in the current video. Or maybe op wants it to be a chaotic feeling.
- flir 2y ago
  Many years ago I did a couple of hundred variations on the mona lisa in photoshop, and stacked them as a gif. As the eyes remained unchanged in each frame, they gave a fixed point you could focus on while the rest of the face strobed chaotically. It was a nice effect.

theendisney4 2y ago

Sounds cool! I would like pictures with people in the room doing weird things energetically like stand on their chair. Dancing. Confetti booze strange outfits. Bordering the offensive then deleted after 30 minutes.

autoexec 2y ago

It's pretty neat, but I'd never have a webcam steaming everything in view to someone else's server on the internet. It'd be cool if the server ran on my own hardware, and ideally in the frame itself. The privacy policy at runpod.io wouldn't even display in my browsers. (their ToS loaded without an issue).

beeflet 2y ago
I wonder if you could do something like this with hardware acceleration (like google coral)
- refulgentis 2y ago
  Unfortunately no, they only have a TensorFlow target and none of the hobbyist / released big model image stuff works with it

imsaw 2y ago

I think the low framerate adds a unique ambience, at least to me, watching the recorded videos. It gives time to appreciate the art style and generated features. It also feels like a time lapse of gradual changes. I could imagine it being annoying to look at in real time though.

tsunamifury 2y ago

Micasso

shib71 2y ago

> Art is ... mostly about surfacing the inner world, and only in part about skill. I like the phrasing of the first part. But what art is "about" is very subjective. For me, part of what I look for in art is intentionality - the notion that the artist has crafted each element toward a purpose, consciously or not. The less an artist contributes to the final piece, the less meaning I assign to it. In this case: I would say that the individual pictures being displayed are not "art" - they have no meaning. But I think the device in whole is a piece of art. That is a creation that surfaces the creator's inner world, because they designed the device, wrote the code, crafted the prompts to achieve pieces that reflected their notion of beauty.

disqard 2y ago
I love this framing (pun unintended) of "art". Besides what you articulated as the "intention", I often think of the "story" behind the art -- whether an idea in the creator's head was expressed via the piece (or not) makes me go "yes, this is Art" (or not). By that token, when I see automated projects like this, I think of the "installation" as art, but the pixels or arrangements generated by the piece itself is less art-like IMO.
d0gsg0w00f 2y ago
Yes. I think we attribute the cost of the human sacrifice into the value of the art. It's a like owning 1/80th of a human soul and hanging it on the wall.

viraptor 2y ago

> It’s unfortunate that art selects so strongly for skill. Can we decorrelate the two? I really like this direction. I understand why some object to the genai approaches, but in practice sometimes I get an idea of something cool and don't have the skills to create it myself. I'm not going to invest months/years to create each of those ideas and they're not important enough to spend hundreds of dollars that a skilled artist would request. Now there's a way people can try generating the thing and may end up enjoying it - and that's great. (At least for personal use, it gets a bit complicated for commercial purposes)

happymellon 2y ago
But unfortunately they did not. Not everyone could build what is presented here, and inadvertently they have validated that it requires skill to produce a machine that distorts your reality. The pictures being produced isn't the art.
d0gsg0w00f 2y ago
This is assuming an unimplemented idea has value. I think the value we attribute to an idea is actually the dissemination of the idea to others. Often a sufficiently radical idea is difficult for others to grasp and therefore an example must be created by the originator of the idea. So when someone says "That's a great idea" what they mean is "That's great work". Time will tell if others see an idea backed by AI work as valuable. Can they even tell? Who knows.

deleted 2y ago

[deleted]

rnxrx 2y ago

This really does change the interaction with art. As a future expansion it might be neat to recognize images on camera that would make for interesting art (i.e. detection of people/animals or recognition of certain styles of composition) as well as being able to choose amongst different styles. It seems sort of akin to some modern art that incorporated TV screens and video to make dynamic installations, like Nam June Paik.

r00fus 2y ago

Is there any way to freeze some of the images? I thought some of them would be awesome to keep (e.g. pfp or lockscreen photo).

melony 2y ago

Some optimization suggestions: - cache the prompting somehow, unless you are doing dynamic stuff with the prompts, the language embeddings generated should be static (this depends on the architecture of the model that you are using, it's only possible with certain setups where the language processing is a separate part in the pipeline) - consider fine-tuning an img to img model with your current outputs instead of using a language-coupled model. My intuition is that this is currently significantly over-engineered on the ML side. - Play around with local hardware acceleration instead of sending everything to the cloud, you also probably don't need particularly high resolution for the images either.

cataPhil 2y ago
Love it thanks, will look into it!

caelinsutch 2y ago

how much does this end up costing to run?

deleted 2y ago

[deleted]

GradientSurfer 2y ago

Nice! I peeked at the code and thought I’d share a few tips for improving the low frame rate: Base64 encoding the JPEG bytes will increase payload size up to ~30% and burns CPU cycles on both client and server. This is unnecessary, as Websocket protocol can send binary payloads (doesn’t need to be text). Consider removing lossy jpg compression as well, ie just send the raw RGB bytes over the network. Then on the server side you can simply call Image.frombuffer(…). StreamDiffusion can achieve high frame rates because of extensive batching in the pipeline. You’re not benefiting from that here as the client is only sending one frame at a time and then waiting for a response. See this example for an idea of how to queue input frames and consume them in batches https://github.com/cumulo-autumn/StreamDiffusion/blob/main/examples/screen/main.py https://github.com/cumulo-autumn/StreamDiffusion/blob/main/e... . Alternatively you could take a look at the SDXL Turbo and Lightning models. They are very fast at img2img but have limited resolution of 512² or 1024² pixels respectively. Which appears a bit lower than what you’re aiming for here, but they can be run locally in real time on a high end consumer grade GPU. For reference I have some code demonstrating this here https://github.com/GradientSurfer/Draw2Img/tree/main https://github.com/GradientSurfer/Draw2Img/tree/main

MrLeap 2y ago
yeah yeah yeah, do all these things, and afterwards, look at 2d interpolation methods that don't require AI for your inbetweens. There's some real fast kernel math that can lerp from one blob to another at 8 billion fps.
- enjeyw 2y ago
  I think you’re getting downvoted because “yeah yeah yeah” is normally a sign that someone is sarcastically dismissing an idea, but the rest of your comment suggests you’re not at all - linerp is a great idea!
  - cataPhil 2y ago
    These are great ideas thank you!
  - 2y ago

randmeerkat 2y ago

> It’s unfortunate that art selects so strongly for skill. Jackson Pollock begs to differ.

aspenmayer 2y ago
Artistic skill lies not merely in the hand, but also in the eye.
- randmeerkat 2y ago
  > Artistic skill lies not merely in the hand, but also in the eye. Precisely. There is no skill in artistry, it’s rather just a developed sense of style, that doesn’t come from a medium or method, it comes from growing to know oneself.
  - aspenmayer 2y ago
    I agree. I think the distinction between arts and crafts is largely one of utility, rather than skill or technique.
    - randmeerkat 2y ago
      > I agree. I think the distinction between arts and crafts is largely one of utility, rather than skill or technique. Which is why I take offense to someone claiming that it selects for and requires skill. Especially because the ones making that argument are usually the ones arguing that they’re incapable of making art. I would tell them they just need to spend more time finding themselves.
      - aspenmayer 2y ago
        > Which is why I take offense to someone claiming that it selects for and requires skill. Especially because the ones making that argument are usually the ones arguing that they’re incapable of making art. I would tell them they just need to spend more time finding themselves. And spend more time (and intentional effort) in making art! It's like people want a shortcut to end result, when "real" artists know that the process of self-discovery is the reason and means through which they make the art in the first place. The map is not the territory. The purpose of the journey is the journey itself; the destination is simply a guiding star. A whole lot of aspirational mapmakers think if they only had "this one weird trick," they'd be gods.

malux85 2y ago

Let’s hook it up to a sentiment and toxicity model, and then if you’re being too negative online, it will start to deform you into a monster, we could have a real Picture of Dorian Gray!

A4ET8a8uTh0 2y ago
I chuckled, but that is actually an interesting ( and very doable ) fun project idea.
CaptainFever 2y ago
This is actually such a cool idea.

bravura 2y ago

FYI, the link to the frame used to mount the display just blocks you: https://www.leroymerlin.fr/produits/decoration-eclairage/decoration-murale/cadre-photo/cadre-noir/cadre-milo-21-x-29-7-cm-noir-inspire-71670942.html https://www.leroymerlin.fr/produits/decoration-eclairage/dec... For those curious, it's a MILO 21 x 29.7 cm black frame. These links work for me: https://www.leroymerlin.pt/produtos/decoracao-e-tapetes/molduras/molduras/moldura-com-passe-partout/moldura-multifotos-inspire-milo-21-x-29-7-cm-preto-19412743.html https://www.leroymerlin.pt/produtos/decoracao-e-tapetes/mold... Also, the screen he uses (HMTECH Raspberry Pi Screen 10.1) is pretty hard to find. Do people have other good recommendations for screens with similar quality and specs? Any idea why this uses infrared light and an infrared camera?

esperent 2y ago
Every time I've looked into building something like this, when I price it out, just using an old Android tablet in kiosk mode and making it a web app ends up being much cheaper.
piva00 2y ago
> Any idea why this uses infrared light and an infrared camera? I'd guess it's to make it work in the dark as well.
- cataPhil 2y ago
  That's correct. Plus it also adds a layer of interaction that's pretty fun, kind of like a wand.
bambax 2y ago
The screen is available on Amazon? But it's a touch screen, which seems overkill; any screen would probably work? One can always build a custom frame around it.
- bravura 2y ago
  Touch screen I think is cool because then you could interact with the art. I did a little research and these Waveshare displays look really beautiful. For a particular size display, they have models in different resolutions (from lower to higher res). But they're only for rpis and windows, not like mac or ipad. The QLED ones are the best. https://www.waveshare.com/product/ai/displays/10.1hp-capqled.htm?___SID=U https://www.waveshare.com/product/ai/displays/10.1hp-capqled...

tiborsaas 2y ago

Really neat idea, I'd also love to have on my bookshelf. > The main issue with the current setup is the low frame rate. I'd call it a feature rather a limitation, it's not bad that I have second to process the image for a bit. I would even increase the update frequency to 5 - 15 minutes and let it capture and generate a new image whenever it detects something changing / moving.

8n4vidtmkvmk 2y ago

I think this would be cooler if the camera was somewhere different than the frame. Looking at an artistic mirror seems a little boring. Maybe make a 2nd one, and put it in someone else's house, and then feed the camera from one into the other. So you can look at the 'reflection' of someone else and have these little moments where you're both looking at the picture at the same time. Heck, make many so you never know who you're looking at. It'll be the Omegle of picture frames.

stefs 2y ago
that's a good idea but something completely different than OP wanted to accomplish. i remember that this has been done already as an art installation in public places so people could see - and interact - with others from around the world. edit: like this here - https://www.inavateonthenet.net/news/article/vc-art-installation-opens-portals-between-cities https://www.inavateonthenet.net/news/article/vc-art-installa...
- bambax 2y ago
  > an art installation in public places so people could see - and interact - with others from around the world Sure, and it wasn't long before this happened: > It took less than a week for people eager to share their ‘assets’ with the world to shut down the visual portals set up between the cities of Dublin and New York on 8 May. Although the project was aimed at bringing people together and connecting cultures, a few visitors to the locations have decided to take the invitation of getting to know others to a whole other level. https://traveltomorrow.com/new-york-dublin-portals-shut-down-after-indecent-exposure/ https://traveltomorrow.com/new-york-dublin-portals-shut-down... (One wonders why flashing is such a big problem that the whole installation needs to be shut down; but it seemed quite obvious from the start that people would try to do this...)
  - 8n4vidtmkvmk 2y ago
    We're getting quite good at nudie filters. Can we not just black out the goods or the whole screen if they're detected?

swayvil 2y ago

>But art is mostly about surfacing the inner world Oh good lord.

more_corn 2y ago
What’s wrong with that? If art is expression shouldn’t everyone have the right to artistic expression? Even people who express themselves with nerdy projects and code and hardware instead of paint and canvas? What do you think art is?

king_magic 2y ago

This is very cool and I applaud you OP.

stonethrowaway 2y ago

> But art is mostly about surfacing the inner world, and only in part about skill. It’s unfortunate that art selects so strongly for skill. Can we decorrelate the two? I don’t think the poor lad knows what Art is. Put another way, we have hundreds of years of recorded philosophical texts and diatribes on what constitutes Art, and what art-making is. Often written by serious practitioners who dedicate their life to it rather than internet-dwelling dabblers and dilettantes. We have people who are deemed artists, not necessarily painters but people who are wired a certain way and are industrious with their abilities. Math geniuses attend certain schools and the other pupils may pick up a thing or two from them, but that doesn’t mean the other pupils are geniuses also. So too, do artists walk among us and may do what we do and we may imitate what they do. But that doesn’t put us on equal footing whatsoever. Art doesn’t select for skill. This is a red herring and a misunderstanding. Art doesn’t select for anything, because if it did it wouldn’t be Art. This is an old somewhat trite topic that, historically, boiled down to no more than a pithy phrase: “Art cannot be taught.” as expounded by many teachers of incredible talent in their own right who have attempted to distill it into teachable material and realized their talent is not transferable as easily as they had hoped. Most of what you read on this subject is nonsense sold to you by grifters who want your money. Now and today more than ever. I’m all for, say, “Art and Fear” and “The Art Spirit” and even a bit of “War of Art” to name some household items on the subject. These are all great recent texts. But let’s take these for what they are: self-help literature, and nothing more. The further back in literature you go the less of this patting-on-the-back attitude you get, and more serious the subject matter is treated (example: read the lectures on Art by the presidents of the Royal Academy, they are numerous, Archive has them all. One president basically tells students to choose a different profession, discussed as an aside topic in a book on portraiture from that time.) Elsewhere in the comments people saying how art is simply good taste seem to be oblivious to the creations of artists that led them to make such a blundering conclusion. Your taste wouldn’t exist had an artist not created a thing to begin with. We’ve used image generators for decades now. It gets the job done. The person using it may be an artist or just someone who wants a dynamic, changing generated image on the wall.

A4ET8a8uTh0 2y ago
<< Archive has them all. Thank you. I added those to my reading list. I don't think I ever delved into the topic. << Often written by serious practitioners who dedicate their life to it rather than internet-dwelling dabblers and dilettantes. And yet, here we have someone not burdened by the serious business of art and gives his personal perspective on it. I am not saying a lot of everything is not mostly crap, because it mostly is, but I found this child-like honesty oddly endearing. << Art doesn’t select for skill. Artist without a skill is just a dreamer, who can't put his vision into place. Barrier of entry has been lowered now, but I am relatively certain that was not always the case.

arathis 2y ago

This is great.

igornadj 2y ago

> Making art is hard. But art is mostly about surfacing the inner world, and only in part about skill. It’s unfortunate that art selects so strongly for skill. Not to sound like a luddite, but I do question the idea that the skill gap is merely an inconvenience. I suspect learning how to paint or make music changes something in yourself which teaches you some deeper life lessons. I've heard the phrase (paraphrased): No great work of art was made by a genius, genius comes to you unexpectedly like a gust of wind. It seems that cultivating these opportunities is the most an artist can do, and removing the skill gap seems to be removing the cultivation, the thing that changes you, the essence. There seems to be a few of these inherent deep workings that we as a people keep coming back to, without knowing what they are or how to discuss them (personally at least!). Not to rain on your parade OP, the project looks fun and super useful to a lot people! Just something I ponder on at times.

terhechte 2y ago
This goes opposite to the saying "Experts say it cannot be done; amateurs accomplish it every day.". Sometimes it's good to have someone with fresh eyes looking and something and not be shaped by decades of prior history.
- NavinF 2y ago
  > "Experts say it cannot be done; amateurs accomplish it every day." I love this because I seem to encounter situations like that every day. Who came up with this saying? Recent example: This guy asked a very simple question about something that's commonly done in industry (wiring two power supplies in parallel and balancing the current between them): https://www.reddit.com/r/AskElectronics/comments/1g84zd7/using_multiple_usb_psus_in_parallel_to_increase/ https://www.reddit.com/r/AskElectronics/comments/1g84zd7/usi... Literally 95% of the replies in that thread are irrelevant bullshit from "experts" that have no idea how redundant server PSUs work. I replied to some under the same username. Meanwhile another guy successfully wired two 100W USB-C ports in parallel to power an entire PC. He had no idea that the resistance of his crappy wires kept the two smps control loops stable and divided the current evenly between the two ports ensuring that neither one trips OCP: https://www.reddit.com/r/UsbCHardware/comments/1g8pser/let_me_trigger_you_guys/ https://www.reddit.com/r/UsbCHardware/comments/1g8pser/let_m...

xlii 2y ago

> But art is mostly about surfacing the inner world, and only in part about skill. I always thought that art is all about emotions, both preserving and creating them. That’s why I find a banana on the wall art, popular music is still art. (Might be school influence but I also tend to ask myself „why”) This invention is definitely work of art, but its output - in my eyes - not. It’s like a cloud. It can make different shapes, and some are funny, some might remind me of my close ones, but it’s still averaged randomness. However the idea to reflect reality to this digital randomness I find without doubt - art. And, well: while not visual or audible, there is software, hardware, code and design which are also art (and difficult to make so counterpoint to thesis in first paragraph ;)).

khafra 2y ago
> I always thought that art is all about emotions, both preserving and creating them. But this leaves the role of intention ambiguous. If I double-park a BMW across two handicap spaces because I'm angry and entitled, is that art? It certainly evokes emotion, and it's also produced by emotion.
- happymellon 2y ago
  If you present it as art, sure. But the best part about art is that other people don't have to consider what you do as art. I would find it hard to believe you are presenting it as art though, as you'll want your BMW back after you've bought whatever crap at the store. I don't have to believe your AI generated slop is art because it doesn't actually convey any emotion. This generation of the picture itself is art, similar to other "experience" art pieces. Its referred to as immersive art.
  - Lerc 2y ago
    >If you present it as art, sure. >But the best part about art is that other people don't have to consider what you do as art. I often think of art in a way that is remarkably consistent with these statements. My view was "Art is an invitation to consider" Sticking a banana to a wall becomes art when you do it to make people to engage or think about it. Engagement is not compulsory, but I think the invitation might be. Sticking a banana to a wall to keep it away from ants is simple utility, not intended to be artistic expression. >I don't have to believe your AI generated slop is art because it doesn't actually convey any emotion. You don't have to engage, but choosing to denigrate is actively hostile engagement. There are plenty of people using AI for expression of ideas. There are also people doing a bunch of dumb things. Lowbrow art is still art in it's own way. Much of it may have very little to say, but I don't think there are very many people who have pretensions that low effort images are much more than a kind of doodling.

unwind 2y ago

I don't have anything to say about the artistic value, but as a part-time tool collector with dreams of being a handy person, this passage was the most interesting: I used a puncher to cut a hole in the frame’s cardboard for the camera (drills didn’t work). I would love some more detail, or just people's interpretations ... in what manner can a drill fail to "work" on cardboard? There can be issues with tearing, perhaps, that I think punching works around nicely.

esperent 2y ago
It's too soft and layered, the drill tends to chew it up. I've had some luck putting duct tape over both sides and drilling through that, but for very clean holes you'd still probably have to clean the edges with a craft knife.
bambax 2y ago
My guess is he tried with a wood drill bit, which will tear cardboard and make a big mess. For cardboard, or plastic, you need a metal drill bit.
- cataPhil 2y ago
  That's right, sorry for not explaining further in the post! The cardboard was so thin than neither wood nor metal drill bits made a clean enough cut.
  - kadoban 2y ago
    I've had success by putting the material between two pieces of wood and drilling through it all. Ideally you clamp it, but you can also just kind of push down on it.

quercus 2y ago

reminds me of the classic 80's "Take on Me" music video

dbspin 2y ago

This is great and all, but just to address the 'art' issue. It's the creation of the 'mirror' that's artistic (to whatever extent running a copy of stable diffusion is a creative act), not looking in the mirror. The enormous number of affordances of 'AI art' will be like this - passively consumptive, creative only in the sense that chatting about a movie or playing a video game can be creative. All of us can already 'surface the inner world' (which I think is actually a pretty great definition of art, or part of one at least) - it's not skill based, it's practice based. Just start drawing, or writing or progaming with Processing or futzing with Ableton or what have you. Skill builds over time, expression needs no automation.

Tepix 2y ago

> The main issue with the current setup is the low frame rate. I think lowering the framerate to something between 0.3 to 1 fps might even be better!

Mumps 2y ago

On the low framerate: why get a pi5 when a Jetson Nano could be had for about the same price?

jansan 2y ago
What do you want a high frame rate for when looking at art? What is the frame rate of the Mona Lisa?
- Mumps 2y ago
  What's the frame rate of a zoetrope, a 1920's silent film, Charlie Chaplin's greatest works, Eraserhead? Not all art is still. So, why the snark?

ogou 2y ago

The interesting thing to me about this, and most other demos, is what people are simulating as examples of art. In this case, it's a very specific Picasso time period. Others tend to center on fantasy or well-known science fiction painters. This makes sense if your only exposure to art is the internet. Maybe a few Wikipedia sessions and a Google search for "modern art". The actual art world is in a huge transition right now and the idea of this Western dominance of art history is being completely re-evaluated. I haven't seen many demos of these tools, or AI art in general, that makes use of any actual modern art from the past 20-30 years. I definitely don't see examples from non-European art history (except Ukiyo-e or anime). That says a lot about what most of you expect from tools like this. It's not just technical, it's culturally normative. Think of it, (without looking it up) can any of you name any living artists that have been exhibited in the Museum of Modern Art in the past 30 years? I see so many strong opinions here about what art is or isn't, but wonder if anybody actually goes to museums or galleries to see what is considered modern art.

xrd 2y ago

I would love to see you add a networking layer so that I could connect my frame to a random one somewhere in the universe. Then I could see another person through this style transfer. They could see me. It would be like an art chat roulette.

tivert 2y ago

You should get a couple of old-work low voltage brackets and brush wall plates (https://www.homedepot.com/p/Carlon-1-Gang-Non-Metallic-Low-Voltage-Old-Work-Bracket-SC100RR/100160916 https://www.homedepot.com/p/Carlon-1-Gang-Non-Metallic-Low-V..., https://www.homedepot.com/p/Commercial-Electric-1-Gang-Brush-Plastic-Wall-Plate-White-5038-WH/207161871 https://www.homedepot.com/p/Commercial-Electric-1-Gang-Brush...), and route the power cable through the wall. It would look much nicer.

pogue 2y ago

I'd love to know the cost of all the materials used in the project and how much time it took you to assemble everything (I didn't see it in the article unless I missed it) This would definitely be a very cool installation piece for a house or commercial property for people to gather around and talk about!