6 ms·
Wow this does so well on text! The original model struggled a lot, it's impressive to see how far they've come.
by srajabi 3y ago
Wow this does so well on text! The original model struggled a lot, it's impressive to see how far they've come.
- mkaic 3y agoI'm quite curious how much of the improvement on text rendering is from the switch to pixel-space diffusion vs. the switch to a much larger pretrained text encoder. I'm leaning towards the latter, which then raises the question of what happens when you try training Stable Diffusion with T5-XXL-1.1 as the text encoder instead of CLIP — does it gain the ability to do text well?
- FL33TW00D 3y agoCheck out figure 4A from the ImageGen paper: https://arxiv.org/pdf/2205.11487.pdf https://arxiv.org/pdf/2205.11487.pdf From that - I would strongly suspect the answer to your question to be yes.
- mkaic 3y agoAh, yes, this seems to be pretty strong evidence. Thanks for pointing that figure out to me!
- minimaxir 3y agoDeepFloyd IF is effectively the same architecture/text encoder as Imagen (https://imagen.research.google/ https://imagen.research.google/), although that paper doesn't hypothesize why text works out a lot better.
- mkaic 3y agoRight, I'm aware of the Imagen architecture, just curious to see further research determining which aspect of it is responsible for the improved text rendering. EDIT: According to the figure in the Imagen paper FL33TW00D's response referred me to, it looks like the text encoder size is the biggest factor in the improved model performance all-around.
- radq 3y agoIt is most likely due to the text encoder - see "Character-Aware Models Improve Visual Text Rendering". https://arxiv.org/abs/2212.10562 https://arxiv.org/abs/2212.10562
- GaggiX 3y agoThe CLIP text encoder is trained to align with the pooled image embedding (a single vector), which is why most text embeddings are not very meaningful on their own (but still convey the overall semantics of the text). With T5 every text embedding is important.
- piyh 3y agoThat alone is a huge milestone.
- simonw 3y agoIt's much better, but it's not perfect. Here's what I got for: > a photograph of raccoon in the woods holding a sign that says "I will eat your trash" https://twitter.com/simonw/status/1651994059781832704 https://twitter.com/simonw/status/1651994059781832704
- Semonto 3y agoIt actually adds quite a nice charm by wording it not correctly
- indymike 3y agoThis is not a problem. The sign was clearly made by the raccoon.
- patapong 3y agoAgreed - to properly test it, we should try: A photograph of an English professor in the woods holding a sign that says "I will eat your trash"