6 ms·

DINOv3

179 points by reqo 1y ago

beklein 1y ago
- Blog post: https://ai.meta.com/blog/dinov3-self-supervised-vision-model/ https://ai.meta.com/blog/dinov3-self-supervised-vision-model... - Paper: https://ai.meta.com/research/publications/dinov3/ https://ai.meta.com/research/publications/dinov3/ - Hugging Face: https://huggingface.co/collections/facebook/dinov3-68924841bd6b561778e31009 https://huggingface.co/collections/facebook/dinov3-68924841b...
ranger_danger 1y ago
I have no idea what this even is.
- n3storm 1y ago
  D3NO?
- kaoD 1y ago
  > An extended family of versatile vision foundation models producing high-quality dense features and achieving outstanding performance on various vision tasks including outperforming the specialized state of the art across a broad range of settings, without fine-tuning
  - ranger_danger 1y ago
    English, doc
  - kevinventullo 1y ago
    To elaborate, this is a foundation model. This basically means it can take an arbitrary image and map it to a high dimensional space H in which ~arbitrary characteristics become much easier to solve for. For example (and this might be oversimplifying a bit, computer vision people please correct me if I’m wrong) if you’re interested in knowing whether or not the image contains a cat, then maybe there is some hyperplane P in H for which images on one side of P do not contain a cat, and images on the other side do contain a cat. And so solving for “Does this image contain a cat?”becomes a much easier problem, all you have to do is figure out what P is. Once you do that, you can pass your image into DINO, dot product with the equation for P, and check whether the answer is negative or positive. The point is that finding P is much easier than training your own computer vision model from scratch.
    - reactordev 1y ago
      If computer vision were semantic search, nailed it. It’s a little more complicated than that but - with this new model, not by much :D
    - hgo 1y ago
      Thanks, I think I understand roughly. Could it be used for for recognizing people? As in identifying what person is in what image?
      - kevinventullo 1y ago
        Caveat: I am not an expert, so this is a semi-educated guess. I imagine it would depend on whether DINOv3 captures the information of whether a given person is in the image, which I think is really a question about training data. So naively, I would guess the answer is yes for celebrities and no for non-celebrities. Partially for data/technical reasons, but also maybe due to the murkier legal expectation of privacy for famous people.
      - joshvm 1y ago
        Foundation models like DINO learn representations of their inputs. That is, they generate very high-dimensional numerical descriptions of what you put into them. The models aren't trained on labelled data, but they're trained on some pretext task like "given this image with a cutout, fill in the cutout" (see Masked Auto-Encoders). So the basic output from a model is a vector - often called an embedding. Literally a 1D list of numbers, O(1k)-dimensional. Your goal is to get an embedding that assigns (well) linearly separable vectors for all the things you want to classify. Vision transformers also output patch tokens, which can be assembled into a low-resolution feature map (w/32, h/32 is common). So what you do with that data depends on the task. Classification can be as simple as linearly classifying the (whole image) embedding. A semantic segmentation task can do the same, but for every pixel. This is why the DINO authors show a PCA representation of a bunch of images, which show that semantically similar objects are grouped together by colour. Object detectors are more complicated, but the key thing is that once you have these pixel-level features, you can use them as input into existing architectures. Now to your question: face recognition is a specific application of object re-identification (keyword: Re-ID). The way most of these models work is from the whole-image embedding. Normally you'd run a detector to extract the face region, then compute the embedding, put it in a vector database and then query for nearest neighbours using something like the cosine distance. I've only worked in this space for animals, but humans are far more studied. Whether DINOv3 is good enough out-of-the-box I don't know, but certainly there's a lot of literature looking at these sorts of models for Re-ID. The challenge with Re-ID is that the model has to be able to produce features which discriminate individuals rather than similar looking individuals. For example with the vanilla model, you probably have a very good tool for visual search. But that's not the same task, because if you give it a picture of someone in a field, you'll get back pictures of other people in fields. That usually requires re-training on labelled imagery where you have a few examples of each person. The short answer is that there are already very good models for doing this, and they don't necessarily even need ML to do a decent job (though it might be used for keypoint detection for facial landmarks).
  - pugworthy 1y ago
    So, a group of AI models that can look at images and understand them, working well for many different tasks without needing extra training?
    - stingraycharles 1y ago
      Exactly that. It’s easy to see why Meta, which Facebook and Instagram, is so heavily invested in these types of AI models.
  - kristopolous 1y ago
    This is still pretty non-specific, however, luckily, they have a landing page: https://ai.meta.com/dinov3/ https://ai.meta.com/dinov3/
- ethan_smith 1y ago
  DINO (Distillation with No labels) is a self-supervised computer vision framework that learns powerful image representations without requiring labeled data. It's particularly valuable for downstream tasks like object detection and segmentation, with DINOv3 now scaling to over 1B parameters and trained on 1.2B images.
barbolo 1y ago
That's awesome. DINOv2 was the best image embedder until now.
llm_nerd 1y ago
You have to share your contact information, including DoB, and then be approved access, to obtain the models, and given that it's Meta I assume they're actually validating it against their All Humans database. They made their own DINOv3 license for this release (whereas DINOv2 used the Apache 2.0 license). Neat though. Will still check it out. As a first comment, I had to install the latest transformer==4.56.0dev (e.g. pip install git+https://github.com/huggingface/transformers https://github.com/huggingface/transformers) for it to work properly. 4.55.2 and earlier was failing with a missing image type in the config.
- Qwuke 1y ago
  Yes, it's pretty disappointing for a seemingly big improvement over SOTA to be commercially licensed compared the previous version.. At least in the press release they're not portraying it as open source just because it's on GitHub/HuggingFace.
  - tough 1y ago
    The new Facebook AI Czar wang hinted on previous interviews that Meta might change their stand on licensing/open source. Seems like the tides are shifting at meta
    - rajman187 1y ago
      This has nothing to do with the newly appointed fellow nor Meta Superintelligence Labs, but rather work from FAIR that would have gone through a lengthy review process before seeing the light of day. Not fun to see the license change in any case
      - dennisloevlie 1y ago
        I remember DINOv2 was originally a commercial licence. I (along with others) just asked if they could change it on a GitHub issue, and after some time, they did. Might be worth asking
Imnimo 1y ago
I think SAM and DINO are the two off-the-shelf image models I've gotten the most mileage out of.
PhilippGille 1y ago
This was submitted earlier: DINOV3: Self-supervised learning for vision at unprecedented scale | https://news.ycombinator.com/item?id=44904608 https://news.ycombinator.com/item?id=44904608
fnands 1y ago
As someone who works on satellite imagery, this part is incredibly exciting: > ViT models pretrained on satellite dataset (SAT-493M) DINOv2 had pretty poor out-of-the-box performance on satellite/aerial imagery, so it's super exciting that they released a version of it specifically for this use case.
cobbzilla 1y ago
Could anyone point to an example or git repo showing a simple implementation? I’m fascinated by this, but am admittedly clueless about how to actually go about building any kind of recognizer or other system atop it.
- fnands 1y ago
  Their repo has some example notebooks: https://github.com/facebookresearch/dinov3/tree/main/notebooks https://github.com/facebookresearch/dinov3/tree/main/noteboo... As for doing it in general, it's a fairly standard vision transformer so anything built on DINOv2 (or any other ViT) should be easy to adapt to v3.
- nonocycle 1y ago
  You can pretty much use it as a drop-in replacement for anything built on top of DINOv2. E.g. if you want to fine-tune a segmentation model you can use EoMT[0] which uses DINOv2 as backbone and replace the backbone with DINOv3. If you just want to run it you can give LightlyTrain a spin [1]. There should also be support in the original EoMT repo soon. The methods in the DINOv3 paper focus on frozen backbones which are usually faster to train but might have lower performance than full fine-tuning. [0]: https://github.com/tue-mps/eomt https://github.com/tue-mps/eomt [1]: https://docs.lightly.ai/train/stable/semantic_segmentation.html#use-eomt-with-dinov3 https://docs.lightly.ai/train/stable/semantic_segmentation.h...
deepsquirrelnet 1y ago
If I’m already using siglip2 for a clustering application, is this enough of a an uplift that I should look at it?

179 points by reqo 1y ago

- Blog post: https://ai.meta.com/blog/dinov3-self-supervised-vision-model/ https://ai.meta.com/blog/dinov3-self-supervised-vision-model... - Paper: https://ai.meta.com/research/publications/dinov3/ https://ai.meta.com/research/publications/dinov3/ - Hugging Face: https://huggingface.co/collections/facebook/dinov3-68924841bd6b561778e31009 https://huggingface.co/collections/facebook/dinov3-68924841b...

ranger_danger 1y ago

I have no idea what this even is.

n3storm 1y ago
D3NO?
kaoD 1y ago
> An extended family of versatile vision foundation models producing high-quality dense features and achieving outstanding performance on various vision tasks including outperforming the specialized state of the art across a broad range of settings, without fine-tuning
- ranger_danger 1y ago
  English, doc
- kevinventullo 1y ago
  To elaborate, this is a foundation model. This basically means it can take an arbitrary image and map it to a high dimensional space H in which ~arbitrary characteristics become much easier to solve for. For example (and this might be oversimplifying a bit, computer vision people please correct me if I’m wrong) if you’re interested in knowing whether or not the image contains a cat, then maybe there is some hyperplane P in H for which images on one side of P do not contain a cat, and images on the other side do contain a cat. And so solving for “Does this image contain a cat?”becomes a much easier problem, all you have to do is figure out what P is. Once you do that, you can pass your image into DINO, dot product with the equation for P, and check whether the answer is negative or positive. The point is that finding P is much easier than training your own computer vision model from scratch.
  - reactordev 1y ago
    If computer vision were semantic search, nailed it. It’s a little more complicated than that but - with this new model, not by much :D
  - hgo 1y ago
    Thanks, I think I understand roughly. Could it be used for for recognizing people? As in identifying what person is in what image?
    - kevinventullo 1y ago
      Caveat: I am not an expert, so this is a semi-educated guess. I imagine it would depend on whether DINOv3 captures the information of whether a given person is in the image, which I think is really a question about training data. So naively, I would guess the answer is yes for celebrities and no for non-celebrities. Partially for data/technical reasons, but also maybe due to the murkier legal expectation of privacy for famous people.
    - joshvm 1y ago
      Foundation models like DINO learn representations of their inputs. That is, they generate very high-dimensional numerical descriptions of what you put into them. The models aren't trained on labelled data, but they're trained on some pretext task like "given this image with a cutout, fill in the cutout" (see Masked Auto-Encoders). So the basic output from a model is a vector - often called an embedding. Literally a 1D list of numbers, O(1k)-dimensional. Your goal is to get an embedding that assigns (well) linearly separable vectors for all the things you want to classify. Vision transformers also output patch tokens, which can be assembled into a low-resolution feature map (w/32, h/32 is common). So what you do with that data depends on the task. Classification can be as simple as linearly classifying the (whole image) embedding. A semantic segmentation task can do the same, but for every pixel. This is why the DINO authors show a PCA representation of a bunch of images, which show that semantically similar objects are grouped together by colour. Object detectors are more complicated, but the key thing is that once you have these pixel-level features, you can use them as input into existing architectures. Now to your question: face recognition is a specific application of object re-identification (keyword: Re-ID). The way most of these models work is from the whole-image embedding. Normally you'd run a detector to extract the face region, then compute the embedding, put it in a vector database and then query for nearest neighbours using something like the cosine distance. I've only worked in this space for animals, but humans are far more studied. Whether DINOv3 is good enough out-of-the-box I don't know, but certainly there's a lot of literature looking at these sorts of models for Re-ID. The challenge with Re-ID is that the model has to be able to produce features which discriminate individuals rather than similar looking individuals. For example with the vanilla model, you probably have a very good tool for visual search. But that's not the same task, because if you give it a picture of someone in a field, you'll get back pictures of other people in fields. That usually requires re-training on labelled imagery where you have a few examples of each person. The short answer is that there are already very good models for doing this, and they don't necessarily even need ML to do a decent job (though it might be used for keypoint detection for facial landmarks).
- pugworthy 1y ago
  So, a group of AI models that can look at images and understand them, working well for many different tasks without needing extra training?
  - stingraycharles 1y ago
    Exactly that. It’s easy to see why Meta, which Facebook and Instagram, is so heavily invested in these types of AI models.
- kristopolous 1y ago
  This is still pretty non-specific, however, luckily, they have a landing page: https://ai.meta.com/dinov3/ https://ai.meta.com/dinov3/
ethan_smith 1y ago
DINO (Distillation with No labels) is a self-supervised computer vision framework that learns powerful image representations without requiring labeled data. It's particularly valuable for downstream tasks like object detection and segmentation, with DINOv3 now scaling to over 1B parameters and trained on 1.2B images.

barbolo 1y ago

That's awesome. DINOv2 was the best image embedder until now.

llm_nerd 1y ago

You have to share your contact information, including DoB, and then be approved access, to obtain the models, and given that it's Meta I assume they're actually validating it against their All Humans database. They made their own DINOv3 license for this release (whereas DINOv2 used the Apache 2.0 license). Neat though. Will still check it out. As a first comment, I had to install the latest transformer==4.56.0dev (e.g. pip install git+https://github.com/huggingface/transformers https://github.com/huggingface/transformers) for it to work properly. 4.55.2 and earlier was failing with a missing image type in the config.

Qwuke 1y ago
Yes, it's pretty disappointing for a seemingly big improvement over SOTA to be commercially licensed compared the previous version.. At least in the press release they're not portraying it as open source just because it's on GitHub/HuggingFace.
- tough 1y ago
  The new Facebook AI Czar wang hinted on previous interviews that Meta might change their stand on licensing/open source. Seems like the tides are shifting at meta
  - rajman187 1y ago
    This has nothing to do with the newly appointed fellow nor Meta Superintelligence Labs, but rather work from FAIR that would have gone through a lengthy review process before seeing the light of day. Not fun to see the license change in any case
    - dennisloevlie 1y ago
      I remember DINOv2 was originally a commercial licence. I (along with others) just asked if they could change it on a GitHub issue, and after some time, they did. Might be worth asking

Imnimo 1y ago

I think SAM and DINO are the two off-the-shelf image models I've gotten the most mileage out of.

PhilippGille 1y ago

This was submitted earlier: DINOV3: Self-supervised learning for vision at unprecedented scale | https://news.ycombinator.com/item?id=44904608 https://news.ycombinator.com/item?id=44904608

fnands 1y ago

As someone who works on satellite imagery, this part is incredibly exciting: > ViT models pretrained on satellite dataset (SAT-493M) DINOv2 had pretty poor out-of-the-box performance on satellite/aerial imagery, so it's super exciting that they released a version of it specifically for this use case.

cobbzilla 1y ago

Could anyone point to an example or git repo showing a simple implementation? I’m fascinated by this, but am admittedly clueless about how to actually go about building any kind of recognizer or other system atop it.

fnands 1y ago
Their repo has some example notebooks: https://github.com/facebookresearch/dinov3/tree/main/notebooks https://github.com/facebookresearch/dinov3/tree/main/noteboo... As for doing it in general, it's a fairly standard vision transformer so anything built on DINOv2 (or any other ViT) should be easy to adapt to v3.
nonocycle 1y ago
You can pretty much use it as a drop-in replacement for anything built on top of DINOv2. E.g. if you want to fine-tune a segmentation model you can use EoMT[0] which uses DINOv2 as backbone and replace the backbone with DINOv3. If you just want to run it you can give LightlyTrain a spin [1]. There should also be support in the original EoMT repo soon. The methods in the DINOv3 paper focus on frozen backbones which are usually faster to train but might have lower performance than full fine-tuning. [0]: https://github.com/tue-mps/eomt https://github.com/tue-mps/eomt [1]: https://docs.lightly.ai/train/stable/semantic_segmentation.html#use-eomt-with-dinov3 https://docs.lightly.ai/train/stable/semantic_segmentation.h...

deepsquirrelnet 1y ago

If I’m already using siglip2 for a clustering application, is this enough of a an uplift that I should look at it?