11 ms·
We (Roboflow) have had early access to this model for the past few weeks. It's really, really good. This feels like a seminal moment for computer vision. I thin
by yeldarb 10mo ago
We (Roboflow) have had early access to this model for the past few weeks. It's really, really good. This feels like a seminal moment for computer vision. I think there's a real possibility this launch goes down in history as "the GPT Moment" for vision.
The two areas I think this model is going to be transformative in the immediate term are for rapid prototyping and distillation.
Two years ago we released autodistill[1], an open source framework that uses large foundation models to create training data for training small realtime models. I'm convinced the idea was right, but too early; there wasn't a big model good enough to be worth distilling from back then. SAM3 is finally that model (and will be available in Autodistill today).
We are also taking a big bet on SAM3 and have built it into Roboflow as an integral part of the entire build and deploy pipeline[2], including a brand new product called Rapid[3], which reimagines the computer vision pipeline in a SAM3 world. It feels really magical to go from an unlabeled video to a fine-tuned realtime segmentation model with minimal human intervention in just a few minutes (and we rushed the release of our new SOTA realtime segmentation model[4] last week because it's the perfect lightweight complement to the large & powerful SAM3).
We also have a playground[5] up where you can play with the model and compare it to other VLMs.
[1] https://github.com/autodistill/autodistill https://github.com/autodistill/autodistill
[2] https://blog.roboflow.com/sam3/ https://blog.roboflow.com/sam3/
[3] https://rapid.roboflow.com https://rapid.roboflow.com
[4] https://github.com/roboflow/rf-detr https://github.com/roboflow/rf-detr
[5] https://playground.roboflow.com https://playground.roboflow.com
- dangoodmanUT 10mo agoI was trying to figure out from their examples, but how are you breaking up the different "things" that you can detect in the image? Are you just running it with each prompt individually?
- rocauc 10mo agoThe model supports batch inference, so all prompts are sent to the model, and we parse the results.
- sorenjan 10mo agoSAM3 is probably a great model to distill from when training smaller segmentation models, but isn't their DINOv2 a better example of a large foundation model to distill from for various computer vision tasks? I've seen it used for as starting point for models doing segmentation and depth estimation. Maybe there's a v3 coming soon? https://dinov2.metademolab.com/ https://dinov2.metademolab.com/
- nsingh2 10mo agoDINOv3 was released earlier this year: https://ai.meta.com/dinov3/ https://ai.meta.com/dinov3/ I'm not sure if the work they did with DINOv3 went into SAM3. I don't see any mention of it in the paper, though I just skimmed it.
- yeldarb 10mo agoWe used DINOv2 as the backbone of our RF-DETR model, which is SOTA on realtime object detection and segmentation: https://github.com/roboflow/rf-detr https://github.com/roboflow/rf-detr It makes a great target to distill SAM3 to.
- sorenjan 10mo ago> It makes a great target to distill SAM3 to. Could you expand on that? Do you mean you're starting with the pretrained DINO model and then using SAM3 to generate training data to make DINO into a segmentation model? Do you freeze the DINO weights and add a small adapter at the end to turn its output into segmentations?
- mchusma 10mo agoThanks for the linkes! Can we run rf-detr in the browser for background removal? This wasn't clear to me from the docs
- yeldarb 10mo agoWe have a JS SDK that supports RF-DETR: https://docs.roboflow.com/deploy/sdks/web-browser https://docs.roboflow.com/deploy/sdks/web-browser