Thank you so much for the video explanation. The walk through makes all the difference. For example that 5:53 prompt engineering explanation is so useful.
Hi! I was wondering if you could let me know how i can use custom images to detect different objects (other than the labels already in the notebook, like camera, hat, light, etc) and how to add their labels so they can be detected I'm a beginner in this field and would really appreciate the help!
It's a very cool concept and surely helpful for some segmentation tasks. However, I see this working mainly with clear and not crowded images. With many tests I did, quite often a lot of items were mislabeled. Nonetheless cool idea and love the channel!
Absolutely! But keep in mind that 3 years ago it was impossible. We just try to highlight cutting-edge models in 2023. I absolutely agree. We are not yet able to get good results for every image.
Hey Peter, could you do a video showing how to integrate SuperGradients/Yolo NAS with Roboflow's Autodistill for custom detections on a live real-time webcam feed. Could you also show maybe in another video how to add custom objects to an existing dataset like the coco dataset? This would be Epic.🔥
Hi Sir I'm Beginner in I saw your Computer vision video's its fully combined and merged can you please update one by one video order that time we can understand easily thank you.
I have noticed you use in the supervision awesome package a method to load datasets in PASCAL-VOC format, are you planning to also support COCO formats (also for export?)?
Incredible video ! I was just reading the Grounded-SAM this morning, and boum you're making a tutorial on it. Great job ! I'm just wondering if I could find ways to use it in a medical imagery task ! What do you think ?
@@Roboflow I would go for automatic segmentation but I'd also like it to be interactive for the user. So maybe combining the two would more appreciated
@@alassanesakande8791 that is our plan for next stage. Allow full auto or human in the loop :) I also think that being able to interactively interact with those labels before you use them to train for example YOLOv8 is required.
for "solar panel counting from UAV image"...which approach is better ? 1. creating bounding box (BB) for solar panel using object detection model and then using BB as input for SAM....or.... 2. segmenting everything in the image from SAM...and then classifying each segment as solar panel and non solar panel.
I guess that it would be great to include in both supervision and autodistill a feature that gets the bounding box given a polyline segmentation from sam
It is in the description. But here is the link: colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/automated-dataset-annotation-and-evaluation-with-grounding-dino-and-sam.ipynb
Thank you for your work, this is exactly what we need urgently, but at the moment I see that it seems to only support saving data in Pascal voc format, do you have any plans to provide an api to convert it to coco format?
Hey there! I really like these videos a lot. Certainly with fast labelling the specific task can be trained supervised. But is there an opportunity in using SAM and/or DINO as a teacher for distillation into a smaller (final) model, even before creating an annotated dataset? Would this be competitive with other self-supervised pretraining methods?
@@sebbecht we didn't explore that rout yet but it would be awesome to test those theories. Thanks for sharing :) I never run out of ideas thanks to conversations like this.
in you previous video on grounding dino, you elaborated on a text prompt as an input, can this be implemented here as well? are you planning on extending this tutoorial (or notebook) to show how to implement it? also, I have noticed that you can also implement stable diffusion tools such as "change do to a monkey". can that also be in the next vid?
Auto labeling with prompts will be part of the auto-distill package that is coming soon. As for stable diffusion, I can't promise anything :/ We have a lot of stuff in the backlog. But maybe I'll play with it on Twitch stream.
thank to this great vid (and notebook) I have tried using it together with SAM and I'm curious to know how can I use a labeled dataset I have (of sea-objects) to learn the model to detect not only a boat/ship but to identify the name of the marine-vessel.
@@Roboflow thanks so much for the reply! am really trying to figure out how to solve this issue: yes! I do have human-labeled dataset for specific classes of marine-vessels e.g., frigatte, corvette, and also some ships with their specific names. My question was if there is a way to fine-tune the grounded-DINO model to identify the objects not as "boat" or "ship" but on more accurate labels
@@kobic8 yes it probably is possible, but you would be much better of if you train model like YOLOv8. Power od GroundingDINO comes from zero shot detection - ability to detect objects that it never saw. If you already have annotated dataset, just train regular object detection model. :)
@@Roboflow but it be "less powerfull" compared to G-DINO, I just thought to tune G-Dino to refine specific labels, so I tought it be btter to somehow get the traning code
You can always convert segmentation into detection. It is just a bit hm... poor usage of resources as it is super time-consuming. What project do you have in your mind?
@@Roboflow I'm working on detecting potato quality on a conveyer belt. I labeled some photos using SAM, but I'm not sure if the polygon labeling actually helps object detection or if a basic rectangle boundary will enough.
Can I convert a multiclass object detection dataset to a segmentation dataset with this? I have only seen the example with the single class Blueberries dataset so im not sure.
Awesome tutorial!!! But while I am running during 6:25, I got error: "NameError: name '_C' is not defined" (after long error description). Anyone can help?
@@Roboflow I have the same error. I ran the colab from top to bottom and got this error at the first annotation part on the line detections = grounding_dino_model.predict_with_classes(.. Any help would be appreciated!
thank u for this incredible vid !💖 but I have a question, when trying to run the following command it told me that " 41 detections.mask = segment(sam_predictor=sam_predictor, image=image, xyxy=filtered_detections.xyxy) 42 43 mask_annotator = sv.MaskAnnotator() NameError: name 'segment' is not defined " and I search for the __init__ in SAM but there isn't found, so is this function is built in sam_anything module or should I wrote it ?
i replaced this command of yours from tqdm.notebook import tqdm for image_name, image in tqdm(object_detection_dataset.images.items()): detections = object_detection_dataset.annotations[image_name] detections.mask = segment( sam_predictor=sam_predictor, image=cv2.cvtColor(image, cv2.COLOR_BGR2RGB), xyxy=detections.xyxy )
@@Roboflow oh I see, thanks, it had been solved. can I ask another question? my dataset is into coco format as it on my PC not roboflow so I converted it into pascal format to be able to follow your steps from converting to segmentation but it didn't work at all. is it a function in supervision to read coco format like pascal? as I searched but it give me errors
@@saharabdulalim hi! We ant to add COCO loading to supervision but it won't happen to soon :/ if you wan to follow those steps now I'd upload dataset to Roboflow. That's probably the fastest way for now.
Thank you so much for the video explanation. The walk through makes all the difference. For example that 5:53 prompt engineering explanation is so useful.
Nice! Looking forward to seeing the new library in action.
I’ll do my best to not disappoint you ;)
wow... excited for the auto distill! :)
That’s what I wanted to hear 💜
Thank you this is exactly what I was waiting for.
I love to hear that! 🔥
Great job as usual!
Thanks a lot! 🙏 we are not slowing down
Hi! I was wondering if you could let me know how i can use custom images to detect different objects (other than the labels already in the notebook, like camera, hat, light, etc) and how to add their labels so they can be detected
I'm a beginner in this field and would really appreciate the help!
Very nice video and explanation, thank you very much!
Awesome video as usual😮👍
Thank you very much… doing my best 🙏🏻
Wow , this is fantastic
Wow, thanks a looot
It's a very cool concept and surely helpful for some segmentation tasks. However, I see this working mainly with clear and not crowded images. With many tests I did, quite often a lot of items were mislabeled. Nonetheless cool idea and love the channel!
Absolutely! But keep in mind that 3 years ago it was impossible. We just try to highlight cutting-edge models in 2023. I absolutely agree. We are not yet able to get good results for every image.
Great video and notebook! However it looks like supervision install step fails with: groundingdino 0.1.0 requires supervision==0.4.0
Hey Peter, could you do a video showing how to integrate SuperGradients/Yolo NAS with Roboflow's Autodistill for custom detections on a live real-time webcam feed.
Could you also show maybe in another video how to add custom objects to an existing dataset like the coco dataset?
This would be Epic.🔥
Hi Sir I'm Beginner in I saw your Computer vision video's its fully combined and merged can you please update one by one video order that time we can understand easily thank you.
Hi, it is Peter from the video? Do you mean videos related to zero-shot annotations?
You're awesome man, thank you so much
I wonder if segment anything can be accelerated or if even it it would run in the google coral edge accelerator.
I heard you can use OpenVINO to run it on CPU. As long as it is Intel CPU.
I have noticed you use in the supervision awesome package a method to load datasets in PASCAL-VOC format, are you planning to also support COCO formats (also for export?)?
Incredible video ! I was just reading the Grounded-SAM this morning, and boum you're making a tutorial on it. Great job ! I'm just wondering if I could find ways to use it in a medical imagery task ! What do you think ?
You want to do full auto or bounding box to mask?
@@Roboflow I would go for automatic segmentation but I'd also like it to be interactive for the user. So maybe combining the two would more appreciated
@@alassanesakande8791 that is our plan for next stage. Allow full auto or human in the loop :) I also think that being able to interactively interact with those labels before you use them to train for example YOLOv8 is required.
As always a very cool video!
Really curious to see Autodistill tool🎉
Does smart polygon tool leverage SAM as well?
Yes it is! We are running SAM in smart polygon since last week 🔥
for "solar panel counting from UAV image"...which approach is better ? 1. creating bounding box (BB) for solar panel using object detection model and then using BB as input for SAM....or.... 2. segmenting everything in the image from SAM...and then classifying each segment as solar panel and non solar panel.
I guess that it would be great to include in both supervision and autodistill a feature that gets the bounding box given a polyline segmentation from sam
we have that already! supervision - roboflow.github.io/supervision/detection/utils/#mask_to_xyxy
thank you so much 😍
Thanks for watching! :)
Any chance for a tutorial on SAM and Roboflow and remote sensing of satellite or uav imagery?
Please tel me more about the idea? What would you like to see?
that's really great waited for that!!. btw why there is no support for tracking annotations formats like MOT/MOTS
I know it took me a lot of time... But this was possibly the most complicated Jupyter Notebook I ever made.
@@Roboflow that's it really great contribution for the community😎 thanks for that
@@gbo10001 we are working on something even beeeeter! 🔥
@@gbo10001 hahaha better than SAM + DINO
Great tutorial!! Is it possible to real time video? something like a webcam?
Thanks a lot! 🙏🏻 model is to slow to run in real time :/ the whole inference for single frame can take around 1-2 seconds.
How can I extract the segmented object produced by SAM?
Masks are stored here `detections.mask`.
great tutorial! can you post the link to the jupyter notebook in the vid bio?
It is in the description. But here is the link: colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/automated-dataset-annotation-and-evaluation-with-grounding-dino-and-sam.ipynb
Thank you for your work, this is exactly what we need urgently, but at the moment I see that it seems to only support saving data in Pascal voc format, do you have any plans to provide an api to convert it to coco format?
Currently the order is YOLO and than COCO. But it might happen next week.
@@Roboflow that's cool! the soon the better, thank you for your work again!
Hey there! I really like these videos a lot. Certainly with fast labelling the specific task can be trained supervised. But is there an opportunity in using SAM and/or DINO as a teacher for distillation into a smaller (final) model, even before creating an annotated dataset? Would this be competitive with other self-supervised pretraining methods?
Hi 👋🏻 you mean SAM and GDINO would generate training examples on the fly during the training?
@@sebbecht we didn't explore that rout yet but it would be awesome to test those theories. Thanks for sharing :) I never run out of ideas thanks to conversations like this.
@@Roboflow my pleasure, I hope you get to explore and share some findings!
@@sebbecht stay tuned :)
Can it be used to annotate for semantic segmentation or only instance?
in you previous video on grounding dino, you elaborated on a text prompt as an input, can this be implemented here as well? are you planning on extending this tutoorial (or notebook) to show how to implement it? also, I have noticed that you can also implement stable diffusion tools such as "change do to a monkey". can that also be in the next vid?
Auto labeling with prompts will be part of the auto-distill package that is coming soon. As for stable diffusion, I can't promise anything :/ We have a lot of stuff in the backlog. But maybe I'll play with it on Twitch stream.
@@Roboflow thanks a lot! any estimation regarding the release date of auto-distill?
@@kobic8 it is close! Reaaaaaaaly close!
@@kobic8 don't want to over promis but I heard something about today :)
thank to this great vid (and notebook) I have tried using it together with SAM and I'm curious to know how can I use a labeled dataset I have (of sea-objects) to learn the model to detect not only a boat/ship but to identify the name of the marine-vessel.
Do you have labels for marine-vessel in your dataset? Or only boat/ship?
@@Roboflow thanks so much for the reply! am really trying to figure out how to solve this issue: yes! I do have human-labeled dataset for specific classes of marine-vessels e.g., frigatte, corvette, and also some ships with their specific names. My question was if there is a way to fine-tune the grounded-DINO model to identify the objects not as "boat" or "ship" but on more accurate labels
@@kobic8 yes it probably is possible, but you would be much better of if you train model like YOLOv8. Power od GroundingDINO comes from zero shot detection - ability to detect objects that it never saw. If you already have annotated dataset, just train regular object detection model. :)
@@Roboflow but it be "less powerfull" compared to G-DINO, I just thought to tune G-Dino to refine specific labels, so I tought it be btter to somehow get the traning code
Can you combine separate polygons into a single object?
I keep getting error messages whenever I used some of the images in my dataset
Hey Peter! Can I use the SAM labelling for object detection as well? or is it only for instance segmentation?
You can always convert segmentation into detection. It is just a bit hm... poor usage of resources as it is super time-consuming. What project do you have in your mind?
@@Roboflow I'm working on detecting potato quality on a conveyer belt. I labeled some photos using SAM, but I'm not sure if the polygon labeling actually helps object detection or if a basic rectangle boundary will enough.
@@snehitvaddi yes, for modern models like YOLOv8 it helps: blog.roboflow.com/polygons-object-detection/
@@Roboflow cool, thanks
@@snehitvaddi use the one thet is faster to annotate? Polygons can be converted to boxes really easily.
I am currently working on pollution detection and classification system project, can I use GDINO and Sam for the same?
What would that be? Images of smoke for example?
@@Roboflow Images of plastic underwater and Oil Pollution in water
6:45 what if I want to detect the umbrella above
Or each of the lights in the umbrella
Can I convert a multiclass object detection dataset to a segmentation dataset with this? I have only seen the example with the single class Blueberries dataset so im not sure.
You can :)
Awesome tutorial!!!
But while I am running during 6:25, I got error: "NameError: name '_C' is not defined" (after long error description). Anyone can help?
Could you give me a bit more info? Do you run it in Google Colab?
@@Roboflow I have the same error. I ran the colab from top to bottom and got this error at the first annotation part on the line detections = grounding_dino_model.predict_with_classes(..
Any help would be appreciated!
@@thegodofrotation-animeamvs7204 I'll do my best to take a look at that. Could you submit new issue here: github.com/roboflow/notebooks/issues
Any update?
How to remove white borders from generated images?
Use cv2.imwrite to save the image on drive www.geeksforgeeks.org/python-opencv-cv2-imwrite-method/amp/ and manually download.
Hi anything for cancer cell application
Hi, Can this also be implemented on custom objects, if so how to implement it
What do you mean by custom object?
thank u for this incredible vid !💖 but I have a question, when trying to run the following command it told me that " 41 detections.mask = segment(sam_predictor=sam_predictor, image=image, xyxy=filtered_detections.xyxy)
42
43 mask_annotator = sv.MaskAnnotator()
NameError: name 'segment' is not defined "
and I search for the __init__ in SAM but there isn't found, so is this function is built in sam_anything module or should I wrote it ?
i replaced this command of yours
from tqdm.notebook import tqdm
for image_name, image in tqdm(object_detection_dataset.images.items()):
detections = object_detection_dataset.annotations[image_name]
detections.mask = segment(
sam_predictor=sam_predictor,
image=cv2.cvtColor(image, cv2.COLOR_BGR2RGB),
xyxy=detections.xyxy
)
Looks to me like you didn’t run all cells in notebook. Segment function is defined in one of the cells in notebook. No need to change the code.
@@Roboflow oh I see, thanks, it had been solved. can I ask another question? my dataset is into coco format as it on my PC not roboflow so I converted it into pascal format to be able to follow your steps from converting to segmentation but it didn't work at all. is it a function in supervision to read coco format like pascal? as I searched but it give me errors
@@saharabdulalim hi! We ant to add COCO loading to supervision but it won't happen to soon :/ if you wan to follow those steps now I'd upload dataset to Roboflow. That's probably the fastest way for now.
@@Roboflow is it possible to upload the whole dataset to RoboFlow?
without annotate every image as I have already the annotation file
wrg