Post cover image

April 13, 2026

Auto-Labeling Unknown Objects: A Vision Model Pipeline for What Models Can’t See

SAM3 + Grounding DINO + VLM — run multiple strategies in parallel, pick the best for your object.

By kyon

12 min read

1. Annotation Is a Grind

Training an object detection model takes labeled data. Lots of it. And drawing bounding boxes by hand is, frankly, miserable work.

Open an image. Scan for the object. Carefully drag a rectangle around it. Next image. And the next. Hundreds, thousands of times. It's tedious, slow, draining, and you inevitably miss things. Nobody wants to do this.

None

Open-vocabulary models — Grounding DINO, SAM3, and their ilk — might save us. Type "cat" and they find cats. Type "traffic light" and they nail it. So can they label your objects automatically too?

If the model already knows the concept, yes. For objects within an open-vocabulary model's training vocabulary, text prompts alone are enough to auto-generate usable annotations.

But what happens when the model has never seen your object?

For this experiment, I happened to pick shrimp in underwater footage. Feed "shrimp" to Grounding DINO, and here's what you get:

"shrimp"  → Recall = 0.000
"fish"    → Recall = 0.761
"crab"    → Recall = 0.843

Zero. Not low confidence — zero detections. The word "shrimp" simply isn't in the model's vocabulary. I tried 11 vision models across both text and visual prompting modes. Not a single one could reliably detect shrimp from reference images alone.

What This Article Covers

I built a pipeline that auto-labels unknown objects using just a handful of reference crop images. No object names. No fine-tuning. No iterative training.

The goal isn't full automation. It's a fundamental change in what the work looks like:

  • Before: Open each image, hunt for the object, draw a bounding box from scratch
  • After: Look at pre-generated candidate crops, hit Accept or Reject

No more searching. No more drawing. Just making calls. That alone is a dramatic improvement in throughput.

A note up front: The results and techniques shown here are specific to underwater shrimp. Different objects will have different sweet spots. The pipeline is designed to be customizable — you run multiple strategies in parallel and iteratively converge on whatever works best for your particular object.

2. Experimental Setup

Dataset

AAU Brackish Underwater Dataset (CC BY 4.0) — underwater footage containing six classes: crab, fish, jellyfish, shrimp, small_fish, starfish. All six fall outside COCO-80, making them genuinely unknown to standard open-vocabulary detectors.

Target: shrimp — 76 instances across 1,467 validation images. Small, translucent, and easily lost in the background.

Hardware & Models

  • GPU: NVIDIA RTX 5090 (32GB VRAM)
  • SAM3: Segment Anything 3 (via Ultralytics, SAM3SemanticPredictor)
  • Grounding DINO v1: Text-guided detection (HuggingFace transformers)
  • Qwen3-VL-8B: Vision-Language Model (for environment detection)

Design Philosophy: Don't Try to Nail It in One Shot

Annotation generation isn't a real-time process. You create the labels once, then train your model as many times as you want. There's no need to optimize for speed.

Instead, run multiple approaches in parallel, compare the outputs, and pick whatever configuration works best for your object. Try SAM3 at several confidence levels. Generate a range of prompt templates. Mix and match verification methods. Look at everything, then decide.

There's no single approach that works universally. Every object has its own sweet spot. This "parallel exploration → interactive selection" mindset is the foundation of the pipeline design.

3. Baseline: 11 Models, Zero-Shot

Before building anything, I benchmarked what's already out there:

Model Mode shrimp F1 Verdict Grounding DINO v1 Text: "shrimp" 0.000 Not in vocabulary Grounding DINO v1 Text: all 6 classes 0.694 (overall) Best overall SAM3 Text 0.392 Moderate DINO-X API Text 0.119 Paid + poor OWL-ViT v2 Visual 0.004 Failed YOLOE26 Text / Visual 0.018 / 0.002 Failed DINOv2 Prototype 0.000 Failed Rex-Omni Visual 0.000 Failed Qwen3-VL-8B VLM grounding 0.019 Failed

Two findings:

  1. Text prompts are the only viable path. All five visual prompting approaches failed outright. Reference images couldn't get the job done for unknown objects.
  2. For known objects, auto-labeling already works. The challenge is exclusively with unknowns.

4. Descriptive Prompts: Detecting What You Can't Name

This is the insight that made everything else possible.

GDino doesn't know "shrimp." But it understands descriptions:

"shrimp"                     → R = 0.000  ❌
"underwater creature"        → R = 0.553  ★
"small translucent creature" → R = 0.368  ★

The model can't look up "shrimp" in its vocabulary, but it can match the visual concept of "underwater creature" against what it sees in the image. GDino's text encoder learned to associate descriptive phrases with visual features during pre-training — it just never encountered the specific word "shrimp."

Automating Prompt Generation

Users shouldn't have to guess which descriptions work. I use a VLM (Qwen3-VL-8B) to analyze reference crops and detect the environment automatically:

Reference crop + full image → VLM → "underwater"
→ Templates: "underwater creature", "underwater animal", "underwater organism"

Why rigid templates instead of free-form VLM descriptions? I'll cover that in Section 6, but the short version: adjectives destroy GDino's discriminative power.

Discovery Is Where It All Starts

Descriptive prompts can now find shrimp. But they also find everything else. GDino returns over 9,000 detections across the dataset, of which roughly 70 are real shrimp.

Here's what matters: you can't annotate what you can't find. No filter, however precise, can rescue a ground-truth object that never made it into the candidate pool. That makes Discovery — the fraction of GT objects captured at the candidate stage — the top priority.

False positives can always be dealt with later. A missed object is gone for good.

5. The Key Shift: Segment First, Then Classify

I tried filtering GDino's full-image detections in every way I could think of — CLIP similarity, confidence deltas, area filters, VLM verification, SVMs. The best any of them managed was getting half the labels right.

The breakthrough came from reframing the problem entirely.

None

The Old Way: Detect → Filter (Didn't Work)

Full image → GDino "underwater creature" → 9,145 detections (TP=69, FP=9,076)
→ Filter → Best result: half the labels are wrong

GDino's full-image detections are noisy: overlapping boxes, multiple objects per box, pure background. Trying to sort the good from the bad in that mess is extremely hard.

The New Way: Segment → Classify (Worked)

Full image → SAM3 segmentation → ~10K clean segments (1 crop = 1 object)
→ Each crop → GDino "underwater creature" → classification score
→ Threshold → High-quality labeling ★★

Why this works: SAM3 produces non-overlapping masks with clean object boundaries. Each crop contains exactly one thing. At that point, GDino isn't doing noisy detection anymore — it's doing binary classification: "Is this crop an underwater creature?" Separating TPs from FPs gets dramatically easier.

SAM3 Configuration Matters

Two settings that made a significant difference:

1. Domain-specific category words are essential.

SAM3 with ["object"] only:                   Recall = 53.9%  ❌
SAM3 with ["animal", "creature", "object"]:  Recall = 78.9%  ✅

SAM3 is a vision-language model. Broad category words like "animal" activate attention toward biological entities in a way that "object" alone doesn't. This is a biology-specific setting — other domains would need different category words.

2. Lowering confidence captures more TPs.

SAM3 conf = 0.15:  Recall = 60.5%, TP = 124
SAM3 conf = 0.10:  Recall = 80.3%, TP = 215 (+73%)

More false positives come along for the ride, but the downstream GDino classification step handles those. The optimal threshold depends on the object — which is why you run multiple levels and compare.

from ultralytics.models.sam.predict import SAM3SemanticPredictor
predictor = SAM3SemanticPredictor(overrides={
    "model": "sam3.pt", "conf": 0.10, "imgsz": 1008
})
predictor.prompts = {"text": ["animal", "creature", "object"]}
results = predictor(image_path)

(Note: SAM3's auto-segmentation mode isn't supported in the Ultralytics API. You need SAM3SemanticPredictor with text prompts.)

6. Full Automation: What VLMs Should (and Shouldn't) Do

The segment→classify pipeline works — but I was hand-picking prompts. To be genuinely reusable, everything needs to run on its own.

The Adjective Trap

First attempt: have the VLM generate rich descriptive prompts like "smooth undersea item" or "pale round organism." Sounds reasonable. But tested against 9,318 SAM3 crops:

Prompt                              GDino Discrimination (TP vs FP mean score gap)
"underwater animal"                 0.144  ★★★  2-word template
"underwater creature"               0.136  ★★★  2-word template
"pale underwater animal"            0.138       +1 adjective
"small underwater creature"         0.122       +1 adjective
"underwater smooth round organism"  0.034  ❌   +3 adjectives (4x worse!)

The more adjectives you add, the worse GDino gets at telling TPs from FPs. Why? GDino's text-image matching is a coarse category judgment. Words like "smooth" and "round" match target crops and background crops equally, collapsing the gap.

The sweet spot for the VLM: just detect the environment.

VLM → ENV = "underwater"
→ Templates: "underwater creature", "underwater animal", "underwater organism"

This is a finding from shrimp specifically — adjectives might be useful for other objects. That's exactly why you generate multiple prompt candidates, let GDino's discriminative screening rank them automatically, and go with whatever the data says works best.

The SAM3 Prompt Contamination Problem

A subtler issue: VLM-generated prompts like "smooth undersea item" were being passed to SAM3 alongside the template prompts. They didn't just fail at classification — they actively misdirected SAM3's segmentation:

SAM3 with base + VLM prompts:     TP = 173
SAM3 with base + templates only:  TP = 215 (+24%)

The fix: only feed template prompts to SAM3. Reserve everything else for the GDino classification step.

Fully Automated Results

Metric Manual Automated Human intervention 3 steps 0 Label quality Good Comparable

The automated pipeline matches the manual version's performance with zero human intervention. The entire flow — from reference crops to labeling data — runs unattended.

7. Discovery: Capturing 96% of Ground Truth

With the pipeline automated, the next question is: how many GT objects actually make it into the candidate pool? That's the Discovery rate, and it sets the ceiling for everything downstream.

What doesn't get found can't get annotated. Discovery is the hard limit.

SAM3 + GDino Complement

SAM3 at conf=0.10 catches 61/76 GT objects. But running GDino directly on full images picks up some of what SAM3 misses — the two models fail in different ways:

SAM3 alone:          61/76 GT (80.3%)
+ GDino complement:  73/76 GT (96.1%)  ★  +16 points

That puts 96% of GT objects in the candidate pool. That's the ceiling for what we can annotate.

The trade-off: GDino complement adds 4,241 extra candidates, only 18 of which (0.4%) are true positives. More candidates, but almost all noise. How you handle those depends on your verification strategy.

Annotation Quality

Discovery is meaningless if the bounding boxes are junk. They aren't:

  • Mean IoU vs. GT: 0.799
  • GT objects with IoU ≥ 0.7: 86%

I visually inspected all 73 overlay images. The bboxes closely follow the shrimp outlines. Pixel-perfect precision isn't the goal — what matters is whether a human reviewer would accept them as-is. They would.

None None None None

The GDino Complement Trade-off

Whether to use GDino complement depends on the object:

  • If SAM3 alone already gives you sufficient Discovery (say, 90%+), skip the complement. It just adds FPs.
  • If SAM3 falls short (80% → 96% in this case), the complement is a major win.

Again, don't lock in a choice upfront — run it both ways and compare.

8. Verification: Separating Signal from Noise

Discovery puts 96% of GT into the candidate pool. Now comes the sorting.

Automated Verification Attempts

I wanted to filter down to TPs automatically. Several approaches:

Method Result Issue CLIP cosine similarity Near-zero discrimination Underwater crops look too similar VLM "Is this a creature?" Too many TPs rejected VLM can't judge small, blurry crops GDino threshold (with supplements) Supplement FPs overwhelm Only 0.4% of supplement candidates are TPs

For shrimp, fully automatic verification hit a wall. But this is a shrimp-specific outcome. For more visually distinctive objects, or domains where CLIP has better coverage, automatic verification may work much better.

A Better Use for GDino Scores

GDino scores aren't reliable enough for automatic filtering, but they're excellent for something else: prioritizing human review.

Higher-scoring candidates are more likely to be TPs. Lower-scoring ones are mostly FPs. Sort by score, review from the top, and stop when you have enough labels.

Strategy GT Coverage Review Time Auto-accept high scores only 30% 0h Auto + review top tier 54% ~1h Auto + review mid tier 83% ~2h Full review 96% ~4h

Where you draw the line depends on how many labels your project needs and how much time you can spare.

9. Not Replacing Humans — Making Them Faster

This pipeline doesn't fully automate annotation.

But that's the wrong question. It's not "Can it replace humans?" — it's "How much faster can it make them?"

Before vs. After

Without the pipeline (traditional annotation):

  1. Open 1,467 underwater images in a viewer
  2. Scan each one for shrimp (small, translucent, easy to miss)
  3. Draw bounding boxes from scratch for each one found
  4. Repeat. Miss some. Lose focus. Make mistakes.

With the pipeline:

  1. Run the pipeline
  2. Review pre-generated candidate crops with bboxes already drawn
  3. Accept or Reject each candidate
  4. Export accepted labels as labeling data

The cognitive load is entirely different. No searching for objects — the pipeline already found 96% of them. No drawing boxes — bboxes at IoU = 0.80 are ready to go. You're just making judgment calls.

This isn't a failure of automation. It's the right division of labor between human and AI. The pipeline handles the heavy lifting — detection and candidate generation. The human handles the easy part — deciding yes or no.

10. Pipeline Overview: Customization Axes

The pipeline has three phases, each with a knob you should tune per object.

None 
PHASE 1: Prompt Generation
  ┌──────────────────────────────────────────────────┐
  │ VLM → Environment detection → Template prompts    │
  │                                                    │
  │ Customization axis: CATEGORY NOUNS                 │
  │   Biology:    creature, animal, organism            │
  │   Industrial: part, component, piece  (untested)   │
  │   Medical:    lesion, tissue, mass    (untested)    │
  └──────────────────────────────────────────────────┘
PHASE 2: Discovery
  ┌──────────────────────────────────────────────────┐
  │ SAM3 primary segmentation                         │
  │ + GDino complement detection (optional)           │
  │                                                    │
  │ Customization axis: COMPLEMENT ON/OFF              │
  │   SAM3 sufficient → complement OFF (fewer FPs)     │
  │   SAM3 insufficient → complement ON (Discovery ↑)  │
  └──────────────────────────────────────────────────┘
PHASE 3: Verification
  ┌──────────────────────────────────────────────────┐
  │ GDino classification scoring → review / auto-accept│
  │                                                    │
  │ Customization axis: VERIFICATION STRATEGY          │
  │   High GDino gap → lean into auto-accept           │
  │   Low GDino gap → lean into human review           │
  └──────────────────────────────────────────────────┘
OUTPUT
  ┌──────────────────────────────────────────────────┐
  │ Labeling data (bboxes + class name)                │
  └──────────────────────────────────────────────────┘

Parallel Exploration → Interactive Selection

You can't know the optimal configuration in advance. Instead:

  1. Run SAM3 at multiple confidence levels (0.10, 0.12, 0.15) — compare the TP/FP trade-off
  2. Generate multiple prompt templates — let GDino's discriminative screening rank them automatically
  3. Try both with and without GDino complement — weigh Discovery against FP volume
  4. Look at the results, then pick a strategy — the GDino gap tells you which verification approach to use

All of this runs offline. A single pipeline run gives you everything you need to make an informed call. From there, you tune interactively based on what the data shows for your particular object.

A different object — say, defect inspection on an assembly line — will have a different optimal setup. That's expected. The pipeline gives you the tools to find it.

What the Pipeline Requires

Input: 1–3 reference crop images + 1 full-scene image + class name

What it does NOT require:

  • An object name the model already knows
  • Model fine-tuning
  • Iterative training loops
  • Real-time inference
  • A large labeled dataset

11. Takeaways

1. Unknown objects need a different approach. Open-vocabulary detection fails for objects outside the model's training vocabulary. Descriptive prompts ("shrimp" → "underwater creature") bridge the gap.

2. Segment first, then classify. Running GDino on full images makes filtering nearly impossible. Segment with SAM3 first, then classify individual crops with GDino, and discrimination improves dramatically.

3. Let the VLM detect the environment — nothing more. Having VLMs generate adjective-heavy descriptions degrades GDino's discriminative power. The best prompt is just "{environment} {category noun}." That said, this is a shrimp-specific finding and may not hold for every object.

4. Discovery sets the ceiling. You can't annotate what you can't find. SAM3 + GDino complement captured 96% of GT objects, with bbox quality at IoU = 0.80 — fully usable for annotation.

5. Don't try to get it right in one shot. Run multiple confidence levels, prompt variants, and complement settings in parallel. Review the results interactively and converge on the best configuration for your object. Annotation is an offline process — spend the compute.

6. Don't replace humans — make them faster. Drawing bboxes from scratch → Accept/Reject on pre-generated candidates. That shift alone slashes annotation cost and cognitive load.

7. Results will differ for every object. Everything in this article is tuned for shrimp. A different object will call for different settings. The customization axes — category nouns, Discovery strategy, verification strategy — exist for exactly this reason.

Appendix

A. Detailed Comparison: 27 Methods Tested

(For readers who want the full picture.)

Verification Methods

Method                        Verdict   Notes
───────────────────────────────────────────────────────────
v4.4 fixed (auto)             ★★★      best fully automated
SAM3(+VLM,c=0.10)→GDino      ★★       best with manual prompts
SAM3(+VLM,c=0.15)→GDino      ★        highest precision
v4.3 template (auto)          ★        first auto success
SAM3→GDino                    ★        first segment→classify
SAM3→CLIP LR                  ★        learning-based
CLIP SVM                      ★        high precision, low recall
───────────────────────────────────────────────────────────
Quad filter                   ❌       best detect→filter result
v5.1 GDino+VLM               ❌       VLM verification limited
CLIP similarity               ❌       ineffective underwater
Multi-prompt intersection     ❌       detects same region

Discovery Methods

Method                   GT Coverage    Notes
──────────────────────────────────────────────────────
SAM3(c=0.05)+GDino       98.7%         FP explosion
SAM3(c=0.10)+GDino       96.1%         ← practical best
GDino v1 t=0.15          90.8%         standalone
SAM3 c=0.10              80.3%         primary method
SAM3 c=0.15              60.5%         conservative

B. Key Negative Results

Dead ends that might save you time:

  1. GDino scores don't predict correctness. A high score doesn't mean it's a TP. You can't threshold your way out.
  2. More reference images can make things worse. Averaging dilutes the discriminative signal.
  3. CLIP couldn't separate underwater crops. Domain-dependent limitation.
  4. Multi-prompt intersection doesn't filter. GDino detects the same spatial region regardless of prompt.
  5. Letting the VLM pick SAM3 base nouns fails. It chooses the wrong words and recall drops.

C. References

Models & Libraries

Dataset

Related

All experiments ran on a single RTX 5090 (32GB VRAM).