Machine Learning System design — Data Labeling Pipelines, With One Content Moderation System Tracked From Raw Comment to Deployed Model (Part 9)
A trust-and-safety team at a video platform spent four months building what they called a state-of-the-art toxicity classifier. They labeled 800,000 comments through a major crowdsourcing vendor, achieved 0.91 inter-annotator agreement (Cohen's kappa above 0.85 is excellent), trained a transformer that hit F1 = 0.94 on holdout. They shipped it. Within two weeks, advocacy groups had filed formal complaints: the model was systematically under-flagging coded language used to harass particular communities. The model was not broken. The labels were. The annotator pool — recruited globally with an English-fluency filter — uniformly didn't recognize specific dog whistles, so they uniformly marked them "not toxic." Inter-annotator agreement was 0.91 because the annotators all shared the same blind spot. The model learned exactly what the labels said. This article walks one comment from a content moderation pipeline — a single 47-character text snippet — through every layer of a labeling system, naming the specific failure each layer is built to prevent.
The Core Insight First
The wrong model architecture does not cause most ML failures. Bad labels cause them.
A 2022 analysis of production ML incidents at large tech companies found that data quality issues — label noise and inconsistency, schema drift, annotator bias — caused more model degradation than any algorithmic choice. Your interviewer at Google or Meta almost certainly knows this. The question is whether you do.
A labeling pipeline is the system that takes raw, unlabeled data and turns it into something a model can actually learn from. That sounds simple. In practice it is a multi-stage operation with four failure modes baked into every stage:
THE FOUR FAILURE MODES TO CATCH
───────────────────────────────
1. NOISE labels themselves are wrong
→ quality control catches this
2. BIAS annotators consistently agree on the WRONG answer
→ a gold-standard validation set catches this
3. DRIFT guidelines mean different things week 1 vs week 12
→ versioned ontology + IAA-over-time catches this
4. STAGNATION labels cover the easy cases, model never learns the hard ones
→ active learning loops catch thisEvery pattern in this article maps to one of these four. Every anti-pattern is a different way of failing to catch one of them.
The Anchor Setup
Domain : video platform · public comments
Task : multi-class toxicity classification (5 classes)
non-toxic / mild / harassment / hate / threats
Volume : 800,000 comments to label · 12M unlabeled in pool
Schema version : v3 (revised 2024-08-15 after legal review)
Annotator pool : 240 crowdworkers via Scale AI · 12 expert moderators
Labels per task : 3 crowd workers + 1 expert review on disagreement
Cost : $0.04 per crowd label · $5.00 per expert label
Quality gates : 5% honeypot rate · 0.85 Cohen's kappa minimum
Gold standard set : 2,400 examples · domain-expert adjudicated
Anchor example : the 47-character comment that broke productionBy the time the model ships, this one comment has been seen by three independent crowdworkers, two senior reviewers, a label model that reconciled votes, and a confidence calibrator that decided whether to trust the consensus. Every stage is a place where the bug hides.
How It Works — Five Stages, Not One
Most candidates describe labeling as "send data to Scale AI, get labels back." That is one step in a much longer chain. The full lifecycle has five distinct stages:
1. INGESTION raw comments queued from product logs and unlabeled pool
│
↓
2. TASK DECOMPOSITION "is this toxic?" broken into atomic decisions
│ → contains slur? targets group? threatens violence?
↓
3. ANNOTATION platform routes tasks to annotators (Scale, Labelbox, ...)
│ IAA scoring runs as labels return
↓
4. QUALITY CONTROL honeypot accuracy · IAA threshold · adjudication queue
│ ONLY labels passing review get written
↓
5. VERSIONED EXPORT labels written with metadata: who, when, schema_version,
confidence, gold-standard agreement
→ consumed by training pipelineThat is the offline lifecycle. The five-stage pipeline is the minimum — but it is also a one-shot process. The actual production system has a sixth stage that closes the loop:
The pipeline is a loop, not a one-shot process. Once the model is deployed, its low-confidence predictions become the next annotation batch. The model tells the system where it is struggling. That feedback loop is what separates a labeling pipeline from a labeling project.
This is the single most important framing in the entire interview. If you describe labeling as something you set up, run once, and then forget about — you have signaled that you have not shipped one of these in production.
The Four Patterns
Each pattern lives at a different point on the cost × throughput × quality triangle. Real production systems combine them.
Pattern 1 — Crowdsourced Annotation with Consensus Voting
Send the same example to multiple independent annotators (typically 3 to 5). Annotators do not see each other's answers. A consensus engine resolves the votes — majority vote at the simple end, Dawid-Skene at the sophisticated end (weights each annotator's vote by their historical reliability). Examples where disagreement exceeds a threshold get routed to an adjudication queue.
The tradeoff: high throughput at low cost, but individual annotator quality is variable. Compensate with redundancy and quality gates (honeypot tasks). This pattern breaks down when the task requires real expertise — radiology scans, legal clauses, dog-whistle detection.
Cost: low. Throughput: high. Quality: medium. Use for: content moderation, sentiment, object detection in everyday images.
Pattern 2 — Programmatic Weak Supervision
You have millions of examples and no budget to hand-label them. Write labeling functions (LFs): short heuristics, regex patterns, keyword lists, or calls to a pre-trained model. Each LF casts a noisy vote on a label. No single LF is reliable — but a label model (Snorkel's generative model is the canonical example) learns each LF's accuracy and correlation with others, then combines their votes into probabilistic soft labels.
The downstream training pipeline consumes the soft labels directly, often weighting examples by label confidence. You never get perfectly clean labels — but you get coverage over the entire corpus in hours instead of weeks.
Interview-winning sentence: "A label model learns each LF's empirical accuracy and the correlations between LFs, then produces calibrated soft labels — not just averaged votes. That calibration is what makes weak supervision actually work."
The discipline is maintaining an LF analysis dashboard: track each function's coverage (what fraction of examples it fires on), conflict rate with other LFs, and empirical accuracy against a small gold-standard validation set.
Cost: very low. Throughput: very high. Quality: noisy (probabilistic). Use for: spam, medical record classification, NLP tasks where regex captures meaningful signal.
Pattern 3 — Active Learning Loops
Flips the usual workflow. Instead of randomly sampling what to label next, the current model scores the unlabeled pool and surfaces the examples it finds most confusing. Those examples — if labeled — would move the decision boundary the most.
Three common selection strategies:
- Least confidence: pick examples where the top predicted class has the lowest probability
- Margin sampling: smallest gap between top two classes
- Query-by-committee: highest disagreement across an ensemble of models
The loop: train → score unlabeled → send most uncertain to annotators → add new labels → retrain → repeat. Batch the uncertainty sampling rather than retraining after every label, because retraining is expensive.
The payoff is significant: active learning can match the performance of random sampling with a fraction of the labeled data. Critical when expert annotation costs $50/example.
The bootstrap problem: you need an initial labeled seed (a few hundred examples) to train the first model. Without that, there is no model to generate uncertainty scores.
Cost: high (expert time). Throughput: low. Quality: high. Use for: expensive annotations (medical imaging, legal review), specialized domain adaptation.
Pattern 4 — Model-Assisted (Pre-labeling) Annotation
Most candidates forget this one. Run an existing model — even a foundation model like GPT-4 or fine-tuned BERT — over your unlabeled data to generate draft labels. A confidence filter splits the output: high-confidence predictions get auto-accepted; low-confidence cases go to a human annotator who sees the draft label and only needs to confirm, correct, or reject.
That "confirm or correct" UI is the throughput multiplier. Annotators working from scratch label ~200 images per hour. Annotators reviewing model drafts hit 600 to 800 per hour, because most predictions are right and a click to confirm is faster than typing a label from scratch.
The compounding flywheel: better labels → better model → better pre-labels → less annotator effort → cheaper to label more data → better labels. Mention this in an interview unprompted and you signal that you have thought about labeling as an engineering problem, not just a procurement one.
Cost: low-medium. Throughput: very high. Quality: high (human-verified). Use for: anytime a working model exists.
Real systems rarely pick one. A common combination: weak supervision generates noisy labels across millions of examples, active learning identifies the highest-uncertainty subset, expert annotators review only those with model-assisted pre-labeling to keep their throughput high. That layered approach is the senior answer.
The Four Anti-Patterns
Anti-Pattern 1 — Treating Labels as Binary
"Once the data is labeled, we feed it into training." Full stop. No mention of confidence, annotator reliability, or disagreement.
The interviewer asks: "What if some of those labels are noisy?" Candidate freezes — or worse, says "we'd just relabel them."
Every label in a real pipeline has a confidence score, an annotator reliability weight, and a history of how many people agreed. Training on labels as if they are ground truth is lying to your model.
The fix: treat labels probabilistically. High-agreement labels from reliable annotators get full weight in training. Low-confidence labels get reduced loss weight (label smoothing, noise-aware loss) or routed back for review. Soft labels are the default, not the exception.
Anti-Pattern 2 — Forgetting That Label Schemas Change
The subtle one. You design a content moderation system, label 500K examples as "toxic" or "not toxic," and six months later your policy team redefines what "toxic" means. Now you have 500K labels built on a definition that no longer exists.
Candidates who do not mention schema versioning are implicitly assuming guidelines are static. They are not. Guidelines evolve as edge cases surface, as legal requirements shift, as the product changes.
The fix: a versioned ontology. Every label is tagged with the schema version it was produced under. When the schema changes, you have an explicit decision: relabel the affected subset, discard, or train on a mixture with schema version as a feature. Mixing schema versions silently is how you corrupt a dataset.
Anti-Pattern 3 — Confusing Agreement with Correctness
This is the bug from the opening anecdote. "We measure inter-annotator agreement to make sure our labels are high quality."
High IAA means annotators agree with each other. It says nothing about whether they are right. For subjective tasks (toxicity, sentiment), a pool of annotators can consistently agree and consistently be biased in the same direction. If your annotator pool skews toward a particular demographic, they uniformly under-flag certain types of harmful content. IAA looks great. Your model learns a biased definition of harm.
The fix: separate agreement metrics from accuracy metrics. Maintain a gold-standard validation set — constructed by domain experts or through formal adjudication — that you use to measure annotator accuracy independently of how much they agree with each other. IAA is necessary but not sufficient. Bringing this up unprompted is one of the clearest senior signals you can send.
Anti-Pattern 4 — Treating Production as Outside the Labeling System
A candidate walks through an elegant offline labeling pipeline. Interviewer asks: "How does this system improve over time?" Candidate says: "We'd periodically collect more data and label it."
That is leaving the best signal on the table. Production generates implicit labels constantly. User corrections, rejection rates, escalation patterns, click-through behavior — all weak supervision signals. A user flagging a recommendation as irrelevant is a label. A moderator overriding an automated decision is a label.
The fix: frame the pipeline as a closed loop. After deployment, low-confidence predictions queue back for human review. User feedback is captured as weak labels. The system gets smarter with every interaction, not only between training runs.
Full Trace — One Comment Through the Whole Pipeline
day 0 · 10:23 │ comment c-1f8a posted on video v-3201
│ "you're really not what we said you'd be lol"
│ 47 chars · ambiguous on its face
│
day 0 · 10:24 │ deployed model scores comment
│ P(non-toxic)=0.32 P(harassment)=0.41 P(hate)=0.27
│ confidence margin = 0.41-0.32 = 0.09 → LOW
│ → routed to annotation queue
│
day 0 · 11:15 │ enters Scale AI task pool
│ schema version = v3
│ task decomposed:
│ Q1: contains slur? → no
│ Q2: targets a group? → ???
│ Q3: implies threat? → ???
│ Q4: requires context to interpret? → yes
│
day 0 · 14:02 │ 3 crowd annotators independently label
│ annotator_a47 → "non-toxic" (confidence 0.7)
│ annotator_b12 → "harassment" (confidence 0.6)
│ annotator_c89 → "non-toxic" (confidence 0.5)
│ consensus by simple majority: non-toxic
│ Cohen's kappa for this triple: 0.31 → BELOW threshold
│
day 0 · 14:03 │ low-IAA flag triggers adjudication
│ → routed to senior reviewer queue
│
day 1 · 09:30 │ expert reviewer e-04 sees it WITH context
│ context: prior comment thread targeting v-3201's creator
│ verdict: harassment (confidence 0.95)
│ + appended note: "common dog-whistle pattern, escalate"
│
day 1 · 09:31 │ written to versioned label store:
│ label = harassment · schema_v3
│ reviewer = e-04 · gold-standard = pending
│ confidence = 0.95
│ annotator_history = stored
│ ALSO: added to gold-standard candidate set
│
day 7 │ aggregated weekly: this example used to update
│ annotator reliability scores for a47, b12, c89
│ → a47 reliability: 0.78 → 0.74
│ → b12 reliability: 0.71 → 0.76
│ → c89 reliability: 0.65 → 0.61
│
day 14 │ batch retraining
│ comment now part of training set
│ + weak label model retrained on full corpus
│ + new pre-labeling heuristic added: "context-dependent"
│
day 30 │ deployed v2 model evaluated against gold standard
│ recall on dog-whistle harassment: 0.42 → 0.71
│ → loop closesThe Question That Remains Open
A labeling pipeline is the system that decides what your model learns. Five stages (ingestion → decomposition → annotation → QC → versioned export) plus a closing feedback loop. Four patterns (crowdsourced, weak supervision, active learning, model-assisted). Four anti-patterns (binary labels, schema drift, agreement ≠ correctness, production-outside-the-loop). One closed loop that compounds.
But the deepest question this layer opens is the one governance and accountability has to answer:
Your labeling pipeline produced 800K labels. Six months later, an external audit asks: "How were these labels produced? Who made the call? Under what guideline? Was the annotator pool demographically representative? Can you reproduce the labels you trained on if a regulator asks?" If the answer to any of those is "we don't know" or "we'd have to dig," your pipeline has a governance gap, not a quality gap. How do you build the layer above labeling that makes every label auditable and every annotator accountable?
Provenance, demographic balance of annotator pools, adjudication audit trails, the right to explanation — these are the next layer of ML infrastructure most teams have not built. Once your labeling pipeline produces millions of labels, the bottleneck stops being throughput and becomes trust.
What's your answer?
This is Part 9 of an ongoing series on ML systems and recommender infrastructure. Part 1: ML system design framework. Part 2: feature stores. Part 3: model serving and inference. Part 4: training pipelines. Part 5: monitoring and observability. Part 6: embedding systems. Part 7: feature engineering at scale. Part 8: experiment platforms. Part 9: this piece.