Home Notes Blog About
Updated:

Semi-Supervised Learning Guide

Semi-supervised learning is the “best of both worlds” approach that combines the power of supervised learning with the abundance of unsupervised data. It’s particularly valuable when you have a small amount of labeled data and a large amount of unlabeled data.

The Semi-Supervised Scenario

The Challenge: Labeling data is expensive, time-consuming, or requires expert knowledge The Opportunity: Unlabeled data is often abundant and cheap to collect

Typical Data Distribution:
📊 Labeled Data:     100-1,000 examples    (expensive)
📊 Unlabeled Data:   10,000-1,000,000 examples (cheap/free)

Semi-Supervised Goal:
Use the small labeled set + large unlabeled set to perform better than
using labeled data alone

Why Semi-Supervised Learning Works

Fundamental Assumptions

1. Smoothness Assumption

If two data points are close in the input space, their outputs should be similar

  • Example: Similar-looking medical scans should have similar diagnoses

2. Cluster Assumption

Data points in the same cluster are likely to have the same label

  • Example: Emails that group together are likely all spam or all not-spam

3. Manifold Assumption

High-dimensional data lies on a lower-dimensional manifold

  • Example: Images of faces vary in systematic ways (pose, lighting, expression)

Semi-Supervised Learning Methods

1. Self-Training (Self-Labeling)

How it works: Train on labeled data, then use the model to label unlabeled data

# Self-Training Process
1. Train initial model on labeled data
2. Use model to predict labels for unlabeled data
3. Add high-confidence predictions to training set
4. Retrain model on expanded dataset
5. Repeat until convergence or no improvement

Advantages:

  • Simple to implement
  • Works with any supervised algorithm
  • Intuitive approach

Disadvantages:

  • Can amplify initial mistakes
  • Sensitive to threshold selection
  • May not work if initial model is poor

Best for: Problems where the initial supervised model has reasonable accuracy

2. Co-Training

How it works: Train multiple models on different feature sets, then have them teach each other

Requirements:

  • Features can be split into independent views
  • Each view should be sufficient for learning

Process:

  1. Split features into two views (e.g., text: words vs. linguistic features)
  2. Train one classifier on each view using labeled data
  3. Each classifier labels unlabeled examples for the other
  4. Add high-confidence predictions to training sets
  5. Iterate

Example: Email classification

  • View 1: Email content (words, phrases)
  • View 2: Email metadata (sender, time, structure)

Advantages:

  • Reduces error propagation compared to self-training
  • Works well when feature views are truly independent

Best for: Problems with natural feature splits (text, multi-modal data)

3. Multi-View Learning

Extension of co-training: Handle multiple feature views systematically

Applications:

  • Web page classification: Text content + link structure + visual layout
  • Medical diagnosis: Symptoms + lab results + imaging data
  • Speech recognition: Audio features + visual lip movements

4. Graph-Based Methods

How it works: Build a graph connecting similar data points and propagate labels

Process:

  1. Create graph where nodes are data points (labeled + unlabeled)
  2. Connect similar points with weighted edges
  3. Propagate labels through graph connections
  4. Points connected to labeled examples get similar labels

Key Algorithms:

  • Label Propagation: Iteratively spread labels through graph
  • Label Spreading: Variant that preserves original labels

Advantages:

  • Naturally incorporates data structure
  • Can handle complex data relationships
  • Works well with manifold assumption

Best for: Problems where similarity relationships are well-defined

5. Generative Models

How it works: Model the joint distribution of features and labels

Approach:

  • Fit a generative model (e.g., Gaussian Mixture Model) to all data
  • Use labeled examples to constrain label assignments
  • Unlabeled data helps estimate feature distributions

Example: Email classification with Naive Bayes

# Generative Semi-Supervised Process
1. Estimate P(word|spam) and P(word|not_spam) from labeled emails
2. Use all emails (labeled + unlabeled) to get better word distributions
3. Apply Bayes rule to classify new emails

Advantages:

  • Principled probabilistic approach
  • Can work with very little labeled data
  • Provides uncertainty estimates

Best for: Problems where generative model assumptions fit the data well

6. Pseudo-Labeling with Deep Learning

Modern approach: Use neural networks for semi-supervised learning

Techniques:

  • Pseudo-labeling: Add confident predictions to training set
  • Consistency regularization: Encourage similar predictions for similar inputs
  • MixMatch: Combines multiple semi-supervised techniques

Process:

  1. Train network on labeled data
  2. Generate pseudo-labels for unlabeled data
  3. Train on both labeled and pseudo-labeled data
  4. Use data augmentation and consistency losses

Real-World Applications

1. Natural Language Processing

Problem: Labeling text is expensive (requires human annotators) Solution: Use large amounts of unlabeled text with small labeled datasets

Examples:

  • Sentiment analysis: Few labeled reviews + many unlabeled reviews
  • Named entity recognition: Few labeled documents + large text corpora
  • Machine translation: Limited parallel sentences + monolingual text

2. Computer Vision

Problem: Image labeling requires expert knowledge or is time-intensive Solution: Leverage abundant unlabeled images

Examples:

  • Medical imaging: Few labeled scans + many unlabeled scans
  • Satellite imagery: Limited ground truth + abundant satellite photos
  • Object detection: Few bounding boxes + many unlabeled images

3. Speech Recognition

Problem: Transcribing audio is time-consuming Solution: Use large amounts of unlabeled audio

Examples:

  • Voice assistants: Few transcribed samples + large audio databases
  • Accent adaptation: Limited accent-specific data + general speech data
  • Language learning: Few perfect pronunciations + many student attempts

4. Bioinformatics

Problem: Biological experiments are expensive Solution: Combine limited experimental data with large biological databases

Examples:

  • Protein function prediction: Few labeled proteins + protein sequence databases
  • Drug discovery: Limited clinical trial data + molecular databases
  • Gene expression: Few labeled samples + large expression datasets

Implementation Strategy

1. Assess Your Data

# Questions to ask:
- How much labeled vs. unlabeled data do you have?
- What's the quality of your initial supervised model?
- Can you split features into meaningful views?
- How similar are your labeled and unlabeled data?

2. Choose the Right Method

  • Good initial model + lots of unlabeled data → Self-training
  • Natural feature splits → Co-training
  • Clear similarity relationships → Graph-based methods
  • Generative model fits data → Generative approaches
  • Deep learning setting → Pseudo-labeling with consistency

3. Validation Strategy

# Semi-supervised validation challenges:
1. Hold out some labeled data for testing
2. Be careful not to use test labels during semi-supervised training
3. Monitor performance on held-out labeled data
4. Check if adding unlabeled data actually helps

Common Pitfalls and Solutions

1. Confirmation Bias

Problem: Model reinforces its own mistakes by pseudo-labeling similar errors Solution: Use confidence thresholds, ensemble methods, or human-in-the-loop validation

2. Distribution Mismatch

Problem: Unlabeled data comes from different distribution than labeled data Solution: Check data distributions, use domain adaptation techniques

3. Poor Initial Model

Problem: If supervised model is bad, semi-supervised learning can make it worse Solution: Ensure reasonable performance on labeled data first

4. Overfitting to Pseudo-Labels

Problem: Model becomes too confident in wrong pseudo-labels Solution: Use soft labels, regularization, or gradually increase pseudo-label weight

Evaluation Considerations

Performance Metrics

  • Compare against supervised baseline: Does adding unlabeled data help?
  • Learning curves: How does performance scale with labeled data size?
  • Robustness: How sensitive is performance to hyperparameters?

Experimental Design

  • Hold-out validation: Reserve labeled data for testing only
  • Multiple runs: Semi-supervised methods can be unstable
  • Ablation studies: Which components contribute most to performance?

Future Directions

  1. Self-supervised pre-training + fine-tuning: Learn representations from unlabeled data first
  2. Active learning integration: Intelligently choose which data to label next
  3. Meta-learning for semi-supervised: Learn to adapt quickly to new domains with few labels
  4. Theoretical understanding: Better characterize when and why semi-supervised learning works

Semi-supervised learning bridges the gap between the label-hungry nature of supervised learning and the abundance of unlabeled data in the real world. When applied correctly, it can significantly improve performance while reducing the cost and effort of data labeling!