Back to blog
Computer VisionAIFootball AnalyticsYOLOv5OpenCVFlaskMachine Learning

Building an AI Referee: Computer Vision for Football Player Detection and Tracking

2026-05-1714 min read
Building an AI Referee: Computer Vision for Football Player Detection and Tracking

Football analytics is rapidly evolving. Clubs and analysts increasingly rely on positional data, tactical insights, and automated event detection to gain a competitive edge. But most commercial tracking systems remain expensive and inaccessible for independent developers.

This project demonstrates how to build an AI referee and football analytics system using open source computer vision tools. The pipeline detects players from broadcast video, tracks them across frames, classifies teams, maps positions onto a tactical pitch, detects collisions, and visualizes everything inside a live dashboard.

The Full Pipeline

Video Frame → YOLOv5 Detection → SORT Tracking → Team Classification → Pitch Mapping → CSV Export → Flask Dashboard

Each stage feeds into the next to create a complete end-to-end football analytics workflow.

Player Detection with YOLOv5

The system uses two YOLOv5 models loaded through PyTorch Hub:

  • yolov5s — detects players using the COCO person class
  • goals.pt — custom fine-tuned model for goal post detection
goal_model = torch.hub.load('../yolov5', 'custom', path='../models/goals.pt', source='local')
player_model = torch.hub.load('../yolov5', 'custom', path='../models/yolov5s.pt', source='local')

Each video frame is processed independently, producing bounding boxes for detected players.

for result in results:
    if result['class'] == 0:
        x1, y1 = int(result['xmin']), int(result['ymin'])
        x2, y2 = int(result['xmax']), int(result['ymax'])
        boxes.append(((x1, y1), (x2, y2)))

Persistent Player Tracking with SORT

Raw object detection alone is insufficient because detections do not maintain identity between frames. The system therefore integrates SORT (Simple Online and Realtime Tracking).

SORT combines:

  • Kalman filtering for motion prediction
  • Hungarian algorithm matching using IoU
  • Automatic track creation and deletion
detections = []
for (x1, y1), (x2, y2) in boxes:
    detections.append([x1, y1, x2, y2, 1.0])

tracker = Sort()
tracked_objects = tracker.update(np.array(detections))

The tracker outputs stable player IDs:

[x1, y1, x2, y2, id]

Team Classification Using SSIM

To distinguish teams and referees, the system compares player jersey crops against reference templates using SSIM (Structural Similarity Index).

for team, template in jersey_templates.items():
    score, _ = ssim(resized_gray, template_gray, full=True)
    if score > max_score:
        max_score = score
        best_match = team

The classification pipeline:

  1. Crop the upper half of the player bounding box
  2. Resize to 40×40 pixels
  3. Convert to grayscale
  4. Compare against jersey templates
  5. Assign the highest similarity score

While lightweight, SSIM performs surprisingly well under consistent broadcast conditions.

Pitch Line Detection

To reconstruct player positions on a tactical map, the system estimates pitch geometry from the broadcast frame.

The pitch detection pipeline includes:

  • HSV masking for green field isolation
  • Canny edge detection
  • Hough Transform line extraction
  • Line clipping and filtering
  • Touchline classification
hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
lower_green = np.array([40, 40, 40])
upper_green = np.array([70, 255, 255])
bitwise = cv2.bitwise_and(image, image, mask=cv2.inRange(hsv, lower_green, upper_green))
edges = cv2.Canny(bitwise, 100, 200)
lines = cv2.HoughLines(edges, 1, np.pi * 1 / 180, 150)

The resulting touchlines allow the system to estimate camera orientation and field perspective.

Perspective Transform and Tactical Coordinates

One of the most challenging components is transforming broadcast pixel coordinates into normalized tactical pitch coordinates.

The implementation uses a simplified pinhole camera model with estimated field of view values derived from average player dimensions.

fov_h = 21 / avg_player_width
fov_v = 24 / avg_player_height
scaled = 2 * tan(fov / 2)

Using camera rotation matrices and ray-plane intersection, player positions are projected onto a virtual pitch.

R = matmul(Rx, Ry)
pinhole = [50 - 100*lft, -7, -70]
vec = R.dot(pinhole)
t = pinhole[1] / vec[1]
point = [pinhole[0] - t*vec[0], pinhole[1] - t*vec[1], pinhole[2] - t*vec[2]]

The final coordinates are normalized into a 0–1 tactical pitch representation suitable for visualization.

Collision Detection

Player proximity analysis enables simple collision and contact detection.

dx = p1['X (Pitch)'] - p2['X (Pitch)']
dy = p1['Y (Pitch)'] - p2['Y (Pitch)']
dist = math.hypot(dx, dy)

if dist <= 1.5:
    active_pairs.add(key)

The system logs collision duration, player IDs, and pitch location into a separate CSV dataset.

Flask Dashboard: The AI Referee Interface

The final component is a Flask-powered dashboard designed like a VAR command center.

app.py → reads output/player_positions.csv → renders index.html

The dashboard includes:

  • Live tactical pitch visualization
  • Player tracking overlays
  • Game clock
  • Match event feed
  • AI-generated alerts
  • Player detail popups
  • Live statistics panels
  • Fan interaction polls

The football pitch itself is drawn entirely with CSS using borders, positioning, and aspect-ratio styling.

Challenges and Limitations

While functional, the system still has several limitations:

  • Broadcast camera movement reduces calibration accuracy
  • SSIM classification struggles under lighting changes
  • No robust ball tracking yet
  • Occlusion handling remains difficult
  • Single-camera reconstruction limits positional precision

Future Improvements

  • Deep ReID embeddings for robust player identity tracking
  • Ball trajectory estimation
  • Offside detection logic
  • Multi-camera 3D reconstruction
  • Automatic event recognition
  • Live stream ingestion
  • Professional camera calibration pipelines

Project Setup

# Clone YOLOv5
git clone https://github.com/ultralytics/yolov5 ../yolov5

# Install dependencies
pip install torch opencv-python numpy pandas flask scikit-image sort-track

# Run the pipeline
cd tests && python main.py

# Launch dashboard
cd .. && python app.py

The project requires custom video footage, YOLO weights, and jersey template images.

Conclusion

This project demonstrates how modern computer vision tools can transform ordinary football broadcast footage into a rich tactical analytics platform. With open source frameworks like YOLOv5, OpenCV, Flask, and SORT, developers can now build systems that were previously available only to elite sports organizations.

From player tracking to tactical mapping, the foundations of an AI referee are more accessible than ever.

Back to all posts