Vision-language incident retrieval

The problem

Identify road accidents and special incidents, such as vehicle fires, from fixed CCTV footage. Real-world incident video is scarce and noisy, so the hard part was turning it into usable training data.

What I built

I built the data pipeline end to end, from collection to evaluation:

Sourced CCTV footage from LTA and from online accident compilations.
Split videos into scenes with TransNetV2 and extracted frames.
Built a Streamlit tool for labelling incident severity.
Marked accident timestamps in Kinovea.
Auto-captioned frames with Gemini Flash.
Formatted the datasets and fine-tuned CLIP-, BLIP- and SigLIP-family encoders for text-image retrieval, with Grounding DINO for object detection.

Eight-step pipeline diagram: CCTV video, scene splitting, frame extraction, manual labels, captioning, dataset formatting, fine-tuning and evaluation — The vision-language training pipeline, from CCTV video to evaluation

Also at Cynapse: road-marking segmentation with Mask2Former (its own project page), rule-based and shape-based segmentation with traditional CV, and visual re-identification under temporal variation (ResNet-50, OSNet).

Result

Diagram: a text query and CCTV frames pass through text and image encoders into a shared embedding space; the closest frames are returned as ranked results, measured by Recall@5 and Recall@10 — How text-to-image incident retrieval works, simplified. An original diagram, not Cynapse material.

Increasing the vision encoder size lifted retrieval to above 80% Recall@5 and 90% Recall@10. The system is deployed by LTA for traffic incident monitoring.

What I learned

Model performance depends heavily on data quality, preprocessing, evaluation choices and practical inference-speed limits.