Computer Vision Expert

Overview

Use this skill to turn vague "make the model see better" requests into explicit computer-vision workflows with measurable inputs, outputs, and failure modes. Prefer simple, testable pipelines and evidence-driven evaluation over fashionable models or premature complexity.

Workflow

1. Detect project state. Consult references/topic_index.md first to match the user problem to a CV task, default library, and guide file. Then inspect the target repo:

• Search the target repo for existing CV source files, model configs, pipeline scripts, and evaluation code.
• Search for existing tests or benchmarks that characterize current behavior.
• If any of these exist, follow the existing-pipeline path: inspect model weights, validation split, and failure logs; establish a per-class baseline before changing anything; tie each proposed change to a specific failing case; prove improvement with before/after metrics on the held-out set.
• If none exist, follow the greenfield path: seed a balanced 100-500 image set with a held-out test split; write a vision contract covering resolution, class taxonomy, FPS budget, and miss-vs-false-alarm tolerance; then build and validate the minimal pipeline.
• Read references/project_workflow.md for the full branching workflow, vision contract spec, and CV review checklist.

2. Define the exact vision task.

• Determine whether the task is classification, detection, segmentation, keypoints, tracking, OCR, retrieval, restoration, or measurement.
• Identify the input domain: still image, video, live camera, document scan, microscopy, satellite, industrial, or another domain.
• Define success in measurable terms such as accuracy, recall, latency, FPS, false positives, localization error, or downstream business impact.
• Read references/task_selection.md when the request is underspecified or multiple CV framings are possible.

3. Choose the simplest viable pipeline.

• Start with a baseline that can be inspected and benchmarked.
• Separate data ingestion, preprocessing, inference, postprocessing, and evaluation.
• Prefer classical CV when the task is geometric, threshold-driven, template-based, or small-data.
• Prefer learned models when invariance, semantic understanding, or scale make heuristic pipelines brittle.
• Read references/pipeline_design.md when choosing between classical CV, hybrid, and model-heavy approaches.
• Reach for these books when they match the task:
  • "Learning OpenCV" for broad OpenCV techniques, feature detection, and matching workflows.
  • "OpenCV Cookbook" for practical implementation patterns and utility code.
  • "Video Object Tracking" for tracking-specific tasks, datasets, and methods.
  • "Multiple View Geometry" for camera models, epipolar geometry, homographies, triangulation, calibration, and multi-view reconstruction.

4. Make data quality explicit.

• Check label quality, class balance, resolution, compression artifacts, lighting, occlusion, and domain shift.
• Treat poor annotations and mismatched evaluation data as first-order causes of failure.
• Do not blame the model first when the dataset is weak or misaligned.

5. Evaluate before optimizing.

• Establish a baseline metric and representative validation set.
• Inspect failures by category rather than averaging everything into one score.
• Measure speed, memory, and deployment constraints alongside accuracy.
• Read references/debugging_and_failure_analysis.md when performance is unstable, errors cluster in strange ways, or the model seems to fail "randomly."
• Read references/testing_and_oracles.md when building the fixture corpus or selecting oracles for a task type.

6. Improve iteratively.

• Change one major factor at a time: data, preprocessing, architecture, thresholding, postprocessing, or evaluation.
• Prefer targeted fixes tied to a known failure mode.
• Keep intermediate visualizations so behavior is explainable.

Implementation defaults

• Use OpenCV for image I/O, geometry, filtering, thresholding, contour work, calibration, and fast visual debugging.
• Use PyTorch-based models when the task needs modern learned vision methods and the environment supports them.
• Save representative outputs with overlays, masks, boxes, or heatmaps so predictions can be inspected visually.
• For video, define frame sampling, buffering, temporal smoothing, and throughput requirements up front.
• For OCR, treat document cleanup, orientation, crop quality, and layout structure as part of the pipeline, not preprocessing trivia.
• If the work is OpenCV-heavy, check the local books before reinventing standard pipelines or utility patterns.
• Load the local-only books first when present; see references/local_books.md and the survey in references/reference_survey.md.

Quality bar

• Favor measurable improvement over architecture churn.
• Favor robust pipelines over benchmark theater.
• Avoid changing data, model, thresholds, and evaluation all at once.
• Avoid shipping a CV system that has never been reviewed on hard negatives and edge cases.
• State what the model cannot do, not only what it can do.

Output expectations

When using this skill, aim to produce:

• A clearly framed CV task with explicit inputs, outputs, and success metrics.
• A pipeline design that can be implemented and debugged in stages.
• Visual inspection artifacts for representative successes and failures.
• A short explanation of the main failure modes and the next best improvement step.