Kinova ColorSort
OpenCV color detection + Kinova Jaco2 arm sorting colored objects into bins.
sigitdanip/kinova-colorsortThe Problem
Industrial pick-and-place tasks require reliable real-time color classification to sort objects into correct bins. The Kinova Jaco2 — a lightweight 6-DOF robotic arm — needs a vision pipeline that can identify colored objects under variable lighting and trigger precise pre-computed trajectories.
This project pairs a standard USB webcam with OpenCV's HSV color space to detect five distinct colors (red, yellow, green, purple, orange). When a dominant color crosses the detection threshold, the Python orchestrator spawns a C++ trajectory executable via subprocess, which commands the Jaco2 over serial to execute the corresponding pick-and-place routine.
The system was developed across three iterations — versions 1 through 3 — with progressive refinement of the HSV mask ranges, pixel-count thresholds, and integration between the Python detection layer and the C++ motion layer.
The System
The vision pipeline captures webcam frames, converts to HSV, applies five parallel color masks, counts pixels per mask, identifies the dominant color, and triggers the corresponding pick-and-place trajectory on the Jaco2 arm via subprocess.
The vision pipeline begins with a webcam feed running at roughly 1 Hz. Each frame is converted from BGR to HSV color space, then five parallel np.array masks isolate yellow, purple, green, red, and orange pixels using empirically tuned range filters.
A cv2.countNonZero() call on each mask yields the pixel count for that color. The dominant color is determined by the mask with the highest non-zero count exceeding a 10,000-pixel threshold, which filters out noise and empty frames.
Once a color is identified, the Python script invokes a pre-compiled Windows .exe (steady, track1, track2, or track3) via subprocess.run(). These executables communicate with the Jaco2 over serial to execute the pick-and-place trajectory for the target bin.
A companion colorpicker/ directory provides calibration utilities for tuning the HSV range boundaries against the specific lighting conditions of the workspace.
Results
H[20-42] S[50-255] V[100-255]
H[140-160] S[128-255] V[255-255]
H[25-102] S[52-255] V[72-255]
H[-10-10] S[214-265] V[245-294]
H[5-15] S[50-255] V[50-255]
HSV ranges tuned empirically with colorpicker/ calibration utility. Purple mask currently dominant in this frame.
The system reliably detects and sorts five colors under consistent indoor lighting. HSV color space provides inherent robustness to brightness variation compared to RGB, making the detection tolerant of minor lighting changes.
Detection runs in a continuous loop with a 1-second sleep between frames, providing adequate throughput for the Jaco2's traversal time between bins. The 10,000-pixel threshold prevents false positives from small color patches or image noise.
Pre-computed trajectories (steady for neutral position, track1-through-track3 for bin-specific placements) keep the motion layer deterministic and safe — no real-time inverse kinematics are computed during operation.
Implementation
The Python orchestrator in colorsort_v3.py represents the third iteration of the detection pipeline. It handles BGR-to-HSV conversion, five parallel color masks, pixel counting, and subprocess orchestration of the C++ trajectory executables.
The current implementation has the trajectory subprocess calls commented out — all five colors are detected and their pixel counts printed to console, allowing dry-run testing before engaging the physical arm.
Limitations include the Windows-only .exe dependency, fixed HSV ranges that may need recalibration for new environments, and the serial-only communication with the arm (no ROS integration). Future work could add a unified ROS 2 node, real-time trajectory computation, and a web-based calibration UI.
Related
The earlier Pixy2-based color tracking robot — 27 citations, same color-detection domain at a smaller scale.