A computer-vision pipeline for detecting enemy armor plates on RoboMaster robots and computing the gimbal angles needed to aim at them. Built around classical OpenCV (light-bar detection + geometric pairing) with a PnP-based pose solver and a serial-link to the gimbal MCU.
Camera frame ──► channel split + threshold + Gaussian blur
──► contour extraction → light-bar candidates (aspect-ratio + area filter)
──► pairwise matching → armor candidates (overlap, area-ratio, slope, distance filters)
──► PnP solve (3D armor model ↔ 2D corners) → tvec / rvec
──► camera-to-gimbal transform → yaw / pitch
──► serial TX (float32, little-endian) → MCU
| Path | Purpose |
|---|---|
armor_detection_py/ |
Core detection pipeline (det.py), image/video test harnesses, and the live-camera + serial driver (cam.py). |
Camera_Calibration/ |
Chessboard-based intrinsic calibration (11×8, 18.1 mm cells) using cv2.calibrateCamera. |
ArmorPNPSolver_cpp/ |
C++ PnP solver: builds the 3D armor model (small 134 mm / large 229 mm), runs SOLVEPNP_AP3P, and converts the result to gimbal yaw/pitch. |
BasicDemo/ |
Hikvision MVS SDK boilerplate (camera enumeration + Qt UI sample). |
opencv_tutorial_py/ |
Reference exercises while learning OpenCV. |
ArmorDetector exposes three preprocessing modes — red-channel, blue-channel, and gray-threshold — selectable for the enemy color. After thresholding and a 5×5 Gaussian blur, light-bar candidates are extracted with these filters:
- area
w·h > 200 - aspect ratio
2 ≤ h/w ≤ 6
Candidates are paired and a pair is accepted as an armor plate only if it satisfies:
- vertical overlap between the two bars
- non-overlapping horizontal extents
- area ratio within 1.5×
- slope (h/w) ratio within 1.5×
- inter-bar distance between
0.4·avg_hand4·avg_h
Once an armor plate is matched, four corner points are reconstructed from the two cv::RotatedRect light bars and fed to solvePnP with the AP3P algorithm. The resulting rotation vector is converted to a rotation matrix via Rodrigues, then to ZYX Euler angles. Gimbal yaw/pitch are derived from the translation vector after applying the camera-to-gimbal offset:
yaw = atan( (x + Δx_yaw) / (z + Δz_yaw) )
pitch = atan( (y + Δy_pitch) / (z + Δz_pitch) )
- Camera: Hikvision industrial camera via the MVS SDK (
MvCameraControl_class), BGR8, 60 fps. - MCU link: USB serial (
/dev/ttyUSB0, 9600 baud). Each frame the detector packs the first armor's center as twofloat32little-endian values (<2f) and writes them to the gimbal controller.
# 1. Install dependencies
pip install -r requirements.txt
# 2. Run on a single image
cd armor_detection_py
python img_test.py --image ./selected_armor_img/issue_red_001.jpg --enemy-color red
# outputs: 1_preprocessed.jpg, 1_light_bars_detected.jpg, 1_armors_detected.jpg
# 3. Run on a video
python video_test.py --video ./video/vid_move_003.mp4 --enemy-color red
# 4. Live mode (Hikvision camera + serial out to gimbal MCU)
python cam.py --port /dev/ttyUSB0 --enemy-color blueimport cv2
from det import ArmorDetector
detector = ArmorDetector(binary_threshold=200, enemy_color="red")
img = cv2.imread("frame.jpg")
for r in detector.detect(img):
x1, y1, x2, y2 = r.armor_xyxy
print(f"armor at ({(x1+x2)/2:.0f}, {(y1+y2)/2:.0f}), id={r.number}")Calibrate the camera first if you plan to use the PnP pipeline:
cd Camera_Calibration
python Camera_Calibration.py # expects ./chessImg/*.jpgEach stage of the pipeline produces a debug visualization. Running img_test.py on selected_armor_img/issue_red_001.jpg yields:
Green boxes are accepted light-bar candidates; red boxes are matched armor pairs.
- Number-classification CNN to replace the
__number_identificationstub - Kalman filter on armor centers for tracking and shoot-prediction
- Finish the C++ PnP solver (declare
TargetData, intrinsics, and gimbal offsets in the header) and bind it to the Python pipeline - Move from 4-point AP3P to IPPE for better stability under partial occlusion
- ROS2 node wrapper for integration with the rest of the robot stack