Engineering App: GimbalPaw
Milestone:
The problem
Connecting person detection to a moving camera requires more than forwarding a bounding box. The application has to coordinate camera frames, target selection, lost-target recovery, and asynchronous device commands without continuing to act on stale observations.
GimbalPaw is an experimental local macOS application for this integration. The native application, currently named OM3 Lab, connects a selected camera to Apple Vision person tracking and Bluetooth Low Energy control of a DJI Osmo Mobile 3 gimbal.
What the application implements
- Camera and vision integration. AVFoundation supplies frames from a selected camera. Apple Vision detects people and tracks the selected target between detection updates. This uses Apple’s existing models, not a custom-trained vision model.
- Stateful tracking and recovery. Target-selection and reacquisition logic work with motion smoothing and predicted positions to coordinate pan and tilt corrections.
- Checks before issuing commands. Session and frame-freshness checks, software motion limits, and cumulative movement budgets constrain command generation. A visible STOP control remains available in the interface.
- Testable coordination. Unit and mocked integration tests cover behaviors such as target reacquisition, stale observations, camera selection, and motion limits. These tests are distinct from validation on physical hardware.
Current scope and limitations
The project demonstrates vision-system integration and device-control software, not new computer-vision research or production robotics. The current BLE control path lacks reliable device-pose feedback and device acknowledgements for writes; STOP is best-effort. Software movement limits are not a validated mechanical safety guarantee.
No measured latency, tracking-accuracy, or hardware-reliability benchmark is claimed here.
Components and attribution
The application uses SwiftUI, AVFoundation, Apple Vision, and CoreBluetooth. OM3 protocol integration builds on the community research credited in the repository, including alkersan/om-research; the project is not affiliated with DJI. The repository is distributed under the MIT License.