A pilot supervising multiple unmanned systems cannot afford to interpret every video stream equally.
The idea is to embed an AI assistant that filters most incoming information, notifying pilots only when something deserves their attention.
My contribution
Designed, implemented and flight-tested from scratch a multi-UAV AI Mission Support Framework prototype for Manned-Unmanned Teaming operations. It processes drone video streams, generates scene descriptions through vision-language models and converts them into operational relevance/danger scores, supporting pilot decision-making and reducing cognitive load. The flight-test prototype ran on a ground-based workstation.
ROS2 perception pipeline
The DJI stream is decoded and published to the shared /video_frames topic, also accepting webcam or ROS2 bag input. LLaVA generates a frame description; BART-MNLI then converts it into a danger score. Video in, danger score out.
