CosFly-Track: The First Large-Scale Multi-Modal Dataset for UAV Visual Tracking (2026)

Key Takeaways

• CosFly-Track — The first large-scale multi-modal dataset purpose-built for UAV visual tracking, filling a critical gap in aerial visual-language navigation
• 240K multi-modal time steps — 7 synchronized data channels including RGB, depth, semantic segmentation, 6-DoF pose, and bilingual instructions
• MuCO optimization engine — Continuous-space multi-constraint trajectory optimizer that outperforms discrete A* planning by 22× in speed while producing smoother, collision-free trajectories
• 53-69 percentage point improvement — Fine-tuning on CosFly-Track boosts tracking SR@1m across 7 VLM architectures
• Dual-trajectory design — Expert + perturbed trajectory pairs enable denoising, DAgger-style correction, contrastive learning, and prediction training paradigms
• Aomway drone video transmission systems benefit from advances in visual tracking and target-following algorithms — stable visual lock is essential for professional aerial cinematography. Aomway FPV transmission modules support real-time 1080p video feedback essential for visual tracking systems

While aerial visual-language navigation datasets are abundant, almost all are designed for static waypoint navigation. Dynamic target tracking — the core scenario for real-world drone cinematography, search and rescue, and surveillance — has long lacked dedicated large-scale training data. CosFly-Track bridges this gap with 2.4 million multi-modal aligned frames and a continuous-space trajectory optimization pipeline.

Authors: Xiangyue Wang, Hanxuan Chen, Songsheng Cheng, Ruilong Ren, Jie Zheng, Shuai Yuan, Tianle Zeng, Hanzhong Guo, Kangli Wang, Ji Pei

Affiliations: Autel Robotics, Nanjing University, Peking University, Southern University of Science and Technology, The University of Hong Kong

Paper: CosFly-Track: A Large-Scale Multi-Modal Dataset for UAV Visual Tracking via Multi-Constraint Trajectory Optimization — arXiv:2605.17776

Dataset: https://huggingface.co/datasets/AutelRobotics/CosFly

Research Background: The Missing Piece in Aerial VLN

In recent years, aerial visual-language navigation (VLN) technology has advanced rapidly, with datasets growing in scale and diversity. However, one critical gap remains: almost all existing datasets are designed for static waypoint navigation — flying from start to a fixed destination.

In real-world applications — tracking pedestrians, sports cinematography, search and rescue — what drones need is visual tracking capability: continuously following a dynamic target, keeping it in view at all times, while avoiding obstacles and respecting flight dynamics. This fundamentally differs from navigation but has had no dedicated large-scale training data.

Generating high-quality tracking trajectories is inherently harder than navigation paths, facing three technical challenges:

1. Discrete planners have inherent flaws: Grid-based algorithms like A* optimize only path length, producing trajectories that violate drone velocity and acceleration limits. Post-processing smoothing introduces residual discretization error.
2. Multiple coupled optimization objectives: Tracking must simultaneously optimize target visibility, viewing angle quality, following distance, and obstacle avoidance — all of which change dynamically as the target moves. Navigation tasks don’t need any of these.
3. Prohibitively high generation costs at scale: Denser urban scenes and longer trajectories cause exponential search-space explosion. Generating tens of thousands of high-quality trajectories is computationally expensive.

To solve these challenges, the team developed a continuous-space multi-constraint optimization pipeline and built a complete industrial-grade dataset generation system. Aomway follows similar principles in its drone FPV transmission and target-tracking system development, ensuring reliable visual lock in dynamic flight conditions.

Key Contributions

1. First dedicated tracking dataset: CosFly-Track is the first large-scale multi-modal dataset purpose-built for UAV visual tracking, containing ~12,000 expert/perturbed trajectory pairs, 2.4 million time steps, 7 time-aligned data channels, and bilingual (Chinese/English) instructions.
2. Industrial-grade generation pipeline: The modular CosFly data production pipeline, powered by the MuCO multi-constraint trajectory optimizer, simultaneously optimizes visibility, viewing angle, obstacle avoidance, and motion feasibility in continuous 3D space.
3. Comprehensive benchmark validation: Systematic evaluation across 7 mainstream VLM models shows tracking performance improvements of 53-69 percentage points over zero-shot baselines after fine-tuning, plus data scaling and multi-dimensional ablation studies.
4. Fully open research resources: The dataset, evaluation scripts, and pre-trained model weights are publicly released to support the entire UAV tracking research community.

Task Definition: Tracking vs. Navigation — Fundamentally Different

The paper provides a clear formal distinction between the two tasks:

• Navigation: Given a fixed target location (or corresponding language instruction), plan a path from start to destination in a static environment. Success is defined as reaching the endpoint.
• Visual tracking: Given a dynamically moving target trajectory, the drone must continuously follow the target, keeping it visible at a reasonable distance while satisfying all kinematic constraints. Success requires stable tracking throughout the entire trajectory — losing the target at any point means failure.

In simple terms: navigation is a single-exam test — just arrive at the destination. Tracking is a continuous exam — every single step counts.

Input and Action Space

The tracking agent receives: current RGB image, historical 6-DoF pose (position + flight attitude), target bounding box and visibility markers, and bilingual (CN/EN) natural language tracking instructions. The agent outputs 6-DoF incremental waypoint predictions — the position and attitude changes for the next step.

Evaluation Metrics

• Waypoint prediction: SR@r (success rate within r meters of endpoint), ADE/FDE (average/final displacement error), RotAcc@d (yaw angle accuracy), position-rotation joint success rate
• Target prediction: mIoU (intersection over union between predicted and ground-truth bounding boxes)

The CosFly Generation Pipeline and MuCO Optimization Engine

CosFly Pipeline Overview

CosFly is a complete modular data production system divided into 6 stages, each independently replaceable for maximum extensibility:

1. Environment preprocessing: Extract and simplify 3D obstacle bounding box maps to reduce computational load
2. Pedestrian trajectory generation: Generate base paths using A* on walkable grids, then apply variable-speed resampling to simulate natural walking rhythms
3. MuCO tracking optimization: The core stage — generates high-quality drone tracking trajectories
4. Dual trajectory augmentation: Generate paired expert and perturbed trajectories
5. Multi-modal rendering: Output RGB, depth, semantic segmentation, and other channels with randomized weather and lighting
6. Bilingual instruction generation: Generate high-quality CN/EN tracking instructions via LLM teacher-student distillation

MuCO: Continuous-Space Multi-Constraint Trajectory Optimizer

MuCO is the technical highlight of this work. It operates entirely in continuous 3D space using gradient-based optimization, fundamentally avoiding the discretization errors and post-processing artifacts of discrete grid planning.

Problem Formulation

Given an N-step pedestrian trajectory and obstacle map, MuCO optimizes the corresponding drone waypoint sequence by minimizing a weighted sum of 9 loss functions.

9 Loss Functions: Comprehensive Tracking Quality Assurance

The 9 loss functions cover three dimensions of trajectory quality:

1. Visibility loss: Tracks the unobstructed ratio of 5-10 sampling points on the target body. BVH (Bounding Volume Hierarchy) acceleration reduces per-query complexity from O(n²) to O(log n), dramatically improving optimization efficiency.
2. Viewing angle loss: Based on the angle between the target’s walking direction and the drone’s observation direction. Includes a direction factor that automatically relaxes constraints when the target turns, preventing overly rigid trajectories.
3. Jerk loss: A third-order difference regularization term constraining trajectory smoothness, ensuring generated trajectories are physically flyable by real drones.

The remaining 6 losses cover: tracking distance, smoothness, safety, pitch angle, altitude, and path length.

Four-Layer Safety Architecture: Guaranteed Collision-Free

MuCO’s four-layer safety architecture ensures 100% collision-free trajectories:

• Soft safety loss: Provides gradient guidance to steer away from obstacles during optimization
• Geometric projection: Projects waypoints that violate obstacle boundaries outwards via altitude increase or lateral deviation
• Velocity repair: Distributes velocity discontinuities caused by projection across subsequent waypoints to maintain smooth motion
• Altitude smoothing: Eliminates vertical oscillations for more stable flight

This decoupled design prevents optimization from getting stuck on heavy safety penalties while guaranteeing fully collision-free output.

Performance: 22× Faster Than Discrete Planning

Compared against the strongest discrete planning baseline (A* with 4D spatiotemporal voxel search + BVH visibility):

• Comparable tracking quality: MuCO’s visibility score is only 7.3% lower, concentrated in extreme occlusion scenarios like narrow alleys. 80% of trajectories maintain visibility above 0.9.
• Shorter, smoother paths: Path length reduced by 13.3%, with vastly superior motion smoothness
• 22× faster: Single trajectory optimization in 247ms vs. A*’s 5.5 seconds
• Scalability: Generating 6,000 trajectories takes A* ~9 GPU hours, while MuCO completes in 25 minutes — true industrial-grade production capacity

CosFly-Track Dataset in Detail

Scale and Statistics

• Expert + perturbed trajectory pairs: ~12,000
• Unique pedestrian paths: ~6,000
• Total time steps: 2.4 million
• Total tracking duration: ~334 hours
• Urban scene coverage: 8+ CARLA town environments
• Weather/lighting conditions: 16 combinations
• Average trajectory length: ~200 steps / 100 seconds

7 Time-Aligned Data Channels

Each trajectory contains 7 strictly time-aligned channels, with 5 recorded per frame:

1. 🖼️ RGB images: 1280×720 resolution
2. 📏 Metric depth maps: pixel-level meter-precision depth
3. 🎨 Semantic segmentation: full semantic category labels
4. 🛸 Drone 6-DoF pose: complete position + attitude
5. 🎯 Target state: world coordinates + visibility markers

Plus 2 trajectory-level annotations:

6. 💬 Bilingual (CN/EN) natural language tracking instructions
7. 📑 Trajectory pair metadata: expert/perturbed labels + time alignment indices

Dual Trajectory Design: Clever Data Augmentation

Each pedestrian path generates two drone trajectories:

• Expert trajectory: The optimal tracking trajectory from MuCO — the high-quality “ground truth”
• Perturbed trajectory: Position + rotation Bernoulli perturbations simulating real-world tracking errors across four perturbation states

Training samples are constructed via sliding window: 5 frames of perturbed observations as input, 5 frames of expert trajectory as supervision. This supports four different training paradigms:

1. Denoising: Recover correct expert actions from perturbed observations
2. DAgger-style correction: Mixed expert and perturbed data improves robustness to distribution shift
3. Contrastive learning: Learn to distinguish high- and low-quality tracking trajectories
4. Prediction: Forecast future optimal waypoints from noisy historical information

Additional Features

• Bilingual instructions enable cross-linguistic research with CN and EN VLMs
• Pipeline validated for migration from CARLA (UE4) to SimWorld (UE5), with modular objective replacement for inspection, patrol, and other tasks
• Deliberately retains <5% imperfect samples for natural scene diversity

Experiments and Benchmark Evaluation

Experimental Setup

Seven mainstream VLMs ranging from 0.8B to 9B parameters were tested. Task: input 5 historical frames + current drone pose + target bounding box, predict next 5 6-DoF incremental waypoints.

Three core experiments:

1. Architecture comparison: Full fine-tuning (frozen vision encoder), 200K training samples
2. Data scaling analysis: LoRA fine-tuning, training data from 250K to 1M samples
3. Ablation studies: Input modality and training paradigm variations

Finding 1: Fine-Tuning Yields Orders-of-Magnitude Improvement

1. Zero-shot models cannot track: All zero-shot models achieve SR@1m of only 25-33%, comparable to outputting zero increments. Tracking is highly task-specific — general VLMs cannot handle it directly.
2. Post-fine-tuning explosion: SR@1m improves by 53-69 percentage points, rotation accuracy by 16-31 percentage points. Direct validation of CosFly-Track’s training value.
3. Model scaling’s diminishing returns on position: Qwen3.5 from 0.8B→9B improves SR@1m by only 0.5 pp, but rotation accuracy by 7 pp. Larger models mainly enhance fine-grained rotation prediction, with limited benefit for coarse position estimation.
4. Vision encoder compatibility varies: Qwen3-VL underperforms Qwen3.5 at similar parameter counts, indicating weaker geometric regression suitability.

Best performer: Qwen3.5-9B achieves 95.6% SR@1m after fine-tuning.

Finding 2: More Data = Better (Not Yet Saturated)

Data scaling experiments show monotonic improvement from 25% to 100% training data — no saturation observed yet. 0.8B and 2B models converge to nearly identical error levels, suggesting the current bottleneck is data distribution richness, not model capacity.

Finding 3: Pose History Is the Decisive Input

• Pose history is the #1 core input: Removing pose information causes FDE to explode by 3.1× and SR@1m to crash from 77.6% to 15-17% — the largest impact of any modality. With pose data included, error levels remain consistent regardless of other modalities.
• Bounding boxes are critical for target prediction: Removing bounding box history drops mIoU by 20%, with high-overlap samples decreasing by 45%.
• RGB offers limited marginal gain: Given complete pose and bounding box data, RGB adds little improvement — though this applies only to in-distribution scenarios. For out-of-distribution cases (sudden target direction changes, intent shifts), visual information remains crucial.

Finding 4: Denoising Training Paradigm Works Best

Denoising (perturbed input → expert ground truth) achieves the best overall performance in both final displacement error and success rate. Training on expert-only trajectories increases yaw prediction error by 1.7× — the model overfits to clean expert trajectories and cannot learn to correct heading deviations.

Additional Findings

• Cross-scene generalization: Multi-map training improves joint success rate by 5.31 percentage points, reduces catastrophic failures by 55.6%
• Transferable to downstream tasks: Multi-modal annotations benefit depth estimation (AbsRel from 0.77→0.045), semantic segmentation (mIoU 0.74→0.86), and object detection

Summary

CosFly-Track is the first large-scale multi-modal dataset purpose-built for urban-environment UAV visual tracking, paired with the continuous-space MuCO trajectory generation engine. It fills the critical gap of dynamic tracking in aerial VLN research.

Extensive experiments demonstrate that fine-tuning on this dataset yields orders-of-magnitude tracking performance improvement. The dual-trajectory design, multi-channel data, and bilingual instructions support diverse training paradigms and research directions.

Limitations

1. Sim-to-real gap: Data generated in CARLA; real-world adaptation needs further study. Real-world data collection is in progress for future releases.
2. Scene diversity: Currently 16 town variants; more diverse environments planned for expansion.
3. Pedestrian behavior models could be improved with social force models for more realistic movement.
4. Pipeline source code not yet open due to corporate policy; full algorithm description available in the paper appendix.

Have questions about this article? Feel free to contact us at [email protected] — we’re happy to help!

Frequently Asked Questions

Q: What makes CosFly-Track different from existing aerial VLN datasets?

A: Existing VLN datasets target static waypoint navigation — flying from point A to B. CosFly-Track is the first dedicated to dynamic visual tracking: continuously following a moving target while maintaining visibility, avoiding obstacles, and respecting flight dynamics. Its 7-channel multi-modal data and dual-trajectory design support training paradigms unavailable in any prior dataset.

Q: Can I use CosFly-Track for non-tracking tasks?

A: Yes. The multi-modal annotations (depth, semantic segmentation, RGB) support downstream tasks including depth estimation, semantic segmentation, and object detection. The paper demonstrates substantial performance gains in all three areas.

Q: How does MuCO compare to traditional trajectory planning?

A: MuCO’s continuous-space gradient optimization avoids the discretization artifacts of grid-based A* planning. It produces 22× faster generation (247ms vs 5.5s per trajectory), 13.3% shorter path lengths, and superior smoothness — while maintaining comparable tracking quality.

Q: Is the CosFly-Track dataset suitable for real drone deployment?

A: The dataset is currently simulation-based (CARLA), so a sim-to-real gap exists. However, the paper demonstrates strong generalization to out-of-distribution scenarios, and the team is actively collecting real-world data. The MuCO optimization engine itself produces physically flyable trajectories with realistic kinematic constraints.

Q: What hardware is needed for models trained on CosFly-Track?

A: The paper tested models from 0.8B to 9B parameters. The 0.8B model achieves competitive tracking performance (SR@1m ~90% after fine-tuning) and can run on modern drone onboard computers with NPU support. For real-time applications, Aomway offers optimized video downlink solutions that integrate with onboard tracking systems. Aomway also offers reference designs for antenna diversity in tracking applications.