Flight experiment with five UAVs.

The Age-of-Information (AoI) metric has been widely studied in the theoretical communication networks and queuing systems literature. However, experimental evaluation of its applicability to complex real-world time-sensitive systems is largely lacking. In this work, we develop, implement, and evaluate an AoI-based application layer middleware that enables the customization of WiFi networks to the needs of time-sensitive applications. By controlling the storage and flow of information in the underlying WiFi network, our middleware can: (i) prevent packet collisions; (ii) discard stale packets that are no longer useful; and (iii) dynamically prioritize the transmission of the most relevant information. To demonstrate the benefits of our middleware, we implement a mobility tracking application using a swarm of UAVs communicating with a central controller via WiFi. Our experimental results show that, when compared to WiFi-UDP/WiFi-TCP, the middleware can improve information freshness by a factor of 109x/48x and tracking accuracy by a factor of 4x/6x, respectively. Most importantly, our results also show that the performance gains of our approach increase as the system scales and/or the traffic load increases.

Tailsitter flying wing aircraft.

This paper proposes a novel control law for accurate tracking of agile trajectories using a tailsitter flying wing unmanned aerial vehicle (UAV) that transitions between vertical take-off and landing (VTOL) and forward flight. The global control formulation enables maneuvering throughout the flight envelope, including uncoordinated flight with sideslip. Differential flatness of the nonlinear tailsitter dynamics with a simplified aerodynamics model is shown. Using the flatness transform, the proposed controller incorporates tracking of the position reference along with its derivatives velocity, acceleration and jerk, as well as the yaw reference and yaw rate. The inclusion of jerk and yaw rate references through an angular velocity feedforward term improves tracking of trajectories with fast-changing accelerations. The controller does not depend on extensive aerodynamic modeling but instead uses incremental nonlinear dynamic inversion (INDI) to compute control updates based on only a local input-output relation, resulting in robustness against discrepancies in the simplified aerodynamics equations. Exact inversion of the nonlinear input-output relation is achieved through the derived flatness transform. The resulting control algorithm is extensively evaluated in flight tests, where it demonstrates accurate trajectory tracking and challenging agile maneuvers, such as sideways flight and aggressive transitions while turning.

Loop trajectory reference and flight experiment.

This paper proposes a novel algorithm for aerobatic trajectory generation for a vertical take-off and landing (VTOL) tailsitter flying wing aircraft. The algorithm differs from existing approaches for fixed-wing trajectory generation, as it considers a realistic six-degree-of-freedom (6DOF) flight dynamics model, including aerodynamics equations. Using a global dynamics model enables the generation of aerobatics trajectories that exploit the entire flight envelope, enabling agile maneuvering through the stall regime, sideways uncoordinated flight, inverted flight etc. The method uses the differential flatness property of the global tailsitter flying wing dynamics, which is derived in this work. By performing snap minimization in the differentially flat output space, a computationally efficient algorithm, suitable for online motion planning, is obtained. The algorithm is demonstrated in extensive flight experiments encompassing six aerobatics maneuvers, a time-optimal drone racing trajectory, and an airshow-like aerobatic sequence for three tailsitter aircraft.

Overview of our proposed algorithm. The BayesOpt model is composed of multiple Gaussian process models, which represent dynamic feasibility and collision avoidance constraints.

We present a modular Bayesian optimization framework that efficiently generates time-optimal trajectories for a cooperative multi-agent system, such as a team of UAVs. Existing methods for multi-agent trajectory generation often rely on overly conservative constraints to reduce the complexity of this high-dimensional planning problem, leading to suboptimal solutions. We propose a novel modular structure for the Bayesian optimization model that consists of multiple Gaussian process surrogate models that represent the dynamic feasibility and collision avoidance constraints. This modular structure alleviates the stark increase in computational cost with problem dimensionality and enables the use of minimal constraints in the joint optimization of the multi-agent trajectories. The efficiency of the algorithm is further improved by introducing a scheme for simultaneous evaluation of the Bayesian optimization acquisition function and random sampling. The modular BayesOpt algorithm was applied to optimize multi-agent trajectories through six unique environments using multi-fidelity evaluations from various data sources. It was found that the resulting trajectories are faster than those obtained from two baseline methods. The optimized trajectories were validated in real-world experiments using four quadcopters that fly within centimeters of each other at speeds up to 7.4 m/s.

Hypothesis pruning is an important prerequisite while working with outlier-contaminated data in many computer vision problems. However, the underlying random data structures are barely explored in the literature, limiting designing efficient algorithms. To this end, we provide a novel graph-theoretic perspective on hypothesis pruning exploiting invariant structures of data. We introduce the planted clique model, a central object in computational statistics, to investigate the information-theoretical and computational limits of the hypothesis pruning problem. In addition, we propose an inductive learning framework for finding hidden cliques that learns heuristics on synthetic graphs with planted cliques and generalizes to real vision problems. We present competitive experimental results with large runtime improvement on synthetic and widely used vision datasets to show its efficacy.

Overview of the proposed algorithm that models dynamic feasibility constraints based on simulation and flight data to efficiently find the time-optimal trajectory.

We consider the problem of generating a time-optimal quadrotor trajectory for highly maneuverable vehicles, such as quadrotor aircraft. The problem is challenging because the optimal trajectory is located on the boundary of the set of dynamically feasible trajectories. This boundary is hard to model as it involves limitations of the entire system, including complex aerodynamic and electromechanical phenomena, in agile high-speed flight. In this work, we propose a multi-fidelity Bayesian optimization framework that models the feasibility constraints based on analytical approximation, numerical simulation, and real-world flight experiments. By combining evaluations at different fidelities, trajectory time is optimized while the number of costly flight experiments is kept to a minimum. The algorithm is thoroughly evaluated for the trajectory generation problem in two different scenarios: (1) connecting predetermined waypoints; (2) planning in obstacle-rich environments. For each scenario, we conduct both simulation and real-world flight experiments at speeds up to 11 m/s. Resulting trajectories were found to be significantly faster than those obtained through minimum-snap trajectory planning.

Tailsitter flying wing aircraft.

We propose a novel control law for accurate tracking of agile trajectories using a tailsitter flying wing micro unmanned aerial vehicle (UAV) that transitions between vertical take-off and landing (VTOL) and forward flight. Our global control formulation enables agile maneuvering throughout the flight envelope, including uncoordinated flight conditions with sideslip. We derive a differential flatness transform for the nonlinear tailsitter dynamics with a simplified aerodynamics model. Using this transform, the proposed controller incorporates accurate tracking of the position reference along with its temporal derivatives velocity, acceleration and jerk, as well as the yaw reference and yaw rate. The inclusion of jerk and yaw rate references through an angular velocity feedforward term increases tracking performance on agile trajectories with fast-changing accelerations. The control design is based on a simplified aerodynamics model that does not require extensive modeling of the aircraft dynamics. By applying incremental nonlinear dynamic inversion (INDI), the controller only depends on a local input-output relation to incrementally update control inputs, resulting in robustness against modeling inaccuracies. We achieve INDI with nonlinear dynamics inversion based on the derived differential flatness transform. The resulting control algorithm is extensively evaluated in flight tests, where it demonstrates accurate trajectory tracking and challenging agile maneuvers, such as uncoordinated sideways flight, aggressive transitions while turning, and differential thrust turning.

Layout of the jet vanes.

Small-scale thrust vector control (TVC) has the potential to enable rocket-powered micro aerial vehicles (MAV) capable of extremely fast and agile maneuvers. Jet vane TVC systems are particularly suitable for this task as they are capable of roll control and of exerting large side forces and moments at low airspeeds where aerodynamic surfaces are ineffective. In this paper, we present a novel small-scale TVC design using servo-actuated jet vanes. Our proposed design attains affordability and ease of manufacturing through use of modern additive manufacturing techniques. Titanium jet vanes are fabricated using selective laser sintering (SLS), and a ceramic heat shield, fabricated using stereolithography (SLA), is also designed. We evaluate the aerodynamic and thermal performance of the proposed design through numerical simulations, including modeling of rocket exhaust composition, computational fluid dynamics (CFD), and conjugate heat-transfer. Additionally, we present a test stand that enables measurement of forces and moments under both static and dynamic jet vane inputs. An initial static fire test is conducted using four vanes arranged in X-formation in the exhaust of a 54 mm commercial solid rocket motor. Experimental results are presented and compared to our numerical simulations.

This work addresses the problem of 6-DoF pose estimation under heavy occlusion. While previous work demonstrates reasonable results in unoccluded situations, robust and efficient pose estimation is still challenging in heavily occluded and low-texture scenarios which are ubiquitous in many applications. To this end, we propose a novel end-to-end deep neural network model recovering object poses from depth measurements. The proposed model enforces pairwise consistency of 3D geometric features by applying spectral convolutions on a pairwise compatibility graph. We achieve comparable accuracy as the state-of-the-art graph matching solver while being much faster. Our approach outperforms state-of-the-art 6-DoF pose estimation methods on LineMOD and Occlusion LineMOD and runs in reasonable time (~5.9 Hz). We additionally verify this method on a synthetic dataset with large affine changes.

Multi-modal trajectory for a hybrid aircraft, which switches between hover mode and coordinated flight mode. The solid and dotted lines indicate the planned and real-life flight trajectories, respectively.

In this paper, we present a motion planning framework for multi-modal vehicle dynamics. Our proposed algorithm employs transcription of the optimization objective function, vehicle dynamics, and state and control constraints into sparse factor graphs, which—combined with mode transition constraints—constitute a composite pose graph. By formulating the multi-modal motion planning problem in composite pose graph form, we enable utilization of efficient techniques for optimization on sparse graphs, such as those widely applied in dual estimation problems, e.g., simultaneous localization and mapping (SLAM). The resulting motion planning algorithm optimizes the multi-modal trajectory, including the location of mode transitions, and is guided by the pose graph optimization process to eliminate unnecessary transitions, enabling efficient discovery of optimized mode sequences from rough initial guesses. We demonstrate multi-modal trajectory optimization in both simulation and real-world experiments for vehicles with various dynamics models, such as an airplane with taxi and flight modes, and a vertical take-off and landing (VTOL) fixed-wing aircraft that transitions between hover and horizontal flight modes.

We study a problem in vision-aided navigation in which an autonomous agent has to traverse a specified path in minimal time while ensuring extraction of a steady stream of visual percepts with low latency. Vision-aided robots extract motion estimates from the sequence of images of their on-board cameras by registering the change in bearing to landmarks in their environment. The computational burden of the latter procedure grows with the range of apparent motion undertaken by the projections of the landmarks, incurring a lag in pose estimates that should be minimized while navigating at high speeds. This paper addresses the problem of selecting a desired number of landmarks in the environment, together with the time parametrization of the path, to allow the agent execute it in minimal time while both (i) ensuring the computational burden of extracting motion estimates stays below a set threshold and (ii) respecting the actuation constraints of the agent. We provide two efficient approximation algorithms for addressing the aforementioned problem. Also, we show how it can be reduced to a mixed integer linear program for which there exist well-developed optimization packages. Ultimately, we illustrate the performance of our algorithms in experiments using a quadrotor.

Overview of the proposed algorithm that models dynamic feasibility constraints based on simulation and flight data to efficiently find the time-optimal trajectory.

We consider the problem of generating a time-optimal quadrotor trajectory that attains a set of prescribed waypoints. This problem is challenging since the optimal trajectory is located on the boundary of the set of dynamically feasible trajectories. This boundary is hard to model as it involves limitations of the entire system, including hardware and software, in agile high-speed flight. In this work, we propose a multi-fidelity Bayesian optimization framework that models the feasibility constraints based on analytical approximation, numerical simulation, and real-world flight experiments. By combining evaluations at different fidelities, trajectory time is optimized while keeping the number of required costly flight experiments to a minimum. The algorithm is thoroughly evaluated in both simulation and real-world flight experiments at speeds up to 11 m/s. Resulting trajectories were found to be significantly faster than those obtained through minimum-snap trajectory planning.

Quadrotor with body-fixed reference system and moment arm definitions.

Autonomous unmanned aerial vehicles (UAVs) that can execute aggressive (i.e., high-speed and high-acceleration) maneuvers have attracted significant attention in the past few years. This article focuses on accurate tracking of aggressive quadcopter trajectories. We propose a novel control law for tracking of position and yaw angle and their derivatives of up to fourth order, specifically velocity, acceleration, jerk, and snap along with yaw rate and yaw acceleration. Jerk and snap are tracked using feedforward inputs for angular rate and angular acceleration based on the differential flatness of the quadcopter dynamics. Snap tracking requires direct control of body torque, which we achieve using closed-loop motor speed control based on measurements from optical encoders attached to the motors. The controller utilizes incremental nonlinear dynamic inversion (INDI) for robust tracking of linear and angular accelerations despite external disturbances, such as aerodynamic drag forces. Hence, prior modeling of aerodynamic effects is not required. We rigorously analyze the proposed control law through response analysis and demonstrate it in experiments. The controller enables a quadcopter UAV to track complex 3-D trajectories, reaching speeds up to 12.9 m/s and accelerations up to 2.1 g, while keeping the root-mean-square tracking error down to 6.6 cm, in a flight volume that is roughly 18 m ×7 m and 3-m tall. We also demonstrate the robustness of the controller by attaching a drag plate to the UAV in flight tests and by pulling on the UAV with a rope during hover.

The increasing popularity of quadrotors has given rise to a class of predominantly vision-driven vehicles. This paper addresses the problem of perception-aware time optimal path parametrization for quadrotors. Although many different choices of perceptual modalities are available, the low weight and power budgets of quadrotor systems makes a camera ideal for on-board navigation and estimation algorithms. However, this does come with a set of challenges. The limited field of view of the camera can restrict the visibility of salient regions in the environment, which dictates the necessity to consider perception and planning jointly. The main contribution of this paper is an efficient time optimal path parametrization algorithm for quadrotors with limited field of view constraints. We show in a simulation study that a state-of-the-art controller can track planned trajectories, and we validate the proposed algorithm on a quadrotor platform in experiments.

This article describes the Blackbird unmanned aerial vehicle (UAV) Dataset, a large-scale suite of sensor data and corresponding ground truth from a custom-built quadrotor platform equipped with an inertial measurement unit (IMU), rotor tachometers, and virtual color, grayscale, and depth cameras. Motivated by the increasing demand for agile, autonomous operation of aerial vehicles, this dataset is designed to facilitate the development and evaluation of high-performance UAV perception algorithms. The dataset contains over 10 hours of data from our quadrotor tracing 18 different trajectories at varying maximum speeds (0.5 to 13.8 ms-1) through 5 different visual environments for a total of 176 unique flights. For each flight, we provide 120 Hz grayscale, 60 Hz RGB-D, and 60 Hz semantically segmented images from forward stereo and downward-facing photorealistic virtual cameras in addition to 100 Hz IMU, ~190 Hz motor speed sensors, and 360 Hz millimeter-accurate motion capture ground truth. The Blackbird UAV dataset is therefore well suited to the development of algorithms for visual inertial navigation, 3D reconstruction, and depth estimation. As a benchmark for future algorithms, the performance of two state-of-the-art visual odometry algorithms are reported and scripts for comparing against the benchmarks are included with the dataset. The dataset is available for download at http://blackbird-dataset.mit.edu/.

3D-printed fuselage halves.

The recent developments in fast additive manufacturing, such as rapid 3d-printing of composite materials, presents opportunities for the manufacturing of small unmanned aerial vehicles (UAVs) that are tailor-made for the specific mission needs. This paper presents a novel framework for mission-oriented, modular design and construction of mini-UAVs using additive manufacturing. The outcome is a manufacturing method which is suitable for a parametric design that can be tailored for the specific mission requirements and rapidly constructed using additive manufacturing techniques. We show how additive manufacturing enables an agile design methodology by allowing fast and efficient iteration of prototypes during the design process.The proposed framework is demonstrated by presenting the iterative design of a tail-sitter hybrid VTOL vehicle, where the wing-tip geometry and the wing dihedral angle went through several changes. Additionally, we have extensively validated the performance of the resulting vehicle design through flight tests in a motion capture facility and outdoors. Finally we have shown several other configurations of vehicles such as quad-rotors, and a fixed-wing that can also take advantage of the proposed manufacturing method.

FlightGoggles rendering of the Abandoned Factory environment, designed for autonomous drone racing.

FlightGoggles is a photorealistic sensor simulator for perception-driven robotic vehicles. The key contributions of FlightGoggles are twofold. First, FlightGoggles provides photorealistic exteroceptive sensor simulation using graphics assets generated with photogrammetry. Second, it provides the ability to combine (i) synthetic exteroceptive measurements generated in silico in real time and (ii) vehicle dynamics and proprioceptive measurements generated in motio by vehicle(s) in flight in a motion-capture facility. FlightGoggles is capable of simulating a virtual-reality environment around autonomous vehicle(s) in flight. While a vehicle is in flight in the Flight-Goggles virtual reality environment, exteroceptive sensors are rendered synthetically in real time while all complex dynamics are generated organically through natural interactions of the vehicle. The FlightGoggles framework allows for researchers to accelerate development by circumventing the need to estimate complex and hard-to-model interactions such as aerodynamics, motor mechanics, battery electrochemistry, and behavior of other agents. The ability to perform vehicle-in-the-loop experiments with photorealistic exteroceptive sensor simulation facilitates novel research directions involving, e.g., fast and agile autonomous flight in obstacle-rich environments, safe human interaction, and flexible sensor selection. FlightGoggles has been utilized as the main test for selecting nine teams that will advance in the AlphaPilot autonomous drone racing challenge. We survey approaches and results from the top AlphaPilot teams, which may be of independent interest. FlightGoggles is distributed as open-source software along with the photorealistic graphics assets for several simulation environments, under the MIT license at http://flightgoggles.mit.edu.

An environment where most of the visual features are situated in the middle. Traditional planning approaches could consider facing forward as the vehicle navigates through a tight turn. Perception-aware planning algorithms, on the other hand, consider facing towards the visual features in the middle of the room while executing the same maneuver. This results in improved state-estimation performance, especially at high speeds.

Recent advances in visual-inertial state estimation have allowed quadrotor aircraft to autonomously navigate in unknown environments at operational speeds. In most cases, substantially higher speeds can be achieved by actively designing motion that reduces state estimation error. We are interested in autonomous vehicles running feature-based visual-inertial state estimation algorithms. In particular, we consider a trajectory optimization problem in which the goal is to maximize co-visibility of features, i.e. features are kept visible in the camera view from one keyframe to the next, increasing state estimation accuracy. Our algorithm is developed for autonomous quadrotor aircraft, for which position and yaw trajectories can be tracked separately. We assume that the desired positions of the vehicle are determined a priori, for instance, by a path planner that uses obstacles in the environment to generate a trajectory of positions with free yaw. This paper presents a novel algorithm that determines the yaw trajectory that jointly optimizes aggressiveness and feature co-visibility. The benefit of this algorithm was experimentally verified using a custom built quadrotor which uses visual inertial odometry for state estimation. The generated trajectories lead to better state estimation which contributes to improved trajectory tracking by a state-of-the-art controller under autonomous high-speed flight. Our results show that the root-mean-square error of the trajectory tracking is improved by almost 70%.

The time optimal path parametrization problem addresses minimizing the traversal time of a specified path by an actuation constrained agent. Recently, an efficient numerical algorithm for solving this problem has been proposed. This letter theoretically establishes convergence of the former algorithm to the optimum for the whole class of problems solved optimally by computationally more demanding approaches based on convex programming. Additionally, we provide a characterization of the optimum, which may be of independent interest.

Quadrotor with body-fixed reference system and moment arm definitions.

In this paper, we propose a novel control law for accurate tracking of aggressive (i.e., high-speed and high-acceleration) quadcopter trajectories. The proposed method tracks position and yaw angle with their derivatives of up to fourth order, specifically, the position, velocity, acceleration, jerk, and snap along with the yaw angle, yaw rate and yaw acceleration. Two key aspects of the proposed method are the following. First, the controller exploits the differential flatness of the quadcopter dynamics to generate feedforward inputs for attitude rate and attitude acceleration in order to track the jerk and snap references. The tracking is enabled by direct control of body torque using closed-loop control of all four propeller speeds based on optical encoders attached to the motors. Second, the controller utilizes the incremental nonlinear dynamic inversion (INDI) method for accurate tracking of linear and angular accelerations despite external disturbances. Hence, no prior modeling of aerodynamic effects is required. We evaluate the proposed control law in experiments under motion capture. Using a 1-kg quadcopter, we are able to track a complex 3D trajectory, reaching speeds up to 8.2 m/s and accelerations up to 2g, while keeping the root-mean-square tracking error down to 4.0 cm, in a flight volume that is roughly 6.5 m long, 6.5 m wide, and 1.5 m tall.

The Blackbird unmanned aerial vehicle (UAV) dataset is a large-scale, aggressive indoor flight dataset collected using a custom-built quadrotor platform for use in evaluation of agile perception. Inspired by the potential of future high-speed fully-autonomous drone racing, the Blackbird dataset contains over 10 h of flight data from 168 flights over 17 flight trajectories and 5 environments at velocities up to 7.0 m/s. Each flight includes sensor data from 120 Hz stereo and downward-facing photorealistic virtual cameras, 100 Hz IMU, ∼190 Hz motor speed sensors, and 360 Hz millimeter-accurate motion capture ground truth. Camera images for each flight were photorealistically rendered using FlightGoggles across a variety of environments to facilitate easy experimentation of high performance perception algorithms. The dataset is available for download at http://blackbird-dataset.mit.edu/.

Flight space (left), and simulated evader and measurement (right).

We consider pursuit-evasion games in which the pursuer is tasked with intercepting the evader using only partial measurements. Motivated by the utilization of visual sensing on board the pursuer, we focus on the case when only bearing measurements are available to the pursuer. The resulting partially-observable interception problem is computationally challenging, and the separation principle does not hold in general. In this paper, we identify a set of maneuvers that improve observability, and we propose an algorithm that utilizes these maneuvers to move the pursuer so that the expected payoff of the differential game is maximized. The algorithm uses in-the-loop uncertainty propagation based on linear covariance analysis to assess the effect of the maneuvers. We evaluate the resulting guidance law in experiments involving a quadcopter in flight representing the pursuer, and a simulated evader.

Value function for Homicidal Chauffeur differential game.

Zero-sum differential games constitute a prominent research topic in several fields ranging from economics to motion planning. Unfortunately, analytical techniques for differential games can address only simple, illustrative problem instances, and most existing computational methods suffer from the curse of dimensionality, i.e., the computational requirements grow exponentially with the dimensionality of the state space. In order to alleviate the curse of dimensionality for a certain class of two-player pursuit-evasion games, we propose a novel dynamic-programming-based algorithm that uses a continuous tensor-train approximation to represent the value function. In this way, the algorithm can represent high-dimensional tensors using computational resources that grow only polynomially with dimensionality of the state space and with the rank of the value function. The proposed algorithm is shown to converge to optimal solutions. It is demonstrated in several problem instances; in case of a seven-dimensional game, the value function representation was obtained with seven orders of magnitude savings in computational and memory cost, when compared to standard value iteration.

To enable algorithmic work in a wide range of visual conditions we have developed a system to replace the UAV's on-board camera with images from a virtual environment. While the UAV is in flight (top) the motion capture pose estimate of the UAV is sent to the Unity game engine running on a TitanX GPU (middle) which can generate the corresponding photorealistic image (bottom) for that pose from a virtual world which is processed and transmitted to the UAV in real time. The system runs fully in real time as if the sensors were on the UAV, allowing experiments and decision making in adverse conditions such as obstacle rich environments or in environments that are difficult to access such as cities.

The development of fast, agile micro Unmanned Aerial Vehicles (UAVs) has been limited by (i) on-board computing hardware restrictions, (ii) the lack of sophisticated vision-based perception and vision-in-the-loop control algorithms, and (iii) the absence of development environments where such systems and algorithms can be rapidly and easily designed, implemented, and validated. Here, we first present a new micro UAV platform that integrates high-rate cameras, inertial sensors, and an NVIDIA Jetson Tegra X1 system-on-chip compute module that boasts 256 GPU cores. The UAV mechanics and electronics were designed and built in house, and are described in detail. Second, we present a novel “virtual reality” development environment, in which photorealistically-rendered synthetic on-board camera images are generated in real time while the UAV is in flight. This development environment allows us to rapidly prototype computing and sensing hardware as well as perception and control algorithms, using real physics, real interoceptive sensor data (e.g., from the on-board inertial measurement unit), and synthetic exteroceptive sensor data (e.g., from synthetic cameras). Third, we demonstrate repeated agile maneuvering with closed-loop vision-based perception and control algorithms, which we have developed using this environment.