Autonomous Vision Blog

KITTI-360

2022-05-30T09:10:00+00:00

Since the 1960s, several major subfields of artificial intelligence like computer vision, graphics, and robotics have developed largely independently. But the community has recently recognized that progress toward self-driving cars requires an integrated effort across different fields. This motivated us to create KITTI-360, successor of the KITTI dataset, a new suburban driving dataset which comprises richer input modalities, comprehensive semantic instance annotations and accurate localization to facilitate research at the intersection of vision, graphics and robotics. To get an overview of the dataset, we invite you to enjoy our (overly) dramatic cinematic video trailer that we have produced.

Online Benchmarks

Recently, we opened up all our evaluation servers for submission. In this blog post, we are excited to share some of the novel and exciting tasks that KITTI-360 considers at the intersection of vision, graphics and robotics.

How to parse the scene?

Semantic scene understanding is one of the key capabilities of autonomous driving vehicles. With KITTI-360, we establish scene perception benchmarks in both 2D image space and 3D domain:

2D Semantic/Instance Segmentation
3D Semantic/Instance Segmentation
3D Bounding Box Detection
Semantic Scene Completion

We evaluated several baselines to bootstrap the leaderboards and assess the difficulties of the tasks:

For some of the tasks, we show interactive plots to illustrate results of the submission on our website. Here is an example for the semantic scene completion task:

How to simulate the scene?

Simulation is an essential tool for training and evaluating autonomous vehicles. Imaging a simulator providing both RGB images and semantic information with no domain gap to the real-world, wouldn’t it be great? Towards this goal, we establish challenging benchmarks:

Novel View Appearance Synthesis
Novel View Semantic Synthesis

For novel view appearance synthesis, we evaluated several state-of-the-art methods, including NeRF-based methods. For novel view semantic synthesis, we evaluate naïve baselines by applying an existing semantic segmentation model to the synthesized images. Here we show results of mip-NeRF as an example:

How can semantics benefit SLAM?

We also establish a semantic SLAM benchmark at the intersection of robotics and computer vision to investigate how semantics can benefit SLAM. Existing works on outdoor semantic SLAM typically evaluate only pose estimation while ignoring the quality of the semantic reconstruction. Considering that the semantic reconstruction is valuable on its own for down-stream tasks, e.g. planning, we evaluate both tasks on KITTI-360:

Localization
Geometric and Semantic Mapping

We evaluate accuracy, completeness and semantic mIoU for geometric and semantic mapping. Here we show the results of a simple baseline by using ORB-SLAM2 for localization and fusing depth maps of semi-global matching for reconstruction:

Download and Submit

Our leaderboard has received a few exciting submissions since the release, including DeepViewAggregation for 3D semantic segmentation and Panoptic Neural Fields for novel view appearance and semantic synthesis. Are you interested in joining us? Download the training and test data on our website and try it out yourself! For each task, we provide evaluation scripts in our utility scripts.

Further Information

To learn more about our work, check out our video here:

You can find more information on our website and our paper. Our annotation tool is also publicly available. We are happy to receive your feedback!

@inproceedings{Liao2022PAMI,
  title = {KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D},
  author = {Liao, Yiyi and Xie, Jun and Geiger, Andreas},
  booktitle = {IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI)},
  year = {2022}
}

On the Frequency Bias of Generative Models

2021-12-10T05:15:00+00:00

Generative adversarial networks (GANs) have enabled photorealistic and high-resolution image synthesis. However, even for state-of-the-art methods, generated images are still straightforward to identify in the frequency domain. Compared to the real images, the average spectrum of the generated images looks quite different, particularly for high frequencies.

This is somewhat surprising because the key objective of GAN training is to make the statistics of the generated images similar to the statistics of the real images. More specifically, GANs are trained in an adversarial game of two players, a generator and a discriminator. The generator generates the images and the discriminator has to distinguish between real and generated images. Over the course of training, the generated images should become indistinguishable from the real images. Here it gets interesting: While the generated images indeed appear photorealistic, their spectrum gives away their identity. In fact, a very simple classifier on the spectrum can distinguish real and generated images almost perfectly. Does this mean that the discriminator, a deep neural network with millions of parameters, is blind to the spectral artifacts? Or is the generator not even able to generate images with the correct spectral statistics? One clue to answering these questions is that the spectral artifacts prevailingly occur at high frequencies. This indicates that low frequencies might be easier to generate or detect for neural networks. Hence, we ask

Is there a frequency bias in the generator and/or the discriminator?

This is a non-trivial question as GAN training involves two players where architectures for both the generator and discriminator, loss functions, as well as the dataset statistics can all affect the generated images. So let us first narrow down potential factors and consider generator and discriminator in isolated testbeds.

Is there a frequency bias in the Generator?

Most existing works suspect that the upsampling operations in the generator cause the spectral artifacts. So let us first investigate the effect of upsampling on the spectrum using a simple reconstruction task.

For a single image, the generator is given a fixed noise input from which it needs to reconstruct the image. To detect a frequency bias, we monitor the reconstructed image, the spectrum, and the error evolution of the spectrum over the course of training. The spectrum error evolution illustrates how the relative error between the ground truth and predicted reduced spectrum evolves during training. Red color indicates a higher predicted spectral density and blue color a lower predicted spectral density than in the training image.

You can see that different upsampling methods bias the generator towards different spectral properties. Due to its interpolation-based nature, bilinear upsampling results in overly smooth generated images. This lack of high-frequency content shows in the reduced spectrum and, indicated by the blue color, in the spectrum error evolution.
With bed-of-nails upsampling the generator instead learns all frequencies approximately equally fast. However, there is a peak at the highest frequencies which indicates checkerboard-like artifacts from inserting the zeros.

Follow-up question: Why do the learnable weights of the generator not compensate for the artifacts from upsampling?
Short answer: The mean-squared error used in our testbed penalizes small (absolute) errors less than large errors. Therefore, it penalizes checkerboard artifacts only slightly. It turns out that when the loss function is sensitive to these artifacts the generator is indeed capable of compensating for them. For more details please check out our paper.
This suggests that checkerboard patterns cannot explain the spectral artifacts alone and raises the question of whether the discriminator is able to detect high frequencies in the first place?

Is there a frequency bias in the Discriminator?

We investigate the spectral properties of the discriminator with a GAN-equivalent of a reconstruction task. Specifically, we train a GAN but parameterize the generator as a learnable tensor with the same dimension as the input image. This allows us to directly visualize the training signal provided by the discriminator. As usual, the generator and discriminator are trained in an alternating fashion. In addition to the original image, we consider a high-pass filtered version of the image to obtain a setting in which the high frequencies dominate the input signal.

As you can see, the discriminator fits frequencies with high magnitudes well but struggles with frequencies of low magnitude. For the original image, low magnitudes correspond to high frequencies whereas for the high-pass filtered image low magnitudes correspond to low frequencies. This shows, that the discriminator does not struggle with high frequencies per se. Instead, it is sensitive to the magnitude of the frequencies. However, for natural images high frequencies typically have the lowest magnitudes, making them difficult to detect for the discriminator.

Follow-up question: Is aliasing a problem in the discriminator because it downsamples the images?
Short answer: Yes. The reconstructed image looks significantly worse than the reconstructed image from the generator testbed. The artifacts in the reconstruction vanish when the convolutional discriminator is replaced with an MLP which uses no downsampling. check out our paper if you want to learn more about different downsampling operations and their effects on the training signal.

So how do we fix this?

Recent works propose several measures to reduce the spectral artifacts in generated images. We found the most effective to be an additional discriminator on the reduced spectrum as proposed in Jung et al.. However, the discriminator is applied to the reduced (1D), and not the full (2D), spectrum and might therefore incentivize a shortcut solution. Indeed, we find that finetuning StyleGAN with this loss can introduce artifacts in the background of the generated images to fool the spectral discriminator.

Conclusion: For now, artifacts in generated images remain an open problem. Our findings suggest that the design of the discriminator plays an important role and deserves more attention in future work. Interesting directions might be to explore alternative downsampling operations and to consider approaches that enable pixel-level supervision for the discriminator.

Curious?

Analyze your own GAN architecture or investigate potential fixes with our generator and discriminator testbeds. For more details on the topic, watch our NeurIPS’21 talk:

Further information (including the paper and supplementary) is available on our project page. If you are interested in experimenting with our testbeds, download the source code of our project and run the examples. We are happy to receive your feedback!

@inproceedings{Schwarz2021NEURIPS,
title = {On the Frequency Bias of Generative Models},
author = {Schwarz, Katja and Liao, Yiyi and Geiger, Andreas},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2021},
doi = {}
}

Animate Implicit Shapes with Forward Skinning

2021-09-29T11:00:00+00:00

Neural implicit surface representation is a recently emerging representation of 3D shapes. They represent the occupancy or SDF field of the 3D shape with multi-layer perceptrons (MLP). These MLPs take a 3D coordinate as input and outputs the corresponding occupancy or SDF value. Such representations have several benefits, including topological flexibility, low memory footprint, and resolution independence, leading to impressive advances in modeling 3D rigid objects and scenes.

However, adapting them to non-rigidly moving objects, such as humans, is not straightforward. In this post, we will explain our approach to animating neural implicit surfaces.

Skinning Polygon Meshes

Let’s first take a look at how polygon meshes are animated. A very common approach is linear blend skinning (LBS). LBS requires a canonical template mesh to represent the surface and a set of handles (bones) to control the posture. Each vertex on the canonical mesh has skinning weights that define how bones influence this vertex. By taking the weighted average of given bone transformations, we know how each vertex should be transformed. The skinning weights are either defined by artists or can be learned from data.

Credit: this image is taken from this tutorial by Ladislav Kavan.

Despite being simple, LBS produces decent animation quality and has become a standard tool in the graphics pipeline. So, can we bring LBS to neural implicit surfaces?

Skinning Neural implicit Surfaces

Unlike meshes where the skinning weights are defined only on vertices, we now need the skinning weights for any point in space - a continuous skinning weights field. This field can be represented as an MLP, which maps 3D coordinates to skinning weights.

Now, in which space should this field be defined? In deformed space or canonical space?

Backward Skinning: Weights in Deformed Space

Previous work chooses to define the skinning weights in deformed space. To know whether a point in deformed space is occupied or not, we first query its skinning weights, compute the transformation of the point via LBS, and then we know the canonical correspondence of that point.

We call such method backward skinning as it maps points in deformed space back to canonical space. Note that in this formulation, we need to know the skinning weights of points in deformed space. In different poses, the same point in the deformed space will have different skinning weights, because the deformed space itself changes with the pose. So the MLP needs to memorize the skinning weights in different poses and cannot generalize to unseen poses, which is an important limitation.

Forward Skinning: Weights in Canonical Space

How about defining the skinning weights field in canonical space? We can then transform a point in canonical space to deformed space.

Because the skinning weights field is defined in the canonical space, it is independent of the pose and naturally does not suffer from generalization.

But why hasn’t forward skinning been used before? We often need to know the occupancy of a specific point in deformed space, which is not straightforward for forward skinning. We know where a given canonical point goes with forward skinning, but not where a given deformed point comes from.

Our approach addresses this technical challenge and enables querying canonical correspondences of deformed points with forward skinning weights. Using our approach, we can generate deformed shapes in any poses, even those unseen during training.

Our Approach

The core of the problem is: given a forward mapping function $d:x \rightarrow x’$, determine it’s inverse mapping $x’ \rightarrow x$. In our case, $d$ is the composition of LBS and an MLP, which is not invertible. Thus, we resort to numerical methods. We use Broyden’s method, an iterative root finding algorithm, to find $x$ that satisfies the condition $d(x)=x’$. The solution $x$ we find is the canonical correspondence of the given $x’$.

In many cases, one deformed point might have multiple canonical correspondences. For example, in the case above, the torso and hand points in canonical space move to the same location in deformed space. This means that the equation $d(x)=x’$ has multiple roots. To find all these roots, we initialize Broyden’s method from multiple initial states. These initial states are obtained by rigidly transforming the deformed point with individual bone transformations. After finding all canonical correspondences, we query their occupancy probabilities from the canonical shape network. We then aggregate these probabilities by taking their maximum - a standard operator for compositing multiple implicit shapes.

To this end, we obtain the occupancy probability of deformed points. We can then compare this to the ground-truth occupancy for training. However, the iterative root finding algorithm is not trivially differentiable for training. We could unroll the iterations and apply standard chain rules to compute gradients, but it would be highly time and memory-consuming. To address this problem, we derive the analytical gradients using the implicit differentiation theorem. This allows us to train our skinning weights and occupancy network from deformed observations using standard back-propagation.

Does it work?

We use our approach to model 3D minimally clothed humans. We train our method using meshes in various poses and ask the model to generate novel poses during inference time. Here is one example:

As shown, our method generalizes to these challenging and unseen poses. In comparison, backward skinning produces distorted shapes for unseen poses. The other baseline, NASA, models human body as a composition of multiple parts and suffers from discontinuous artifacts at joints.

Our method can also represent other objects, such as clothed humans and animal:

Further Information

Thank you for reading! We invite you to try out our code and to check out our paper/video/project page for more technical details. If you find our work helpful, please consider citing it as

@inproceedings{chen2021snarf,
  title={SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes},
  author={Chen, Xu and Zheng, Yufeng and Black, Michael J and Hilliges, Otmar and Geiger, Andreas},
  booktitle={International Conference on Computer Vision (ICCV)},
  year={2021}
}

Novel approaches for fast rendering of neural coordinate-based representations

2021-09-09T06:00:00+00:00

Neural coordinate-based or implicit representations like the one used in Neural Radiance Fields (NeRF) offer very good compression but are due to their slow decoding speed inherently unsuited for real-time applications. In this blog post, I discuss several recent works that address this limitation of neural 3D representations.

The video above was directly rendered in real-time from the neural representation adopted by KiloNeRF.

Neural coordinate-based representations

Since their introduction (Chen et al., 2019, Mescheder et al., 2019, Park et al, 2019) to the computer vision community, neural coordinate-based representations for 3D scenes are being used in an ever-increasing number of works. Here the scene is encoded with help of a (deep) neural network that gets as input a 3D coordinate and gives out scene properties like color, occupancy or volumetric density at the given 3D coordinate. This leads to a highly compressed representation that can be directly learned from various kinds of supervision signals like multi-view images. NeRF demonstrated that high-quality results for novel view synthesis (NVS) can be achieved by making use of such a coordinate-based representation. A fascinating aspect is that a scene can be encoded with only 5 MB of memory and this holds even though NeRF models color in a view-dependent way, which means that the scene function is even 5-dimensional (3 dimensions for position and 2 dimensions for viewing direction). In contrast, storing a scene uncompressed, i.e. with a 5-dimensional table (view-dependent voxel grid), would consume tens of gigabytes of disk storage. However, the high compression ratio of NeRF and other neural representations comes with a steep price. Directly rendering such a representation, which typically requires dense evaluation of the neural network, is painfully slow and therefore these representations are unsuited for real-time applications. Recently, a lot of works tried to address this issue of coordinate-based neural representations and NeRF in particular.

KiloNeRF

For each point where we want to evaluate the NeRF a forward pass through a Multi-Layer Perceptron (MLP) is required. This means that we have to execute a sequence of expensive vector-matrix multiplications to decode each individual point. Naturally, decoding would be faster if we could get away with using a smaller MLP, i.e. one with a smaller number of neurons and/or hidden layers. However, if we just use a smaller MLP without any other measures the reduced MLP’s capacity is not sufficient anymore to faithfully represent the whole scene. The simple idea underlying KiloNeRF is to first subdivide the scene into a coarse voxel grid and assign a separate MLP to each voxel cell. Since each of these MLPs only has to represent a small part of the scene tiny MLPs suffice to represent the scene with high detail.

In combination with standard optimization techniques from volumetric rendering, we demonstrate that real-time rendering with a resolution of 800x800 can be achieved. Without sacrificing any visual fidelity KiloNeRF requires less than 100 MB of memory to store a scene. While this is more than the 5 MB that a single MLP requires, compression is still sufficiently high for many practical purposes.

@INPROCEEDINGS{Reiser2021ICCV,
  author = {Christian Reiser and Songyou Peng and Yiyi Liao and Andreas Geiger},
  title = {KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs},
  booktitle = {International Conference on Computer Vision (ICCV)},
  year = {2021}
}

Tabulation-based approaches

Another conceptually straightforward idea is to convert NeRF’s representation into a discrete representation after training. However, naively converting a radiance field into a 5-dimensional table would require tens of gigabytes of memory, which exceeds what current GPUs offer. Three recent works (Garbin et al., 2021, Yu et al., 2021, Hedman et al., 2021) show that by cleverly dealing with the view-dependent color the storage impact can be significantly decreased. For instance, PlenOctrees make use of spherical harmonics, which allow compact encoding of functions defined on a sphere. In a Sparse Neural Radiance Grid (SNeRG), view-dependent color is encoded as a low-dimensional feature vector that can be efficiently decoded with a tiny MLP. A clear advantage of these methods over KiloNeRF is that the tabulation-based approaches enable real-time rendering of higher resolution outputs on older/cheaper hardware. There is still ample potential for optimization of KiloNeRF by making use of the Tensor Cores present in more recent consumer GPUs, which might also enable KiloNeRF to render HD images in real-time. A disadvantage of the tabulation-based approaches is that they signficiantly increase GPU memory and/or hard disk consumption as these methods require several gigabytes of memory. For small and medium-sized scenes this does not yet pose a big problem since current GPUs offer sufficient amounts of memory, but scaling to larger scenes quickly leads to running out of memory. SNeRG demonstrates that with help of standard image codecs like PNG or JPEG their representation can be compressed to less than 100 MB of memory. This greatly helps with decreasing disk consumption and transmission times. However, since PNG/JPEG decoding has to happen prior to rendering GPU memory consumption cannot be reduced with this additional compression step. Therefore scalability remains an aspect where KiloNeRF’s approach might be advantageous. That being said, there are numerous works (Rodríguez et al., 2014) from the volumetric rendering community that encode volumes such that transient decompression is possible, i.e. the full volume does not need to be present in decompressed form on the GPU for rendering. It might be thus interesting to use these codecs on tabulated data and see how this compares in terms of compression ratio and decoding speed to KiloNeRF.

Neural Geometric Level of Detail

Similar to KiloNeRF, Neural Geometric Level of Detail (NGLOD) also speeds up inference by using smaller MLPs to represent the scene. However, NGLOD does not employ MLPs that are only responsible for a particular region of the scene. Rather, each of their MLPs is responsible for the whole scene. To reduce the required parameter count of their MLPs they condition on local features, that are tri-linearly interpolated before being fed into the MLP. To be able to compactly store the scene their representation supports multiple levels of detail (LODs), where the higher LODs are only defined for geometrically complex regions. Each LOD is represented by a separate MLP. Their approach enables real-time rendering and leads to high compression ratios. However, NGLOD is trained from 3D supervision and only encodes geometry, while in the NVS setting only multi-view supervision is available and both geometry and view-dependent color needs to be encoded. It would be interesting to adapt NGLOD for this more difficult setting and see how it compares to KiloNeRF.

Light Field Networks

In NeRF, to render each pixel a ray is cast through that pixel and the representation has to be evaluated for many query locations along the ray. Light Field Networks (LFN) and Neural 4D Light Fields do away with this requirement for multiple samples along the ray by directly learning the mapping from ray origin and ray direction to the pixel’s RGB value. This does not impose the same inductive biases as the volumetric rendering employed by NeRF and therefore some additional learned or handcrafted priors are necessary. The network is only evaluated once per pixel and therefore rendering is greatly sped up. Since the scene is represented by a single MLP the storage requirements are as low or even lower than NeRF’s. However, these approaches assume that the color along each ray is constant, which limits generality.

Outlook

There is some promising progress in adopting neural coordinate-based representations for real-time applications. An interesting avenue for future work is to scale these approaches to larger scenes.

Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks

2021-06-20T08:30:00+00:00

Automatically, abstracting 3D shapes into semantically meaningful parts without any part-level supervision is hard. Existing approaches either lead to semantic abstractions with few simple shapes (eg. superquadrics) or they yield geometrically accurate reconstructions with a large number of primitives, while sacrificing semantic interpretability, namely primitives do not represent identifiable parts. In this work, we attempt to combine the simplicity and semanticness using an expressive flow-based 3D primitive representation which enables efficient computation of both the implicit surface function and the mesh. Our expressive 3D primitive representation allows to parse 3D objects into geometrically accurate and semantically consistent part arrangements without any part-level supervision. Here, we show the reconstructed primitives on various ShapeNet objects, D-FAUST humans and FreiHAND hands using only 5 primitives.

Our key insight is that a primitive should be a non trivial genus-zero shape with well defined implicit and explicit representations. These two characteristics are crucial in order to allow us to efficiently combine primitives and accurately represent arbitrarily complex geometries.

Why wasn’t this possible before?

Existing primitive-based representations seek to infer semantically consistent part arrangements across different objects and provide a more interpretable alternative, compared to more powerful representations that focus solely on the object geometry. They rely on simple shapes such as cuboids, superquadrics, 3D anisotropic gaussians or more general convex shapes for reconstructing the 3D geometry as a set of parts. Due to their simple shape parametrization, existing primitives require a large number of primitives to accurately capture the 3D geometry. However, increasing the number of parts results in less meaningfull abstractions, where parts do not correspond to identifiable parts.

For example, in the previous image, for the majority of the primitives, it is really hard to say whether a primitive corresponds to a human part or a plane part. This is expected since existing primitives have limited expressivity, due to their simple parameteriazation, namely a primitive cannot represent an arbitrarily complex shape, such the legs the and hands of a human using a single primitive. As a result, in order to represent complex geometries existing primitive-based representations that rely on simple shapes, require a large number of primitives that do not correspond to semantically meaningful parts.

Existing primitive-based representations can be arranged based on their semantic interpretability, namely whether the reconstructed primitives have a semantic interpretation, and their representation power, namely whether the reconstructed shape is geometrically accurate. We identify that there exists a trade-off between the reconstruction quality and the number of parts in primitive-based methods and we propose a novel primitive-based representation that addresses this trade-off.

In contrast to prior work that considers convex shapes as primitives, we propose to define primitives using an Invertible Neural Network (INN) which implements homeomorphic mappings between a sphere and the target object. The INN allows us to compute the inverse mapping of the homeomorphism, which in turn, enables the efficient computation of both the implicit surface function of a primitive and its mesh, without any additional post-processing.

Our Approach

Our main idea is to define each primitive as the deformation of a sphere that is conditioned on the input via a predicted latent vector. This formulation does not impose any constraint on the shape of each primitive and thus our primitives can capture arbitrarily complex genus-zero geometries. In particular, we define this deformation as a homeomorphic mapping between shapes implemented with an Invertible Neural Network (INN). A homeomorphism is a continuous map between two topological spaces $Y$ and $X$ that preserves all topological properties. In our setup, a homeomorphism $\phi_{\theta}: \mathbb{R}^3 \to \mathbb{R}^3$ and and $\phi_{\theta}:Y \to X$, $\phi_{\theta}^{-1}: X \to Y$ are continuous bijections.

Intuitively a homeomorphism is a continuous stretching and bending of Y into a new space X. Implementing a homeomorphism with an INN allows us to efficiently compute the implicit and explicit representation of the predicted shape and impose various constraints on the predicted parts such as normal consistency, parsimony etc.

Due to the lack of primitive annotations, we train our model by minimizing the geometric distance between the target and the predicted shape. Our supervision comes from a watertight mesh of the target object in the form of surface samples, i.e. randomly sampled points on the surface of the target object) and occupancy pairs i.e. points inside and outside the bounding boxes that contains the target object accompanied by their occupancy labels.

During training, we optimize our network using a Chamfer loss between points on the surface of the predicted shape and the target object, as well as with an occupancy loss that ensures that the free and the occupied space of the predicted and the target shape coincide. Moreover, we define three auxiliary losses that enforce firstly, that the orientation of the normals of the predicted and the target shape is consistent, secondly that primitives do not overlap which ensures that they describe different parts of the object and finally a loss that prevents degenerate primitive arrangements, where some primitives are very small and do not contribute to the reconstruction.

How well do Neural Parts work?

Here, we compare the part-based reconstructions with different number of primitives using Neural Parts and CvxNet. Neural Parts accurately capture the 3D geometry of the human using even a single primitive, whereas CvxNet fail to capture the 3D geometry even when increasing the number of parts to 10. For Neural Parts, it can be easily seen that reconstructions with 5, 8 and 10 primitives are very similar, namely Neural Parts achieve similar performance regardless of the number of primitives. This is a very important property as it allows us to select the appropriate abstraction level based on our application without having to compromise abstraction to improve reconstruction.

We evaluate our model on the 3D reconstruction task on the D-FAUST dataset and show that Neural Parts result in semantically more meaningful abstractions, such as legs and arms, using only 5 primitives, whereas simpler primitives such as CvxNet, SQs and H-SQs require an order of magnitude more parts to accurately represent the target object. As a result, simpler primitive representations yield less meaningful abstractions. In addition, we also note that Neural Parts preserve their semantic identity through a wide range of human motion, since the same primitive is consistently used for representing the same human part while humans are moving.

We further evaluate our model on FreiHAND dataset, and observe that it faithfully captures fine details of organic shapes such as bent fingers using a single primitive, while prior work focuses primarily on the structure of the predicted shape and misses out fine details such as fingers.

We finally validate our model on various ShapeNet objects and observe that it results in more accurate reconstructions than CvxNet with 5 primitives. When increasing the number of primitives to 25, CvxNet improves in terms of reconstruction quality but the predicted primitives lack a semantic interpretation. For the case of chairs and lamps, CvxNet with 5 primitives do not have enough representation power, thus entire object parts are missing.

Further Information

To learn more about our work, check our video here:

More information (including the paper, and the code and our pretrained models is available on our project page. Make sure to check out our interactiva demo for visualizing the reconstructed primitives!

@inproceedings{Paschalidou2021CVPR,
    title = {Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks},
    author = {Paschalidou, Despoina and Katharopoulos, Angelos and Geiger, Andreas and Fidler, Sanja},
    booktitle = {Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
    month = jun,
    year = {2021}
}

SMD-Nets: Stereo Mixture Density Networks

2021-06-02T00:00:00+00:00

Introduction

Stereo matching indicates the problem of finding dense correspondences in pairs of images in order to perform 3D reconstruction. Recently, deep learning based stereo matching approaches are taking over, becoming one of the best performing methods as witnessed by the leaderboard across different datasets. However, although the performance of CNN-based algorithms has gained great improvement, some difficult problems still exists in the disparity estimation task, in particular:

Over-smoothing effects at edge boundaries, causing bleeding artifacts in 3D point clouds
Disparity estimates at pixel locations of a fixed resolution image grid

In the following figure, we illustrate the boundary bleeding problem that characterizes classical deep stereo networks (for clarity we visualize the disparity for a single image row):

We can notice how standard networks for stereo regression predict disparity values at discrete spatial locations and continuously interpolate object boundaries. This effect, which can be observed in the figure below, causes artifacts (i.e. flying pixels) when disparities are converted to point clouds that could be potentially detrimental to subsequent applications such as 3D object detection.

Our Approach

How can we solve the over-smoothing problem? Our key idea in this project is to alter the output representation adopting a bimodal mixture distribution such that sharp discontinuities can be regressed despite the fact that the underlying neural networks are only able to make smooth predictions. We can visualize this as follows:

In this example, the first mode models the background while the second mode models the foreground disparity close to the discontinuity. When the probability density (illustrated in gray) of the foreground mode becomes larger than the probability density of the background mode, the most likely disparity sharply transitions from the background to the foreground.

With this simple idea, we are able to produce sharp and accurate depth discontinuities and, thus, notably alleviate bleeding artifacts near object boundaries!

How can we estimate disparities at any continuous spatial location? Also in this case, the solution is straightforward. For every continuous 2D location of the image domain, we bilinearly interpolate the features computed by a standard stereo network and employ a multi-layer perceptron to map the abstract feature representation to the parameters of our bimodal mixture distribution. This strategy allows us to predict disparity maps at arbitrary spatial resolution!

Is this method designed for a specific stereo network? No, it isn’t. Our framework is compatible with a wide class of 2D and 3D stereo architectures existing in the literature. Moreover, it can be effectively adopted for other exciting tasks such as monocular depth estimation and active depth!

Stereo Super-Resolution

Our continuous formulation allows us to exploit ground truth at higher resolution than the input, which we refer to as stereo super resolution.

Here, a comparison of our model that estimates disparities at 128Mpx (top) and a standard stereo network that predicts disparity maps at 0.5Mpx resolution (bottom), both taking stereo pairs at 0.5 Mpx as input.

UnrealStereo4K

Motivated by the lack of large-scale, realistic and high-resolution stereo datasets, we introduce a new large-scale synthetic binocular stereo dataset with ground truth disparities at 3840×2160 resolution, comprising photo-realistic renderings of indoor and outdoor environments.

Results

Some qualitative results of the proposed SMD-Nets using PSMNet as stereo backbone. From left to right, the input image from the UnrealStereo4K test set, the predicted disparity and the corresponding error map.

Further Information

To learn more about work, check our video:

You can find more information (including the paper and supplementary) on our project page.

If you are interested in experimenting with our approach yourself, you can clone the source code of our project page and train your own models.

@INPROCEEDINGS{Tosi2021CVPR,
author = {Fabio Tosi and Yiyi Liao and Carolin Schmitt and Andreas Geiger},
title = {SMD-Nets: Stereo Mixture Density Networks},
booktitle = {Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2021}
} 

Sensor Fusion for Self Driving

2021-05-11T18:45:00+00:00

Consider a scenario where the ego-vehicle (shown in green) is about to enter an intersection. To safely navigate the intersection, the ego-vehicle needs to understand the relation between the traffic lights on the right (shown in yellow) and vehicles on the left (shown in red), e.g. if the traffic light state is green then the vehicles on the left will move towards the right side of the intersection. If the ego-vehicle is unable to capture this relationship, it could lead to dangerous collisions. Therefore, in a dense urban setting with multiple vehicles and traffic lights, it is imperative for any vehicle to model the dependencies between different entities in the entire 3D scene, i.e., the ego-vehicle needs to capture the global context of the 3D scene.

The ego-vehicle perceives the environment through different sensors - cameras and LiDAR being the most popular ones. While a camera provides dense perceptual information of the scene, it lacks reliable 3D information and is highly susceptible to variation in weather conditions. On the other hand, LiDAR consists of 3D information but LiDAR measurements are typically very sparse (in particular at distance) and do not contain important information such as traffic light state. Hence, image-only and LiDAR-only methods are likely to fail in complex urban scenarios.

This limitation can be mitigated by integrating information from camera and LiDAR sensors since their benefits are complementary to each other. This leads to several research questions which we investigate in this work:

How to integrate information from multiple sensors?
To what extent should they be processed independently?
What kind of fusion mechanism to use for maximum gain?

Geometric Fusion

Prior works on sensor fusion are mostly focused on geometry-based fusion between camera and LiDAR sensors. In this method, points in 3D space (LiDAR point cloud) are projected to pixels in image space (camera input) and information is aggregated from the projected locations. In particular, the features (extracted using a convolutional neural network) corresponding to these projected locations are combined together. This is referred to as geometric feature projection in the above figure. This has been shown to be very effective on vision tasks such as object detection, motion forecasting and depth estimation but it has not been extensively explored in the context of end-to-end driving.

Limitation of Geometric Fusion

We observe that geometric fusion underperforms in complex urban scenarios involving dense traffic resulting in collision with other agents in the scene.

We hypothesize that this happens due to the lack of global context since features are aggregated from a local region in the projected 2D or 3D space. In the illustraion shown below, for the traffic light region in the image (shown in yellow), geometric fusion aggregates features from the blue region in LiDAR point cloud since these points project to the yellow region in image space. However, to safely navigate the intersection, it is essential to aggregate features from the red region in LiDAR point cloud since it overlaps with vehicles moving from left to the right.

Our Approach

Our key idea is to use attention-based feature fusion which helps the network to aggregate features from relevant regions in both the sensors. These relevant regions are spread across the entire input space rather than being restricted to just the projected locations (as in geometric fusion). This helps to capture the global context of the entire 3D scene. In particular, we use Transformers for this purpose - Multi-Modal Fusion Transformer (TransFuser).

We consider RGB image and LiDAR BEV (topdown view of the LiDAR point cloud) as inputs to our model. These are then processed by convolutional neural networks (in particular, ResNet modules) resulting in intermediate feature maps of different resolutions. We then use transformers to combine the image and LiDAR features at multiple resolutions. TransFuser outputs a feature vector which is then passed to a GRU-based autoregressive waypoint prediction network. The predicted waypoints are then fed to PID controllers which output the vehicle control.

Results

We conduct extensive experiments using CARLA in dense urban setting involving complex scenarios such as pedestrians emerging from occluded region to cross the road at random locations, other vehicles running red light at intersections and unprotected turnings.

Driving Performance in New Town

Driving Performance in New Weathers

Infraction Analysis

Our experiments indicate that TransFuser incurs fewer collisions and red light violations and can navigate difficult scenarios.

Further Information

To learn more about work, watch our video:

For more detailed information, check out our project page, paper and code. We are happy to receive your feedback!

@inproceedings{Prakash2021CVPR,
    title = {Multi-Modal Fusion Transformer for End-to-End Autonomous Driving},
    author = {Prakash, Aditya and Chitta, Kashyap and Geiger, Andreas},
    booktitle = {Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
    year = {2021}
}

GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields

2021-04-24T08:30:00+00:00

TL;DR: We incorporate a compositional 3D scene representation into the generator model such that at test time, we can control individual objects in the scene during image synthesis.

Introduction

Generative models are great! In the typical setting, we train our generator model on a large collection of images of some domain $X$, and at test time, we can draw new samples from our prior distribution $p_z$ and get new outputs in the same domain $X$:

Let’s imagine our domain $X$ are images of cars. Once training has finished, we can then generate millions of images of new cars! In particular convolutional neural network-based methods have led to impressive results for this task, e.g., the following cars are generated using the state-of-the-art StyleGAN2:

The Challenge

In practice, we often do only want to generate new images, but we would like to have control over the synthesis process. But how do we need to change the model to control the scene, i.e., how can we move the car to the right or the top?

It turns out that this problem is quite hard! One reason for this is that if we would like to easily move objects around, rotate them, and change the camera viewpoint, the model needs to perform some form of reasoning in 3D. Further, the model has to encapsulate individual objects in the scene to allow for controlling them individually. It remains unclear how we can achieve this when our model operates only in 2D.

Our Approach

Our main idea is to incorporate a compositional 3D scene representation into the generator model:

More specifically, in each forward pass we sample individual latent codes for the objects in the scene as well as the background. These give us individual feature fields in a canonical space. A feature field is a function which maps a 3D point and viewing direction to a density value and a feature vector. We then sample a pose for each object such that we can composite them into a single scene. Next, we sample a camera pose and volume render a feature image of the scene. A 2D neural renderer then upsamples the feature image and outputs the final RGB rendering:

Note that although we train our model from raw, unposed image collections, we are able to incorporate a 3D bias into our generator model. Further, we facilitate the decomposition of the scene into individual objects while no supervision is necessary!

Sounds cool, but does it actually work?

Here we show some results for moving the object around and rotating it when we train on a collection of images of cars:

Note how the background doesn’t change while we move the car around. Our model clearly disentangles fore- from background!

Similarly, we can also change the latent shape and appearance codes while we keep everything else fixed. Here we show a shape interpolation (top) as well as an appearance interpolation (bottom):

We can also interpolate between different backgrounds while keeping the car fixed:

Finally, we can even add more cars to the scene! Keep in mind that we trained our model on a dataset which contains only images of single cars:

Further Information

To learn more about GIRAFFE, watch our video here:

For more detailed information, check out the project page, the paper, and the supplementary. If you are interested in experimenting with GIRAFFE yourself, download the source code and run the demo!

@inproceedings{Niemeyer2020GIRAFFE,
    author    = {Michael Niemeyer and Andreas Geiger},  
    title     = {GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields},
    booktitle   = {Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
    year      = {2021},
}

How to evaluate tracking with the HOTA metrics

2021-03-02T08:30:00+00:00

How to evaluate tracking with the HOTA metrics

HOTA (Higher Order Tracking Accuracy) is a novel metric for evaluating multi-object tracking (MOT) performance. It is designed to overcome many of the limitations of previous metrics such as MOTA, IDF1 and Track mAP.

This short blog post gives an overview of the most important aspects of HOTA in three parts:

How to calculate the HOTA metrics.
How to compare trackers with the HOTA metrics.
How does HOTA compare to alternative tracking metrics.

For more details on HOTA, check out the IJCV 2020 paper and the metrics code on GitHub.

Part 1: How to calculate the HOTA metrics.

HOTA can be thought of as a combination of three IoU scores. It divides the task of evaluating tracking into three subtasks (detection, association and localization), and calculates a score for each using an IoU (intersection over union) formulation (also known as a Jaccard Index). It then combines these three IoU scores for each subtask into the final HOTA score.

Below we look at how the IoU score is calculated for each of these three subtasks.

Localization:

Localization measures the spatial alignment between one predicted detection and one ground-truth detection. Localization IoU (Loc-IoU) is typically used in many evaluation metrics to measure localization accuracy. It is calculated as the ratio of the overlap (intersection) between the two detections and the total area covered by both of them (union). This can be seen in the diagram below.

This concept trivially extends from bounding boxes to segmentation masks. As can be seen, when the Loc-IoU score here increases, the predicted and ground-truth detections are better spatially aligned and the localization is improved.

We can measure the overall Localization Accuracy (LocA) by averaging the Loc-IoU over all pairs of matching predicted and ground-truth detections in the whole dataset (we describe below how we obtain these matches):

Detection:

Detection measures the alignment between the set of all predicted detections and the set of all ground-truth detections. Detection IoU (Det-IoU) is also commonly used for measuring detection accuracy. Here we need to define which detections between the set of predicted and ground-truth detections are intersecting. To do this, we define a localization threshold (e.g. Loc-IoU > 0.5) above which we say that two detections intersect. However one predicted detection may overlap with more than one ground-truth detection (and vice-versa). To deal with this we run the Hungarian algorithm to determine a one-to-one matching between predicted and ground-truth detections. These matching pairs of detections are called True Positives (TP), and can be thought of as the intersection between the two sets of detections. Predicted detections that don’t match are False Positives (FP) and ground-truth detections that don’t match are False Negatives (FN). The detection IoU is then given by:

We can see that this has fundamentally the same structure as Loc-IoU. It is the intersecting area (the matches or TPs) divided by the total area (the union over all detections). Whereas Loc-IoU measures the alignment between a single predicted and ground-truth detection, the Det-IoU now measures the alignment between the set of all predicted detections and the set of all ground-truth detections. This set-based IoU formulation is also commonly called a Jaccard Index.

We can measure the overall Detection Accuracy (DetA) by simply calculating the Det-IoU using the count of TPs, FNs and FPs over the whole dataset:

Association:

Association measures how well a tracker links detections over time into the same identities (IDs), given the ground-truth set of identity links in the ground-truth tracks. We can measure this by taking a predicted detection and ground-truth detection that are matched together (using the Hungarian matching as explained above), and measuring the alignment between this predicted detection’s whole track and the ground-truth detection’s whole track. This alignment can again be represented in an IoU formulation.

The intersection between two tracks can be measured as the number of True Positive matches between the two tracks, we call these True Positive Associations (TPA). Any remaining detections in the predicted track (which are either matched to other ground-truth tracks or none at all) are False Positive Associations (FPA) and any remaining detections in the ground-truth track False Negative Associations (FNA). The Association IoU (Ass-IoU) can then be calculated in a similar way as seen previously:

This now measures the alignment between two tracks, which gives us a measure for every pair of matching detections (TPs) how good the association is for this detection.

We can see a visual example of the definitions TPA, FNA and FPA below:

The red square indicates the matched TP pair of a prediction and ground-truth detection, for which we want to find an association score. In order to measure how good the temporal association aligns between these detections, we find all the detections in these two tracks which match between these tracks (TPAs in green) and all the detections where they don’t match (FPAs in yellow and FNAs in brown).

We can measure the overall Association Accuracy (AssA) by averaging the Ass-IoU over all pairs of matching predicted and ground-truth detections in the whole dataset:

Building HOTA from three different IoU scores:

Naturally, all three components (localization, detection and association) are important for tracking success, so it’s important to measure all of them. However, we often want a single metric which we can use for ranking how well trackers perform overall. This metric is HOTA, which is a combination of all three IoU scores defined above:

Note that earlier, both the DetA and AssA were defined using a Hungarian matching based on a certain Loc-IoU threshold (α). Since the DetA and AssA score both depend on the Loc-IoU values, we calculate these score over a range of different α thresholds. For each threshold value, we calculate the final score as the geometric mean of the detection score and the association score. By then integrating over the different α thresholds, we include the localization accuracy into the final score.

The use of the geometric mean for combining detection and association ensures that both are evenly weighted in the final score, and that the score goes to 0 if either detection or association go to zero. Furthermore, this has the interpretation that the HOTA score can be thought of as a Det-IoU formulation where each of the TPs in the numerator are weighted by the Ass-IoU for this TP. E.g. an average of the Ass-IoU scores over the union of all detections.

Part 2: How to compare trackers with the HOTA metrics.

Armed with the family of HOTA metrics, we can now evaluate multi-object tracking in way that was not possible before. We can now not only see how good a tracker is, but also where it is good, which is crucial for understanding the underlying behavior of a tracker when both deciding on a tracker for an application, and when investigating how to improve current trackers.

As an example, let’s look at the results of the top 20 methods (as of when this was written) on the KITTI tracking leaderboard for pedestrians:

The rank (red number) gives the order of the methods, ordered by the overall HOTA score. However, now we can see how well methods perform in each of the dimensions of detection (x-axis) and association (y-axis) separately, with the curves in the background showing how the overall HOTA score increases as both detection and association scores increase.

The top 3 ranked trackers have a very similar overall HOTA scores (46.3%, 45.9% and 45.7%), but we can see from this plot that there is a clear difference in where one is better than the other. Tracker 1 is the best at association, while Tracker 2 is better at detection and Tracker 3 is in between the two for both. If you wanted to select a tracker for a particular application, you could now decide whether association or detection was more important for your application and pick the most appropriate tracker accordingly. In fact, each of these three trackers lie on the Pareto optimal front (red dashed line) such that for some level of trade-off between association and detection accuracy each is the best choice. With the HOTA metrics there is now not only a single best tracker at the top of the leaderboard, but rather a number of best trackers with different trade-offs along the Pareto optimal front (Hat tip to Jack Valmadre and Alex Bewley for the idea of plotting Pareto fronts).

If the author of Tracker 2 wishes to improve his tracker, these results now indicate that he can improve it by looking into how Tracker 1 performs association, while in contrast the author of Tracker 1 might want to look into how Tracker 2 (or better Tracker 4) performs detection.

We can go further than just comparing detection and association. Because each are designed using an IoU formulation, they are both naturally decomposable into one component that only measures recall, and one component that only measures precision. We can perform this decomposition and plot the results to get even further insight into the tracking results:

The above plots have tracker numbers ordered by the overall HOTA score still, so the same number refers to the same tracker as above. Detection recall (DetRe) measures how well a tracker finds all the ground-truth detections, whereas detection precision (DetPr) measures how well a tracker manages to not predict extra detections that aren’t there. From the first plot above, we can see that Tracker 1 and Tracker 3 have a similar detection accuracy overall, but Tracker 3 generally finds more of the ground-truth objects (higher recall), but also predicts more detections that are wrong (lower precision).

Precision and recall are commonly used for evaluating detection, but now with the HOTA metrics we can extend these concepts to also measure association. Association recall (AssRe) measures how well trackers can avoid splitting the same object into multiple shorter tracks. In contrast, association precision (AssPr) measures how well tracks can avoid merging multiple objects together into a single track. E.g. Tracker 15 is more likely to split tracks into multiple smaller ones than tracker 20, but it is better at not merging tracks together. Like detection precision and recall, there is a natural trade-off between association precision and recall when designing trackers.

The HOTA metrics allows meaningful analysis and comparison between trackers over all four of these dimensions (missing detections, extra detections, splitting tracks and merging tracks) while also combining all of these scores meaningfully into an overall score for ranking trackers.

Finally, HOTA also allows the analysis of localization accuracy:

In the first plot above we compare HOTA(0) (HOTA at the single lowest alpha threshold, so to not include the influence of localization accuracy, in this case at alpha=0.05) against the localization accuracy LocA(0) (LocA at the same threshold). We can see that Tracker 3 performs slightly better than Tracker 1 at HOTA(0), e.g. when allowing detections to match even if they have only a little overlap, Tracker 3 has overall better detection + association, however the localization of these matched detections is worse, such that when we calculate the final HOTA score by calculating over the range of localization thresholds, Tracker 1 has a higher score. This shows how HOTA is able to decompose and combine tracker behavior not only for detection and association but also for localization.

In the second plot, we compare the top 5 trackers’ HOTA scores across the range of different alpha thresholds. All trackers get worse when increasing the alpha threshold, but the rate at which they get worse is interesting and is useful for comparing behavior between trackers.

Part 3: How does HOTA compare to alternative tracking metrics.

Previously, three other main metrics have been used for evaluating multi-object tracking, these are MOTA, IDF1 and Track mAP. We won’t go into the details of each one here, but rather walk through a simple example which highlights the differences between HOTA and the previous metrics:

In this example, we have single ground-truth object, which is present in all frames of a 100 frame video. We then have three different trackers (A, B and C) with increasing detection accuracy and decreasing association accuracy. In this example, every predicted detection is a TP (e.g. matches with the ground-truth).

Which tracker is better out of A, B and C? This is not a well-defined question, because each tracker is very different, some are better at detection, others at associations, so it depends which of these are more important for your application. Since often both detection and association are very important, HOTA is designed to equally weight each of them in calculating an overall score. This can be seen in the above example. The detection score (DetA) increases, while the association score (AssA) decreases, such that for all three the combined HOTA score remains the same.

In comparison it can be seen that the MOTA score is heavily biased towards measuring detection at the expense of ignoring association. In contrast the IDF1 score is heavily biased towards measuring association, at the expense of ignoring detection. HOTA finds a perfect balance between these two extremes by equally weighting both detection and association, while allowing analysis of each component separately with the DetA and AssA sub-scores.

Finally the Track mAP metric is the least informative of all the metrics in our example because it requires predicted tracks to have more than 50% overlap with ground-truth tracks to count at all. Thus all three trackers in our example have a Track mAP score of 0, as if the tracker didn’t run at all.

Summary:

In this short blog post we’ve introduced the HOTA metrics, a new family of metrics for evaluating multi-object tracking. We’ve walked through how HOTA is calculated as the combination of three separate IoU scores for detection, association and localization respectively. We’ve seen how we can evaluate and analyze trackers along these different dimensions using different HOTA sub-metrics, and how each of these can be split into a recall and precision component for more fine-grained analysis. Finally we’ve briefly seen how these HOTA metrics compare with other metrics previously used for multi-object tracking evaluation, and highlighted some of the reason why using HOTA is often preferable.

For more details on HOTA, check out the IJCV 2020 paper (especially for detailed comparison and analysis between different metrics) and the metrics code on GitHub. This code can be used to evaluate your own trackers with HOTA, while also providing all of the analysis and plots in this article. Code is also present for other metrics for easy comparison, and it is easy to add new benchmarks and new metrics. HOTA metrics is currently rolling out to a number of different tracking benchmarks, check the GitHub readme for updates.

@article{luiten2020IJCV,
  title={HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking},
  author={Luiten, Jonathon and Osep, Aljosa and Dendorfer, Patrick and Torr, Philip and Geiger, Andreas and Leal-Taix{\'e}, Laura and Leibe, Bastian},
  journal={International Journal of Computer Vision},
  pages={1--31},
  year={2020},
  publisher={Springer}
}

Counterfactual Generative Networks

2021-02-09T08:00:00+00:00

What do you see in the following images?

If you answered Camel, Cow, Camel, and Cow, well done! In many instances, machine learning models are not that smart yet. If we collect a dataset for this Camel-Cow classification task, images in the dataset will typically depict cows on green pastures and camels in the desert. The most straightforward correlation a classifier can learn is the connection of the label “cow” to a grass-textured background and of the label “camel” to a sandy background. There are many more such examples, known as shortcuts, e.g., Clever Hans or the Tank Legend. Learning those shortcuts becomes problematic if the test data does not follow the training distribution - in our example, an image of a cow in the desert. In the causality literature, such a correlation is called spurious.

Causality also aims to answer counterfactual questions like “How would this image look like with a different background.” If we were able to answer this question, we could generate images of unseen combinations, i.e., cows in the desert, and supply them to our classifier. Training on this counterfactual data would increase robustness against spurious correlations.

Generative Models struggle too!

So, let’s generate counterfactual images! Unfortunately, it is not that easy with standard generative models like a VAE. Consider the simple example of colored MNIST, a digit-classification dataset, where each digit is strongly correlated with a specific color (left). If we train a VAE on this dataset, its latent space will look like on the right:

There are nice clusters of red zeroes, yello eights, etc. But there is not a single blue zero – the different factors of variation (FoV) in the data are not disentangled.

Building a Counterfactual Generative Network

How do we get direct control over these factors? And how can we scale our generative model beyond toy datasets so we generate realistic-looking images? The main idea is to structure the generative network into several subnetworks, so-called independent mechanisms. Each of them is responsible for one FoV. In the image below, there are three mechanisms, shape $\mathbf{m}$, foreground $\mathbf{f}$, and background $\mathbf{b}$. We composite their output into the final image $\mathbf{x_{gen}}$. During training, all mechanisms receive the same label, here “ ring-tailed lemur.” We refer to the whole model as Counterfactual Generative Network (CGN).

By choosing appropriate losses for the optimization, we can disentangle the signal, e.g., generating a background without any object. With the help of pre-trained models (BigGAN, U2-Net), we scale the CGN to ImageNet, a large dataset that remained out of reach for previous disentanglement methods.

Generating Counterfactuals

To generate counterfactual images, we can now sample a noise vector, generate an image, and ask: “How would this image look like if the object had a cheetah texture?” Answering this questions is as simple as supplying the cheetah label to the texture mechanisms to generate the cheetah texture. We have now full control over the FoVs for the generative process:

The counterfactual images with randomized labels can look quite funny and intriguing:

Training on these images results in the desired effect of less reliance on backgrounds or textures. Another cool find is that a CGN does interesting things during training, like generating high-quality binary object masks and unsupervised image inpainting.

From left to right: mask, foreground, background, and composite image over the course of training.

Fun & Further Information

In GAN literature it is common to visualize the latent space with a latent walk, i.e., sampling and interpolating between points in the latent space. Latent walks are way more interesting if we sync them to music! So, see below the Counterfactual Latent Dance, a musical video / latent walk with a CGN.

Another fun application is to use the generated textures for image stylization. We walk through the latent space of the texture mechanism and optimize the input image to follow the style of the generated texture:

Further information (including the paper and code is available on our project page. If you want to generate counterfactual images directly in your browser, you can try the Colab.

@inproceedings{Sauer2021ICLR,
  title = {Counterfactual Generative Networks},
  author = {Sauer, Axel and Geiger, Andreas},
  booktitle = {International Conference on Learning Representations (ICLR)},
  year = {2021}
}