Event camera is a kind of sensor that records signals when the scene exhibits illumination changes [6, 7]. An event camera reports signals (events) asynchronously when the log intensity change exceeds a preset threshold τ. We have witnessed the rise of event cameras due to their distinctive advantages over conventional active pixel cameras, e.g., higher frame rate, higher dynamic range and lower power consumption. These properties attracted the use of event cameras in many computer vision tasks, e.g., tracking [1, 10], deblurring [2, 11], optical flow estimation [3, 12], SLAM [4, 13, 14], video frame interpolation [5, 11]. However, the unique data structure of event cameras renders the existing computer vision tools and algorithms unusable, which places a major obstacle against the application of event cameras. Many works have been focusing on bridging events and conventional cameras by reconstructing intensity frames from events [15–20], thus allowing modern vision algorithm to take place. Rebecq et al. [8] proposed E2VID, a recurrent network to reconstruct videos from a stream of events and trained it on a large amount of simulated event data. Scheerlinck et al. [9] proposed FireNet, which simplified the neural architecture in Ref. [8] with a smaller number of parameters while maintaining similar quantitative results. Although many studies have attempted to reconstruct the intensity image from the event, none of these methods can recover the intensity and color information well. Therefore, the absence of color information in the reconstructed image degrades the performance for downstream tasks. In our application, the color mismatch makes the existing self-supervised stereo matching algorithm based on photometric consistency invalid.

Stereo matching is the process of linking the pixels in different views that correspond to the same point of the scene. It follows a long line of research works. Early works involve searching and matching corresponding pixels on the epipolar line [21, 22]. Recently, deep learning based methods have dominated the field of stereo matching due to their superior performance and usability. Zbontar and LeCun [23] are among the first to use a convolutional network for computing stereo matching cost in image pairs. Following this, a number of studies were proposed to improve the performance, e.g., inner product layer [24], encoder-decoder architecture [25], 3D convolution cost-volume module [26], spatial pyramid pooling and 3D hourglass convolution [27], guided attention cost-column [28], PatchMatch module for sparse cost volume representation [29], intra-scaling cost aggregation [30]. With the development of various sensors, multi-model and cross-spectral stereo matching has become an emerging topic [31–36]. But none of them is suitable for calculating the correspondence between intensity images and events or event reconstruction images. Concurrent with our work, Mostafavi et al. [37] investigated stereo matching with event-intensity cameras on both views and proposed an event-intensity network that refines image details using events. Our work is essentially different in purpose and method; we use only one intensity and one event camera and train our model self-supervised.

Learning-based stereo methods are data-hungry. They often require a lot of ground truth data for training. Over the past few years, self-supervised models have been developed to learn stereo matching without ground truth annotations. They are usually built on the principles of disparity smoothness prior and re-projection photometric consistency constraints. Garg et al. [38] tackled monocular depth estimation by minimizing the loss between the source image and backwards-warping from the subsidiary stereo image. Similarly, Godard et al. [39] included a left-right consistency to enforce disparity prediction. They further proposed a new minimum re-projection loss and auto-masking loss to improve the performance [40]. Zhou et al. [41] adopted left-right check to guide the training and pick suitable matching as training data. Zhi et al. [32] proposed a self-supervised learning framework for cross-spectral stereo matching. They introduced a material-aware loss function to handle regions with unreliable matching. However, their method involves the translation between intensity and near-infrared images and is thus unsuitable for our setting.

The multi-modal intensity-event stereo matching problem is first formulated as follows. As shown in Fig. 1, an intensity camera and an auxiliary horizontally displaced event camera are used in our setting. In this work, we assume that the camera on the left is the image camera, and the one on the right is the event camera. Let Il be the left view intensity image and {Emr}m∈N be the event stream obtained by the right event camera within a short amount of time before the intensity image is captured. The underlying problem can be considered a data association problem, that is, to find correspondences between the points in the left image and right events. The correspondences are formed as the final disparity map, which is also the output of the stereo matching problem.

However, the right signal (event) is not in the same modality as the left signal (intensity images), failing the existing methods with the given problem setting. A reconstruction network is employed at first to convert the event stream {Emr}m∈N to a roughly reconstructed image Ir. In our work, we use two popular event reconstruction models, E2VID [8, 42] and FireNet [9]. Note that these models are replaceable. Given these two images, we can adapt the existing image stereo models to predict disparity between images, which is also the disparity between the left image and the right events. However, the event camera does not record the value of the pixels and only records the pixels changing. Thus the rough reconstruction Ir only contains usable edge information but are unreliable in color and detail and still in different modalities with Il (see the reconstruction result in Figure 3).

As stated above, we need a multi-modal stereo network to handle inputs with different modalities (either the right view is event voxel or reconstructed images). Although we can apply the previous convolutional stereo matching networks theoretically, the difference in modalities still poses challenges. Most stereo networks are composed of feature extraction, correlation and aggregation sub-models, and the feature extraction model usually share weights for both two views. This weight sharing strategy is effective originally but poses limitations for images with different modalities. We make the minor changes to these networks to make images with different modalities using modality-specific feature extraction sub-modules. This design has two advantages. First, the feature extraction models dedicated to different modalities avoid confusion between different images. Second, the new design allows us to swap the modalities of the left and right views and predict disparity of the other view, as long as we swap them together with the feature extraction branches. The second property is essential for the cross-consistency loss, which we will describe in Section 3.3. Note that this structure also allows the event voxel to be used directly as input.

To achieve the goal of self-supervised learning, we define the loss function as the combination of the following four parts: where the λs denote the hyper-parameters that control the loss weights. Next, we describe each component of the loss function.

In this section, we experimentally evaluate the stereo matching performance of the proposed method. We use both synthetic and real data in our experiments. For the experiments based on synthetic data, we employ the Stereo Blur Dataset proposed by Zhou et al. [46], which contains 20,637 blurry-sharp stereo image pairs from 126 diverse sequences and their corresponding bidirectional disparities. To reliably synthesize events, we first increase the sequence frame rate from 60 fps to 2,400 fps via a high-quality frame interpolation algorithm [47] and then applying the V2E event simulator [48] to the high frame rate sequences. We use the officially split method, where 89 sequences are used for self-supervised training, and 37 sequences are used for testing. For real sensor data, we use the MVSEC [49] dataset, which contains the stereo intensity images and events captured by DAVIS 240C. MVSEC also provides ground truth depth captured by LiDAR. We use the official split method for the MVSEC dataset. Our method is implemented using Pytorch [50] framework and trained using NVIDIA V100 GPUs. For the stereo network design, we build our multi-modal networks by modifying DeepPruner [29] and AANet [30] according to Sec. 3.2. Note that our framework is compatible with the most alternative architectures of the reconstruction and the stereo networks. For optimization, we use Adam [51] with β1 = 0.9, β2 = 0.999 and learning rate 1 × 10−4. We set the weighting of the different loss components to λgd = 1, λsm = 0.1, λcc = 0.025 and λitn = 0.005. The settings of ρ and t are experimental. The final value of ρ is 1 on the synthetic dataset and 1.5 on the real world data. The value of t is 2 for all datasets. The overall self-supervised training costs about 2 days.

We first quantitatively demonstrate the effectiveness of the proposed method in Tables I and II. The metrics are averaged end-point error (EPE) and >1-pixel, >3-pixel and >5-pixel error. Under the proposed setting, only limited methods can be used to obtain disparity maps as there is no multi-modal stereo matching model for event-intensity setting. We included the monocular depth model [40] for comparison, which was not fine-tuned on the target data, as we cannot obtain intensity images from both left and right views at the same time in this setting theoretically. We also consider the monocular event depth model [52], but the advantage of the event model is to combat high-speed motion and low-light situations that the intensity camera cannot handle, and its effect cannot be compared with the results predicted using the intensity images. We can have the following observations. Firstly, the proposed method can achieve much better results compared with monocular disparity estimation results using monodepthv2 [40] that is only based on the left image. It indicates that the information of another view plays an essential role in stereo matching. The unsatisfactory effect prevents us from using this disparity map to align events with the intensity image. We then show the results of directly performing stereo matching using the reconstructed right image and left image (marked as the “baseline experiments”). As can be seen, since the reconstructed images have a huge color difference against the left view image, it isn’t easy to obtain a good result by employing the existing stereo vision models. The visualization results in Figure 5 also speak to similar conclusion. However, introducing the modified stereo network and self-supervised learning using only gradient structure loss can improve the stereo estimation results. Even if all the loss functions are used, the network architecture is still a significant obstacle to improving performance. Third, the proposed self-supervised learning method unleashes the full potential of the overall framework’s effect, which proves the advantage of our self-supervised learning strategy, that is, learning from unlabeled data. We visualize the results of the proposed method in Fig. 5 and Figure 6. As one can observe, we can only get poor matching results in these scenarios based on the pre-trained stereo network. Our full model produces accurate object boundaries and better preserves the overall scene structures. We also provide the upper bound performance obtained by a fine-tuned DeepPruner network using both sides’ intensity images as a reference. It can be seen that the information lost by the event is detrimental to the final matching result. But the purpose of our method is not to rely on events to obtain better matching results but to make it possible to calculate reasonable disparity under the proposed intensity-event setting.

To study the effects of each component in the proposed method, we conduct several ablation studies. All the experiments are conducted using FireNet reconstruction and a modified AANet stereo model. We first examine the use of gradient structure loss function. We train our model using the four alternative loss functions described in Section 3.3 and their convergence curves are shown in the left figure of Figure 7. As can be seen, all pixel-wise loss functions fail to converge. They can only provide very limited information, and continuous optimization of these losses will bring adverse effects and make training fail. The structure loss directly calculated on images performs well initially, but it could no longer provide adequate supervision as the training progressed. The proposed gradient structural loss can produce a relatively reliable convergence curve, although it is still tricky to improve performance with training steadily. We next involve the proposed cross-consistency loss and internal disparity loss in training. The results are shown in right figure of Fig. 7. It is surprising that although it cannot be compared with competitors initially, the proposed losses make the training process more stable. The proposed method can steadily and continuously improve performance without any ground truth data. We visually explain why the proposed cross-consistency loss is so effective. Figure 8 shows some comparisons between models trained with and without the cross-consistency loss. The results speak to the fact that the inclusion of this loss improves the quality of the result. This loss term helps outline the precise edge and generate sharp, accurate shapes in the disparity map.

We next demonstrate the use of a disparity map to establish a connection between the intensity camera and the event camera. Due to the displacement between these two cameras, even if we can obtain the left view intensity image and the right view events, the disparity between them makes it difficult to make full use of the advantages of the two sensors, as shown in Figure 9(b). In this case, many algorithms and applications that require alignment of events and images cannot be implemented, e.g., [11, 53]. We first obtain the disparity map using the proposed multi-modal stereo method. Each value in the disparity map indicates the number of pixels that need to be shifted horizontally. We warp events by changing the x coordinate in each event tuple (xm, ym, tm, pm), where xm, ym, tm denote the spatial-temporal coordinates, and pm ∈{− 1, +1} denotes the polarity of the event. The warped event are visualized in Fig. 9(c). It can be seen that the warped right view events and the left view image are well aligned both spatially and temporally. Obtaining such a connection between two sensors allows many downstream tasks. Here we show the application potential by the event-based video interpolation task. Motivated by the physical model of event that the residuals between a blurry image and sharp frames are the integrals of events, Lin et al. [11] propose to estimate the residuals for the sharp frame restoration based on events. Our reconstruction result shows good fidelity performance, which further proves the application value of the proposed problem setting.

Finally, we show some failure cases and analyze the potential limitations. We show two failure cases in Figure 10 and both of them are from MVSEC. The direct reconstruction using E2VID contains only very limited information, resulting in the failure of stereo matching. This shows a possible flaw of the proposed method, that is, it still faces significant challenges when reconstruction results are very vague. A possible solution is to design a stereo network to perform stereo matching between event streams and intensity images directly. In that case, the loss functions described in this paper can still provide good self-supervised learning results.