Publicité

30 Apr 2021•0 j'aime## 0 j'aime

•119 vues## vues

Soyez le premier à aimer ceci

afficher plus

Nombre de vues

0

Sur Slideshare

0

À partir des intégrations

0

Nombre d'intégrations

0

Télécharger pour lire hors ligne

Signaler

Ingénierie

3D object detection, monocular camera, autonomous driving

Yu HuangSuivre

Chief Scientist, Global AI Technical Officer, Autonomous Driving à AI startupPublicité

Publicité

Publicité

3-d interpretation from single 2-d image IVYu Huang

Fisheye Omnidirectional View in Autonomous DrivingYu Huang

3-d interpretation from single 2-d image for autonomous driving IIYu Huang

3-d interpretation from single 2-d image for autonomous drivingYu Huang

Pose estimation from RGB images by deep learningYu Huang

LiDAR-based Autonomous Driving III (by Deep Learning)Yu Huang

- 3D Interpretation from Single 2D Image for Autonomous Driving III Yu Huang Yu.huang07@gmail.com Sunnyvale, California
- Outline • Towards Generalization Across Depth for Monocular 3D Object Detection • RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving • Monocular 3D Object Detection with Decoupled Structured Polygon Estimation and Height-Guided Depth Estimation • Exploring the Capabilities and Limits of 3D Monocular Object Detection - A Study on Simulation and Real World Data • Object-Aware Centroid Voting for Monocular 3D Object Detection • Monocular 3D Detection with Geometric Constraints Embedding and Semi- supervised Training • Learning Monocular 3D Vehicle Detection without 3D Bounding Box Labels
- Towards Generalization Across Depth for Monocular 3D Object Detection • This work advances the state of the art by introducing MoVi-3D, a single- stage deep architecture for monocular 3D object detection. • MoVi-3D builds upon an approach which leverages geometrical information to generate, both at training and test time, virtual views where the object appearance is normalized with respect to distance. • These virtually generated views facilitate the detection task as they significantly reduce the visual appearance variability associated to objects placed at different distances from the camera. • As a consequence, the deep model is relieved from learning depth-specific representations and its complexity can be significantly reduced. • In particular, in this work thanks to virtual views generation process, a lightweight, single-stage architecture suffices to set new state-of-the-art results on the popular KITTI3D benchmark.
- Towards Generalization Across Depth for Monocular 3D Object Detection Aim at predicting a 3D bounding box for each object given a single image (left). In this image, the scale of an object heavily depends on its distance with respect to the camera. For this reason the complexity of the detection increases as the distance grows. Instead of performing the detection on the original image, perform it on virtual images (middle). Each virtual image presents a cropped and and scaled version of the original image that preserves the scale of objects as if the image was taken at a different, given depth.
- Towards Generalization Across Depth for Monocular 3D Object Detection Illustration of the Monocular 3D Object Detection task. Given an input image (left), the model predicts a 3D box for each object (middle). Each box has its 3D dimensions s = (W;H;L), 3D center c = (x; y; z) and rotation (alpha).
- Towards Generalization Across Depth for Monocular 3D Object Detection • The goal is to devise a training/inference procedure that enables generalization across depth, by indirectly forcing the models to develop representations for objects that are less dependent on their actual depth in the scene. • The idea is to feed the model with transformed images that have been put into a canonical form that depends on some query depth. • After this transformation, no matter where the car is in space, obtain an image of the car that is consistent in terms of the scale of the object. • Clearly, depth still influences the appearance, e.g. due to perspective deformations, but by removing the scale factor from the nuisance variables，able to simplify the task that has to be solved by the model. • In order to apply the proposed transformation，need to know the location of the 3D objects in advance.
- Towards Generalization Across Depth for Monocular 3D Object Detection 3D viewport Compute the top-left and bottom-right corners of the viewport, namely (Xv，Yv，Zv) and (Xv + Wv，Yv – Hv，Zv) respectively, and project them to the image plane of the camera, yielding the top-left and bottom-right corners of a 2D viewport. Crop it and rescale it to the desired resolution wv x hv to get the final output. It is a virtual image generated by the given 3D viewport.
- Towards Generalization Across Depth for Monocular 3D Object Detection • The goal of the training procedure is to build a network that is able to make correct predictions within a limited depth range given an image generated from a 3D viewport. • A ground-truth-guided sampling procedure：repeatedly draw (without replacement) a ground-truth object and then sample a 3D viewport in a neighborhood thereof so that the object is completely visible in the virtual image. • The location of the 3D viewport is perturbed with respect to the position of the target ground-truth object in order to obtain a model that is robust to depth ranges up to the predefined depth resolution Zres, which in turn plays an important role at inference time. • In addition, let a small share of the virtual images to be generated by 3D viewports randomly positioned in a way that the corresponding virtual image is completely contained in the original image. • A class-uniform sampling strategy：allows to get an even number of virtual images for each class that is present in the original image.
- Towards Generalization Across Depth for Monocular 3D Object Detection Training virtual image creation. We randomly sample a target object (dark-red car). Given the input image, object position and camera parameters, compute a 3D viewport that we place at z = Zv. Then project the 3D viewport onto the image plane, resulting in a 2D viewport. Finally crop the corresponding region and rescale it to obtain the target virtual view (right).
- Towards Generalization Across Depth for Monocular 3D Object Detection • Since have trained the network to be able to predict at distances that are twice the depth step, reasonably confident not missing objects, in the sense that each object will be covered by at least a virtual image. • Also, due to the convolutional nature of the architecture adjust the width of the virtual image in a way to cover the entire extent of the input image. • By doing so have virtual images that become wider as increasing the depth, following the rule （W is the width of the input image）： • Finally perform NMS over detections that have been generated from the same virtual image.
- Towards Generalization Across Depth for Monocular 3D Object Detection Inference pipeline. Given the input image, camera parameters and Zres，create a series of 3D viewports placing every Zres/2 meters along the Z axis. Then project these viewports onto the image, crop and rescale the resulting regions to obtain distance-specific virtual views. Finally use these views to perform the 3D detection.
- Towards Generalization Across Depth for Monocular 3D Object Detection It consists of two parallel branches, the top one devoted to providing confidences about the predicted 2D and 3D bounding boxes, while the bottom one is devoted to regressing the actual bounding boxes. White rectangles denote 33 convolutions with 128 output channels followed by iABNsync.
- Towards Generalization Across Depth for Monocular 3D Object Detection
- Towards Generalization Across Depth for Monocular 3D Object Detection
- RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving • It proposes an efficient and accurate monocular 3D detection framework in single shot. • This method predicts the nine perspective keypoints of a 3D bounding box in image space, and then utilize the geometric relationship of 3D and 2D perspectives to recover the dimension, location, and orientation in 3D space. • In this method, the properties of the object can be predicted stably even when the estimation of keypoints is very noisy, which enables us to obtain fast detection speed with a small architecture. • Training uses the 3D properties of the object without the need for external networks or supervision data. • This method is the first real-time system for monocular image 3D detection while achieves state-of the-art performance on the KITTI benchmark. • Code will be released at https://github.com/Banconxuan/RTM3D.
- RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving Overview of proposed method: first predict ordinal keypoints projected in the image space by eight vertexes and a central point of a 3D object. then reformulate the estimation of the 3D bounding box as the problem of minimizing the energy function by using geometric constraints of perspective projection.
- RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving An overview of proposed keypoint detection architecture: It takes only the RGB images as the input and outputs main center heatmap, vertexes heatmap, and vertexes coordinate as the base module to estimate 3D bounding box. It can also predict other alternative priors to further improve the performance of 3D detection.
- RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving Illustration of keypoint feature pyramid network (KFPN).
- RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving
- RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving
- Monocular 3D Object Detection with Decoupled Structured Polygon Estimation and Height-Guided Depth Estimation • Since the location recovery in 3D space is quite difficult on account of absence of depth information, this work proposes a unified framework which decomposes the detection problem into a structured polygon prediction task and a depth recovery task. • Different from the widely studied 2D bounding boxes, the proposed structured polygon in the 2D image consists of several projected surfaces of the target object as better representation for 3D detection. • In order to inversely project the predicted 2D structured polygon to a cuboid in the 3D physical world, the following depth recovery task uses the object height prior to complete the inverse projection transformation with the given camera projection matrix. • Moreover, a fine-grained 3D box refinement scheme is proposed to further rectify the 3D detection results.
- Monocular 3D Object Detection with Decoupled Structured Polygon Estimation and Height-Guided Depth Estimation The overall framework (Decoupled-3D) decouples the monocular 3D object detection problem into sub-tasks. The overall network consists of three parts. (Top row) The 2D structured polygons are generated with a stacked hourglass network. (Middle row) Object depth stage utilizes 3D object height as a prior to recover the missing depth of the object. (Bottom row) 3D box refine stage rectifies coarse 3D boxes using bird’s eye view features in 3D-ROIs.
- Monocular 3D Object Detection with Decoupled Structured Polygon Estimation and Height-Guided Depth Estimation Structured polygon estimation aims to estimate the 2D locations of the projected vertices
- Monocular 3D Object Detection with Decoupled Structured Polygon Estimation and Height-Guided Depth Estimation Height-Guided Depth Estimation. Combine object height H and corresponding pixel value h to estimate object depth
- Monocular 3D Object Detection with Decoupled Structured Polygon Estimation and Height-Guided Depth Estimation 3D Box Refinement. Rectify coarse boxes with bird’s eye view map Note： Depth Net DOR（“Deep Ordinal Regression Network for Monocular Depth Estimation”）
- Monocular 3D Object Detection with Decoupled Structured Polygon Estimation and Height-Guided Depth Estimation
- Monocular 3D Object Detection with Decoupled Structured Polygon Estimation and Height-Guided Depth Estimation
- Exploring the Capabilities and Limits of 3D Monocular Object Detection - A Study on Simulation and Real World Data • Recent deep learning methods show promising results to recover depth information from single images by learning priors about the environment. • In addition to the network design, the major difference of these competing approaches lies in using a supervised or self-supervised optimization loss function, which require different data and ground truth information. • This paper evaluate the performance of a 3D object detection pipeline which is parameterizable with different depth estimation configurations. • It implement a simple distance calculation approach based on camera intrinsics and 2D bounding box size, a self-supervised, and a supervised learning approach for depth estimation. • It evaluate the detection pipeline on simulator data and a real world sequence from an autonomous vehicle on a race track. • Advantages and drawbacks of the different depth estimation strategies are discussed
- Exploring the Capabilities and Limits of 3D Monocular Object Detection - A Study on Simulation and Real World Data 3D object detection pipeline with three alternative configurations
- Exploring the Capabilities and Limits of 3D Monocular Object Detection - A Study on Simulation and Real World Data • Distance calculation using the 2D bounding box height, and the known height of the real world race car as a geometric constraint. “known height assumption” • Depth estimation for the whole image using the supervised DenseDepth network. The distance to each object is calculated as the median depth value in the bounding box crop. Explicit knowledge about the objects, like height information, is not required in this approach. • Depth estimation for the whole image using the self-supervised struct2depth network. The distance to each object is calculated as the median depth value in the bounding box crop. Explicit knowledge about the objects, like height information, is not required in this approach.
- Exploring the Capabilities and Limits of 3D Monocular Object Detection - A Study on Simulation and Real World Data
- Object-Aware Centroid Voting for Monocular 3D Object Detection • This paper propose an end-to-end trainable monocular 3D object detector without learning the dense depth. • Specifically, the grid coordinates of a 2D box are first projected back to 3D space with the pinhole model as 3D centroids proposals. • Then, a object-aware voting approach is introduced, which considers both the region-wise appearance attention and the geometric projection distribution, to vote the 3D centroid proposals for 3D object localization. • With the late fusion and the predicted 3D orientation and dimension, the 3D bounding boxes of objects can be detected from a single RGB image. • The method is straightforward yet significantly superior to other monocular-based methods.
- Object-Aware Centroid Voting for Monocular 3D Object Detection 3D Object Detection Pipeline. Given an image with predicted 2D region proposals (yellow box), the regions are divided into grids. Each grid point with (u;v) coordinate is projected back to 3D space by leveraging the pinhole model and the class-specific 3D height H, resulting in 3D box centroid proposals. With the voting method inspired by both appearance and geometric cues, 3D object location is predicted.
- Object-Aware Centroid Voting for Monocular 3D Object Detection The Architecture. 2D region proposals are first obtained from the RPN module. Then, with the 3D Center Reasoning (left), multiple 3D centroid proposals are estimated from the 2D RoI grid coordinates. Followed by the Object-Aware Voting (right), which consists of geometric projection distribution (GPD) and appearance attention map (AAM), the 3D centroid proposals are voted for 3D localization. For the 3D dimension and orientation, they are estimated together with 2D object detection head.
- Object-Aware Centroid Voting for Monocular 3D Object Detection • For the objects on driving road, they are horizontally placed without the pose angles of yaw and pitch with respect to the camera. • Besides, the 3D dimension variance of each class of objects (such as Car) is quite small. • These constraints lead to the idea that the apparent heights of objects on image are approximately invariant when objects are in the same depth. • Recent survey also points out that the positions and apparent size of object in an image are applicable to infer the depth on KITTI dataset. • Therefore, the 3D object centroid can be roughly inferred with the simple pinhole camera model.
- Object-Aware Centroid Voting for Monocular 3D Object Detection • Specifically, divide each 2D region proposals into s X s grid cells and project the grid coordinates back onto 3D space. • Since each grid point indicates the probable projection of the corresponding 3D object centroid, get multiple 3D centroid proposals P3d where the i-th centroid proposal P3d(Xi;Yi;Z) is computed by Examples and statistics on KITTI training set.
- Object-Aware Centroid Voting for Monocular 3D Object Detection • Specifically, use a single 1X1 convolution followed by sigmoid activation to generate appearance attention map from the feature maps of RoI pooling layer. • The activated convolution feature map from the image indicates the foreground semantic objects due to the classification supervision in 2D object detection, leading to the object- ware voting. • This voting component comes from the distribution of the offset between the projected 3D centroid and the 2D box center. • It is demonstrated the 2D box center can be modeled as Gaussian distribution with ground truth as expectation • To dynamically learn the distribution, the 2D grid coordinates and image features of RoI are concatenated together as input of a fully-connected layer to predict the offset with Kullback Leibler (KL) divergence as loss function to supervise the learning
- Object-Aware Centroid Voting for Monocular 3D Object Detection The object-aware voting can be formulated as the element-wise multiplication with both the normalized probability maps Mapp and Mgeo as follows In the training stage, the 3D localization pipeline is trained with smooth L1 loss
- Object-Aware Centroid Voting for Monocular 3D Object Detection 3D dimension prediction loss function comparing predictions and the ground truth are defined in the logarithm space through the smooth L1 loss In 3D orientation estimation, use Multi-Bin to disentangle it into residual angle prediction and angle bins classification. 3D orientation estimation loss is formed as Loss functions for the joined training of multi-tasks of 2D and 3D object detection as
- Object-Aware Centroid Voting for Monocular 3D Object Detection
- Object-Aware Centroid Voting for Monocular 3D Object Detection
- Object-Aware Centroid Voting for Monocular 3D Object Detection Qualitative results. Red: detected 3D boxes. Yellow: ground truth. Right: birds’ eye view (BEV) results.
- Monocular 3D Detection with Geometric Constraints Embedding and Semi-supervised Training • A single-shot and keypoints-based framework for mono 3D objects detection, KM3D-Net. • Here, design a fully convolutional model to predict object keypoints, dimension, and orientation, and then combine with perspective geometry constraints to compute position. • Further, reformulate the geometric constraints as a differentiable version and embed it into the network while maintaining the consistency of model outputs in an E2E fashion. • Then propose a semi-supervised training strategy where labeled training data is scarce. • In this strategy, enforce a consensus prediction of two shared-weights KM3D-Net for the same unlabeled image under different input augmentation conditions and network regularization. • In particular, unify the coordinate-dependent augmentations as the affine transformation for the differential recovering position and propose a keypoints-dropout module for the network regularization. • This model only requires RGB images without synthetic data, instance segmentation, CAD model, or depth generator.
- Monocular 3D Detection with Geometric Constraints Embedding and Semi-supervised Training Overview of KM3D-Net which output keypoints, object dimensions, local orientation, and 3D confidence, followed by differentiable geometric consistency constraints to predict position.
- Monocular 3D Detection with Geometric Constraints Embedding and Semi-supervised Training Overview of unsupervised training. It leverages affine transformation to unify input augmentation and devise keypoints dropout for regularization. These two strategies make KM3D-Net output two stochastic variables with the same input. Penalizing their differences is training goal.
- Monocular 3D Detection with Geometric Constraints Embedding and Semi-supervised Training
- Monocular 3D Detection with Geometric Constraints Embedding and Semi-supervised Training
- Learning Monocular 3D Vehicle Detection without 3D Bounding Box Labels • The training of deep-learning-based 3D object detectors requires large datasets with 3D bounding box labels for supervision that have to be generated by hand-labeling. • A network architecture and training procedure for learning monocular 3D object detection without 3D bounding box labels. • By representing the objects as triangular meshes and employing differentiable shape rendering, define loss functions based on depth maps, segmentation masks, and ego- and object-motion, which are generated by pre-trained, o-the-shelf networks. • In comparison to SOA methods requiring 3D bounding box labels for training and superior performance to conventional baseline methods.
- Learning Monocular 3D Vehicle Detection without 3D Bounding Box Labels A mono 3D vehicle detector that requires no 3D bounding box labels for training. The right image shows that the predicted vehicles (colored shapes) fit the GT bounding boxes (red). Despite the noisy input depth (lower left ), the method is able to accurately predict the 3D poses of vehicles due to the proposed fully differentiable training scheme. It show the projections of the predicted bounding boxes (colored boxes, upper left).
- Learning Monocular 3D Vehicle Detection without 3D Bounding Box Labels The proposed model contains a single-image network and a multi-image network extension. The single- image network back-projects the input depth map into a point cloud. A Frustum PointNet encoder predicts the pose and shape then decoded into a predicted 3D mesh and segmentation mask through differentiable rendering. The multi-image network architecture takes three images as the inputs, and the single-image network is applied individually to each image. This network predicts a depth map for the middle frame based on the vehicle‘s pose and shape. A pre-trained network predicts ego-motion and object-motion from the images. The reconstruction loss is computed by differentiably warping the images into the middle frame.
- Learning Monocular 3D Vehicle Detection without 3D Bounding Box Labels In order to train without 3D bounding box labels we use three losses, the segmentation loss Lseg, the chamfer distance Lcd, and the photometric reconstruction loss Lrec. The first two are defined for single images and the photometric reconstruction loss relies on temporal photo-consistency for three consecutive frames
- Learning Monocular 3D Vehicle Detection without 3D Bounding Box Labels Qualitative comparison of MonoGRNet (1st row), Mono3D(2nd row), and this method (third row) with depth maps from BTS. It show GT bounding boxes for cars (red), predicted bounding boxes (green), and the back- projected point cloud. In comparison to Mono3D, the prediction accuracy is increased specifically for further away vehicles. The performance of MonoGRNet and this model is comparable.
- Learning Monocular 3D Vehicle Detection without 3D Bounding Box Labels

Publicité