2019/08/20 Azure Kinect DK Meetup Vol.00 https://tmcn.connpass.com/event/143242/

1. Azure Kinect DK C/C++開発概要(仮)

2. Self Introduction
杉浦 司 (Tsukasa Sugiura)
- Freelance Programmer
- Microsoft MVP for Windows Development
- Point Cloud Library Maintainer
- @UnaNancyOwen

3. Azure Kinect DK Sensor

4. Azure Kinect SDK Introduction
Azure Kinect Sensor SDK
(k4a / k4arecord)
Azure Kinect Body Tracking SDK
(k4abt)
- Color, Depth, Infrared, IMU, Point Cloud
- Open Source Library hosted on GitHub
(Depth Engine is Closed Source)
- Cross Platform (Windows, Linux)
- C API, C++ and C# Wrapper
- Body Index Map, Skeleton (26 Joints/Person)
- Deep Learning based Pose Estimation
- Closed Source Library
- Cross Platform (Windows, Linux)
- C API, (C++ Wrapper)

5. How to Install Azure Kinect SDK?
Azure Kinect Sensor SDK
- Install Pre-Built SDK using Installer
- Build and Install SDK from Source Code
Azure Kinect Body Tracking SDK
- Install SDK using Installer
- Install NVIDIA GPU Driver and Visual C++ 2015 Runtime
for ONNX Runtime (CUDA Backend)
Download Azure Kinect Body Tracking SDK | Microsoft Docs
https://docs.microsoft.com/en-us/azure/Kinect-dk/body-sdk-download
About Azure Kinect Sensor SDK | Microsoft Docs
https://docs.microsoft.com/en-us/azure/Kinect-dk/about-sensor-sdk

6. How to Generate Project with Azure Kinect SDK?
cmake_minimum_required( VERSION 3.6 )
project( Solution )
add_executable( Project main.cpp )
# Azure Kinect Sensor SDK (Official Support)
find_package( k4a REQUIRED )
find_package( k4arecord REQUIRED )
# Azure Kinect Body Tracking SDK (Un-Official Support)
set( CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}" )
find_package( k4abt REQUIRED )
if( k4a_FOUND AND k4arecord_FOUND AND k4abt_FOUND )
target_link_libraries( Project k4a::k4a )
target_link_libraries( Project k4a::k4arecord )
target_link_libraries( Project k4a::k4abt )
endif()
CMake Module for Azure Kinect Sensor SDK | GitHub Gists
https://gist.github.com/UnaNancyOwen/5ce9115ba8b71d12982e5aa9f99788f1#file-findk4abt-cmake
Include CMake and MS Build files in the MSI #370 | GitHub/Azure-Kinect-Sensor-SDK
https://github.com/microsoft/Azure-Kinect-Sensor-SDK/issues/370

7. Azure Kinect Sensor SDK Overview
Capture Image
Playback
Device

8. Azure Kinect Sensor SDK Programming
// Azure Kinect Sensor SDK
#include <k4a/k4a.h>
#include <k4a/k4a.hpp> /* C++ Wrapper */
// Get Connected Devices
const int32_t device_count = k4a::device::get_installed_count();
if( device_count == 0 ){
throw k4a::error( "Failed to found device!" );
}
// Open Default Device
k4a::device device = k4a::device::open( K4A_DEVICE_DEFAULT );
// Start Cameras with Configuration
k4a_device_configuration_t configuration = K4A_DEVICE_CONFIG_INIT_DISABLE_ALL;
configuration.color_format = K4A_IMAGE_FORMAT_COLOR_BGRA32;
configuration.color_resolution = K4A_COLOR_RESOLUTION_1080P;
configuration.depth_mode = K4A_DEPTH_MODE_NFOV_UNBINNED;
configuration.wired_sync_mode = K4A_WIRED_SYNC_MODE_STANDALONE;
configuration.synchronized_images_only = true;
device.start_cameras( &configuration );

9. Azure Kinect Sensor SDK Programming
// Get Capture
k4a::capture capture;
const std::chrono::milliseconds timeout( K4A_WAIT_INFINITE );
const bool result = device.get_capture( &capture, timeout );
if( !result ){
break;
}
// Get Color Image
k4a::image color_image = capture.get_color_image();
cv::Mat color = k4a::get_mat( color_image );
// Get Depth Image
k4a::image depth_image = capture.get_depth_image();
cv::Mat depth = k4a::get_mat( depth_image );
- k4a::get_mat() is utility function that defined in util.h
Utility for Azure Kinect Sensor SDK | GitHub Gists
https://gist.github.com/UnaNancyOwen/9f16ce7ea4c2673fe08b4ce4804fc209#file-util-h

10. Azure Kinect Sensor SDK Programming
// Show Image
depth.convertTo( depth, CV_8U, -255.0 / 5000.0, 255.0 );
cv::imshow( "color", color );
cv::imshow( "depth", depth );
// Wait Key
const int32_t key = cv::waitKey( 30 );
if( key == 'q' ){
break;
}
// Close Device
device.close();
// Clear Handle
color_image.reset();
depth_image.reset();
capture.reset();

11. Azure Kinect Sensor SDK Programming
// Initialize Transformation
k4a::calibration calibration = device.get_calibration( configuration.depth_mode, configuration.color_resolution );
k4a::transformation transformation = k4a::transformation( calibration );
// Transform Color Image to Depth Camera
k4a::image transformed_color_image = transformation.color_image_to_depth_camera( depth_image, color_image );
cv::Mat transformed_color = k4a::get_mat( transformed_color_image );
// Transform Depth Image to Color Camera
k4a::image transformed_depth_image = transformation.depth_image_to_color_camera( depth_image );
cv::Mat transformed_depth = k4a::get_mat( transformed_depth_image );
// Transform Depth Image to Point Cloud
k4a::image xyz_image = transformation.depth_image_to_point_cloud( depth_image, K4A_CALIBRATION_TYPE_DEPTH );
//k4a::image xyz_image = transformation.depth_image_to_point_cloud( transformed_depth_image, K4A_CALIBRATION_TYPE_COLOR );
cv::Mat xyz = k4a::get_mat( xyz_image );
- Transform Image to Other Coordinate System
Color Image → Depth Camera
Depth Image → Color Camera
Depth Image → Point Cloud
Add overload functions that returns k4a::image in k4a::transformation #596 | GitHub/Azure-Kinect-Sensor-SDK
https://github.com/microsoft/Azure-Kinect-Sensor-SDK/pull/596
// Close Transformation
transformation.destroy();

12. Azure Kinect Sensor SDK Programming
// Azure Kinect Sensor SDK
#include <k4arecord/playback.h>
#include <k4arecord/playback.hpp> /* C++ Wrapper from Azure Kinect Sensor SDK v1.2.0 */
// Open Playback File
k4a::playback playback = k4a::playback::open( "./path/to/file.mkv" );
// Get Capture
k4a::capture capture;
bool result = playback.get_next_capture( &capture );
if( !result ){
break;
}
// Close Playback
playback.close();
- NOTE: k4arecorder (Azure Kinect Sensor SDK v1.1.x) has a bug that first 2-frames are empty.
Move C++ wrapper for playback #493 | GitHub/Azure-Kinect-Sensor-SDK
https://github.com/microsoft/Azure-Kinect-Sensor-SDK/pull/493

13. Azure Kinect Sensor SDK Programming
Azure Kinect Office Sample Recordings | Microsoft Download Center
https://www.microsoft.com/en-us/download/details.aspx?id=58385

14. Azure Kinect Body Tracking SDK Overview
Capture Queue
Playback
Device
Frame Body

15. Azure Kinect Body Tracking SDK Programming
// Azure Kinect Body Tracking SDK
#include <k4abt.h>
#include "k4abt.hpp" /* Un-Official C++ Wrapper */
//#include <k4abt.hpp> /* Official C++ Wrapper Provide from Next Release */
// Create Tracker
k4abt::tracker tracker = k4abt::tracker::create( calibration );
if( !tracker ){
throw k4a::error( "Failed to create tracker!" );
}
// Enqueue Capture
tracker.enqueue_capture( capture );
// Pop Tracker Result
k4abt::frame body_frame = tracker.pop_result();
- Official C++ wrapper is coming soon …
C++Wrapper for Azure Kinect Body Tracking SDK | GitHub Gists
https://gist.github.com/UnaNancyOwen/41b578f5d272aa6f22cf8ff6565fb71b#file-k4abt-hpp

16. Azure Kinect Body Tracking SDK Programming
// Get Body Index Map
k4a::image body_index_map_image = body_frame.get_body_index_map();
cv::Mat body_index_map = k4a::get_mat( body_index_map_image );
// Draw Body Index Map
cv::Mat body_index = cv::Mat::zeros( body_index_map.size(), CV_8UC3 );
body_index.forEach<cv::Vec3b>(
[&]( cv::Vec3b& pixel, const int32_t* position ){
const int32_t x = position[1], y = position[0];
const uint32_t index = body_index_map.at<uint8_t>( y, x );
if( index != K4ABT_BODY_INDEX_MAP_BACKGROUND ){
pixel = colors[index % colors.size()];
}
}
);
- Background Index is 255 (K4ABT_BODY_INDEX_MAP_BACKGROUND)

17. Azure Kinect Body Tracking SDK Programming
// Get Body Skeleton
std::vector<k4abt_body_t> bodies = body_frame.get_bodies();
// Get Image that used for Inference
k4a::capture body_capture = body_frame.get_capture();
k4a::image skeleton_image = body_capture.get_color_image();
cv::Mat skeleton = k4a::get_mat( skeleton_image );
// Draw Body Skeleton
for( const k4abt_body_t& body : bodies ){
for( const k4abt_joint_t& joint : body.skeleton.joints ){
k4a_float2_t position;
const bool result = calibration.convert_3d_to_2d( joint.position, K4A_CALIBRATION_TYPE_DEPTH, K4A_CALIBRATION_TYPE_COLOR, &position );
if( !result ){
continue;
}
const int32_t id = body.id;
const cv::Point point( static_cast<int32_t>( position.xy.x ), static_cast<int32_t>( position.xy.y ) );
cv::circle( skeleton, point, 5, colors[id % colors.size()], -1 );
}
}
- Get Image that used for Inference
- Convert Joint Position (3D) to 2D for Draw Circle on Image

18. Azure Kinect Body Tracking SDK Programming
// Show Image
cv::imshow( "body index", body_index );
cv::imshow( "skeleton", skeleton );
// Wait Key
const int32_t key = cv::waitKey( 30 );
if( key == 'q' ){
break;
}
// Close Tracker
tracker.destroy();
// Clear Handle
body_index_map_image.reset();
skeleton_image.reset();
body_capture.reset();
body_frame.reset();