Camera 2.0 in Android 4.2

Camera 2.0
The New Camera Hardware Interface in Android 4.2

Balwinder Kaur, Senior Member, Technical Staff, Aptina Imaging
Ashutosh Gupta, Principal Software Engineer, Aptina Imaging

Android Builder’s Summit, San Francisco, CA
2.18.2013
© 2013 Aptina Imaging Corporation. All rights reserved. Products are warranted only to meet Aptina’s production data sheet specifications. Information, products, and/or
specifications are subject to change without notice. All information is provided on an “AS IS” basis without warranties of any kind. Dates are estimates only. Drawings not to
scale. Aptina and the Aptina logo are trademarks of Aptina Imaging Corporation. All other trademarks are the property of their respective owners.

1 | © 2013 Aptina Imaging Corporation

Agenda
• Camera Use Cases
‣ Prominent Use Cases
‣ Limitations of existing APIs
‣ Overview of android.hardware.Camera

• Android Framework : Camera Service
‣ Native Camera service
‣ New - Under the hood – Camera 2.0

• It’s all about the Camera hardware !
‣ Camera Hardware Abstraction Layer
‣ Camera Device Driver

• Challenges in Camera HAL development
• Emerging Trends
• Q&A


Section I
Android Camera APIs


Prominent Camera Use Cases
• Main Use Cases
‣ Live Preview of Camera Stream
• Live Preview + copy of the Frame returned to the application

‣ Capture a frame
‣ Video Recording of a Camera Stream

• Secondary Use Cases
‣ Configuring the Camera
‣ Snapshot during video recording
‣ Event Callbacks: Shutter Clicked, AutoFocus Achieved

• Information Related Use cases
• Receiving more than an image back . e.g. face detection data
• Meta Data of an Image


Limitations in Camera API
• No support for Burst Mode Photography

• No support for Panoramic Shots

• Very limited support for frame metadata

• No per-frame control of the camera or the image
processing pipeline

• No access to control subsystems within the camera e.g.
Focus, Flash, Image Sensor


Overview of android.hardware.Camera
6 Classes 8 Callback Interfaces
• Camera.AutoFocusCallback
• Camera
• Camera.ErrorCallback
• Camera.CameraInfo
• Camera.FaceDetectionListener
• Camera.Parameters
• Camera.OnZoomChangeListener
• Camera.Size • Camera.PictureCallback
• Camera.Face • Camera.PreviewCallback

• Camera.Area • Camera.ShutterCallback
• Camera.AutoFocusMoveCallback


Camera class
Contains all the methods for the Camera Lifecycle
• Open & Release
• Access to the Camera Controls
• Preview
‣ Direct Live Preview to the display or a texture
‣ Get Preview Frame in a Callback

• Capture
‣ Callbacks: Shutter, JPEG, RAW, “Postview”

• Lock & Unlock
• Actions: startAutoFocus, startSmoothZoom & startFaceDetection


Camera.Parameters
Class for Camera Controls
① Mandatory Feature Set
‣ getSupportedPreviewSizes + set/get

② Optional Feature Set
‣ isVideoStabilizationSupported + set/get

③ Custom Feature Set
‣ Camera.Parameters class provides a “dumb” pipe to the
hardware for custom controls
‣ set (“your_param_string”, value); get(“your_param_string”);

Auto White Balance, Scene Modes, Focus Modes, Preview Sizes, Picture Sizes,
Thumbnail Sizes, Preview Format, Picture Format, Anti-Banding


… the rest of the Camera Classes
• Camera.CameraInfo
‣ For each camera, front or back facing, orientation of the camera
image

• Camera.Size
‣ width and height of the image

• Camera.Face
‣ face-id, co-ordinates for left eye, right eye, mouth, outer bounds of
the face

• Camera.Area
‣ Rectangular bounds with a weight
‣ Metering Regions for 3A : Auto Focus, Auto White Balance, Auto
Exposure


Camera 2 Internals
Released as part of Android 4.2
Camera 1 => android.hardware.Camera

Camera 2 => android.hardware.ProCamera

New Camera HAL implementation based on
Camera 2.0 – Samsung exynos5 based

No application that actually uses Camera 2.0
Disclaimer: All reference to Camera2 in this slide deck is based
on reverse engineering AOSP code !


Section II
Android Framework - Camera


High Level Architecture


Android High Level Architecture

Source: Android Anatomy and Physiology, Google IO 2008


Hardware Abstraction Layer

Camera

Source: Android Anatomy and Physiology, Google IO 2008


Camera Subsystem

Application

Application framework Hardware Independent

Camera Service
Camera HAL
Implementation

Camera Device Driver Hardware Dependent

Camera Hardware

HAL = Hardware Abstraction Layer


Process View

App App App

Binder IPC
ICamera

Camera Service

Back Facing Front Facing
Binder IPC
Surfaceflinger ISurface Camera Hardware Camera Hardware
Object Object

Media server
System Call System Call

Kernel Driver Kernel Driver


Inside the Camera App
Application Code
android framework code Camera.java

JNI

android_hardware_Camera.cpp

IBinder Interfaces

Camera Service
libcameraservice.so

Media server

JNI Layer


android_hardware_Camera
• Allocates Memory from the Java memory heap for JPEG
images.

• If a Copy of the Preview Frame is requested by the app,
then the copy from native to java buffers is done here.

• Creates a persistent context for callbacks from native
code to Java (JNICameraContext)

• Holds references to the Java Camera, Face and Area
objects.


Camera Service
- with Camera 2.0


Camera Service
Application Code

JNI


ICameraClient | ICameraService | ICamera

Camera Service
libcameraservice.so
camera.h

Camera HAL
implementation
Media server

Camera Service with Camera 2
Application Code

JNI


ICameraClient | ICameraService | ICamera

Camera2Client
Camera Service libcameraservice.so
Camera2Device
camera.h camera2.h

Camera HAL Camera2 HAL
implementation implementation
Media server

Camera Service
• Runs in the media server process

• It is a shared library libcameraservice.so

• Queries System for Camera HAL Api version - accordingly creates a
CameraClient or Camera2Client for the application

• For Camera 1, libcameraservice does:
Permission check android.permission.CAMERA
Ensures only one Client connects to a Camera Hardware Object
Ensures each Process connects to a single Camera Hardware Object
Redirects callbacks back to the app layer
Accessed over IBinder Interface
Number of Cameras Available
CameraInfo Details


Camera2Client
• Implements android.hardware.camera API on top of camera device HAL version 2

• Has processors that run in their own threads

Recording
Manages Preview & Recording Streams Streaming Processor Preview Stream
Stream

Still Image capture output processing JPEG Processor Capture Stream

Processes Callback Callback Processor Callback Stream

ZSL Queue Processing ZSL Processor ZSL Stream

Metadata processing for output frames Frame Processor

Manages Capture Sequences for ZSL Capture Sequencer
regular and Burst modes


Camera2Device
• Manages Input and Output Streams between the Camera HAL
and the Service
• Functions
‣ Retrieves Static Metadata Information of the Camera
‣ Sends requests for Capture and Streaming
‣ Manages input and output streams including re-processing streams
and metadata buffers
‣ StreamAdaptor between ANativeWindow Interface and the
Camera HAL stream ops
• ANativeWindows is Android Native Window interface
‣ Sends Notifications (Error, Shutter, AutoFocus, AutoExposure and
AWB)


Metadata
• Camera2 has a lot of emphasis on retrieving metadata from the camera and
making it available to the application
• Two kinds of Metadata
‣ Static & Frame-Based
‣ Static does not change and is available without opening the camera device (Camera Info
Class)
‣ *_INFO is the static metadata
ANDROID_LENS, ANDROID_LENS_INFO, ANDROID_SENSOR, ANDROID_SENSOR_INFO,
ANDROID_FLASH, ANDROID_FLASH_INFO, ANDROID_HOT_PIXEL, ANDROID_HOT_PIXEL_INFO,
ANDROID_DEMOSAIC, ANDROID_DEMOSAIC_INFO, ANDROID_NOISE, ANDROID_NOISE_INFO,
ANDROID_SHADING, ANDROID_SHADING_INFO, ANDROID_GEOMETRIC,
ANDROID_GEOMETRIC_INFO, ANDROID_COLOR, ANDROID_COLOR_INFO, ANDROID_TONEMAP ,
ANDROID_TONEMAP_INFO, ANDROID_EDGE, ANDROID_EDGE_INFO, ANDROID_SCALER,
ANDROID_SCALER_INFO, ANDROID_JPEG, ANDROID_JPEG_INFO, ANDROID_STATS,
ANDROID_STATS_INFO, ANDROID_CONTROL, ANDROID_CONTROL_INFO,
ANDROID_QUIRKS_INFO etc.

• Extensible - there is provision for Vendor Specific Tags


Android Open Source Project (AOSP) Structure
• Android Framework
‣ Java: frameworks/base/core/java/android/hardware
‣ JNI: frameworks/base/core/jni

• Camera Service
‣ frameworks/av/services/camera/libcameraservice/

• IBinder Interfaces
‣ frameworks/av/include/camera/ICamera.h etc.

• IBinder Implementation
‣ frameworks/av/camera/ICamera.cpp etc.

• Camera HAL Interface
‣ hardware/libhardware/include/hardware/camera2.h etc.

• Camera HAL Implementation
‣ hardware/<vendor>/camera (typically)

• Camera Metadata
‣ system/media/camera/include/system/camera_metadata_tags.h


Section III
Its all about the Camera Hardware !


Camera Hardware Abstraction Layer
Review of a Typical Implementation


Camera Stack – Camera HAL
Upper Camera Stack …

Camera
SurfaceFlinger (HAL)
/Overlay Buffers User
Vendor Specific HAL
Implementation

Camera Driver Kernel

Image Sensor Image Sensor Hardware
Processer


Android CameraHAL Library
• The Camera Hardware Abstraction Layer (HAL) is a library that is
specific to the camera hardware platform
‣ Written by hardware vendors (Qualcomm, Samsung, TI, others)

• CameraHAL maps Android Camera Service calls to driver functions
‣ Ice Cream Sandwich (ICS) uses camera.h
‣ Jelly Bean (and above) use camera2.h

• CameraHAL low level interface communicates with the kernel level
driver
‣ It can support interfaces including Video for Linux 2 (V4L2) or OpenMax
(OMX)
‣ Communicates with the driver through file I/O calls (open, close,
input/output controls (IOCTL), etc)


Sample ICS CameraHAL
Functional Diagram

Camera Service I/F
Memory Manager

Display Surface
Manager
CameraHAL

Event
Notification
Manager

Camera Manager

User

Source: TI OMAP4 /dev/videoX
git.omapzoom.org


Sample JB CameraHAL
Functional Diagram

Camera Service I/F
Stream Manager

CameraHAL
Operations Handler Metadata Handler

Reprocess Stream
Management

User

Source: Samsung Exynos 5 /dev/videoX


JB CameraHAL Block Diagram Discussion
• Block Functions
‣ CameraHAL
• Initialization, thread creation, function dispatch

‣ Stream Manager
• Handle stream events, coordinate buffer and stream usage,
manage state machine

‣ Metadata Handler
• Acquire per shot metadata, convert to Android format

‣ Reprocess Stream Management
• Setup and manage reprocess streams


JB CameraHAL Changes
• For CameraHAL 2.0, much of the CameraHAL 1.0 functionality is
moved to libcameraservice
• CameraHAL 2.0 targeted at HW specific functionality
• New CameraHAL functionality currently not passed up to
applications
• Image metadata has more importance
• Reprocessing introduced – Process an already captured image
stream
• Stream based rather than stream function based
‣ CameraHAL 1.0 has methods for specific stream types (e.g.,
start_preview(), take_picture())
‣ CameraHAL 2.0 has generalized stream methods (e.g., allocate_stream())


Camera Device Driver


Camera Stack – Camera Driver
Upper Camera Stack …

Camera
SurfaceFlinger (HAL)
/Overlay Buffers User
Vendor Specific HAL
Implementation


Image Sensor Image Sensor Hardware
Processer


Android Kernel Camera Driver
• The kernel driver presents a standard interface for
different types of camera hardware
‣ Camera hardware specific attributes are handled by the low
level kernel driver
‣ Image Sensor Processor (ISP) vs. SOC (smart) sensor -
differences are handled at the driver level

• For Android, Video for Linux 2 (V4L2) is used in many
implementations
‣ V4L2 has been in existence for many years
‣ OpenMax (OMX) is also used for a low level driver interface by
some vendors.


V4L2 Kernel Driver Block Diagram

V4L2 Driver Interface
- IOCTL support/dispatch
- V4L2 driver infrastructure

Controlling Interface
- Support for different device configurations
- Control device flow

Buffer/Memory Management Camera HW Management
- Memory allocation (if needed) - One of these blocks for each camera
- Buffer management type
- Buffer queue/de-queue - Device discovery
- Device initialization
- Power management
- Set/get device specific parameter
- Enable/disable image streaming


V4L2 Kernel Driver Resources
• Memory
‣ Memory can be either driver-allocated or user-provided
‣ The image transfers from the camera to memory through hardware Direct Memory Access
(DMA)
‣ Hardware memory management may be used to avoid contiguous memory requirement

• Interrupts
‣ Camera ports support for interrupts on events such as frame start, finish, focus events,
etc.

• Camera Control: I2C/SPI
‣ I2C (Inter-Integrated Control) is used for writing or reading camera registers
‣ SPI (Serial Peripheral Interface) is a faster alternative to I2C

• Control Signals/GPIO
‣ All controlled by the low level driver

• Power
‣ Sensor power management is critical to embedded device operation
‣ Sensors support standby mode where settings are maintained while power usage is reduced


V4L2 Driver Buffer Management
• One or more buffers are supported
• User buffers or kernel-allocated buffers
are supported Buff_0

• Buffers are treated the same for preview,
capture, video (output resolution does not Buff_x Buff_1
matter)
• Buffers are queued to a circular list
Buff_3 Buff_2
• Buffer filling starts when the V4L2
Stream_On command is executed
• Once filled, the CameraHAL de-queues a
buffer, processes the buffer, then re-
queues the buffer
• The Stream_Off command causes all
buffer to be released

Typical V4L2 Preview Sequence (1)
• V4L2 preview start up sequence is given below

V4L2 Call Driver Events Hardware Events
VIDIOC_S_FMT – set format Set image format and size Set both resolution and output
pixel format
VIDIOC_G_PARM – get Get a camera driver or hardware Read camera parameter
parameter parameter
VIDIOC_S_PARM – set Set a camera driver or hardware Write camera parameter
parameter parameter
VIDIOC_CROPCAP – get Return camera cropping None
cropping capabilities capabilities
VIDIOC_S_CROP – set Set cropping rectangle Set camera cropping rectangle
cropping
VIDIOC_REQBUFS – request Request buffer support from the None
camera buffer driver (user vs. kernel)
Loop: VIDIOC_QUERYBUF For kernel allocated buffers, return None
– query buffer caps buffer characteristics
V4L2_MMAP – map buffers For kernel allocated buffers, None
to user space memory map to user space


Typical V4L2 Preview Sequence (2)
• V4L2 preview start up sequence (cont)
Loop: VIDIOC_QBUF – Queue buffers in the circular queue none
queue buffers
VIDIOC_STREAM_ON – start Start image capture state Enable image output
streaming

• V4L preview shut-down sequence
VIDIOC_STREAM_OFF – stop Stop streaming, deallocate Disable image output
streaming buffer


V4L2 Media Controller Architecture
• Designed to support dynamically re-connectable
hardware blocks.

• Allows for greater programmer control

• Introduces the notion of entities, pads and links


V4L2 Media Controller Architecture(Cont.)


Section IV
Challenges in Camera HAL Development


Challenges in Camera HAL Development
• Memory Management

• Various Implementation of Camera Driver

• Color Format Conversion

• Buffer Synchronization between Camera & Display

• Support for Advanced Features


Are all Camera HALs equal ?
No

• Supported Features depend on hardware capabilities

• Supported Features depend on the implementation in
the Camera HAL

• Performance

• Reliability

• Extensions to the standard Android Feature Set


A Peek into the Future


Emerging Trends
• Computer Vision Applications go mainstream
‣ APIs on Object Tracking, Gesture Recognition become more
common place

• Computation Photography application
‣ Developers get fine grained control of flash and camera

• High Dynamic Range
‣ Ability to capture larger exposure range

• 3D Imaging
‣ Use of 2 cameras to generate a 3D image


What’s coming ?


Q&A


References
• http://developer.android.com/
• http://www.codeaurora.org
• http://omappedia.org
• http://source.android.com
• http://stackoverflow.com/questions/10775942/android
-sdk-get-raw-preview-camera-image-without-
displaying-it
• http://www.cjontechnology.com/blog/?p=14
• http://graphics.stanford.edu/projects/camera-2.0/


About
Aptina enables imaging everywhere.
A leading innovator of CMOS imaging technology, Aptina
delivers excellent pixel performance, sensor functionality and camera
system capability to a world going visual.

Aptina Camera Software Stack (Access) is Aptina’s
implementation of the Android Camera HAL with some custom
extensions.

Contact Information

Balwinder Kaur bkaur@aptina.com

Ashutosh Gupta ashutosh@aptina.com


Camera 2.0 in Android 4.2

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (20)

Similaire à Camera 2.0 in Android 4.2

Similaire à Camera 2.0 in Android 4.2 (20)

Dernier

Dernier (20)

Camera 2.0 in Android 4.2

Notes de l'éditeur