Material presented at the 10th Korea Android Conference, Oct. 2012.

1. Skia & FreeType
Android 2D Graphics Essentials
Kyungmin Lee
Software Platform Lab., LG Electronics
snailee@gmail.com
The 10th Kandroid Conference

2. Goal
S/W 플랫폼에 있어 GUI(Graphical User Interface) 시스템은 해당 플랫폼의 성능을
결정짓는 매우 중요한 요소이다. Android 역시 새로운 버전을 발표할 때마다 GUI
시스템의 성능을 꾸준히 개선하여 최근에 릴리즈된 Jelly Bean에 와서는 iOS에
필적할만한 성능을 보여주고 있다. 이번 세션에서는 이벤트 처리 기술과 더불어 GUI
시스템의 한 축을 이루고 있는 2D Graphics 기술을 Android에서 어떻게 활용하고
발전시켜 왔는지에 대해 살펴본다.
1. Skia: 2D Graphics Library
 Skia in Android
 Skia API Overview
 Skia Rendering Pipeline
 Skia Backends: Raster, OpenGL, PDF, ...
2. FreeType: Font Engine
 FreeType in Android
 FreeType API Overview
 Font Rasterization: Scan-conversion, Hinting, …
3. Android 2D Graphics Essentials
 Evolution of Drawing Models:
GPUI (UI on the GPU), Display List, Display List Properties, View Layers, …
 Android 2D Graphics Architecture
(From Conference Program Overview)
The 9th Kandroid Conference

3. Skia is Greek for “shadow”
http://code.google.com/p/skia/
Skia is a complete 2D graphic library for drawing Text, Geometries, and Images.
• 3x3 matrices w/ perspective
• antialiasing, transparency, filters
• shaders, xfermodes, maskfilters, patheffects
Mike Reed
• subpixel text
Device backends for Skia currently include:
• Raster
• OpenGL
• PDF
• XPS
• Picture (for recording and then playing back into another Canvas)
(From Project’s Home)
The Skia Graphics Engine is a compact open source graphics library written in
C++. It was originally developed by Skia Inc., which was subsequently acquired by
Google in 2005, who then released the software as open source licensed under
the New BSD License free software license. Now known as skia, it is currently used
in Mozilla Firefox, Google Chrome, Chrome OS, Chromium OS, and Android; and
the Skia library is present on the BlackBerry PlayBook though the extent of its
usage is unclear. Skia's drawing has back-ends for a standard CPU-based software
rasterizer, PDF, and OpenGL
The 10th Kandroid Conference
(From Wikipedia)

4. Skia in Android
Skia (svn)
2006-09
2006-09
r1
r2
2006-11
r3
2008-11
r4
2009-03
r100
2011-03
r1000
2011-07
r2000
2012-01
r3000
2012-05
r4000
2012-08
r5000
2012-10
r6022
Skia
in Android
…
2008-10
…
???
~700
total commit #
Added
Removed
The 10th Kandroid Conference

5. Skia API Overview
A Canvas encapsulates all of the state about drawing into a device (bitmap).
This includes a reference to the device itself, and a stack of matrix/clip values.
For any given draw call (e.g. drawRect), the geometry of the object being drawn
is transformed by the concatenation of all the matrices in the stack. The
transformed geometry is clipped by the intersection of all of the clips in the stack.
While the Canvas holds the state of the drawing device, the state (style) of the
object being drawn is held by the Paint, which is provided as a parameter to
each of the draw() methods. The Paint holds attributes such as color, typeface,
textSize, strokeWidth, shader (e.g. gradients, patterns), etc.
Drawing basic primitives include rectangles, rounded rectangles, ovals, circles,
arcs, paths, lines, text, bitmaps and sprites. Paths allow for the creation of more
advanced shapes. Each path can be made up of multiple contours (or
continuous sections), each consisting of linear, quadratic, and cubic segments
Drawing
Primitives
Canvas
Paint
The 10th Kandroid Conference
Bitmap

6. Skia API Overview
SkRect rect;
rect.set(0, 0, 150, 120);
canvas->drawRect(rect, shapePaint);
canvas->drawRoundRect(rect, 15, 15, shapePaint);
canvas->drawOval(rect, shapePaint);
canvas->drawCircle(60, 60, 60, shapePaint);
// Arc without a wedge
canvas->drawArc(rect, 0, 255, false, shapePaint);
// Arc with a wedge from the center
canvas->drawArc(rect, 0, 255, true, shapePaint);
canvas->drawLine(0, 0, 150, 120, shapePaint);
const char str[] = "Hello World!";
canvas->drawText(str, strlen(str), 75, 145, textPaint);
// Load a bitmap from an image and draw it only if the load succeeds
SkBitmap bitmap;
if (SkImageDecoder::DecodeFile("app/native/icon.png", bitmap)) {
canvas->drawBitmap(*bitmap, 0, 0, NULL);
}
SkPath path;
path.moveTo(50, 50);
// Specify endpoint using absolute coordinates
path.lineTo(100, 100);
// Specify endpoint using a relative offset from the last point
path.rLineTo(50, 50);
// Specify endpoint using absolute coordinates
path.quadTo(120, 125, 150, 150);
// Specify endpoint using a relative offset from the last point
path.rQuadTo(20, 25, 50, 50);
// Specify endpoint using absolute coordinates
path.cubicTo(175, 175, 140, 120, 200, 200);
// Specify endpoint using a relative offset from the last point
path.rQuadTo(25, 25, -10, -30, 50, 50);
The 10th Kandroid Conference
canvas->drawPath(path, shapePaint);

7. Skia Rendering Pipeline
SkCanvas
draw*(…, SkPaint)
Shading
Rasterization
Apply
Shaders
Scan
Conversion
Initial Path
Initial Mask
Apply
Path Effects
Rasterizer
Apply
Color Filters
Apply
Mask Filters
Apply
Stroke Styles
SRC Image
Mask
Path Generation
Initial
SRC Image
Transfer
Path
Blending
Intermediate
Image
Modified
DST Image
Apply
XferMode
DST Image
http://www.xenomachina.com/2011/05/androids-2d-canvas-rendering-pipeline.html
The 10th Kandroid Conference

8. Skia Backends
To render with software Skia
1) Create a native window and then
2) Wrap a pointer to its buffer as an SkBitmap
3) Initialize an SkCanvas with the bitmap
SkCanvas
SkDevice
SkBitmap
To render with hardware-accelerated Skia
1) Create a GLES2 window or framebuffer and
2) Create the appropriate GrContext,
SkGpuDevice, and SkGpuCanvas
SkGpuCanvas
SkGpuDevice
GrContext
The 10th Kandroid Conference

9. A Free, High-Quality,
FreeType? and Portable Font Engine
http://code.google.com/p/skia/
FreeType is a software library written in C that implements a font rasterization
engine. It is used to render text on to bitmaps and provides support for other fontrelated operations.
(From Wikipedia)
FreeType 2 is a software font engine that is designed to be small, efficient, highly
customizable, and portable while capable of producing high-quality output (glyph
images). It can be used in graphics libraries, display servers, font conversion tools,
text image generation tools, and many other products as well.
Note that FreeType 2 is a font service and doesn't provide APIs to perform higherlevel features like text layout or graphics processing (e.g., colored text rendering,
‘hollowing’, etc.). However, it greatly simplifies these tasks by providing a simple,
easy to use, and uniform interface to access the content of font files.
By default, FreeType 2 supports the following font formats: TrueType (and
collections), Type 1, CID-keyed Type 1, CFF, OpenType (both , TrueType and CFF
variants), SFNT-based bitmap, X11 PCF, Windows FNT, BDF (including anti-aliased
ones), PFR, Type 42 (limited support)
David Turner
Robert Wilhelm
Werner Lemberg
(From Project’s Home)
The 10th Kandroid Conference

10. FreeType in Android
FreeType2
2000-11
…
2.0.0
…
2007-07
2.3.5
2008-06
2.3.6
2009-03
FreeType2
in Android
2.3.9
???
2008-10
2009-06
2010-02 2.3.12
2010-03
2010-08
2.4.2
2010-11
2.4.4
2010-07
2.4.6
2010-10
2.4.7
2010-11
2.4.8
2012-03
2010-09
2.4.9
2011-01
2011-08
2011-11
2011-12
2012-03
2012-06 2.4.10
commit #
~1100
since 2007-07
~50
The 10th Kandroid Conference
Added
Removed

11. FreeType API
Glyph Outline
(Lines + Curves)
Quadratic Bé zier Curve
Cubic Bé zier Curve
The 10th Kandroid Conference

12. FreeType API
FT_Library
FT_Face
FT_GlyphSlot
FT_Error
char*
int
library;
face;
slot;
error;
filename, text;
pen_x, pen_y, n, num_chars;
1
Initialize the library
2
Load a font face
... /* initilization code */
error = FT_Init_FreeType( &library );
if ( error ) ... /* error handling code */
error = FT_New_Face( library, filename, 0, &face );
3
if ( error ) ... /* error handling code */
error = FT_Set_Char_Size( face, 50 * 64, 0, 100, 0 );
if ( error ) ... /* error handling code */
slot = face->glyph;
4
for ( n = 0; n < num_chars; n++ ) {
error = FT_Load_Char( face, text[n], FT_LOAD_DEFAULT );
if ( error ) continue; /* ignore errors */
error = FT_Render_Glyph( face, text[n], FT_RENDER_MODE_NORMAL );
if ( error ) continue; /* ignore errors */
5
draw_bitmap( &slot->bitmap, pen_x + slot->bitmap_left,
pen_y - slot->bitmap_top );
pen.x += slot->advance.x >> 6;
}
Set the character size
(e.g., 50pt at 100dpi)
Load a glyph image
into the slot
Convert the glyph’s
outline to a bitmap
6
Draw the bitmap to
a target surface
7
Release resources
show_image();
FT_Done_Face
( face );
FT_Done_FreeType( library );
The 10th Kandroid Conference

13. Font Rasterization: Hinting
Font hinting (also known as instructing) is the use of mathematical instructions
to adjust the display of an outline font so that it lines up with a rasterized grid. At
low screen resolutions, hinting is critical for producing a clear, legible text. It can
be accompanied by antialiasing and (on liquid crystal displays) subpixel
rendering for further clarity.
Hints are usually created in a font editor during the typeface design process
and embedded in the font. A font can be hinted either automatically (through
processed algorithms based on the character outlines) or set manually. Most font
editors are able to do automatic hinting, and this approach is suitable for many
fonts. However, commercial fonts of the highest quality are often manually hinted
to provide the sharpest appearance on computer displays. Verdana is one
example of a font that contains a large amount of hinting data, much of which
was accomplished manually by type engineer Tom Rickner.
(From Wikipedia)
The 10th Kandroid Conference

14. How Views are Drawn in Android pre-3.0?
CPU Rasterization + GPU Composition
ViewRootImpl
performDraw()
View
draw(Canvas)
View
onDraw(Canvas)
Canvas
draw*(…, Paint)
Android Framework (android.jar)
SkCanvasGlue
draw*(…, SkPaint)
libandroid_runtime.so
SkRasterizer
rasterize(…)
SkDraw
draw*(..., SkPaint, ...)
SkDevice
draw*(..., SkPaint, ...)
SkCanvas
draw*(…, SkPaint)
SkBlitter
blit*(…)
libskia.so
http://developer.android.com/guide/topics/ui/how-android-draws.html
The 10th Kandroid Conference

15. Why New Drawing Models in Android post-3.0?
CPU Rasterization + GPU Composition
GPU Rasterization + GPU Composition
GPUI
Increased
Number of
Pixels
Display List
Display List
Properties
View Layers
…
Source: Accelerated Android Rendering, Google I/O 2011
The 10th Kandroid Conference

16. GPUI – UI on the GPU (since 3.0)
android:handwareAccelerated="true"
*GPUI
GPU Rasterization + GPU Composition
turned on by default from Android 4.0
ViewRootImpl
performDraw()
View
draw(Canvas)
View
onDraw(Canvas)
HardwareCanvas
draw*(…, Paint)
Android Framework (android.jar)
android_view_GLES20Canvas
_draw*(…, SkPaint)
libandroid_runtime.so
OpenGLRenderer
draw*(…, SkPaint)
libhwui.so
GLES2/gl2.h
glXXX(…)
libGLESv2.so
The 10th Kandroid Conference

17. Display List in General
A display list (or display file) is a series of graphics commands that define an
output image. The image is created (rendered) by executing the commands.
…
A display list can represent both two- and three-dimensional scenes. Systems
that make use of a display list to store the scene are called retained mode
systems as opposed to immediate mode systems.
(From Wikipedia)
Source: MSDN, http://msdn.microsoft.com/en-us/library/windows/desktop/ff684178(v=vs.85).aspx
http://en.wikipedia.org/wiki/Display_list
http://developer.android.com/guide/topics/graphics/hardware-accel.html
The 10th Kandroid Conference

18. Display List in Android (since 3.0)
A display list records a series of graphics related operation and can replay them
later. Display lists are usually built by recording operations on a
android.graphics.Canvas. Replaying the operations from a display list avoids executing
views drawing code on every frame, and is thus much more efficient.
<<enumeration>>
DisplayList::Op[Non-Drawing]
GLES20RecordingCanvas
draw*(…, Paint)
+Sav e
+Restore
+RestoreToCount
+Sav eLayer
+Sav eLayerAlpha
+Translate
+Rotate
+Scale
+Skew
+SetMatrix
+ConcatMatrix
+ClipRect
android_view_
GLES20Canvas_draw*(…)
SkWriter32
write*(…)
DisplayListRenderer
draw*(…, SkPaint)
SkReader32
read*()
DisplayList
replay(…)
void* buffer
GLES20Canvas
drawDisplayList(…)
draw*(…, SkPaint)
OpenGLRenderer
drawDisplayList(…)
The 10th Kandroid Conference
GLES2/gl2.h
glXXX(…)
<<enumeration>>
DisplayList::Op[Drawing]
+DrawDisplayList
+DrawLayer
+DrawBitmap
+DrawBitmapMatrix
+DrawBitmapRect
+DrawBitmapData
+DrawBitmapMesh
+DrawPatch
+DrawColor
+DrawRect
+DrawRoundRect
+DrawCircle
+DrawOv al
+DrawArc
+DrawPath
+DrawLines
+DrawPoints
+DrawText
+DrawTextOnPath
+DrawPosText
+ResetShader
+SetupShader
+ResetColorFilter
+SetupColorFilter
+ResetShadow
+SetupShadow
+ResetPaintFilter
+SetupPaintFilter
+DrawGLFunction

19. Display List Properties (since 4.1)
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
bool mClipChildren;
float mAlpha;
int mMultipliedAlpha;
bool mHasOverlappingRendering;
float mTranslationX, mTranslationY;
float mRotation, mRotationX, mRotationY;
float mScaleX, mScaleY;
float mPivotX, mPivotY;
float mCameraDistance;
int mLeft, mTop, mRight, mBottom;
int mWidth, mHeight;
int mPrevWidth, mPrevHeight;
bool mPivotExplicitlySet;
bool mMatrixDirty;
bool mMatrixIsIdentity;
uint32_t mMatrixFlags;
SkMatrix* mTransformMatrix;
Sk3DView* mTransformCamera;
SkMatrix* mTransformMatrix3D;
SkMatrix* mStaticMatrix;
SkMatrix* mAnimationMatrix;
bool mCaching;
Without DisplayList
Properties
With DisplayList
Properties
Source: For Butter or Worse, Google I/O 2012
The 10th Kandroid Conference

20. View Layers (since 3.0)
Layers = Off-screen Buffers or Caches
View.setLayerType(int type, Paint p) [API Level 11]
type
View is
H/W-Accelerated
View is NOT
H/W Accelerated
LAYER_TYPE_NONE
•
•
Rendered in H/W
Directly into surface
•
•
Rendered in S/W
Directly into surface
LAYER_TYPE_HARDWARE
•
•
Rendered in H/W
Into hardware texture
•
•
Rendered in S/W
Into bitmap
LAYER_TYPE_SOFTWARE
•
•
Rendered in S/W
Into bitmap
•
•
Rendered in S/W
Into bitmap
*Measured
DisplayList::Op
+ DrawLayer
+ DrawBitmap
when drawing a ListView with Android 3.0 on
a Motorola XOOM
Source: Accelerated Android Rendering, Google I/O 2011
The 10th Kandroid Conference

21. Android 2D Graphics Architecture
Application (View)
Render
Script
Canvas
OpenGL ES
HWUI
EGL
Skia
AGL
HGL
Pixel Flinger
GPU
Surface
SurfaceFlinger
OpenGL ES
HW
Composer
•
•
HGL = Hardware OpenGL ES
AGL = Android’s Software
OpenGL ES
EGL
AGL
GPU
Overlay
HGL
Pixel Flinger
Frame Buffer
The 10th Kandroid Conference

22. Android 2D Graphics Libraries

23. What-How-Why-Why not?
http://sangminpark.wordpress.com/2011/09/02/what-how-why-why-not/
What?
Why?
Why not?
When?
Who?
How?
Past
Present
The 10th Kandroid Conference
Future

24. References
•
•
•
•
•
•
•
•
•
How about some Android graphics true facts?, Dianne Hackborn, Nov. 2011,
https://plus.google.com/105051985738280261832/posts/2FXDCz8x93s
Android 3.0 Hardware Acceleration, Romain Guy, Mar. 2011, http://androiddevelopers.blogspot.kr/2011/03/android-30-hardware-acceleration.html
Android 4.0 Graphics and Animations, Romain Guy and Chet Haase, Nov. 2011,
http://android-developers.blogspot.kr/2011/11/android-40-graphics-and-animations.html
Android Graphics, Animations and Tips & Tricks, Romain Guy and Chet Haase, Dec. 2010,
http://www.curious-creature.org/2010/12/02/android-graphics-animations-and-tips-tricks/
Learning about Android Graphics Subsystem, Bhanu Chetlapalli, Apr. 2012,
http://developer.mips.com/2012/04/11/learning-about-android-graphics-subsystem/
iOS vs. Android ICS: Hardware Accelerated Graphics Pipelines, Nathan de Vries, Nov.
2011, http://atnan.com/blog/2011/11/10/ios-vs-android-ics-hardware-acceleratedgraphics-pipelines/
Android's 2D Canvas Rendering Pipeline, Laurence Gonsalves, May 2011,
http://www.xenomachina.com/2011/05/androids-2d-canvas-rendering-pipeline.html
How FreeType's rasterizer work, David Turner, Feb. 2007, FreeType2 source/docs
Android Graphics, Jim Huang, Sep. 2011, http://0xlab.org/~jserv/android-graphics.pdf
The 10th Kandroid Conference

25. The 10th Kandroid Conference