Intro to Image processing @DHoxSS 2015 - Introductory strand of the Digital Humanties Summer school, July 2015

1. Working with Digital Images
Dr Ségolène M. Tarte
Digital.Humanities@Oxford Summer School – 23rd July 2015
University of Oxford, UK

2. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Why images?
• To preserve, conserve, and curate
• To analyze, study, and interpret
• To document, present, and disseminate
And because they:
• Are portable
• Can be processed without damage to the pictured
object
• Can give access to new information:
– Multispectral imaging: seeing beyond visible light
– Faces and surfaces hidden in exhibitions
– etc…

3. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
0 0 0 0 0 0 0 0 0 0 0 127
0 255 255 255 0 0 255 0 0 255 0 0
0 255 0 0 255 0 255 0 0 255 0 0
0 255 0 0 255 0 255 255 255 255 0 0
0 255 0 0 255 0 255 0 0 255 0 0
0 255 255 255 0 0 255 0 0 255 0 0
0 0 0 0 0 0 0 0 0 0 0 127
What are digital images?
For a grey-scale image (8bit):
• An array of integers with values between 0 and 255
(or 256 values between 0 and 1)
– 0 is black
– 255 (1) is white
• Each cell in the array
is a pixel, with:
– Coordinates (x, y)
– A pixel value v
between 0 and 255
– A pixel size defining the
resolution of the image
0 0 0 0 0 0 0 0 0 0 0 127
0 255 255 255 0 0 255 0 0 255 0 0
0 255 0 0 255 0 255 0 0 255 0 0
0 255 0 0 255 0 255 255 255 255 0 0
0 255 0 0 255 0 255 0 0 255 0 0
0 255 255 255 0 0 255 0 0 255 0 0
0 0 0 0 0 0 0 0 0 0 0 127

4. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
What are digital images?
For a colour image:
• Up to 4 arrays, or channels, storing values
according to a given model of colour space
• Examples of colour spaces:
– HSL – Hue Saturation Lightness
– RGB – Red Green Blue
– HSV – Hue Saturation Value
– RGBA – Red Green Blue Alpha
– CMYK (for printing) – Cyan Magenta Yellow blacK

5. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Two models of colour spaces
RGB
• Red value, r
• Green value, g
• Blue value, b
Note: if r=g=b, then the
colour is on the grey scale
HSL
• Hue, h
• Saturation, s
• Lightness, l
Notes: if s=0, then the
colour is on the grey scale;
if l=0, the colour is black; if
l=1, the colour is white

6. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Discrete light receptors in the retina:
– Cones – photopic vision:
• 6-7 million
• Highly sensitive to colour
(specialised red, green, blue cones)
• Concentrated around the fovea
• Bright-light vision
• Fine details
– Rods – scotopic vision:
• 75-150 million
• Sensitive to low levels of illumination
• Large area of distribution on the retina
• Dim-light vision
• Overall picture of the field of view
Elements of human visual perception

7. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Elements of human visual perception
• Mach Band effect
• Optical illusions

8. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Digital images and visual perception
• Parallel between RGB colour space and the cones
of our visual system
– RGB appropriate for fine details detection
• Perception of brightness is adaptive and important
in detection of changes
– HSL’s saturation channel and grey-scale images
appropriate for feature detection
• Visual perception is context dependent and
encapsulates (implicit) expectations and
knowledge
– Choosing how to look at images and how to process them
(as well as, upstream, how to capture them!!) is
interpretative

9. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Woodgrain removal: asset or hindrance?

10. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Digital is not neutral!
Digitized versions of an artefact are digital avatars of
the artefact
Digital avatars:
(1) Are encoded
(2) Are embedded into the real
(3) Influence the real
– Express a certain form of presence of the artefact (re-
materializaton)
– Are contingent on the intention of the act of digitization
– Have an expected performative value

11. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
As for texts, images require:
• Provenance
– Who made the image?
– From what?
– How?
– Why?
– When?
• Processing principles
[// Editorial principles]
– For what purpose was
the image
produced/modified?
– Was it
modified/processed?
– If so, how and why?
– Processing is political
[“All mark-up is political”]

12. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Histogram-based processing
• A histogram visualizes the distribution of grey levels
in the image:
– to each grey levels value v (bin in the histogram)
corresponds the count N of pixels with this grey level value v
(N is the height of the histogram bar, for the bin v)
Note: All principles of processing presented hereafter will deal with 8bit grey-scale images but can
be applied to colour images by applying to each channel of the adopted colour model]
0 0 0 0 0 0 0 0 0 0 0 12
7
0 25
5
25
5
25
5
0 0 25
5
0 0 25
5
0 0
0 25
5
0 0 25
5
0 25
5
0 0 25
5
0 0
0 25
5
0 0 25
5
0 25
5
25
5
25
5
25
5
0 0
0 25
5
0 0 25
5
0 25
5
0 0 25
5
0 0
0 25
5
25
5
25
5
0 0 25
5
0 0 25
5
0 0
0 0 0 0 0 0 0 0 0 0 0 12
7

13. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Example
Image size: 6048x4032
LinearcountscaleLogarithmiccountscale

14. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Brightness and contrast adjustments
• Brightness: shifts the histogram towards the whites
(255) to brighten; shifts the histogram to the blacks
(0) to darken.
– The pixels are redistributed in the bins of the histogram:

15. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Brightness and contrast adjustments
• Contrast: redistributes the pixel colours so that they
span more grey values for more contrast, (resp.
less grey values, less contrast)
– The pixels are redistributed in the bins of the histogram:

16. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Image segmentation
• Image segmentation is the action of determining
region(s) of particular interest (ROI) in an image,
e.g. script, brush strokes
• The crucial task: translate into image/pixels terms
what a region of interest is:
– Specific structures (usually called features in image
processing terms)
– Areas sharing a given property, a form of similarity

17. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Image segmentation
• Two main strategies to identify ROIs:
– Feature detection: detect features, i.e. where there are
discontinuities in the grey values (like at the edges of the
Mach bands)
Example:
• Finding blobs, lines, and edges
– Region identification: define regions, i.e. where there is a
form of continuity/similarity between pixels
Example:
• Finding areas, patches

18. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Feature detection
• Related to brightness perception
• Easier done after having transformed the image into so-called
Fourier space (which deals with frequencies)
– Useful filters:
• Sobel filter, differential filter, Canny edge detector (edges)
• Laplace filter, Difference of Gaussians (blobs)
• Hough transform (ridges)
These filters work by identifying specific “behaviours” of the image
expressed in Fourier space, it then isolates those behaviours in Fourier
space and returns the corresponding areas in image space.
– Other filters:
• High-pass: sharpening (keep the fine details)
• Low-pass: smoothing and blurring (keep the larger areas)

19. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Feature detection
[ generated in Gimp 2.8 – SobelFilter]

20. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Feature detection
[ generated in Gimp 2.8
– Despeckle + DoG 14-
12]

21. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Region identification
• Related to colour/grey-level perception
• Thresholding
– Histogram-based classification of pixels into
“foreground/background” by mapping selected values onto
black or white
• Region growing (magic wand / fuzzy selection –
colour selection)
– Starts at a so-called “seed” point, defined manually
– Based on a similarity criterion (allowed colour variation)
and (for the fuzzy selection) connectivity of the “similar”
pixels

22. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Thresholding

23. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Region growing

24. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Multiple images, getting more information
• Multi-spectral imaging
• Changing illumination conditions: Reflectance
Transformation Imaging (RTI)

25. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Multispectral imaging (MSI)
• MSI can typically span
wavelengths in the range
~380 nm to 1100 nm
Da Vinci’s adoration of the Magi

26. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
MSI
• MSI also relies on the light absorption and
reflective properties of the components of the
artefact being imaged
– The (mineral & organic) chemical components react
differently to different wavelengths
– Possibility to isolate components

27. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
MSI

28. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
RTI: Allowing procedural mimesis
– Capture the physical characteristics of the artefact that
power the sense-making process
• Rely on properties of the visual system
• Mimic a physical-world interpretation strategy of the
experts
Pitch-and-yaw motion
in raking light
• Exaggeration of
highlights and shadows
• Visual system extracts
(interpolates) volumetric
information (shadow-
stereo principle)
• An aspect of materiality

29. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
How RTI works
• Multiple image capture
– 76 LEDs
– One picture per LED
• Create a Polynomial Texture
Map (PTM; hence *.ptm
files)
– Extract a base RGB image
– Based on a luminance model of
light fit the changes of
illumination to a quadratic
surface

30. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
PTM – the LRGB format principle
For each pixel in a .ptm file, are stored:
• RGB as in other formats
– A red value
– A green value
– A blue value
• And a light channel L
– Does not store the 76 values of each of the captured
images
– Instead fits a (quadratic) surface to these 76 values
• Only requires to store the 6 coefficients describing the
surface as a function of light position
• Also allows to simulate light positions for which no
picture was originally captured
[L(lu,lv )= a0lu
2
+a1lv
2
+a2lulv +a3lu +a4lv +a5]

31. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
A Proto-Elamite tablet Louvre, Sb 02801; Source: http://cdli.ucla.edu/

32. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Information in the difference
• Take advantage of the
shadow stereo principle,
i.e., of the motion of the
shadows and highlights
depending on the light
position
32

33. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
How the blend modes work (Gimp / Photoshop)
• Layers are stacked
– Their order is important
• To each layer is associated a mode
– This mode defines how the current layer is combined
with the layer below it
• Depending on the nature of the blend mode, swapping
two layers (and their associated blend modes) will
drastically affect the results
– It can be useful to have an extra empty background layer:
• e.g. a black layer if looking to combine all images and
only see the lighter pixels
• or, a white layer if looking to combine all images and
only see the darker pixels

34. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Multiple images
• In a set of images of an object taken from the same
vantage point, new information lies in how those
images vary
– Explore the differences by using the blend modes:
• Difference
• Subtract
• Darken only
• Lighten only

35. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Using processed images
• Processing images is modifying them
• Processing images is interpreting them
It’s ok to modify images if we’re clear about what we’re
looking for and why we use one method or another when
processing
Understanding the (often black-boxed) image processing
options helps justify choices and make expectations
explicit – the act of interpretation then becomes more
sharable, reproducible, and teachable…

36. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Processing IS interpreting
• It’s important to not mislead your audience
– Make your processing obvious
• As a process
• Give details of what has been done and
why (expose methodology & methods)
• As a result
• Avoid smoothing stitching of images
• Use non-photo-realistic colours as much
as possible - it’ll then be obvious you have
done something to the images

37. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Example of the “Artemidorus papyrus”
• “A strange papyrus” with
– Text – including portions of Artemidorus’ geography
– Maps
– Drawings
• Controversies around its authenticity
– “It’s a fake”: and the forger is… C. Simonides(19th cent)
– It can’t possibly be a fake, in spite of its strangeness
Theory: “the three lives of the papyrus”:
1. early Roman period de luxe copy of the 2nd book of
Artemidorus’ geophraphy (2nd cent BC) with maps
2. re-used in an artist’s workshop: verso with mythological
and real animals (sketch book)
3. Verso blanks filled in with drawings of heads, hands, and
feet (sketch book) [Gallazzi & Kramer, 1998]

38. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Making the intangible tangible: P. Artemid.
Virtual access to the papyrus only
– IR images
– Mirror-images through ink transfers
• Virtually evaluate how the papyrus was rolled
• Virtually compute its length
• Virtually reposition the fragments
– Re-materialization of some aspects of the papyrus
[Tarte, 2012]
[D’Alessio, 2012]
[Latour & Lowe, 2011]
(in collaboration with Prof. D’Alessio (KCL), and Dr Elsner (Oxford))

39. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Example of the “Artemidorus papyrus”

40. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Example of the “Artemidorus papyrus”
Correspondences between recto
and verso pictures:
Measurements between original
and transfer images to simulate the
rolled papyrus
• 12.5cm at the level of V25 on the verso, which
corresponds on the recto to approximately 40cm
inwards of the left end of section (b+c)
• 13.2cm at the level of column (iv)
• 15.3 cm to 4cm at the level of the hands (R16,
R18) at the right end of section (b+c)

41. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Example of the “Artemidorus papyrus”
Κροκóττας
An Indian wild beast, hybrid between
wolf and dog – possibly a hyena

42. Digital.Humanities@Oxford Summer School
23rd July 2015, University of Oxford, UK
S. Tarte  Introduction to the Digital Humanities:
Digital version of the hand-out:
https://www.dropbox.com/s/32yks9fjxw142n6/Refere
nceList-Images.docx?dl=0