1. Artefact conceives, designs, and develops technology products and services for the 21st century
artefactgroup.com
Face Remapping: When Words Are Not Enough
Daniel C. Robbins – Strategy Director
INTRODUCTION
A parent tears up in frustration after having spent an hour repeating a simple lesson for her
autistic child. A grandma feels her distant grandkids drifting away because they don’t speak the
same language. A rape victim is terrified that if people in her village find out who she is, she will
be made an outcast.
These three, very different tableaus, attest to the significance of human relations. We live, work, and play
in relation to each other. While we often think and create on our own, as social creatures, the activities
that form and maintain our relationships coalesce around face-to-face contact. Each of these tableaus
rests on the ability of people to express themselves not only with speech but also with the subtle non-
verbal cues of facial expressions, gaze direction, and posture. Without these cues, our communications
become flat, lose their impact, and ultimately fail in their power to connect. Immediacy suffers.
When we are at our best, we can infer interest, attention, concern, and general mood from each other’s
faces. But what happens when an autistic person can’t easily interpret and respond to facial
expressions? What happens when language differences get in the way of emotional connections across
gaps in age and distance in a far-flung family? What happens when a person in an extreme situation
needs to hide her face and is thus denied the ability to convey the full power of her situation? What
happens when real people lose out on the power of nuanced expression and face-to-face
communication?
THE LIMITS OF EXISTING COMMUNICATION TECHNOLOGIES
Current digital communication technologies have allowed us to bridge time, space, and imagination. We
can send and receive communications when and where we want. But where we have scaled our ability
to communicate, we have also diluted our face-to-face connections. A parent can download a
therapeutic app for their autistic child but that app can’t yet present realistic human faces. Real-time
translation systems can decipher words from another language but they can’t yet provide a natural
connection with the speaker. And current video technologies can shield witnesses but they can’t yet
convey the subtleties and emotional impact of their testimony.

2. Artefact conceives, designs, and develops technology products and services for the 21st century
artefactgroup.com
PUSHING THE LIMITS: FACE REMAPPING
Face Remapping is a new set of technologies that has the potential to bring the human connection back
to modern communications. Imagine the ability to perfectly manipulate the facial expression and
appearance of anyone whose face is conveyed electronically. From the subtle to the brazen: a frown can
be made into a smile, lips can be shown to speak a different language, or a ten-year-old girl can
animate visage of the latest pop star. Brand new developments in Face Remapping mean that anyone
will be able create these manipulations. Face Remapping brings together progress in face tracking, face
model generation, machine learning, and new sensors to create a potentially disruptive technology. A
capability that was, until recently, only possible in high-end special effects studies and academic labs is
now rapidly evolving to the point at which designers, technologists, and even regulators should take
notice. The most exciting academic work includes the papers “Real-time Expression Transfer for Facial
Reenactment,” and “What makes Tom Hanks look like Tom Hanks?” Market interest is shown by recent
acquisitions: Apple snatched up FaceShift and PrimeSense and Intel’s purchased Omek. The big players
are also investing heavily in related technologies, such as face recognition, translation, and even emotion
sensing. If we step back, we can see how these targeted investments might be the first steps along the
way towards using Face Remapping as a means of putting a more human face on digital agents such as
Siri, Google Now, Cortana, and Alexa.
Face Remapping can seamlessly modify video of one person’s expressions and lip motions to match that
of another person.
Illustration by Daniel C. Robbins

3. Artefact conceives, designs, and develops technology products and services for the 21st century
artefactgroup.com
THE “ICK” FACTOR
Even without peeking at the latest episode of “Black Mirror” or chuckling over a Bruce Sterling polemic,
it’s easy to fathom the nefarious side of Face Remapping. It’s no great leap to see that the video
production studio in our pockets, otherwise known as a smartphone, can not only snap family photos
but will soon be able to spoof a celebrity speech, co-worker gaff, or distort the telling of an unfolding
public event. This gives anyone the ability to rewrite transmitted and recorded reality. What is now a
harmless fascination with snapshot filters could become a dangerous fascination with undetectable
manipulation of video of people’s faces. While a first response might be to call for regulation and
technological counter-measures, as an alternative, 21st century design perspectives can elucidate
positive ways to embrace Face Remapping.
THE POWER OF POSITIVE EXAMPLES
A society carves out boundaries through regulation but also through selective purchasing, grass-roots
promotion, and unanticipated adaptations. We can decide to be more comfortable with banks using big
data to detect suspect credit card transactions than we are with insurance companies charging us more
based on lifestyle choices revealed by that same data. Now is the time to put on our intellectual running
shoes so, that in parallel with the research labs, corporate investments, and technology refinements we,
as a society, can define the positive boundaries for this new technology. 21st century design principles
mandate that we dig deep to identify people who currently experience compromised communications.
Face Remapping can meaningfully empower these people by giving them better access to natural, face-
based interactions.
CONCEPTS
We will sketch out three design concepts that suggest how Face Remapping can open up richer forms of
personalized human interaction for an autistic child, a remote family member, and even a victim in a
sensitive court case. Each of these concepts was chosen to help those who are underserved by current
technological solutions and where it’s acceptable to modify facial representations.
EMOTION COACH FOR AUTISM
Children with autism benefit when they are repetitively coached as to how to read and respond to facial
expressions and basic interactions. Ideally, it’s a trained professional, caregiver, or parent who brings
these learning experiences to life but this kind of human coaching can’t happen as frequently or for as
long duration as needed. While there are a bevy of smartphone and tablet autism apps they typically rely

4. Artefact conceives, designs, and develops technology products and services for the 21st century
artefactgroup.com
on simplistic cartoon graphics. When video of actual humans is provided, the content sets are limited
and fixed.
Face Remapping could bring a human touch to coaching applications for autistic children. We propose
an application where an autistic child learns
how to better read emotions by practicing
with demonstration video segments that are
remapped onto a wide variety of faces,
including people they know. The creator of
the app need only would record video
segments of a small number of actors. Face
Remapping is then used transfer these
expressions onto any face, including the faces of
people the autistic child routinely encounters:
a regular bus driver, other teachers, or even
extended family. Augmenting human coaching
with a tireless and personalized digital app
via Face Remapping, reduces the load on
caregivers and frees them up for more
intensive and cognitive tasks.
Consider what happens in the future when we infuse this class of application with the intelligent
systems being developed for digital agents such as Siri and Alexa. Instead of drilling with fixed
sequences, a future system would be more interactive and respond back to the expressions and
utterances of the child with autism. The pace, style, and content of the presented segments could
dynamically adapt to the child’s progress.
TRANSLATION FOR SPREAD OUT FAMILIES
In our migratory world, staying in touch with far flung family is vitally important. Whether these global
movements are initiated by armed conflict, social strife, political turmoil, economic hardships, or even a
desire for a better education, it is the children who then lose fluency in the language of their roots. While
current systems, such as Skype Translator, can help scattered families stay in touch, the
communications are stilted. Attention is divided: we listen to the synthesized voice, we read the sub-
titles, all while trying to concentrate on the face of a remote family member. This takes family members
out of the moment and the emotional connection is compromised.
Face Remapping helps create a wide array of emotion
coaching videos for children with autism.
Illustration by Kenneth Tay (Artefact)

5. Artefact conceives, designs, and develops technology products and services for the 21st century
artefactgroup.com
Face Remapping could bring a human
touch to increasingly common remote
family communications by adding
perfect lip-sync to the translated video from
remote family members. The result is a
conversation between family members
that conveys face, pose,
expressions, and eye gaze. Face Remapping
would allow natural
communications between family members,
regardless of where they live or what
language they speak.
Imagine a future where we combine
360 cameras and other sensors with the
family translation systems. Family
conversations move beyond the formal. A
grandchild takes her grandparent on a rich
visual tour of her new life, weaving in her daily wanderings, new friends, and little discoveries, all the
while maintaining a natural conversation deepened by subtle facial cues and expressions.
PROTECTING IDENTITY
Victims of violence often face multiple layers of injustice: the initial crime, reprisals, and even social
stigma from their communities. Victims of ethnic and domestic violence, along with their children, can be
ostracized and abandoned by their communities if they seek out justice. If the victim attempts to protect
her identity via video masking or visual screens she will then have an even harder time conveying the
weight of her plight and garnering empathy.
Face Remapping adds perfect lip-synch to translated video
conversations for families who live far apart and speak
different languages.
Illustration by Kenneth Tay (Artefact)

6. Artefact conceives, designs, and develops technology products and services for the 21st century
artefactgroup.com
Face Remapping could help bring a more
human touch to highly charged, constrained,
and consequential court testimony.
The victim would give her testimony via
video that could be real-time or prerecorded.
Face Remapping would be used to transfer
the nuanced expressions and lip motions of
the victim onto a neutral face that matched
the general age, race, and gender
characteristics of the speaker. The
presented face presented could either be that
of an actor, composited from multiple
representative face components, or entirely
synthesized. A corpus of neutral faces, that
were appropriate to different locales, could
be built up over time to provide a safer way for
victims to testify. Face Remapping
would enable the victim to convey her full conviction and determination without fear of people in the
court or community directly identifying her.
In the future Face Remapping could be applied in positive ways to other sensitive situations. Imagine if
smartphones, by default, anonymized faces of people in crowds before uploading to social media.
Protestors and bystanders alike, would be less vulnerable to scraping of social media for purposes of
identification and reprisal. We could have a future where identity via social media was always opt-in –
enforced not merely by voluntary policy, but through a technical intervention at the time of media
creation.
CONCLUSION
Face Remapping is a new technology and it is too early to say exactly how it will play out. From the dire
uses of TNT to the innocuous rise of the selfie, technologists are not oracles of unintended futures. But
with Face Remapping it’s easy to conjure up dystopian misuses. While we implore a vigilance against
abuse, such as political and social spoofing, we also apply 21st century design perspectives to identify
constructive uses that genuinely bring a more natural and human quality back to our vast array of digital
communications. We invite an enthusiasm for empowering scenarios, such as coaching, translating, and
protecting. These small design sketches serve to increase our understanding of the positive and negative
implications of Face Remapping. Our design examples point the way toward a preferable future where
exciting new technologies can be used in ways that make the world a better place.
Face Remapping protects a victim’s identity by changing her
appearance while still conveying the emotional power of her
testimony.
Illustration by Kenneth Tay (Artefact)