The document presents an Analytical Framework for Creativity in Multimodal Texts (AFCMT) that analyzes creativity across four dimensions: type of creativity (pattern-forming vs. pattern-reforming), formula of construction (implicit vs. explicit), reference style (endo-referenced vs. exo-referenced), and medium (sign language, music, graphics, etc.). The framework is applied to examples from sign language used by gorilla Koko, the song "Pen-Pineapple-Apple-Pen", and social media graphics in Hong Kong. The document concludes by proposing future applications of the framework to other artistic media.

1. Creativity & Multimodality: an Analytical Framework
for Creativity in Multimodal Texts (AFCMT)
1
Locky, LawLocky Law
PhD candidate
The Hong Kong Polytechnic University
Lx3h@yahoo.com
RIDCH Conference 2017
1-3th June, 2016 (Thur-Sat)

2. 2

3. Content
 Introduction
 Creativity, Cognition and Language
 Analytical Framework for Creativity in
Multimodal Texts (AFCMT)
 Implicit (Assumed) & Explicit (Known)
 Endo-referenced & Exo-referenced
 Sample Analysis
 Gorilla Koko and Sign Language
 MTV and Song
 Graphics and Digital Arts
 Key References 3

4. 4
Is this ‘creative’?
Magnification X 10000
Introduction

5. 5
Is this ‘creative’?

6. 6
Is this ‘creative’?

7. 7
Is this ‘creative’?

8. 8
Is this ‘creative’?

9. Creativity and
Cognition
Boden (2009, pp. 24-25) defines the three types of creativity as follows:
 “Combinational creativity produces unfamiliar combinations of
familiar ideas, and it works by making associations between ideas that
were previously only indirectly linked…
 Exploratory creativity rests on some culturally accepted style of
thinking, or “conceptual space”... The space is defined (and constrained)
by a set of generative rules. Usually, these rules are largely, or even
wholly, implicit…the person moves through the space, exploring it to
find out what’s there…
 In transformational creativity, the space or style itself is transformed by
altering (or dropping) one or more of its defining dimensions. As a
result, ideas can now be generated that simply could not have been
generated before the change.”
 Boden (2009, p. 25) admits that “there is no clear-cut distinction between
exploratory and transformational creativity”.
9

10. Creativity and
Language
not dissimilar from Carter (2004)’s linguistic classification of creativity.
 pattern-forming creativity: “creativity via conformity to language
rules rather than breaking them, creating convergence, symmetry
and greater mutuality between interlocutors”, which is practically the
linguistic counterpart of combinational creativity,
 pattern-reforming creativity: “creativity by displacement of
fixedness, reforming and reshaping patterns of language” (Vo &
Carter, 2010, p. 303),
10

11.  When creativity is created by explicitly showing the formula for
creation, then the instructions of creativity construction is
‘known’;
 otherwise, when creativity is created implicitly without showing
the formula of creation, then the instruction of creativity
construction is ‘assumed’
 because it is assumed by the creator that the target of an instance
of creativity has the level of competence to comprehend or decipher
the formula of creativity creation without explicitly showing the steps
involved in the creation process.
11
the IEEE in AFCMT
Implicit (Assumed) & Explicit (Known)

12. Inspired by the concept of ‘given’ and ‘new’ from Halliday’s (1967)
information status, i.e. information recoverability from preceding
discourse.
 Endo-referenced creativity makes reference to a source which is
recoverable within the preceding or same ‘text’,
 Exo-referenced creativity makes reference to a source which is
unrecoverable within the preceding or same ‘text’, which means
that such reference is made to an external source.
12
the IEEE in AFCMT
Endo-referenced & Exo-referenced

13. 13
Types of
creativity
Formula of
construction
Reference style
Exo-referenced Endo-referenced
Pattern-
forming
Implicit
Direct use / quoting of
external resources such as famous
lines, quotes, speeches, quotes,
sayings, idioms, metaphor, song
lyrics, classic paintings, movie
scenes and dialogues without
explicit citation of the source and
explicitly showing the formula of
repetition (Assumed).
Repeating / playing along with
existing resource / someone’s
creation to the user or witnesses of
such use of it without explicitly
showing the formula of repetition
(Assumed).
Explicit
Direct use / quoting of
external resources such as famous
lines, quotes, speeches, quotes,
sayings, idioms, metaphor, song
lyrics, classic paintings, movie
scenes and dialogues by explicit
citation of the source by explicitly
showing the formula for repetition
(Known).
Repeating / playing along with
existing resource / someone’s
creation to the user or witnesses of
such use of it by explicitly showing
the formula of repetition (Known).
Pattern-
reforming
Implicit
Direct creation of New /
neologism without explicit
citation / indication of the source
and explicitly showing the formula
for creation (Assumed).
Direct creation of New /
neologism using existing resources
without explicitly showing the
formula for creation (Assumed)
Explicit
Creation of New / neologism
by explicit citation / indication of
the source and by explicitly
showing the formula for creation
(Known).
Creation of New / neologism
using existing resources and by
explicitly showing the formula for
creation (Known).

14. A Conversation With Koko (1999)
 humans create or construct portmanteau neologisms in
English in which two or more words are blended
together using parts of each word,
 Koko compounds signs in order to invent new words to
represent meanings which are absent from her
vocabulary.
14
Sample Analysis
Gorilla Koko and Sign Language

15. 15

16. 16
Endo-referenced
(from existing gestural lexicon)

17. MTV and Song
17

18. [Intro]
P-P-A-P
[Verse 1]
I have a pen, I have an apple
Uh! Apple pen!
[Verse 2]
I have a pen, I have pineapple
Uh! Pineapple pen!
[Verse 3]
Apple pen, Pineapple pen
Uh! Pen-Pineapple-Apple-Pen
Pen-Pineapple-Apple-Pen
18
Endo-referenced
Endo-referenced
Endo-referenced

19. “Re-creativity” and Social Media
 In Hong Kong, Cantonese is the first language of around 90% of the
population (Census and Statistics Department, 2012) and it is used in
both formal and informal settings in verbal communication.
 While traditional Chinese characters are the official logographs used
in the city (GovHK, 2017), written Cantonese is also commonly used
in social media.
 As for English, it is used as an official language by the executive
authorities, legislature and judiciary (GovHK, 2008) and is used by
41% of the population (Census and Statistics Department, 2012).
 In terms of Information Technology usage, the Household broadband
penetration rate and Fibre-to-the-home / building (FTTH/B)
household penetration rate as of November 2016 is 86.4% and 71.8%
respectively (OFCA, 2017).
 Mobile subscriber penetration rate as of November 2016 is 234.7%,
which is equivalent to 17,241,608 subscribers (OFCA, 2017).
19

20. “Re-creativity” and Social Media
20

21.  ‘葡萄乘淑儀 / 葡萄X淑儀’ (noun phrase)
 noun ‘葡萄’ (‘grape’) is taken from the Chinese version
of the expression ‘sour grapes’ (‘吃不到的葡萄是酸的’)
from Aesop’s fables The Fox and the Grapes (狐狸與葡萄)
 transitive verb ‘X’ or ‘乘’ is from ‘cross’ or ‘crossover’
borrowed from ‘crossover’ marketing strategy, meaning
to “generate new brand or product by uniting with two
brands in different areas. ” (Ji & Shen, 2013, p. 1364)
 proper noun ‘淑儀’ (‘Regina (Ip)’)
21
Exo-referenced

22. ‘葡萄乘淑儀 / 葡萄X淑儀’ (noun phrase)
22
Exo-referenced

23. 23
Implicit &
Exo-referenced
Implicit &
Endo-referenced
Explicit &
Exo-referenced
Explicit &
Endo-referenced

24. 24
House 1

25. Conclusion
 It is hoped that this paper has provided a new perspective for
researchers of linguistics in the analysis of multimodality and has
assisted studies in realization of meanings in modes and in
intersemiotics between modes.
 For future research, it will be exciting to see how this proposed
analytical framework for creativity in multimodal texts can be
used to explain creativity in other forms of the arts, such as
culinary arts, photography, cinematography, music, sculpting,
classical paintings and social media “re-creativity”.
25

26. 26

27. Key References
 Boden, M. A. (1998). Creativity and artificial intelligence. Artificial Intelligence,
103, 347-356.
 Boden, M. A. (2004). The Creative Mind: Myths and Mechanisms (2nd ed.).
London: Routledge.
 Boden, M. A. (2009). Computer Models of Creativity. AI Magazine, 30(3), 23-
34.
 Carter, R. (1999). Common language: corpus, creativity and cognition.
Language and Literature, 8(3), 195-216.
 Carter, R. (2004). Language and Creativity: The Art of Common Talk. London:
Routledge.
 Gardner, H. (2008). Art, mind, and brain: A cognitive approach to creativity.
Madison, Wisconsin: Basic Books.
 Halliday, M. (1967). Notes on Transitivity and Theme in English: Part 2.
Journal of Linguistics, 3(2), 199-244.
 Halliday, M. (1993). Towards a Language-Based Theory of Learning.
Linguistics and Education, 5(2), 93-116.
 Halliday, M., & Matthiessen, C. M. (1999 [2006]). Construing Experience
Through Meaning: A Language-based Approach to Cognition. London, New
York: Continuum.
27

28. 28

29. Cognition
Language
Creativity
29
Creativity
Cognition Language
Boden (1994,1998, 2004, 2009)
Halliday and Matthiessen (1999 [2006])
Carter (2004); Vo & Carter (2010)

30. Computational
creativity
 Elgammal and Saleh (2015) unintentionally merges Boden
and Carter.
 Use computer algorithm to rank 1,710 images of art work
from 1412-1996 according to their quantifiable creativity
score.
 H-creativity in paintings along the historical timeline
 “to be creative it is not enough to be novel, it has to be
influential as well (some others have to imitate it)”
(Elgammal & Saleh, 2015, p. 41) in line with Carter’s
hypothesis of creativity in common talk
[See
http://www.huffingtonpost.com/2015/07/06/creativity-
algorithm_n_7700002.html]
30

31. 31
Is this ‘creative’?
Graphics and Digital Arts

32. 32
Is this ‘creative’?

33. Example
Symbol
type
Symbol
Creativity
Type
Formula of
construction
Reference
style
Source -- -- --
Unit
pattern-
reforming
Implicit
exo-
referenced
Logo
pattern-
forming
Implicit
endo-
referenced
33