The Mobile Information Society
Conference, May 24-25, 2002
Hungarian Academy of Sciences
|
The Mobile Information Society |
|
|||
|
Conference, May 24-25, 2002
|
||||
|
|
|
|
|
||
Pictorial Meaning and Mobile Communication
Summary
Thinking in Images
Convention and Resemblance
Knowledge and Visual Communication
MMS Arrives
Pictorial Communication and Mobile Communities
A picture, as the saying goes, is worth a thousand words. Words are
of course often spoken hurriedly when voiced through the mobile phone,
and they have to be used sparsely when composing an SMS message. The assumption
that in mobile communication pictures could be usefully employed should
then not come as a surprise. But why, exactly, would a picture be worth
a thousand words? Sometimes, indeed, the opposite seems to be the case.
Words are needed to index, explain, and disambiguate pictures, as well
as to express abstract concepts, logical relations, and linguistic modalities.
In my talk I will, first, present arguments to the effect that images,
not words, are the primordial stuff of thinking, and that, consequently,
physical pictures, too, are likely instruments of thought, and can serve
as convenient vehicles of communication. Secondly I will argue that although
pictorial communication is seldom entirely successful if not accompanied
by words, and any visual language needs the background of convention, pictures
can indeed function as natural symbols due to their resemblance to the
objects and facts represented. Thirdly I will stress that, precisely because
they resemble what they represent, pictures are eminently suited for conveying
visual information. However, the employment of pictures for the communication
of knowledge was impeded, throughout the millenia of alphabetic literacy,
by the limited means for the creation and duplication of graphics. This
has changed dramatically with the new capabilities we enjoy thanks to computers.
And with the advent of multimedia messaging, as I will then point out,
devices by which to create and communicate pictures will become ubiquitous.
By way of conclusion I will suggest that with visual elements re-entering
the process of communication, and with communication remaining continuous
even over great physical distances, personal relationships can retain or
regain an intimacy that has been largely lost in the world of modern communications,
an intimacy recalling the condition of close communities.
The story commences with Plato and Aristoteles, with important new beginnings
in the late 1960s and early 1970s - I will mention one of them in a minute
- but let me here start by referring to the book
Descartes' Error
by a leading neurophysiologist, Antonio Damasio, published in 1994.(1)
It is in the form of images, Damasio holds, that the factual knowledge
required for reasoning and decision-making is present to our minds. Images
are not stored as facsimile pictures of things, or events, or words, or
sentences. We are all aware, writes Damasio, that in recalling a face,
or an event, we generate not an exact reproduction but rather some sort
of re-interpretation, a new version of the original which will in addition
evolve over time. On the other hand however we all equally have the sensation
that we can indeed conjure up, in our mind's eye, approximations of images
we previously experienced. Images form the main content of our thoughts.
Of course "hidden behind those images, never or rarely knowable by us",
there are numerous processes that guide the generation and deployment of
images. "Those processes ... are essential
for
our thinking but are not a content of our thoughts."(2)
Another important book appearing in 1994 was the linguist Ray Jackendoff's
Patterns
in the Mind.(3)
Jackendoff, too, reminds us that the brain does not actually store images;
however, he certainly believes that research on mental imagery is not anymore,
as he puts it, "forbidden fruit", and in fact he begins his argument, not
quite
unseriously, by drawing a picture of a mental image!
Recall that throughout the twentieth century the view that visual images play a substantial role in rational thought, and that pictures are important carriers of information, was a minority position in philosophy. The position was defended by Russell, who in 1919 wrote: "The habit of abstract pursuits makes learned men much inferior to the average in the power of visualization, and much more exclusively occupied with words in their 'thinking'."(4) And he added: "The 'meaning' of images is the simplest kind of meaning, because images resemble what they mean, whereas words, as a rule, do not."(5) Russell's view was echoed by H.H. Price in 1953, in his Thinking and Experience. As Price wrote: "We have the misfortune to live in the most word-ridden civilization in history, where thousands and tens of thousands spend their entire working lives in nothing but the manipulation of words. The whole of our higher education is directed to the encouragement of verbal thinking and the discouragement of image thinking. Let us hope that our successors will be wiser, and will encourage both."(6) Price here also made the telling remark: "some people are almost incapable of drawing".(7)
A philosopher who was certainly capable of drawing, and indeed employed
graphics to explain difficult points in his arguments was Ludwig Wittgenstein.
His Nachlaß contains some 1400 sketches and diagrams. He held
that word languages on the one hand, and the language of pictures on the
other, function jointly, acting on each other; that pictures, like words,
are instruments embedded in our life. However, while words are predominantly
conventional, pictures are in essential 
respects
natural carriers of concrete meanings. Let me here just give two brief
quotes from his volume entitled
Philosophical Grammar. The first:
"Thinking is quite comparable to the drawing of pictures." The second:
"for the picture to tell me something it isn't essential that words should
occur to me while I look at it; because the picture should be the more
direct language."(8)
A fundamental work arguing for the pictorial nature of thought was Rudolf
Arnheim's 1969 book Visual Thinking. "Nobody denies", wrote Arnheim,
"that language helps thinking. What needs to be questioned is whether it
performs this service substantially by means of properties inherent in
the verbal medium itself or whether it functions indirectly, namely, by
pointing to the referents of words and propositions, that is, to facts
given in an entirely different medium. Also, we need to know whether language
is indispensable to thought. - The answer to the latter question is 'no'.
Animals, and particularly primates, give clear proof of productive thinking.
... However, animal thinking may be inferior to that of humans in one important
respect. It may be limited to coping with directly given situations."
Arnheim's book had an influence on Merlin Donald's work Origins
of the Modern Mind, published 1991. (Another important influence on
Donald I should here mention came from Dunbar's earlier writings.) In his
talk yesterday Csaba Pléh gave a detailed account of Donald's theory.
This theory distinguishes three evolutionary transitions in the development
of humankind. The first transition, from apes to Homo erectus, was
characterized by "the emergence of the most basic level of human representation,
the ability to mime, or re-enact, events". The second transition, from
Homo
erectus to
Homo sapiens, completed the biological evolution
of modern humans. "The key event during this transition", writes Donald,
"was the emergence of the human speech system, including a completely new
cognitive capacity for constructing and decoding narrative." To the third
transition, the emergence of "external symbolic storage", Donald allots
"three broadly different modes of visual symbolic invention", which he
designates as "pictorial, ideographic, and phonological". Of these, the
pictorial mode emerged first; and the point Donald makes is that this signaled
the beginnings of "a new cognitive structure", already enabling some primitive
forms of "analytic thought", i.e. "formal arguments, systematic taxonomies,
induction, deduction".(9)
Striving to enrich mobile communication by a visual language involves
a twofold task. First, a unified system of appropriate conventions has
to be introduced. Secondly, the potential of pictorial likeness as a natural
dimension for conveying meaning should be exploited. Note however that
the border between resemblance and conventionality is not a sharp one.
Natural gestures and facial expressions, for instance, can become conventional;
and conventional signs can come to be treated as natural ones. Wittgenstein
has some interesting remarks on this. Thus he wrote in connection with
a familiar pictorial convention: "Das Symbol des gesprochenen Wortes Schriftzeichen
in einer Schlinge die aus dem Mund des Sprechers kommt. Dies Bild erscheint
uns ganz natürlich, obwohl wir doch dergleichen nie gesehen haben."(10)
("The symbol of the spoken word: characters in a loop which emerges from
the mouth of the speaker. This picture strikes us as quite natural, although
we have never seen anything like it.) Indeed it is not just the speech
bubble as such that we have come to experience as a natural sign, but also
its particular varieties. As William Horton indicates, convention and intuition
both play a role in the family of speech balloon symbols. He presents speech
balloons of various shapes, and prompts us to consider what meaning they
convey. "What kind of message", he asks, "would you expect each of these
speech balloons to deliver?"(11)
And indeed the various forms do suggest to us different moods, sentiments,
meanings - even though we might have never encountered them before, have
never learnt any conventions relating to them. Speech bubbles have been
with us for centuries; they have evolved from ancient and medieval speech
bands.
 |
 |
In the so-called Part II of the Philosophical Investigations
Wittgenstein makes it quite clear that for him there are some kinds of
pictures which convey unambiguous meanings even though we have never been
taught how to interpret them. He introduces the example of a "picture-face",
and remarks: "In some respects I stand towards it as I do towards a human
face. I can study its expression, can react to it as to the expression
of the human face. A child can talk to picture-men or picture-animals,
can treat them as it treats dolls."(12)
There are pictures we do not interpret at all, but react to, as Wittgenstein
puts it, in an
immediate way. Whether
we do so react, can be influenced by "custom and upbringing"(13),
such influence however is sometimes a very slight one. Wittgenstein's view
has been corroborated by John Kennedy's findings in his 1974 book A
Psychology of Picture Perception.(14)
Understanding photographs or line drawings generally does not presuppose
any previous training in pictorial conventions. At the same time Kennedy
points out that static pictures are not always unequivocal. He refers to
a certain crowd scene about which it has been noted that some African people
tend to interpret it as showing people fighting, whereas other African
people may see the same scene as part of a dance. "Frozen pictures tend
to be ambiguous, of course", Kennedy writes, "and the viewer's culture
can be expected to predispose him toward one imaginative story rather than
another".(15) Writing
at the time he did, it is perhaps understandable that the idea of animations
as a means of disambiguation does not occur to Kennedy; both Price and
Wittgenstein however did hit on that idea when confronted by the problem
of ambiguous pictorial meaning.
Knowledge and Visual Communication
In his paper "Visualization and Cognition" Bruno Latour points to "writing
and imaging craftmanship"(16)
as the ultimate ground of modern science. Through the technologies of writing
and pictorial representation the objects of cognition become mobile,
and at the same time immutable; they can be collected, presented,
and combined with one another in the power centres of knowledge.(17)
A work Latour is particularly indebted to is the brilliant book by William
Ivins, Prints and Visual Communication, published in 1953, and now
at long last translated into Hungarian. The observation with which Ivins
begins is that "the backward countries of the 
world
are and have been those that have not learned to take full advantage of
the possibilities of pictorial statement and communication. … many of the
most characteristic ideas and abilities of our western civilization have
been intimately related to our skill exactly to repeat pictorial statements
and communications."(18)
Ivins makes the fundamental point that the lack of a proper technology
for duplicating pictures was a major obstacle to the development
of science throughout most of Western history. Picture printing was invented
around 1400 a.d. Ivins argues that this was a much more revolutionary invention
in the history of communication than that of typography half a century
later. Pictures became more or less exactly repeatable. However, they were
still a long way from being faithful copies of particular natural objects.
Ivins points out that when Lessing wrote his famous treatise on the Laocoon
group, he did not, because he could not, have reliable illustrations at
his disposal. Until the age of photography, as Ivins stresses, there existed
no technology of exactly repeatable pictorial representations of particular
objects.
It was perhaps the main discovery of twentieth-century philosophy that
all
knowledge, ultimately, is based on practical knowledge. Now pictures are
better at teaching practical knowledge than are texts. It was not
by chance that Otto Neurath, the ardent advocate of the logical positivist
thought of a unified science, conceived of the idea of complementing his
planned compendia by an international picture language.(19)
Neurath was working towards an "International System Of Typographic Picture
Education", abbreviated as isotype, an interdependent and interconnected
system of images, to be used together with word languages, yet having
a visual logic of its own. Isotype was to use distinctive conventions,
shapes, colours, and so on. "Frequently it is very hard", Neurath wrote,
"to say in words what is clear straight away to the eye. It is unnecessary
to say in words what we are able to make clear by pictures."(20)
Neurath particularly stressed that the elaboration of his picture language
was meant to serve a broader task, that of establishing an international
encyclopaedia of common, united knowledge.(21)
However, he never even came near to realizing his lofty aims. His experiments,
conducted from the 1920s to the 40s, turned out to be technologically premature.
The icons 
elaborated
within the framework of the isotype program have served as models for those
international picture signs we today daily encounter at airports and railway
stations, but - because they are so crude, and so cumbersome to produce
- they could not form the basis of a true visual
language. With
the iconic revolution we today witness, such a language is clearly becoming
feasible.
Ivins recalls that during his time at the Metropolitan Museum he again and again had to experience "how inadequate words are as tools for description, definition, and classification of objects each of which is unique". Words can never, as he puts it, "catch the personality of objects which we know by acquaintance", whereas "pictures or images" can.(22)
Pictorial communication has obvious advantages; but also obvious drawbacks. In his book The Search for the Perfect Language Umberto Eco cites with approval all the usual arguments against visual languages - ambiguity, lack of grammar, the need for conventions, limited applicability. "One could say", he writes, "that there is only a single system which can claim the widest range of diffusion and comprehensibility: the images of cinema and television. One is tempted to say that this is certainly a 'language' understood around the earth." However, he adds, "if there is no difficulty involved in receiving cinematic or televised images, it is extremely difficult to produce them. Ease of execution is a notable argument in favour of verbal languages. Anyone who wished to communicate in a strictly visual language would probably have to go about with a camcorder, a portable television set, and a sackful of tapes, resembling Swift's wise men who, having decided that it was necessary to show any object they wanted to designate, were forced to drag enormous sacks behind them."(23)
As I have tried to make clear in the foregoing, I do not advocate communication
in a strictly visual language. But I certainly believe that complementing
verbal - voiced or written - communication with a pictorial dimension can
enhance the effectivity of information exchange. And Swift's wise men are
facing a task that is 
increasingly
easy to perform. MMS - multimedia messaging service - in the introduction
of which Hungary has witnessed pioneering steps just a few weeks ago, constitutes
a significant new phase. MMS allows users of mobile phones to take snapshots
with
built-in
cameras and immediately post them; to create line drawings, edit pictures,
add text to graphics, and send the complex messages thus created. Such
capabilities deserve our attentions on at least four counts. Having as
it were a drawing block handy all the time, with the continuous possibility
of communicating sketches and having them applied to practical tasks at
the receiving end, goes at least a small way towards helping to solve the
problem Price complained about, namely that few people know how to draw.
The collection and combination of those mobile and at the same time immutable
objects of cognition Latour regards as the foundation of Western intellectual
superiority will cease to be the exclusive prerogative of "power centres
of knowledge" once they really become mobile, and once "imaging craftmanship"
becomes a widespread art. Being able to take photos and send them away
on the spot surely alleviates the predicament Ivins pointed at when he
wrote that mere words cannot capture the unique characteristics of particular
objects. And the disequilibrium Eco alludes to, that producing images is
so much more cumbersome than viewing them, will become less marked once
the creating and disseminating of images becomes a common everyday skill.
Pictorial Communication and Mobile Communities
There is a memorable phrase by Neurath: "Words make division, pictures make connection".(24) Pictures connect people who are otherwise divided by speaking different languages. But let us observe that even among those who share the same tongue pictures have a greater potential to create common bonds than do words. The reason for this has been known since ancient times. Saint Bonaventure in the thirteenth century summed up a millennium of argument about the institution of images in the Church when he said, first, that the illiterate might learn from sculptures and from pictures as if from books, and second, that people who are not excited to devotion when they hear of Christ's deeds might at least be excited when they see them in figures and pictures.(25) The essential fact behind both observations of course is that understanding images, thinking in images, having feelings in connection with images, and even communicating in images - namely in mimetic patterns - is more basic to human nature than thinking and communicating in words. Again, this is not to say that words are dispensable. The significance of MMS is precisely that it combines images and words - as well as the spoken and the written. Communicating synchronously in voice, writing, and graphics has the potential to create and maintain a higher level of human cohesion than could be achieved by any of these dimensions by themselves. It was Karl W. Deutsch who applied the notion of complementarity, originally a concept in communications theory, to the issues of social communication.(26) In my introductory talk yesterday I referred to the distinction Deutsch made between society and community. Communities are characterized precisely by patterns of communication that display a high level of complementarity between information conveyed through various channels. Multimedia messaging, the synchronous-complementary transmission of speech, text and pictures seems to me to be not just a social activity but an activity sustaining those very types of human communication that make up genuine communities.
NOTES
1. Damasio, Descartes' Error: Emotion, Reason, and the Human Brain. New York: Grosset / Putnam, 1994.
3. Patterns in the Mind: Language and Human Nature, New York: Basic Books, 1994.
4. Bertrand Russell, "On Propositions: What They Are and How They Mean" (1919). Aristotelian Society Supplementary Volume, 2, pp.1-43. I am here quoting from J.G. Slater (ed.), The collected papers of Bertrand Russell, Volume 8: The Philosophy of Logical Atomism and Other Essays, 1914-19. London: George Allen & Unwin, 1986, pp.284f.
6. London: Hutchinson's Universal Library, p.252.
8. Philosophical Grammar, pp. 163f. - For a fuller treatment see my "Wittgenstein's Philosophy of Pictures" (2001).
9. Merlin Donald, Origins of the Modern Mind: Three Stages in the Evolution of Culture and Cognition, Cambridge, MA: Harvard University Press, 1991, pp.16f. and 278, 284, 273.
10. Pages 4r-5r of MS 159 (1938).
11. William Horton, The Icon Book: Visual Symbols for Computer Systems and Documentation (New York: John Wiley & Sons., Inc., 1994, p.69). On the emergence and the varieties of the speech bubble as a comics and cartoon convention, cf. also Carl G. Liungman, Dictionary of Symbols (New York: Norton & Co., 1991, pp.358f., original Swedish edition 1974), and Robert E. Horn, Visual Language: Global Communication for the 21st Century (Bainbridge Island, WA: MacroVU, 1998, pp.141f.).
12. Ludwig Wittgenstein, Philosophical Investigations, Oxford: Blackwell, 1953, p.194.
14. John M. Kennedy, A Psychology of Picture Perception: Images and Information, San Francisco: Jossey-Bass, 1974, see esp. pp.47-84.
16. Bruno Latour, "Visualization and Cognition: Thinking with Eyes and Hands", Knowledge and Society: Studies in the Sociology of Culture Past and Present, vol., 1986, p.3.
17. Ibid., p.7. - "Economics, politics, sociology, hard sciences", Latour writes, "do not come into contact through the grandiose entrance of 'interdisciplinarity' but through the back door of the file. ... domains which are far apart become literally inches apart", p.28. Latour refers to the "new branches of science and technology that can accelerate the mobility of traces, perfect their immutability, enhance readability, insure their compatibility, quicken their display: satellites, networks of espionage, computers", ibid., p.30.
18. William M. Ivins, Jr., Prints and Visual Communication, London: Routledge and Kegan Paul, 1953, p.1.
19. "Was wir Wissenschaft nennen, kann als die typische Art des Argumentierens angesehen werden, die den Menschen aller Nationen, reich und arm, gemeinsam ist. Diskussionen über Sonne, Mond, Sterne, Anatomie, Geographie, Freude und Schmerz können in jeder Zivilisation geführt werden; Theologie und Rechtsausdrücke haben andererseits hauptsächlich lokalen Charakter. - Es ist wichtig, das, was den Menschen gemeinsam ist, in einer Sprache auszudrücken, die möglichst einfach und neutral ist. Eine Bildersprache, die Hieroglyphensprache, hat den Vorteil, von der Wortsprache unabhängig zu sein, ist besonders geeignet, faktische Information auf vereinfachte Weise zu vermitteln, und hat eine gewisse Neutralität." (Otto Neurath, "Visual Education: Humanisation versus Popularisation" [1945], here quoted from the German translation: "Bildpädagogik: Humanisierung gegen Popularisierung", in Otto Neurath, Gesammelte bildpädagogische Schriften, Rudolf Haller - Robin Kinross, eds., Wien: Hölder-Pichler-Tempsky, 1991, p.649.)
20. Otto Neurath, International Picture Language, London: 1936, repr. University of Reading: Dept. of Typography & Graphic Communication, 1980, p.26.
23. Umberto Eco, The Search for the Perfect Language, Oxford: Blackwell Publishers, 1995, pp.174ff.
24. International Picture Language, loc. cit., p.18.
25. I here follow David Freedberg, The Power of Images: Studies in the History and Theory of Response, Chicago: University of Chicago Press, 1989, pp.162ff.
26. See esp. his Nationalism and Social Communication: An Inquiry into the Foundations of Nationality, London and New York: 1953, pp.69ff.