Creativity and Sensation:

The Case for 'Synaesthetic Media'

John A Waterworth

Department of Informatics
Umeå University
S-901 87 UMEÅ, Sweden
tel. +46 90 786-67-31


The most salient and vital aspect of interacting with computer systems is consistently overlooked. That is the importance of computer systems as perceptual rather than conceptual tools. Insofar as people interact with them, computer systems function primarily as sensual transducers, which I term 'synaesthetic media', and not as so-called 'cognitive artifacts'. Synaesthetic media are the result of focusing design efforts on the sensational possibilities of human-computer interaction (HCI). My claim is that such computer tools can serve as powerful supporters of human creativity. Rather than expending more effort on the fruitless quest for 'cognitive artifacts', we need to recognise that we are already creating synaesthetic media and to direct our HCI design efforts accordingly.


Cognitive psychology has been the most important influence on our understanding of the process and significance of human-computer interaction. But modern cognitive psychology has, until recently, been diverted from a core aspect of its historical concerns - the nature of human experience. The term 'cognition' has come to mean only the storage and processing of symbolic information, in a way that is presumed to be analogous to how a digital computer might perform these functions (a school of thought often referred to as "cognitivism"). Because of this, and because of the relatively large number of cognitivist psychologists who have influenced its development, HCI is widely construed as the art and science of designing systems which directly support rational problem-solving activities in the minds of their users, a design activity often called 'cognitive ergonomics'. The resultant systems are currently termed 'cognitive artifacts' or even, quasi-medically, 'cognitive prostheses' [1]; they are supposed to improve the quality of human thinking, to make us "smarter" than we otherwise would be [2].

The cognitivist assumption underlies much work in artificial intelligence (AI) and cognitive science. However, neither of these two subject areas is synonymous with, nor even imply, the cognitivist assumption. Indeed, there has been a strengthening trend away from cognitivism in recent years [3, 4, 5]. But in the HCI community the dominant bias is still towards cognitivism. This is unfortunate, indeed, since if creativity depends on the inter-relatedness of reason and the non-computational aspects of cognition (in particular, sensation), then it follows that cognitivism cannot account for creativity and that 'cognitive artifacts' cannot increase creativity; nor can they support much of real human cognition, which embraces the non-computational aspects of sensation, imagination, emotion, and fantasy as well as more plausibly computational faculties such as mental problem solving (reasoning).

My claim is that most computer artifacts now in interactive use function not as 'cognitive' tools (tools which help the user process information better) but as sensual enhancers, as essentially perceptual artifacts. And this trend will get progressively more pronounced. Put simply, computer tools do enhance perceptual, experiential capabilities directly, they do not enhance reflective, reasoning capabilities (except indirectly through perceptual enhancement or by replacing those abilities with AI). We cannot help people to think better, but we can allow them to experience more. HCI design is thus mostly a matter of doing sensual ergonomics. (In an earlier paper I named this activity 'sensory ergonomics' [6], but that term suggests fine-grain analysis of human sensory systems, which is not my aim here.) By "sensual" I mean that which relates to perceptions arising from rich stimulation of the sensory systems - of the senses rather than reason - deriving from the sensualist (or sensationalist) assumption that knowledge derives ultimately from bodily sensation, not only from purely mental constructions of reality.


Recent developments in information technology, in particular of hypermedia, multimedia and virtual reality, have challenged the traditional view of HCI and, indeed, of what information technology is all about. The fact that these technologies do not seem to be producing tools that are much use for supporting 'cognition' is seen as a problem to be designed around, or explained away. Norman [2], for instance, points to what he sees as the problem that we tend to produce experiential artifacts and not reflective artifacts (i.e. we produce artifacts that stimulate the senses rather than aiding thought). But what is a reflective artifact in his terms? He gives examples of experiential artifacts - telescopes, petrol gauges, movies and recordings - but where are the reflective artifacts? What would they be like? Do 'experiential artifacts' such as telescopes and microscopes not provide the experiences that make it more worthwhile for us to be reflective? Is it not simpler and more accurate to suggest that most, if not all, interactive artifacts provide us with alternative experiences of the world, experiences we could not have without the artifact? We then may reflect on the experience, or not, according to our predilections, purposes and the nature of the experience.

The only convincing 'reflectors' are natural language and other schemes for information representation, including visualisation and its non-visual analogues (such as 'auralisation'). But visualisation is not a 'cognitive artifact'; it does not (nor does it attempt to) support rational problem solving. Rather, it presents existing information in a way that changes the sensual experience of observers, as compared to the original formulation of that information. In other words, visualisation and related techniques of media-translation act as sensory transducers that change human perception not human problem solving. This change may then result in the observer having new insights into the information presented, insights that will affect how he reasons about the phenomena. Such creativity arises from support for rich sensual experience, not from enhanced reasoning abilities.

The development of hypertext brought a kind of crisis to the field of HCI and a re-examination of what it is all about [7]. Initially, the non-rational nature of hypertext was explained away by suggesting that the loose linking of fragments of information in hypermedia in some way mirrors how information is really organised in the human mind. Since we can use this mind to do 'cognitive' work, it was assumed that hypermedia, which is essentially similar, must be useful to help us do mental information processing. The fact that it clearly wasn't useful for this kind of task was not, for some time, allowed to get in the way of what must, on narrowly 'cognitivist' assumptions, be the case. The problem of how one loosely organised network could communicate well-structured arguments to another - what, for example, would be the medium of exchange? - was simply ignored and often still is [8]. Some saw the delinearisation of nuggets of knowledge as liberating but, as thought is often considered to be linear, the resulting loss of narrative integrity was seen by others as mere incoherence. Hypermedia was not merely non-linear, it also had sound, video action, 3D graphics and animation; all of which might be expected to support the idea that the senses, rather than the intellectual faculties, are the main beneficiaries of this new technology. Multimedia was then seen as the general class of which hypermedia is a member; a class favouring the immediate gratification of "mediacy" rather than the deferred and effortful fruits of 'educated literacy' [9]. Put baldly, multimedia was seen as encouraging experience, not reflection, and this was, and still is, generally seen as a problem. Might this be not because the nature of knowledge has changed, but rather that we expect the wrong kind of thing from our computerised artifacts?

Virtual reality brought us, amongst many other possibilities (both negative and positive), the "concretization" of knowledge. An example is that of 'Newton World' [10], an immersive VR where children can experience Newton's laws of physics directly. They can vary such factors as an object's coefficient of restitution and internal friction, then experience from various viewpoints, including that of the objects themselves, what happens when objects of differing masses collide. One claim is that this kind of experiential learning permits children intuitively to grasp concepts which they find extremely difficult to understand (or simply do not believe) when presented in more traditional ways, say as a set of rules and associated formulae. But some complain that this is not the way to teach physics, that the VR experience is both misleading and not generalisable, since "to experience is not to learn". This clearly is a primarily experiential system, but aimed at producing an understanding of difficult concepts. In other words, experience is enriched, and this is meant to lead to better reflection on the topic in focus.

And now we have 'cyberspace', i.e. the Net and its popular multimedia component, the Web. Here we have hypermedia, multimedia, and, increasingly, VR rolled into one. As if that weren't problematic enough, we also have social interaction on a world-wide scale. It is no accident that computer artifacts are become both more sensational in nature, and more involved in how we communicate and cooperate with others, at exactly the same time. We communicate in contexts by interpreting events in terms of what is relevant in a situation [11]. Communication is often concerned with sharing experiences, not merely facts. This is borne out by work [12, 13] on what has been called 'Distributed Cognition', which focuses on how co-operating, communication and true cognition in time-sensitive practical situations are intimately connected. Artifacts, and other people, present information to the senses according to what is appropriate for the task; the necessary, rapid communication involves inherently shared sensations and awareness of that sharing.

The development of new technologies, in particular multimedia systems, virtual realities and cyberspace, has been seen as problematic because of their emphasis on sensation and communication. This trend, away from reflection, away from a focus on thought and understanding, this de-intellectualisation of knowledge, is seen as dangerous. But it is also an opportunity to experience the phenomena more richly before we reflect on their meaning. In doing sensual ergonomics, we are giving normal people the gift of synaesthesia.


I am using the word synaesthesia to mean the experience of information that is usually perceived in one form in a radically different form. In this paper I am making a simplistic distinction between 'sensation' and 'perception', to the effect that a sensation is what impinges on the sense organs, and a perception is what is experienced most directly as a result (see Figure 1). There is a normal mapping between these two such that a sensation of sound waves impacting on the auditory apparatus is perceived as audible sound. But in synaesthesia, a sensation that normally produces one kind of perception actually produces another kind, so that the sensation produced by sound waves might be perceived as varying colour or touch textures [14, 15].

Figure 1. Synaesthesia: Cross-modal mapping between sensations and perceptions

Synaesthesia has been known throughout history; it is relatively rare, especially in adults, and it is often associated with creativity. While, at times, the experience of synaesthesia can be problematic for the individual concerned, it is generally seen by them as life-enhancing. People with this condition (or ability) seem to have a richer experience of reality than the rest of us.

Synaesthetes experience sounds as visual phenomena (colours, graphics, patterns), visual stimulation as sounds, have colour associations of rooms or letters, tactile experiences of sounds, and so on. Naturally-occurring synaesthesia in people is, however, not under their control to any great extent. And it tends to fade with age. The key lesson of synaesthesia in people is that reality has no particular form. It does have content, which may be experienced in a variety of ways. To apprehend reality as fully as we can, we need to experience it in as many forms as possible.

From the perspective of synaesthesia in people, we can start to see multimedia, virtual reality and other recent developments in information technology not as failed attempts to support cognition, but as a promising start to the enhancement of perceptual experience, to broadening the experience of reality of users of these systems. In other words, what we are developing with these new technologies are artifacts that can mimic synaesthesia but that are under the control of their users. I call such technology 'synaesthetic media' (see Figure 2). They change how information is perceived, by changing the modality of sensations produced by its display. (I use the word "media" rather than, say, "artifact" to reinforce my emphasis on form, rather than content, and to stress the communicative aspects of sensation.)

Figure 2. Synaesthetic media: Selectable modalities of information display

There are many examples of work to develop the capability to take information in one modality, transform and present it in one or more other modalities (see Figure 2). A relevant project with which I was involved [16] developed a computer system that recorded electric piano performances as MIDI data. The obvious way to display these data would be as sound, to be experienced as a repeat of the acoustic events that were produced during a performance. But the system we developed aims to be a medium of communication between a piano student and his or her teacher. It does this by allowing piano performances to be displayed visually, as graphical annotations to the score that was played. The display can be chosen to reflect different aspects of the performance, such as dynamics, tempo and articulation. The displays comprise a synaesthetic medium of communication between teacher and student that, unlike the original performance, is not ephemeral. It is significant that the visual displays were found to enhance listening ability; both students and teachers heard things in a performance after viewing the displays that they would otherwise not have detected. Their musical perception was enhanced. Other recent work has included the visualisation of numerical data, the 'auralisation' of information, the graphical display of motor performance, the use of gestures for musical composition, and associating information with colour and/or spatial location. These can all be seen as attempts to support direct perceptual experience and de-emphasise 'cognitive' interpretations by mimicking natural synaesthesia with computer technology.


Synaesthetic media are already appearing. They allow us to choose the sense modalities through which information is perceived, in an analogous way to that in which human synaesthesia alters how sensations are perceived (but without the perceiver having a choice). In most cases, we have not seen them for what they are, and have not seen this development as a general trend. Instead, we worry about not being reflective enough, and about decontextualisation, anti-intellectualism, and other perceived "problems". We try to find ways in which the computer systems that include these features really support 'cognition' not new experiences. Rather than being seen as a problem for 'cognitive ergonomics', however, recent developments such as these present wonderful opportunities for supporting human creativity. They enable us to become more aware that what we are doing when interacting with computers is essentially perceptual, not 'cognitive' in nature, and to start to do sensual ergonomics consciously and well. As an HCI researcher, I think the need is quite urgent. The HCI community is becoming irrelevant with the advent of new technology; for example, hypermedia research has had little or no impact on the way the Web (by far the most popular hypermedia system in use) has developed. It is time to revise HCI design aims based on a better understanding of the nature of computer artifacts. We need to design for creativity, and I suggest work towards richly synaesthetic media as a practical approach to that worthy aim.

Resistance to the idea that computer systems could and, in particular, should support sensation is long-standing. Our educational tradition equates cognitive with "good", and sensual-experiential with "bad". Reason attempts to make up for what we cannot, or should not, experience directly. We have been pursuing what follows from this since the Enlightenment, and in recent years have tried to support the endeavour with information technology. But, as I have suggested, computer systems are better at directly supporting sensation than 'cognition' (rational problem-solving) in their users. Information technology is now poised to deliver a richness of experience that was previously unavailable to most people. We need not reject reason in the face of this change, but should see it as only part of human mental life. The other, the experiential, is by its nature irrational, in the sense of being beyond rational debate. These two are merged in true cognition; and hence arises the potential for creativity.

What I am saying is not really new. We are starting to see a realisation that existing approaches to developing 'cognitive artifacts' need to be rethought; they do not even support that ostensibly most intellectual group of users, "knowledge workers" [17]. Current computer tools do not help such people in their work, they hinder them. Various commentators have talked about expanded perceptual capacities in the new information age, but this has not previously been seen as the obverse of the inadequacy of 'cognitive artifacts'. And these two developments have not, until now, been seen as intimately related nor as pointing to a possible way of designing computer technology to support human creativity.

We are at a crucial point in the development of technology and its impact on society. To me, the purpose of our new technology is to broaden our channels of sensation (and communication) to experience reality more fully; to make us more creative in the face of life's challenges. I think it is plausible to suggest that progress towards this goal may depend more on technologically-expanded sensual experience than on computer-supported reason.


This paper summarises a presentation to the Second International Creativity and Cognition Symposium, which was held at Loughborough University and College of Art and Design from 29 April to 2 May 1996.


1. Wright, P., "Cognitive Overheads and Prostheses: Some Issues in Evaluating Hypertexts", Proceedings of Hypertext '91 (New York: ACM, 1991) p1-12.

2 Norman, D. A., Things that Make Us Smart (Reading, Mass: Addison-Wesley, 1993).

3. Varela, F. J., Thompson, E. and Rosch, E., The Embodied Mind: Cognitive Science and Human Experience (Cambridge, USA: MIT Press, 1991).

4. Johnson, M., The Body in the Mind (University of Chicago Press, 1987).

5. Johnson, M., Moral Imagination: Implications of Cognitive Science for Ethics (Chicago: University of Chicago Press, 1993).

6. Waterworth, J. A., "HCI Design as Sensory Ergonomics", Proceedings of IRIS-18 (Gothenburg Studies in Informatics, Report 7, 1995) p743-753.

7. Waterworth, J. A. and Chignell, M. H., "A Manifesto for Hypermedia Usability Research", Hypermedia 1, No. 3, 205-234 (1989).

8. Ginige, A., Lowe, D. R. and Robertson, J., "Hypermedia Authoring", IEEE Multimedia 2, No. 4, 24- 35 (1995).

9. Waterworth, J. A., Multimedia Interaction: human factors aspects (Chichester, UK: Ellis Horwood, 1992).

10. Dede, C., "Evolving from Multimedia to Virtual Reality", Educational Multimedia and Hypermedia Annual (Charlottesville, VA: Association for the Advancement of Computing in Education, 1993) p123-130.

11. Sperber, D. and Wilson, D., Relevance (Oxford: Basil Blackwell, 1986).

12. Hutchins, E., "The Technology of Team Navigation", Intellectual Teamwork: the Social and Technological Foundations of Cooperative Work (Hillsdale, NJ: Lawrence Erlbaum, 1990).

13. Heath, C. and Luff P., "Collaboration and Control: Crisis management and multimedia technology in London Underground line control rooms", Computer Supported Cooperative Work 1, 69-94 (1992).

14. Marks, L., The Unity of the Senses: Interrelations among the Modalities (New York: Academic Press, 1978).

15 Cytowic, R. E., "Synesthesia: Phenomenology and Neuropsychology", Psyche 2 (1995).

16. Smoliar, S. W., Waterworth, J. A. and Kellock, P. R., "pianoFORTE: a System for Piano Education beyond Notation Literacy", Proceedings of ACM Multimedia 95 Conference (New York: ACM, 1995) p457-465.

17. Kidd, A L, "The Marks are on the Knowledge Worker", Proceedings of CHI'94 Conference on Human Factors in Information Systems (New York: ACM, 1994) p186-191.