Department of Informatics,

Umeå University,

S-901 87 UMEÅ, Sweden

jwworth@informatik.umu.se

Extended Abstract

This presentation is in two related parts. The first considers the range of applications currently addressed through VR and suggests a spectrum of techniques for evaluation, ranging from objective "outside-in" assessments, to subjective "inside-out" experience. The second part relates this range to two 3-dimensional models: one of information exploration, the other of states of mind. It is suggested that presence in a broad range of VR applications can be characterised and then assessed in terms of these models.

The range of applications currently addressed through VR stretches from very constrained practical tasks for dedicated professionals - such as surgeons wanting to plan a minimally-invasive route for brain surgery - to artistic experiences that radically, if temporarily, change how people feel about themselves, their bodies, and the world around them. It makes little sense to try to apply the same evaluation techniques across this broad range. In HCI studies, it has been traditional to think of evaluation as something that revolves around users' purposes in terms of the tasks they wish to perform. However, this is too narrow a focus for recent technologies such as multimedia and VR. To address this broad range of applications of VR, we can put forward a spectrum of techniques for evaluation, ranging from objective outside-in assessments, to subjective inside-out experiences. I will briefly illustrate this spectrum with a few examples in the following text, and the main points are summarised in Table 1 below.

The most distant, outside-in perspective on VR evaluation is typified by an HCI or other expert not involved in developing or designing a particular VR application making an expert assessment. This is somewhat analogous to an outsider evaluating a factory or university on the basis of short reports from the institution. The expert does not experience the VR, does not observe others experiencing the VR, and makes no objective measures of performance in the VR - although he may rely on reports of such measurements carried out by others. This sort of approach underlies at least parts of a recent survey of applications of VR in medicine [1].

Attempts to extract objective measures of performance in virtual reality represent a step inwards from the expert perspective, above. These are often carried out by the designers and developers themselves, to evaluate the success or otherwise of a particular design or implementation. In [2], we describe the design and evaluation of two types of selector tool for use in medical VR applications (HCI aspects of the environment are described more fully in [3]). In this account, we provide details of the background behind the research and focus on design aspects of selector tools for use in a particular task context, that of medical surgery planning. The evaluation is typical of a traditional HCI approach, since it assumes that there are criteria - such as time and accuracy - against which performance can be measured.

The range of VR, and of related technologies such as hypermedia, goes beyond tasks, however, and seems to imply a more subjective approach to evaluation, not least because pleasurable experiences are seldom assessed along the same dimensions as work-related tasks. To put it simply, it's generally good to complete work tasks quickly, but often preferable for pleasant experiences to last as long as possible. VR can serve as both a task-related tool and a provider of experiences, along with other fairly recent innovations such as educational games, Web-surfing, and socialising in MUDs. In these examples, it is very hard to correlate the quality of the environment or tool with time or other performance measures. While it might be good to explore an information space very briefly, because that means you quickly found what you wanted (your purpose was satisfied), a long browse session might mean you have found lots of interesting stuff you weren't even looking for (you had no definite purpose, but you had a satisfying experience). I suggested the potential value of "traveller's tales" for evaluation of interactive experiences where a task-based approach is not appropriate [4].

But the problem of evaluation is more than just tasks versus experiences. In artistic and some recreational applications of VR, the effect may be to render the immersant more or less incapable of giving a coherent account of his or her experience. People who have experienced emotionally very powerful installations such as Char Davies' Osmose will testify to the difficulties of expressing the nature of the event. In this respect, VR is rather like a recreational drug, and the more powerful and effective it is the less likely we are to be able to give coherent subjective accounts.

There are, however, some possibilites for objective measures, such as skin resistance, heart and breathing rate, pupil dilation, but these are almost never unambiguous.

Table 1 - Spectrum of VE Applications and Evaluation Types*

*Note that this does not pretend to be comprehensive, and there will be much combining and crossing-over between cells; for example, physiological measures might be disambiguated with subjective reports of experienced duration of a session, and so on.

It will never be possible to define presence, but it may be possible to identify dimensions of variation affecting the overall sense of presence in particular VRs. In [6], a model of human information exploration was characterised along three axes. This model has stood the test of time, with a few slight changes of terminology. We now think of information exploration activities as being located in a space defined by the dimensions of structural responsibility, interaction loop speed, and specificity of user requirements. These dimensions reflect how individuals deal with the information that is the product of exploration, and in this, a consideration of consciousness and the emphasis of conscious activities is central. This is not surprising of course, since mind can be viewed as an exploring system with the function of delivering relevant information to consciousness. So we find that the dimensions of this information exploration system - the mind - map nicely onto those of our model (Figure 1).

Figure 1a and b: Dimensions of Mind and of Information Exploration

Combining these models with our spectrum of assessment techniques can provide a view of both degree and characteristics of presence, informed by a consideration of the information exploration activities going on in the mind of the user(s).

References

[1] Waterworth, J A (1998) - VR in Medicine: aSurvey of the State of the Art.

[2] Serra, L and Waterworth, J A - 'Designing Virtual Selectors for Surgeons'. Applied Ergonomics, 28 (4), 1996, 269-275.

[3] Serra, L, Poston, T, Ng, H, Chua, B C and Waterworth, J A - 'Interaction techniques for a virtual workspace'. Paper and video presented at the International Conference on Artificial Reality and Tele-Existence (ICAT) /Conference on Virtual Reality Software and Technology (VRST) '95, Makuhari Messe, Japan, November 1995.

[4] Waterworth, J A - 'Personal Spaces: 3D Spatial Worlds for Information Exploration, Organisation and Communication'. In R. Earnshaw and J. Vince (eds.): 'The Internet in 3D: Information, Images, and Interaction'. San Diego, USA: Academic Press, 1997.

[5] Waterworth, J A - 'Technology in Support of Returning - From Conscious Doing to Consciously Being'. International Conference on "Science and the Primacy of Consciousness". Lisbon, Portugal, April 1998.

[6] Waterworth, J A and Chignell, M H - 'A Model of Information Exploration'. Hypermedia, 3 (1), 1991, 35-58.