Analysis of Assets for Virtual Reality Applications in Neuropsychology

Abstract

Virtual Reality (VR) technology offers new opportunities for the development of innovative neuropsychological assessment and rehabilitation tools. VR-based testing and training scenarios that would be difficult, if not impossible, to deliver using conventional neuropsychological methods are now being developed that take advantage of the assets available with VR technology. If empirical studies continue to demonstrate effectiveness, virtual environment applications could provide new options for targeting cognitive and functional impairments due to traumatic brain injury, neurological disorders, and learning disabilities. This article focuses on specifying the assets that are available with VR for neuropsychological applications along with discussion of current VR-based research that serves to illustrate each asset. VR allows for the precise presentation and control of dynamic multi-sensory 3D stimulus environments, as well as providing advanced methods for recording behavioral responses. This serves as the basis for a diverse set of VR assets for neuropsychological approaches that are detailed in this article. We take the position that when combining these assets within the context of functionally relevant, ecologically valid VEs, fundamental advancements can emerge in how human cognition and functional behavior is assessed and rehabilitated.

I. Introduction

The field of neuropsychology has grown exponentially over the last three decades. Neuropsychologists have been leaders in providing an understanding of brain organization and brain behavior relationships, giving new insight into the nature and consequences of brain damage, disease and developmental disorders, as well as normal aging processes. Neuropsychologists have developed a wide range of measures to assess cognitive, sensory, and motor abilities, as well as behavioral and self-regulatory functions. The field has maintained high standards with regard to ensuring that neuropsychological (NP) measures are reliable and have adequate construct validity. However, a continuing and important challenge for neuropsychologists has been to find ways to better measure, understand, and predict everyday functional capacities (Wilson, 1997). Borrowing principals and themes from cognitive neuroscience, there has been a tendency to explain behavior by attempting to break it down into separate cognitive abilities or component parts. As a result, though perhaps theoretically useful, many NP tasks themselves appear quite dissimilar to the demands of everyday life. Given the potential mismatch between NP test demands and those of everyday functioning, the predictability of many commonly used NP measures for aspects of adaptive functioning and real life performance has been called into question. Some neuropsychologists have advocated 'top down' tasks, which require integration of a number of cognitive abilities and higher levels of self monitoring (Shallice & Burgess, 1991) to better emulate real life demands. Though such tasks are a step in the right direction, they fail to assess the impact of precise presentation and timing of subtle changes to stimuli and do not analyze response characteristics in any detail. Control or measurement of these aspects of tasks and task performance may be quite important in the prediction of actual everyday abilities and real life function.

Another domain in which cognitive and behavioral assessment play a critical role is in rehabilitation. The identification of useful rehabilitation goals and the measurement of meaningful rehabilitation outcomes are critically dependent on an accurate and reliable assessment of real-world adaptive functioning. Whereas NP assessment may be undertaken for multiple purposes, including diagnosis and description, rehabilitation planning and rehabilitation outcome assessment are critically dependent on tools and techniques that closely predict the individual's ability to function within natural contexts, with all their attendant stimuli and multiplicity of demands. Indeed, the most consistent concern with respect to rehabilitation techniques has been limitations in the ecological validity of the actual rehabilitation activities and resultant limitations in generalization of new abilities, knowledge, and/or skills (Carney, et al. 1999; Park & Ingles, 2001; Ylvisaker & Feeney, 1998).

II. Why Virtual Reality?

The development of virtual reality (VR) technology holds the potential to address many of these areas of concern. By its nature, VR is designed to simulate naturalistic environments. Within these environments, researchers and clinicians can present more ecologically relevant stimuli imbedded in a meaningful and familiar context. Rather than try to predict functional implications from a decontextualized measure of attention, for example, one can look at the effects of systematically increasing ecologically relevant attentional demands in a virtual environment (VE), such as a classroom, office, or store. VR technology allows for exquisite timing and control over distractions, stimulus load and complexity, and can alter these variables in a dynamic way contingent on the response characteristics of the client. Response characteristics in terms of accuracy, timing, and consistency can also be collected to allow a finer and detailed analysis of responses.

When discussion of the potential for VR applications in neuropsychology first emerged in the mid-1990s (Pugnetti, Mendozzi, Motta, Cattaneo, Barbieri, & Brancotti 1995; Rizzo, 1994; Rose, Attree & Johnson, 1996), the technology to deliver on the anticipated "visions" was not in place. Consequently, during these early years VR suffered from a somewhat imbalanced "expectation-to-delivery" ratio, as most users trying systems during that time will attest. The "real" thing never quite measured up to expectations generated by some of the initial media hype, as delivered for example in the films "The Lawnmower Man" and "Disclosure"! Yet the idea of producing simulated virtual environments that allowed for the systematic delivery of ecologically relevant cognitive challenges was compelling and made intuitive sense. As well, a long and rich history of encouraging findings from the aviation simulation literature lent support to the concept that testing and training in highly proceduralized VR simulation environments would be a useful direction for neuropsychology to explore (Johnston, 1995; Rizzo, 1994). Within this context, a small group of researchers began the initial work of exploring the use of VR technology for applications designed to target cognitive/functional performance in populations with CNS dysfunction. While a good deal of this early work employed non-head mounted display flatscreen environments, these less immersive systems produced encouraging results (Cromby et al., 1996; Rose et al., 2001; Stanton, Foreman & Wilson, 1998). This work demonstrated the unique value of the technology, served to inform future applications and created a demand for the assets available with more immersive VR approaches.

Over the last few years, revolutionary advances in the underlying VR enabling technologies (i.e., computation speed and power, graphics and image rendering technology, display systems, interface devices, immersive audio, haptics tools, wireless tracking, voice recognition, intelligent agents, and authoring software) have supported development resulting in more powerful, low-cost PC-driven VR systems. Such advances in technological "prowess" and accessibility have provided the hardware platforms needed for the conduct of human research within more usable and useful VR scenarios. From this, current research efforts to develop more accessible VR systems have produced applications that are delivering encouraging results on a wide range of cognitive, physical, emotional, social, vocational and psychological human issues and research questions (Blascovich et al., 2002; Rizzo, Buckwalter & van der Zaag, 2002a; Weiss & Jessel, 1998; Zimand et al., in press).

III. Analysis of VR Assets

What makes VR application development in this area so distinctively important is that it represents the potential for more than a simple linear extension of existing computer technology for human use. This was recognized early on in a visionary article ("The Experience Society") by VR pioneer, Myron Kruegar (1993), in his prophetic statement that, "…Virtual Reality arrives at a moment when computer technology in general is moving from automating the paradigms of the past, to creating new ones for the future". (p. 163). By way of VRs capacity to place a person within an immersive, interactive computer generated simulation environment, new possibilities exist that go well beyond simply automating the delivery of existing paper and pencil testing and training tools on a personal computer. However, while encouraging on a theoretical level, the value of this technology for neuropsychology still needs to be substantiated via systematic empirical research with normal and clinical populations that can be replicated by others. To accomplish this first requires specification as to the real assets that VR offers that add value over existing methodologies, as well as further exploration of its current limitations. Non-immersive computerized testing and training tools have been available for some time and a case can be made that they offer some of the same features found with immersive head mounted display VR. As well, in spite of the many claims that computers would revolutionize cognitive rehabilitation in the late 1980's, the manifested value of these tools have been questioned by some (Robertson, 1990). Therefore it becomes imperative that research be conducted to determine the incremental value of VR-specific assets (i.e., immersive, naturalistic and/or supra-normal human computer interaction) over already existing methods. To address these issues we will discuss the assets that are available with VR along with examples of NP assessment and rehabilitation research and findings from related fields that illustrate the relevance of these assets. Challenges that still need to be addressed will also be discussed.

1. The capacity to systematically deliver and control dynamic, interactive 3D stimuli within an immersive environment that would be difficult to present using other means.

One of the cardinal assets of any advanced form of simulation technology involves the capacity for systematic delivery and control of stimuli. This asset provides significant opportunities for advancing NP methods. In fact, one could conjecture that the basic foundation of all human research methodology requires the systematic delivery and control of an environment and the subsequent capture and analysis of the behavior that occurs within the environment. In this regard, an ideal match appears to exist between the stimulus delivery assets of VR simulation approaches and the requirements of NP assessment and rehabilitation. Much like an aircraft simulator serves to test and train piloting ability, VEs can be developed to present simulations that assess and rehabilitate human cognitive and functional processes under a range of stimulus conditions that are not easily controllable in the real world. This "Ultimate Skinner Box" asset can be seen to provide value across the spectrum of NP approaches, from analysis at a molecular level targeting component cognitive processes (e.g., selective attention performance contingent on varying levels of stimulus intensity exposure), to the complex targeting of more molar functional behaviors (e.g., planning and initiating the steps required to prepare a meal in a chaotic setting).

This asset can be seen to allow for the hierarchical delivery of stimulus challenges across a range of difficulty levels. For example, an individual's rehabilitation could be customized to begin at a stimulus challenge level most attainable and comfortable for them, with gradual progression of difficulty level based on that individuals' performance. The rehabilitation of driving skills following traumatic brain injury is one example where individuals may begin at a simplistic level (i.e., straight, non-populated roads) and gradually move along to more challenging situations (i.e., crowded, highway roads) (Schultheis & Mourant, 2001). This asset would also provide the opportunity to identify, implement and modify individual compensatory strategies that can be tested at various hierarchical levels of challenge within a VE modeled after a targeted real world environment. Repeated practice could result in "successful learning" and produce positive reinforcement of compensatory strategy use that could potentially enhance the generalization of these strategies to everyday activities. As well, the successful execution of many everyday activities often requires the integration of a variety of cognitive functions, and subsequent component evaluation of these complex behaviors is often challenging to clinicians and researchers. By providing options for stimulus control within a VE, the impact of specific component cognitive assets and limitations may be better isolated, assessed and rehabilitated.

Enhanced stimulus control also can result in better consistency of stimulus presentations. Naturally occurring changes in "everyday" real world settings typically make the exact repetition of assessment unfeasible and this inconsistency can negatively impact on the standardization of defining and measuring specific behaviors. For example, current assessment and rehabilitation approaches of everyday functional skills, such as ambulation in the community, are currently limited by the inability to control and repeat exact stimuli in relevant settings (i.e., in the street, within office buildings). Subsequently, assessment and rehabilitation is typically conducted within a more controlled environment (e.g., gymnasium), which may not reflect the actual demands of ambulation in the "real world". The application of VR to this approach would address this limitation by allowing assessment and rehabilitation in more functionally relevant VEs (e.g., city streets) while still allowing clinicians and researchers full control over stimulus presentations. This level of control could serve to improve consistency across assessments and interventions and allow for increased standardization and validation of methods for assessing complex behaviors. Examples of such VR applications include the development of "virtual cities" and other complex environments for assessing and rehabilitating wayfinding (Brown, Kerr & Bayon, 1998), the use of public transportation (Mowafty & Pollack, 1995) and a wide range of other instrumental activities of daily living (see review in Rizzo et al, 2002a).

2. The capacity to create of more ecologically valid assessment and rehabilitation scenarios.

Traditional NP assessment and rehabilitation has been criticized as limited in the area of ecological validity, that is, the degree of relevance or similarity that a test or training system has relative to the "real" world (Neisser, 1978). While existing NP tests obviously measure behaviors mediated by the brain, controversy exists as to how performance on analog tasks relates to complex performance in an "everyday" functional environment. By designing virtual environments that not only "look like" the real world, but actually incorporate challenges that require 'real world' functional behaviors, the ecological validity of cognitive/functional performance assessment and rehabilitation could be enhanced. As well, the complexity of stimulus challenges found in naturalistic settings could be presented while still maintaining the experimental control required for rigorous scientific analysis and replication. Thus, VR derived assessment results could have greater predictive validity/clinical relevance and a more direct linkage to both restorative and functional NP rehabilitation approaches.

A number of examples illustrate efforts to enhance the ecological validity of assessment and rehabilitation by designing VEs that are "replicas" of relevant archetypic functional environments. This has included the creation of virtual cities (Brown et al., 1998; Costas, Carvalho & de Aragon, 2000), supermarkets (Cromby et al., 1996); homes (Rose et al., 2001); kitchens (Christiansen et al., 1998; Davies et al., 1998;), school environments (Stanton et al., 1998; Rizzo et al., 2000), workspaces/offices (McGeorge et al., 2001; Schultheis & Rizzo, 2002); rehabilitation wards (Brooks et al., 1999) and even a virtual beach (Elkind et al., 2001). While these environments vary in their level of pictorial or graphic realism, this factor may be secondary in importance, relative to the actual activities that are carried out in the environment for determining their value from an ecological validity standpoint. Interestingly, when in a virtual environment, humans often times display a high capacity to "suspend disbelief" and respond as if the scenario was real. It could be conjectured that the "ecological value" of a VR task that needs to be performed may be well supported in spite of limited graphic realism and less immersion (such as in flatscreen systems). In essence, as long as the VR scenario "resembles" the real world, possesses design elements that replicate key real life challenges and the system responds well to user interaction, then ecological validity is enhanced beyond existing analog approaches. Evidence to support this view can be drawn from clinical VR applications that address anxiety disorders. While a number of the successful VR scenarios designed for exposure-based therapy of specific phobias would never be mistaken for the real world, clients within these VEs still manifest physiological responses and report subjective units of discomfort levels that suggest they are responding "as if" they are in the presence of the feared stimuli (Wiederhold & Wiederhold, 1998).

This point is also illustrated in a number of examples where VR has been applied to target executive functioning and wayfinding. In the mid-90's, using graphic imagery that would be considered primitive by today's standards, Pugnetti et al. (1995; 1998) developed a head mounted display delivered VR scenario that embodied the cognitive challenges that characterize the Wisconsin Card Sorting Test (WCST). The scenario consisted of a virtual building within which users were required to use environmental clues to aid in the correct selection of appropriate doorways needed to pass from room to room through the structure. The doorway choices varied according to the categories of shape, color and number of portholes. Similar to the WCST, the correct choice criteria were changed after a fixed number of successful trials, and the user was then required to shift cognitive set, look for clues and devise a new choice strategy in order to successfully pass into the next room. In one study, Pugnetti et al. (1998) compared a mixed group of neurological patients (multiple sclerosis, stroke, and traumatic brain injury) with normals' performance on both the WCST and on this head mounted display executive function system. Results indicated that the VR results mirrored previous anecdotal observations by family members of everyday performance deficits in the patient populations. Though the psychometric properties of the VE task were comparable to the WCST in terms of gross differentiation of patients and controls, weak correlations between the two methods suggested that the methods measured different aspects of these functions. A detailed analysis of the VR task data indicated that specific preservative errors appeared earlier in the test sequence compared to the WCST. The authors suggested that "…this finding depends on the more complex (and complete) cognitive demands of the VE setting at the beginning of the test when perceptuomotor, visuospatial (orientation), memory, and conceptual aspects of the task need to be fully integrated into an efficient routine" (p.160). The detection of these early "integrative" difficulties for this complex cognitive function may be particularly relevant for the task of predicting real world capabilities from test results.

This was further evidenced in a detailed single subject case study of a stroke patient using this system. In this report (Mendozzi, Motta, Barbieri, Alpini & Pugnetti, 1998), results indicated that the VR system was more accurate in identifying executive function deficits in a highly educated patient two years post-stroke, who had a normal WCST performance. The VR system, although using graphic imagery that would never be mistaken for the real world, was successful in detecting deficits that had been reported to be limiting the patient's everyday performance, yet were missed using existing NP tests. These results are in line with the observation that patients with executive disorders often perform relatively well on traditional NP tests of 'frontal lobe function', yet show marked impairment in controlling and monitoring behavior in real-life situations (Shallice & Burgess, 1991).

Similar findings were recently reported by McGeorge et al. (2001) in a study comparing real world and virtual world "errand running" performance in five traumatic brain injury patients and five matched normal controls. The selection of the patient sample for this study was based on staff ratings that indicated poor planning skills. However, the patient and control groups did not differ significantly from normative values on the Behavioural Assessment of the Dysexecutive Syndrome (BADS) battery (Wilson, Alderman, Burgess, Emslie & Evans, 1996). Video taped performance of subjects was coded and compared while performing a series of errands in the University of Aberdeen psychology department (real world) and within a flatscreen VR scenario modeled after this environment. Performance in both the real and virtual environment, as defined as the number of errands completed in a 20-minute period, was highly correlated (r = .79; p<.01). Interestingly, while the groups did not differ on age-corrected standardized scores on the BADS, significant differences were found between the groups in both the real world and virtual testing. This finding suggests several things. First, performance in the real and virtual world was functionally similar, second, patient and control groups could be discriminated equally using real and virtual tests while this discrimination was not picked up by standardized testing with the BADS, and third, that both measures of real and virtual world performance showed concordance with staff observations of planning skills. That these results support the view that VR testing may possess higher ecological value is in line with the observation by Shallice and Burgess (1991) that traditional NP tests do not demand the planning of behavior over more than a few minutes, or the prioritization of competing subtasks and may result in less effective prediction of real world performance.

In the area of rehabilitation, a number of studies have supported the ecological value of VR training for wayfinding in both developmentally disabled teenagers navigating a supermarket (Cromby et al., 1996) and for school navigation in children in wheelchairs with limited experience in independent wayfinding (Stanton et al., 1998). Further initial support for the ecological value of VR wayfinding training, can be found in a case study by Brooks et al. (1999). In this report, a female stroke patient with severe amnesia showed significant improvements in her ability to find her way around a rehabilitation unit following training within a VE modeled after the unit. This was most notable given that prior to training, the patient had lived on the unit for two months and was still unable to find her way around, even to places she had visited regularly. In the first part of the training, improvements on two routes were seen after a three-week period of VE route practice lasting only 15 minutes per weekday and retention of this learning was maintained throughout the patient's stay on the unit. In the second part of the study, the patient was trained on two more routes, one utilizing the VE, and the other actually practicing on the "real" unit. Within two weeks the patient learned the route practiced in the VE, but not the route trained on the real unit, and this learning was maintained throughout the course of the study (Brooks et al., 1999). The authors account for this success as being due in part to the opportunity in the VE for quicker traversing of the environment than in the real world, which allowed for more efficient use of training time. Another factor in this success may be that the VE training did not contain the typical distractions normally present when real world training is conducted that might have impeded route learning. It might be found that the gradual fading in of distractions would be useful for inoculating the patient to the potentially deleterious impact of their inevitable presence in the real world and further enhance the ecological value of this form of rehabilitation.

These findings lend support to the view that due to the similarity of VR testing or training tasks with the demands of the real world, the enhancement in ecologically validity promotes the generalizability of such results to functional real world performance. Thus, VR assessment results could have enhanced clinical relevance and serve as a basis for the development of both restorative and contextual cognitive rehabilitation approaches. However, before this vision can be fully reached, technological advances need to occur in the area of human-computer interaction interfacing devices. Current technology is still limited in the degree to which a user can naturalistically interact with the challenges presented in a VE. From a human-computer interaction perspective, a primary concern involves how to design more naturalistic and intuitive tools for human interfacing with such complex systems. In order for persons with cognitive impairments to be in a position to benefit from VR applications, they must be able to learn how to navigate and interact within the environment. Many modes of VR interaction (i.e., data-gloves, joy sticks, 3D mice, etc.), while easily mastered by unimpaired users, could present problems for those with cognitive difficulties. Even if patients are capable of using a VR system at a basic level, the extra non-automatic cognitive effort required to interact/navigate could serve as a distraction and limit the assessment and rehabilitation processes. In this regard, Psotka (1995) hypothesizes that facilitation of a "single egocenter" found in highly immersive interfaces serves to reduce "cognitive overhead" and thereby enhance information access and learning. This is an area that needs the most attention in the current state of affairs for VR applications designed for populations with CNS dysfunction and an excellent review of these tools and issues can be found in Bowman, Kruijff, LaViola & Poupyrev (2001).

3. The delivery of immediate performance feedback in a variety of forms and sensory modalities.

The capacity for systematic delivery and control of stimuli presented to users in a VE can serve as a significant asset for the development of NP assessment and rehabilitation scenarios. This asset can also be harnessed to provide immediate performance feedback to users contingent on the status of their efforts. Such automated delivery of feedback stimuli can appear in graded (degree) or absolute (correct/incorrect) forms and can be presented via any or multiple sensory modalities (though mainly audio, visual, tactile is used) depending on the goals of the application and the needs of the user. This is an intuitively essential component for rehabilitation efforts as performance feedback is generally accepted to be necessary for most forms of learning or skill acquisition (Sohlberg & Mateer, 1989; 2001).

While VR-based feedback can be presented to signal performance status in a form that wouldn't naturally occur in the real world (e.g., a soft tone occurring after a correct response), more relevant or naturalistic sounds can also be creatively applied to enhance both ecological validity and the believability of the scenario. For example, in an Internet delivered VR application designed to help children with learning disabilities practice escape from a house fire (Strickland, 2001), the sound of a smoke detector alarm raises in volume as the child gets near to the fire's location. As the child successfully navigates to safety, the alarm fades contingent on her choosing the correct escape route. An efficacy study of this application is currently in progress (Dorothy Strickland, personal communication, Aug. 21, 2002).

The potential value of virtual performance feedback for NP rehabilitation applications can also be conjectured from applications designed to support physical therapy in adults following a stroke (Jack et al., 2001; Deutsch, Latonio, Burdea, & Boian, 2001). These applications use various glove and ankle VR interface devices that translate the user's movements into a visible and somewhat relevant activity that is presented graphically on a flatscreen display. For example, in one application, as the user performs a prescribed hand exercise designed to enhance fractionation (independence of finger motion), the image of a hand appears on the display, playing a piano keyboard, reflecting the actual hand movements of the client. In a similar application, the appropriate hand movement moves a "wiper" that serves to reveal an interesting picture along with display of a graphic rendering of a performance meter representing range of movement. These features not only serve as a mechanism for providing feedback regarding the ongoing status of targeted movement, but could be potentially used as a motivator. Results from this lab with stroke patients, presented in a series of seven case studies, reported positive results for rehabilitating hand performance across range, speed, fractionation and strength measures (Jack et al., 2001). In one noteworthy case, functional improvement was reported in a patient who was able to button his shirt independently for the first time post-stroke following two weeks of training with the VR hand interface. As well, by making the repetitive and sometimes boring work of physical therapy exercise more interesting and compelling, patients reported enhanced enjoyment leading to increased motivation.

For assessment purposes, although performance feedback is not typically a component of traditional testing, there may be a well-matched place for it in the emerging area of "dynamic" testing (Sternberg, 1997). In a critique of traditional cognitive and ability performance testing, Sternberg posits that dynamic interactive testing provides a new option that could supplement traditional "static" tests. The dynamic assessment approach requires the provision of guided performance feedback as a component in tests that measure learning. This method appears well suited to the assets available with VR technology. In fact, VEs might be the most efficient vehicle for conducting dynamic testing in an ecologically valid manner while still maintaining an acceptable level of experimental control.

4. The provision of "cueing" stimuli or visualization tactics designed to help guide successful performance to support an error-free learning approach.

The capacity for dynamic stimulus delivery and control within a VE also allows for the presentation of cueing stimuli that could be used for "error-free" learning approaches in cognitive rehabilitation scenarios. This asset underscores the idea that in some cases it may not be desirable for VR to simply mimic reality with all its incumbent limitations. Instead, stimulus features that are not easily deliverable in the real-world can be presented in a VE to help guide and train successful performance. In this special case of stimulus delivery, cues are given to the patient prior to a response in order to help guide successful error-free performance. Error-free training in contrast to trial and error learning has been shown to be successful in a number of investigations with such diverse subjects as pigeons to persons with developmental disabilities, schizophrenia, as well as a variety of CNS disorders (see Wilson and Evans, 1996 for review).

The basis for these findings regarding error-free learning may lie in reports that indicate that in persons with neurologically based memory impairment, certain memory/learning processes often remain relatively intact. Procedural, or skill memory, is one such cognitive operation (Cohen & Squire, 1980; Charness, Milberg, & Alexander, 1988). This type of memory ability concerns the capacity to learn rule-based or automatic procedures including motor tasks, certain kinds of rule based puzzles, and sequences for running or operating equipment, tools, computers, etc. (Sohlberg & Mateer, 1989;). Procedural memory can be viewed in contrast to declarative, or fact-based memory, which is usually more impaired following CNS insult and less amenable to rehabilitative improvement. Additionally, patients often demonstrate an ability to perform procedural tasks without any recollection of the actual training. This is commonly referred to as implicit memory (Graf & Schacter, 1985) and its presence is indicative of a preserved ability to process and retain new material without the person's conscious awareness of when or where the learning occurred.

Virtual Reality, by way of its interactive and immersive features, could provide training environments that foster cognitive/functional improvement by exploiting a person's preserved procedural abilities. Hence, cognitive processes could be restored via procedures practiced successfully and repetitively within a VE that contains functional real-world demands. Whether the person actually had any declarative recall of the actual training episodes is irrelevant, as long as the learned process or skill is shown to generalize to functional situations. Error-free learning strategies could be well integrated into a VE by way of thoughtful presentation of cueing stimuli within dynamic stimulus presentations. The real challenge would then be to somehow translate difficult declarative (and semantic) tasks into procedural learning activities, with the goal being the restoration of the more complex higher reasoning abilities.

Very few studies have examined the direct or specific effects of providing such cueing stimuli (compared to trial and error training) within a VE. In the only VR-based head-to-head comparison of this type, Connor et al. (2002), has reported a series of case studies on the use of a haptic joystick mediated "Trails B" type-training task. In the error-free condition, the haptic joystick restricted movement on a flatscreen Trails type task such that the patient was not allowed to make navigation errors. Mixed findings were reported, but error-free training resulted in significant response speed improvements compared to errorful training in some cases. Other studies have reported on the inclusion of an error-free component embedded within an overall VR training approach with more encouraging findings. For example, in the Brooks et al. (1999) case study previously cited, error-free training for wayfinding in a rehabilitation ward was one component in a VR training system that produced positive transfer to the real ward. Harrison et al. (2002) also reported the use of cueing stimuli in a VR system designed to train maneuverability and route-finding in novice motorized wheelchair users. This scenario provided a series of arrows that were presented with the caption "Go this way" to guide successful route navigation whenever the user would stray into areas where invisible "collision boxes" were programmed in the environment. Two patients with severe memory impairments took part in route finding training over the course of seven days. Post testing on the real routes produced mixed results with the patients successfully learning two subsections of the test routes but failing to eradicate errors on two further subsections of the routes. The investigators felt that further collision detection refinement of the system will be required to support accurate prompt delivery to patients before a more systematic group test will be possible.

Cueing stimuli have also been incorporated into a VE designed for executive function assessment and training in the context of a series of food preparation tasks within a virtual kitchen scenario (Christiansen et al., 1998). This scenario consists of a head mounted display VE of a kitchen in which patients have been assessed in terms of their ability to perform 30 discrete steps required to prepare a can of soup and make a sandwich. Various auditory and visual cues can be presented to help prompt successful performance. However the specific effect of this cueing has not been isolated, nor was a system in place to prevent errors from actually occurring (although successful usability findings--30 patients-minimal side effects--has been reported along with acceptable test/retest reliability coefficients for use of this system (Christiansen et al., 1998)). These researchers report ongoing enhancements to the system regarding the delivery of more complex challenges and increased flexibility in the presentation of cueing stimuli.

Generally, it appears that the provision of cueing stimuli to support error-free rehabilitation in a VE is promising in concept and supported by findings using traditional methods. However, empirical support in the form of systematic group VR data is still lacking. Part of the difficulty up to now has been due to programming challenges for tracking the user's position in the VE as was reported in Harrison et al. (2002), and for accurately providing prompts and restricting errors in an automatic fashion. This has become less of a problem with recent advances in collision detection and "physics" software, but still may be difficult as programming is typically not the clinician's primary skill. However, as better 'end-user' programming technologies come along, this may become less of an issue.

Technology challenges aside, VR-based research that could adequately explore the error-free "cueing" issue would require at least an errorless condition that could be compared with trial and error methods as seen in Connor et al. (2002). As well, the sensory mode of cue presentation should be explored to determine whether auditory cueing could be a useful option relative to the types of visual cues typically seen in the form of word captions and arrow pointers. If auditory cueing was found to be of equal effectiveness, it would reduce graphic requirements in system programming. Importantly, auditory prompting may also better resemble and "provoke" self-talk instruction methods that might support generalization to the real world of self-generated subvocal prompting on the part of the patient. If key prompting statements could be specified in advance, it would be possible to pre-record the patient speaking supportive cues in their own voice. When these cues are played back at strategic choice points within the VE, the patient could be directed more naturalistically by this form of "inner-voice" guidance. Also, since the early reports in this area have thus far mainly focused on spatial navigation and object localization, cues have been limited to the visual mode for pointing direction and labeling objects. Perhaps more complex tasks could be trained with inclusion of auditory cueing in this manner that could support error free training for the type of integrative problem solving required for effective executive functioning.

Finally, if the user-interfacing tools could be effectively designed, it would be possible to incorporate the use of various electronic compensatory devices into the VR interaction strategy as a method to deliver prompts in a fashion similar to how the patient is being encouraged to use these devices in the real world. This could take the form of assessing what level of "augmentive" information could actually be used by patients to assist in compensatory strategies aimed at improving day-to-day functional behavior and for training patient's effective use of these devices under a variety of environmental challenges.

5. The capacity for complete performance capture and the availability of a more naturalistic/intuitive performance record for review and analysis.

The review of a client's performance in any assessment and training activity typically involves examination of numeric data and subsequent translation of that information into graphic representations in the form of tables and graphs. Sometimes videotaping of the actual event is used for a more naturalistic review and for behavior rating purposes. These methods, while of some value, are typically quite labor intensive to produce and sometimes deliver a less than intuitive method for visualizing and understanding a complex performance record. These challenges are compounded when the goal of the review is to provide feedback and insight to clients whose cognitive impairments may preclude a useful understanding of traditional forms of data presentation. VR offers the capability to capture and review a complete digital record of performance in a virtual environment from many perspectives. For example, performance in a VE can be later observed from the perspective of the user, from the view of a third party or position within the VE and from what is sometimes termed, a "God's eye view", from above the scene with options to adjust the position and scale of the view. This can allow a client to observe their performance from multiple perspectives and repeatedly review their performance. Options for this review also include the modulation of presentation as in allowing the client to slow down rate of activity and observe each behavioral step in the sequence in "slow motion".

Advanced programs to do this have already been developed by the military to conduct what is termed After Action Reviews (Morrison & Meliza, 1999). In military VR applications that often include multiple participants in a shared virtual space, a computerized After Action Review tool can allow the behavior of any participant to be reviewed from multiple vantage points at any temporal point in the digital training exercise. This is now standard procedure for military simulation training, but has had limited application in traditional NP approaches. With the exception of less naturalistic review of paper and pencil results and the occasionally review of a client's video taped performance from a single fixed position, the capacity to provide more intuitive "first-person" perspective views to clients has not been feasible with existing technology.

Thus far, this VR asset has begun to appear as a feature for reviewing navigational performance in a number of wayfinding and place learning applications (Astur, Oriz & Sutherland, 1998; Jacobs, Laurance & Thomas, 1997; Skelton et al., 2000). This has mainly been used in applications where a tracked movement record is vital for measuring the dependent variable of exploratory behavior. Systematic studies of the clinical use of this form of performance record review have yet to appear in the literature, although the capacity to present this information exists with most applications, but requires additional programming to extract and display it. In this regard, the first author's lab has developed a visual record review method for replaying children's head movements while they are tracking stimuli within a virtual classroom. This application (Rizzo et al., 2000; 2002b) takes data from a magnetic field tracking device positioned on top of the head mounted display and represents the captured movement via a virtual representation of a person's head. The head faces outward on the screen and "straightforward" head position represents the attentive gaze at the virtual blackboard where target hit stimuli are displayed to the child. During playback, it is possible to observe the child's head movements during discrete periods when distracting stimuli are presented around the classroom. Head movements away from the center of the screen represent the child's actual movements to follow the distracting stimuli on each side of the classroom instead of the face forward position required to view the target stimuli. This presentation format delivers an extremely intuitive understanding of the distractibility of children diagnosed with Attention Deficit Hyperactivity Disorder (ADHD) during VR classroom performance testing. In the initial prototype of this system, we can deliver side-by-side concurrent performance of both a non-diagnosed and ADHD child and observe the stark contrast in their head turning away from the target stimuli during distraction periods. Thus far in selected case comparisons, the non-diagnosed children are noticed to turn in the direction of the distraction very briefly, but nearly immediately return to the on-task position. By contrast, the children with ADHD are often observed to look away and then continue off task for varying extended periods of time resulting in subsequent omission performance errors. The "head to head" playback of these head movements serves to underscore in an intuitive manner, the significant findings of off task head position that were revealed from the complex statistical analyses of this movement data. Integration of this form of intuitive performance record review could serve to provide insight for understanding the behavior of ADHD children to professionals, parents and perhaps even the tested child, in a manner not possible with graphs and data tables. This is an asset in which VR may add value across all areas of performance testing and training that is not readily available with existing traditional tools.

6. The capacity to pause assessment, treatment and training for discussion and/or integration of other methods.

In the assessment and rehabilitation of complex behavior and/or functional activities, feedback is often an integral component. Similar to the previous asset regarding the availability of a naturalistic performance record, VR allows for a cumulative record to be reviewed at any point in the testing and training sequence. Specifically, immediate external therapist response to client performance is one form of feedback that is commonly seen in the rehabilitation of clinical populations. This may be of particular value for clinical populations who have memory difficulties that require more frequent review and feedback during a training session. While this may be possible through "traditional" approaches (i.e., one can always pause analog NP testing and training), VRs unique assets offer the opportunity to pause or "freeze time" in the middle of a functional "real-world" simulated task. This can result in additive learning benefits, whereby you can "stop and evaluate" not only individual performance, but also by examining what environmental elements may be effecting performance. For example, during activities in a VR kitchen for the completion of a simple task (i.e., making a can of soup), performance may be paused for the correction of errors (missed procedure steps), evaluation of safety elements of the task (where are the sharp objects) or discussion of perceptual difficulties (inappropriate visual scanning).

Thus, the ability to pause performance "mid-digitalstream" may also foster better processing and discussion of decision-making elements of performance. This may be useful for individuals with frontal lobe damage who have compromised executive skills and subsequently may benefit from an on the spot review of their step-by-step decision-making process. In addition, VR may allow for the clinician to monitor performance and provide problem solving guidance to test out potential alternative solutions, that when integrated into the rehabilitation intervention, may help increase client self-awareness of assets and limitations. For some tasks, the opportunity of combining immediate feedback and processing/discussion, obtainable through VR, may offer safety options not possible in the real world. For example, in driver re-training for individuals with cognitive compromise, the ability to pause performance mid-task and provide guidance may support an increased level of "awareness", which may serve to enhance learning and recall. Participants experiencing an "accident" in a driving VE, can be immediately "pulled over" and assisted in identifying errors that lead to the accident. This may result in fostering a heightened client awareness of the rehabilitation experience due to the immediacy and better specificity of performance feedback.

7. The design of safe testing and training environments that minimize the risks due to errors.

As alluded to in the VR Driving example presented above, when developing certain functionally based assessment and rehabilitation approaches, one must consider the possibility of safety risks that may occur during activities designed to test and train abilities in the real world. Driving would probably represent one of the more risk-laden activities that a client with CNS dysfunction would undertake in order to achieve functional independence. However, even simple functional activities can lead to potential injury when working with persons having CNS-based impairments. Such potential risks can be seen in the relatively "safe" environment of a kitchen (i.e. burns, falls, getting cut with a knife) as well as in more naturally dangerous situations such as in street crossing, the operation of mechanical/industrial equipment and driving a motor vehicle. Additionally, the risk for client/therapist injury and subsequent liability concerns, may actually limit the functional targets that are addressed in the rehabilitation process. These "overlooked" targets may actually put the client at risk later on as they make their initial independent efforts in the real world without having such targets addressed thoroughly in rehabilitation.

This is an area where VR provides an obvious asset by creating options for clients to be tested and trained in the safety of a simulated digital environment. The value of this has already been amply demonstrated in the predecessor field of aviation simulator research where actual flying accidents dropped precipitously following the early introduction of even very crude aircraft simulation training (Johnston, 1995). Thus far, this asset has served as a driving force for VR system design and research with clinical and "at-risk" normal populations. Such applications include: street crossing with unimpaired children (McComas, MacKay & Pivak, 2002), populations with learning and developmental disabilities (Strickland, 2001; Brown et al., 1998), and adult traumatic brain injury groups with neglect (Naveh, Katz & Weiss, 2000); kitchen safety (Rose, Brooks & Attree, 2000); escape from a burning house with autistic children (Strickland, 2001); preventing falls with at risk elderly (Jaffe, 1998); use of public transportation (Mowafty et al., 1995) and driving with a range of clinical populations (Liu, Miyazaki & Watson, 1999; Rizzo, Reinach, McGehee & Dawson, 1997; Schultheis et al., 2001). In addition to the goal of promoting safe performance in the real world, some researchers have reported positive results for building a more rational awareness of limitations using a VR approach. For example, Davis & Wachtel, (2000), have reported a number of instances where older adults, post-stroke, had decided not to continue making a return to driving a primary immediate goal after they had spent time in a challenging VR driving system. It is expected that the VR driving literature will grow as more attention is focused on preventing risk in both novice and aged populations.

Finally, one concern that may exist with this asset involves the potential that practice of activities that are dangerous in real life, within the safety of a VE, might create a false sense of security or omnipotence that would put the client at risk upon subsequent action in the real world. In essence, can safe transfer of training occur in the real world when the consequences of errors are prevented from occurring in the VE? This is a very challenging concern that will need to be considered carefully. Perhaps, one option would be to provide a noxious sound cue, contingent on the occurrence of dangerous errors in the VE, as a means to condition a proper attitude of caution in clients. This concern further underscores the need for a professional to closely monitor client activity in order to recognize possible patterns of risk taking behavior that could emerge when using such VEs.

8. The capacity to improve availability of assessment and rehabilitation by persons with sensorimotor impairments via the use of adapted interface devices and tailored sensory modality presentations built into VE scenario design.

One of the current challenges in neuropsychology concerns the adaptation of NP assessment and rehabilitation methods for use by clients with significant sensory and motor impairments. And when such adaptations are attempted, the question often arises as to how much does a client's performance reflect centrally-based cognitive dysfunction vs. artifacts due to more peripheral sensorimotor impairments. VR offers two ways in which this challenge may be addressed in the testing and training of cognitive and everyday functional abilities in persons with sensorimotor impairments.

One approach places emphasis on the design of adapted human-computer interface devices in a VE to promote usability and access. The thoughtful integration of adapted interface devices between the person and VR system could assist those with motor impairments to navigate and interact in functional testing and training VR applications (beyond what might be possible in the real world). Such interface adaptations may support actuation by way of alternative or augmented movement, speech, expired air, tracked eye movement and by way of neurofeedback-trained biosignal activity. While an extensive literature exists in the area of interface design for persons with disabilities (Barrett, McCrindle, Cook & Booy, 2002) and on concerns about an emerging "Digital Divide" (LaPlant, 2001), those domains are beyond the scope of this article. However, two examples should serve to illustrate this potential in the VR area. One basic example involves the use of a gaming joystick to navigate in a VE that was found effective for teaching wayfinding within a VE modeled after an amnestic client's rehabilitation unit (Brooks et al., 1999). These authors partially attributed the observed positive training effects to the client's capability for quicker traversing of the VE using a joystick compared to what her ambulatory impairments would allow in the real environment. This strategy supported efficient use of training time. A more technically complex approach uses "biosignals", as seen in the use of the "Cyberlink" system (Doherty, Bloor & Cockton, 1999). Initial results using this system suggested that persons with extreme motor and language impairments following stroke and traumatic brain injury were able to communicate using an EEG/EOG/EMG-driven cursor on a flatscreen computer. With continued advances in adapted interface technology, these approaches could support VE navigation and interaction in persons with motor impairments and serve to promote better access to cognitive and functionally-based assessment and rehabilitation. As well, by minimizing the impact of peripheral impairments on performance, centrally-based performance components may be more efficiently tested and trained.

A second approach to this challenge has been to tailor the sensory modality components of the VE around the needs of persons with visual impairments. The few efforts in this area have mainly attempted to build simulated structures around the use of enhanced 3D sound (Lumbreras & Sanchez, 2000) and tactile stimuli (Connor, 2002: see Asset 4). For example, Lumbreras et al. (2000), aiming to design computer games for blind children, created a 3D audio VR system referred to as "AudioDOOM". In this application, blind children use a joystick to navigate the mazelike game environment exclusively on the basis of 3D audio cues (i.e., footstep sounds, doors that "creak" open, echoes, etc.) while chasing "monsters" around the environment. Following varied periods of time in the VE, the children are then given "Legos" to construct their impression of the structure of the layout. The resulting Lego constructions are often noteworthy in their striking resemblance to the actual structure of the audio-based layout of the maze. Children using this system (who never actually have "seen" the physical visual world) often appear to be able use the 3D sound cues to create a spatial-cognitive map of the space and then accurately represent this space with physical objects (i.e., Legos, Clay, Sand). Examples of some of these constructions are available on the Internet (http://www.dcc.uchile.cl/~mlumbrer/audiodoom/audiodoom.html). While still in the "proof of concept" stage, it would be possible to conceive of such 3D audio-based environments as providing platforms for testing and training of persons with visual impairments at any age.

Finally, the use of haptic simulation tools has been investigated as a method to create VE applications for persons with visual impairments (Jansson, 2000). However, the technology to deliver convincing touch-based simulations is still in the very early stages of development and readers interested in further details are referred to McLaughlin, Hespanha, & Sukhatme (2002).

9. The introduction of "gaming" features into VR rehabilitation scenarios as a means to enhance motivation.

Plato was reputed to have said, "You can discover more about a person in an hour of play than in a year of conversation." (cited in Moncur & Moncur, 2002). This ancient quote may have particular relevance for future applications of VR in neuropsychology. Observing and/or quantifying a person's approach or strategy when participating in a gaming activity may provide insight into cognitive functioning similar to the types of challenges found in traditional performance assessments. However, a more compelling clinical direction may involve leveraging gaming features and incentives for the challenging task of enhancing motivation levels in clients participating in rehabilitation. In fact, one possible factor in the mixed outcomes found in cognitive rehabilitation research may be in part due to the inability to maintain a client's motivation and engagement when confronting them with a repetitive series of retraining challenges, whether using wordlist exercises or real life functional activities. In this regard, an understanding of gaming features and their integration into VR-based rehabilitation systems to enhance client motivation may be a useful direction to explore for a number of reasons.

There is general agreement that the peak ages for traumatic brain injury are in the 15-24-year age range (Lezak, 1995). This same age group also makes up the largest percentage of users of commercial interactive computer gaming applications and this popularity is also extending to other age groups at a rapid pace (Lowenstein, 2002). In fact, the computer gaming industry has now surpassed the "Hollywood" film industry in total entertainment market share, and in the USA--sales of computer games now outnumber the sale of books (Digiplay Initiative, 2002). As such, it appears that gaming applications have become a standard part of the "digital homestead" as delivered on PCs and specific gaming boxes (i.e., Playstation, X-Box, etc.). From this, interactive gaming has become well integrated into the lifestyles of many people who at some point may require rehabilitative services. For this segment of the population, familiarity with and preference for interactive gaming could become useful assets for enhancing client motivation and engagement when designing VR-based rehabilitation tasks.

Thus far, the integration of gaming features into a VE has been reported to enhance motivation in adult client's undergoing physical therapy following a stroke (Jack et al, 2001). As well, Strickland (2001) reports that children with autism were observed to become very engaged in the VR safety training applications that she has developed which incorporate gaming features. Further anecdotal observations suggest that children diagnosed with Attention Deficit Hyperactivity Disorder often have a fascination for the type of stimulus environments that occur with computer/video games (Greenhill, 1998). Parents are often puzzled when they observe their children focusing on video games intently, while teacher reports indicate inattention in the classroom. Additionally, in the first author's clinical experience, it was observed that some of the young adult traumatic brain injury clients, who had difficulty maintaining concentration on traditional cognitive rehabilitation tasks, would easily spend hours at a time playing the computer game "SimCity". These observations suggest that designers of rehabilitation tasks might benefit from examining the formulas that commercial game developers use in the creation of interactive computer games. These formulas govern the flow and variation in stimulus pacing that provide linkage to a progressive reward and goal structure. When delivered within a highly interactive graphics-rich environment, users are observed to become extremely engaged in this sort of game play. Neuroscience research in the area of Rapid Serial Visual Presentation (RSVP) may provide some scientific insight into the human attraction to these fast paced stimulus environments. In this regard, Biederman (2002) suggests that a gradient of opiate-like receptors in the portions of the cortex involved in visual, auditory, and somatosensory perception and recognition drives humans to prefer experiences that are novel, fast, immersive, and readily interpreted. This may partly underlie the enhanced motivation that is observed for the types of activities that are presented in interactive gaming environments. While many reasons may contribute to the allure of current interactive computer gaming, a proper discussion of these issues is beyond the scope of this article. However, the potential value of gaming applications in general education and training is increasingly being recognized and an excellent presentation of these topics can be found in Prensky (2001) along with an extensive gaming bibliography that is available at the Digiplay Initiative (2002). As VR systems in neuropsychology begin to enter the mainstream, investigation on how to integrate gaming features within rehabilitation applications is likely to become an area of intense interest in the future.

10. The integration of virtual human representations (avatars) for systematic applications addressing social interaction.

Over the last few years, continuing advances in the underlying VR enabling technologies has allowed for the creation of more realistic and compelling virtual environment structures. One needs only to look at the recent offerings from the interactive computer gaming industry to appreciate the enhanced level of realism that is afforded by the current state of computer graphics technology. This graphics revolution has also driven the creation of ever more realistic virtual human representations, commonly referred to as "avatars". A compelling case can be made for "populating" VEs with avatars for clinical purposes. More believable virtual humans inhabiting VEs would open up possibilities for assessment and rehabilitation scenarios that target social interaction, naturalistic communication and awareness of social cues. As well, avatars could perhaps serve as "personal" guides that provide instruction or feedback to users operating in a VE. The existence of avatars in VEs could also serve to enhance the realism of VR scenarios that may in turn promote the experience of presence. Such enhanced presence or suspension of disbelief while in a VE might serve to increase psychological engagement in a training scenario, and hence, could foster better generalization to the real world.

However, while advances in avatar design can be readily appreciated in the highly processed fixed forms typically found in computer gaming and in the film, Final Fantasy, the creation of believable characters that can support realtime interaction within a VE is still a non-trivial endeavor. Indeed, Alessi & Huang (2000) has pointed out that until recently, "virtual humans" have mainly appeared in mental health scenarios to "…serve the role of props, rather than humans." (p. 321). This has been mainly due to challenges for both the creation of avatars that can dynamically communicate non-verbal implicit signals via facial and body gestures and in the capacity to drive such avatar expression/interaction with some form of artificial intelligence. Research on these issues is actually quite active from a basic science perspective (Rickel, Marsala, Gratch, Hill, Traum & Swartout, 2002; Rizzo, et al., 2001a), but high development costs and technical challenges have thus far limited progress for all but the most basic direct clinical applications.

In the clinical area, VEs populated with avatars have mainly been designed for use in exposure therapy for specific anxiety disorders. For example, early research in this area is investigating the use of video and computer graphics methods to render virtual humans for treatment of public speaking and social phobias (Anderson, Rothbaum & Hodges, in press; North, North & Coble, 2002; Pertaub, Slater & Barker, 2002; Rizzo, Neumann, Pintaric, & Norden, 2001b). These are application areas that require the presence of human representations to effectively target the specific fear structure in treatment. Some of these applications have used two-dimensional photographic paste-ups of human forms (Moore, Wiederhold, Wiederhold & Riva, 2002; Riva et al., 1999). Such applications allow the clinician to select the number of avatars that appear in the scenario (i.e., supermarkets, parties, auditoriums, subways, etc.) in order to hierarchically target anxiety as part of an exposure based treatment approach. At a somewhat higher level of complexity, graphics-based avatars capable of dynamic expressions have been used by Pertaub et al., (2002) to target public speaking anxiety. In this study, subjects gave an oral presentation in a VR conference room with avatars in the audience whose eyes were programmed to follow the speaker's movement. The avatars were also programmed to dynamically display facial and body cues that represented states of appreciation/interest, boredom/hostility and neutral cues. Random autonomous behaviors (i.e., twitches, blinks, nods, etc.) were also programmed into these continuously animated avatars. Results with non-phobics revealed significantly lower self-reported speech confidence when in the presence of the negative audience and higher negative ratings by females when using a head mounted display compared to flatscreen delivery. A second study reported in this article indicated that high speech anxiety subjects had more discomfort (as measured by heart rate and self-reported anxiety) simply when speaking in the presence of avatars compared to a no avatar condition. For our purposes, these results indicate that subjects were reacting to avatars "as if" they were real members of an audience. Along these lines another research group has used avatars in a group of scenarios as part of a research program that is replicating traditional social psychology studies on social distance, behavioral facilitation/inhibition and conformity. This work has also revealed the occurrence of a similar suspension of disbelief, with subjects' responding to graphics-based virtual humans in a manner similar to previous experiments using real people (Blascovich et. al., 2002).

Other groups have begun experimenting with the incorporation and VR delivery of dynamic video clips of humans for public speaking anxiety (Anderson et al., 2000) and for social phobia and anger management (Rizzo et al., 2001b). Early case study results on the head mounted display speech anxiety applications that use "pasted-in" videos of audiences that vary in size and demeanor have been positive (Anderson et al., 2000). Our social phobia and anger management scenarios using 360-degree panoramic video (Rizzo et al., 2001b) has produced 15 test scenarios (party and work scenarios) that are currently being evaluated. The incorporation of 2D video in a VE may provides more realistic rendering of actual scenes, but also has some limitations, among them restrictions in the user's capacity to explore and navigate "within" the environment as is possible in 3D graphics. Also, once video is captured, it becomes a "fixed" medium that can limit the flexible control of events that is needed for some types of training applications.

The integration of avatars is a potential asset for VEs designed to target NP issues, although this "asset" has rarely been implemented in any systematic fashion for applications with persons having CNS dysfunction. This may primarily be due to an early emphasis on testing and training performance on "tasks" in VEs that mainly involve navigation and perception/interaction with objects. For example, Rizzo et al. (2000; 2002b) has incorporated a virtual teacher within a VR classroom designed to assess attention processes, but this avatar is only capable of delivering instructions and providing verbal commands for various cognitive challenges using fixed audio file inputs. Other avatars appear in the VR classroom to serve as "distracters" during testing, by way of their position in adjacent seats and via their entrance in and departure from the classroom scenario. In an environment similar in concept to the VR classroom, this same approach is being developed in a VR Office scenario designed to assess a wider range of cognitive processes (Schultheis & Rizzo, 2002). In this application, avatars that represent co-workers and supervisors exist in the office as distracters and to deliver verbal commands to look out for and report the occurrence of various target stimuli at a later time as part of a prospective memory assessment. Avatars that represent animals that have anthropomorphic features have also been used in VEs as guides to assist children with learning disabilities on streetcrossing, yard safety and escape from a burning house (Strickland, 2001).

These sorts of applications illustrate the types of first steps that have been taken for VR avatar integration in NP applications. However, with technological advances, it is likely that avatars could play a more dynamic role in VR assessment and rehabilitation applications. Already, advanced research is demonstrating the feasibility of developing avatars that are "fueled" with Artificial Intelligence, aimed at fostering more "authentic" real time interaction between "real" humans and virtual characters for training purposes. For example, Rickel and Johnson (1999) have reported success in the implementation of an avatar with artificial intelligence named "Steve" who serves the role as "instructor" for a virtual training environment targeting the operation and maintenance of equipment on a battleship. As well, similar avatar applications for testing and actual training of tactical decision-making performance for crisis responses in US Army peacekeeping operations are under development (Rickel et al., 2002). These applications could be said to emulate the type of interactions that occur with holographic characters as has been portrayed on the 'holodeck" in various versions of the science "fiction" TV franchise/series "Star Trek". With these research efforts in mind, it is reasonable to consider that future avatar-based VEs could be designed to address self-awareness, social interaction, emotional and vocational targets in persons with CNS dysfunction. This would allow for VE application development that is in line with a "holistic" conceptualization of NP rehabilitation (Prigatano, 1997), while at the same time serving to better integrate cognitive and learning theory based approaches (Wilson, 1997) within a unified assessment and rehabilitation platform.

11. The potential availability of low cost libraries of VEs that could be easily accessed by professionals.

The future evolution of VR as a useful and usable tool in neuropsychology will be driven by three key elements. First, continuing advances in the underlying enabling technologies necessary for VR delivery, along with concomitant hardware cost reductions, will allow VR to become more available and usable by independent clinicians and researchers. Second, this potential for increased access and the impact of market forces will result in further development of new VR applications that target a broader range of clinical and research targets. And finally, continued research aimed at determining reliability, validity and utility will help establish certain VR applications as mainstream NP tools. Contingent upon the occurrence of these events, it will be possible that in the future, neuropsychologists will be able to purchase a VR system that provides them with a suite of environments (i.e., home, classroom, office, community, etc.) within which, a variety of testing and training tasks will be available. This has already occurred in the area of VR/Anxiety Disorder applications with no less than three companies marketing systems in this manner. Internet access to libraries of downloadable VR scenarios will become a likely form of distribution. Data mining, scoring and report writing features will also become available similar to what currently exists with certain standardized tests. As well, highly flexible "front end" interface programs will allow clinicians and researchers to modify stimulus delivery/response capture parameters within some VEs and tailor system characteristics to more specifically meet their targeted purposes. This level of availability could provide professionals with unparalleled options for using and evolving standard VR applications in the service of their clients and for scientific aims.

In anticipation of these possibilities, a parallel objective within ongoing VR research is to determine the most efficient and ethical mechanisms for creation and distribution of VEs. Although promising in concept, the evolution of VR as a clinical tool raises numerous questions regarding the application of technology with clinical populations. One concern is the potential impact on the patient-therapist relationship. Earlier applications of computers in cognitive remediation were met with criticism from professionals who argued that the introduction of computers was equivalent to the removal of the therapist. As well, will greater access to VR applications via the Internet encourage individuals to undertake self-treatment without feeling the need for professional guidance? Will slick marketing of costly VR rehabilitation programs that lack evidence for effectiveness entice naïve clients and deliver no tangible benefit? For a review of some of the ethical issues relevant to the use of VR in clinical practice and overall societal impact, the reader is referred to Rizzo, Schultheis & Rothbaum, (2002).

Such concerns underscore the need for the careful definition of VR as a clinical tool, not unlike the various instruments currently used by therapists for assessment and rehabilitation (e.g., psychometric tests, biofeedback procedures). Ultimately, the goal of providing "low cost libraries" of VR applications will be driven by both scientific evidence and market forces. While this will provide options for professionals, the clinical judgment as to what VR applications are appropriate should remain an individualized decision between an informed patient and clinician, as is the case with currently available NP methods.

12. The option for self-guided independent testing and training by clients when deemed appropriate.

Independent self-assessment and "home-based" skills practice by clients are common components of most forms of rehabilitation. Generally, it is accepted that by having clients do "homework", that this will promote generalization of skills learned in treatment proper, to everyday behavior. The widespread increase in access to personal computing over the last decade has also encouraged the autonomous use of computerized cognitive self-help software by clients (for better or worse!). As such, it is likely that the independent use of VR will also become more common as access to systems and software expands in the future. Notwithstanding the potential for shoddy VR applications to reach the marketplace with little evidence to support their efficacy or value, the option for independent VR use (when "guided" by an appropriate professional) could be viewed as an asset for a number of reasons. When compared with existing computerized testing and training formats, VR is distinguished by its capacity to provide higher levels of both immersion and interactivity between the user and the VE. These unique features are seen to enhance the suspension of disbelief required to generate a sense of presence within the VE. When this psychological state of presence occurs, it is conjectured to create a user-experience that may influence task performance (Sadowski & Stanney, 2002). This user experience may produce behaviors that are different from what typically occurs in persons undergoing traditional testing and training due to the user's attention being more occupied "within" the VE. As well, the user experience may be less "self-conscious" due to the perceived removal of the test administrator from the immediate personal and attentional space. This experience could provide a unique qualitative window into how people perform tasks when operating in a more independent and autonomous fashion. For example, if clients were allowed to freely interact within a functional VE (i.e., office, mall, home, etc.) that contained very subtle test challenges, the recording and later observation of more naturalistic client behaviors would be possible. This could include observing how individualized compensatory or problem-solving strategies are spontaneously employed when challenged with a complex situation. As well, this may also be of value for monitoring decision-making in the assessment of potentially hazardous real world skills in a VE, such as driving an automobile.

Another approach might involve the development of a scenario that allows for repeated visits in which the user must monitor and make adjustments in controllable events occurring in the VE over longer periods of time (with progress "saved" at the end of each visit and carried over to the next). This would be akin to what occurs in the game "SimCity" and might provide insight into the factors that influence client performance extended over long periods of time. Similarly, for the retraining of specific skills, a client's autonomous interaction with the dynamic features of a VE could help capitalize on the established benefits of active learning over more passive approaches. In this area, differential learning effects have been reported in VR training, with active interaction better supporting route learning (Rose et al. 2001) and mental rotation training (Rizzo, et al. 2001c) compared to passive observation training. Such studies lend support for a rehabilitation perspective that underscores the significance of empowering individuals by providing active opportunities for error and experience. While traditional NP rehabilitation may value the opportunities for self-guided evaluations and practice, oftentimes feasibility is limited for a variety of reasons (i.e., limited therapy time, safety concerns). In this regard, the use of VEs may provide a mechanism for allowing safe, repeatable, self-guided, independent testing and training.

While this asset may offer a useful option for clinical assessment and retraining, it is not without the potential for risk. That is, while immersion and interactivity may enhance the "realism" of a VE, these same features may also create difficulties for certain individuals with psychiatric conditions or cognitive impairments that produce distorted reality testing. Specifically, such conditions could result in increased vulnerability for negative emotional responses during or following VR exposure. Although such incidents have yet to be reported in the VR literature, insurances for monitoring behavior and responses during VR exposure become an ethical responsibility for the professional. While current therapeutic uses of VR still require a clinician to be present, future applications may not have this requirement and this potential "opportunity" again underscores the need for advance consideration of the ethical issues pertinent to the use of VR as a clinical tool.

IV. Conclusions

It is our view that the use of computer-based VR simulation technology will play an increasing role in how NP assessment and rehabilitation is done in the future. Advances in the underlying enabling technologies and continuing cost reductions in system hardware are expected to make it possible for VR to shortly become a mainstream tool in this area. With this view in mind, this article has aimed to provide a detailed specification of the assets that are available when using VR for NP approaches. VR is not a panacea for all challenges in neuropsychology. It may also be unlikely that any one application would make use of all of the assets specified in this article. However, it is hoped that neuropsychologists with an interest developing and/or applying VR applications will find this detailed asset specification useful for targeting areas that could maximize value in their area of expertise. Although some overlap between certain assets may seem to appear at first glance, each asset is seen to represent a unique facet that could be harnessed to address specific challenges in neuropsychology. Also, by integrating examples of current NP rationales and applications with specific reference to component VR assets, it is hoped that clinicians and researchers may use this information to communicate more effectively with computer science-based VR system developers. The task of building really good VR/NP systems that are both usable and useful is a challenging endeavor that requires a multidisciplinary mix of domain-specific knowledge. This can be best accomplished by combining an informed view of what is possible with the technology with what makes the most sense from a clinical perspective. With proper attention to these issues, it is hoped that effective VR system development will bring the benefits of the information age to those with impairments due to CNS dysfunction.

References:

Alessi, N.E., & Huang, M.P. (2000). Evolution of the virtual human: From term to potential application in psychiatry. CyberPsychology and Behavior, 3, 321-326.

Anderson, P., Rothbaum, B. O. & Hodges, L. F. (in press). Virtual reality in the treatment of social anxiety: Two case reports. Cognitive and Behavioral Practice.

Astur, R.S., Oriz, M.L. & Sutherland, R.J., (1998). The characterization of performance by men and women in a virtual Morris water task: A large and reliable sex difference. Behavioral Brain Research, 93, 185-190.

Barrett, J., McCrindle, R.J., Cook, G.K., Booy, D.A., Accessible interface design: A review of access to information and communication technology for older and disabled people, Technical Report, 2002, available from Department of Computer Science, The University of Reading.

Biederman, I. (2002). A Neurocomputational Hypothesis of Perceptual and Cognitive Pleasure. Invited paper presented at the 9th Joint Symposium on Neural Computation. California Institute of Technology, Pasadena, May, 2002. 

Blascovich, J., Loomis, J., Beall, A.C., Swinth, K.R., Hoyt, C.L. & Bailenson, J.N. (2002). Immersive Virtual Environment Technology as a Methodological Tool for Social Psychology. Psychological Inquiry,13, 103-124.

Bowman, D., Kruijff, E., LaViola, J.J. & Poupyrev, I. (2001). An introduction to 3-D user interface design. Presence: Teleoperators and Virtual Environments, 10, 96-108.

Brooks, B.M., McNeil, J.E., Rose, F.D., Greenwood, R.J., Attree, E.A. & Leadbetter, A.G. (1999). Route learning in a case of amnesia: A preliminary investigation into the efficacy of training in a virtual environment. Neuropsychological Rehabilitation, 9, 63-76.

Brown, D. J., Kerr, S. J., & Bayon, V. (1998). The development of the Virtual City: A user centred approach. In Sharkey, P., Rose, D. & Lindstrom, J. (Eds.), Proceedings of the 2nd European Conference on Disability, Virtual Reality and Associated Techniques (pp. 11-16). Reading UK: University of Reading.

Carney, N., Chesnut, R.M., Maynard H., Mann, N.C., Patterson, P. & Helfand, M. (1999). Effect of cognitive rehabilitation on outcomes for persons with traumatic brain injury: A systematic review. Journal of Head Trauma Rehabilitation. 14, 277-307.

Costas, R., Carvalho, L. & de Aragon, D. (2000). Virtual city for cognitive rehabilitation. In Sharkey, P., Cesarani, A., Pugnetti, L. & Rizzo, A. (Eds.), Proceedings of the 3rd International Conference on Disability, Virtual Reality, and Associated Technology (pp. 305-313). Reading, UK: University of Reading.

Charness, N., Milberg, W., & Alexander, M. P. (1988). Teaching an amnesic a complex cognitive skill. Brain Cognition, 8, 253-272.

Christiansen, C., Abreu, B., Ottenbacher, K., Huffman, K., Massel, B., & Culpepper, R. (1998) Task performance in virtual environments used for cognitive rehabilitation after traumatic brain injury. Archives of Physical Medicine and Rehabilitation, 79, 888-892.

Cohen, N. J. & Squire, L. R. (1980). Preserved learning and retention of pattern-analyzing skill in amnesia: dissociation of "knowing how" and "knowing that." Science, 210, 207-209.

Connor, B. Wing, A.M., Humphreys, G.W., Bracewell, R.M. & Harvey, D.A. (2002). Errorless Learning Using Haptic Guidance: Research in Cognitive Rehabilitation Following Stroke. In Sharkey, P. Lanyi, C.S. & Standen, P. (Eds.), Proceedings of the 4th International Conference on Disability, Virtual Reality, and Associated Technology. Reading, UK: University of Reading, 77-86.

Cromby, J., Standen, P., Newman, J., Tasker, H. (1996). Successful transfer to the real world of skills practiced in a virtual environment by student with severe learning disabilities. In: Sharkey, P.M. (Ed) Proceedings of the 1st European Conference on Disability, Virtual Reality and Associated Technologies. Reading, UK: University of Reading, 305-313.

Davies, R. C., Johansson, G., Boschian, K, Lindén, A., Minör, U., & Sonesson, B. (1998). A practical example using virtual reality in the assessment of brain injury. In Sharkey, P., Rose, D. & Lindstrom, J. (Eds.), Proceedings of the 2nd European Conference on Disability, Virtual Reality and Associated Techniques. Reading UK: University of Reading, 61-68.

Davis, E.S. & Wachtel, J, (2000). Interactive Driving Simulation as a Tool for Insight Development and Motivation in a Rehabilitation Setting. . Paper presented at the 8th Annual Medicine Meets Virtual Reality Conference, Newport Beach, CA. January 22, 2000.

Deutsch, J.E., Latonio, J., Burdea, G. & Boian, R. (2001) Post-Stroke rehabilitation with the Rutgers Ankle System A Case Study. Presence: Teleoperators and Virtual Environments, 10, Presence: Teleoperators and Virtual Environments, 10, 416-430.

Digiplay Initiative, (2002). Website found at: http://www.digiplay.org.uk/index2.php

Doherty, E., Bloor, C. & Cockton, G. (1999). The "Cyberlink" brain-body interface as an assistive technology for persons with traumatic brain injury: Longitudinal results from a group of case studies. CyberPsychology and Behavior, 3, 249-260.

Elkind, J.S., Rubin, E., Rosenthal, S., Skoff, B. & Prather, P. (2001). A simulated reality scenario compared with the computerized Wisconsin Card Sorting test: An analysis of preliminary results. Cyberpsychology and Behavior, 4, 489-496.

Graf, P. & Schacter, D. L. (1985). Implicit and explicit memory for new associations in normal and amnesic patients. Journal of Experimental Psychology: Learning, Memory, & Cognition, 11, 501-518.

Greenhill, L.L. (1998). Diagnosing Attention-Deficit/Hyperactivity Disorder in Children. Journal of Clinical Psychiatry 59 (Suppl. 7), 31-41.

Harrison, A., Derwent, G., Enticknap, A., Rose, F.D. & Attree, E.A. (2002). The role of virtual reality technology in the assessment and training of inexperienced powered wheelchair users. Disability and Rehabilitation, 24, 599-606.

Jack, D., Boian, R., Merians, A., Tremaine, M., Burdea, G., Adamovich, S., Recce, M. & Poizner, H. (2001). Virtual Reality-Enhanced Stroke Rehabilitation. IEEE Transactions on Neurological Systems and Rehabilitation Engineering, 9, 308-318.

Jacobs, W.J., Laurance, H.E., & Thomas, K. (1997). Place learning in virtual space I: Acquisition, overshadowing and transfer. Learning and Motivation. 28, 521-541.

Jaffe, D.L. (1998). Use of virtual reality techniques to train elderly people to step over obstacles. Paper presented at the Technology and Persons with Disabilities Conference, Los Angeles, CA.

Jansson, G. (2000). Basic issues concerning visually impaired people's use of haptic displays. In Sharkey, P., Cesarani, A., Pugnetti, L. & Rizzo, A. (Eds.), Proceedings of the 3rd International Conference on Disability, Virtual Reality, and Associated Technology. Reading, UK: University of Reading, 33-38.

Johnston, R. (1995). Is it live or is it memorized? Virtual Reality Special Report. 2, 53-56.

Krueger, M.W. (1993). The experience society. Presence: Teleoperators and Virtual Environments, 2, 162-168.

LaPlante, W.P., (2001). A proposal for a research agenda in computer human interaction. Paper presented at The ACM SIGCHI 2001 State of the Science Exchange. Available at: http://www.icdri.org/technology/indexbp.htm

Lezak, M. D. (1995). Neuropsychological Assessment. New York: Oxford University Press.

Liu, L., Miyazaki, M. & Watson, B. (1999). Norms and validity of the DriVR: A virtual reality driving assessment for persons with head injuries. Cyberpsychology & Behavior, 2, 53-67.

Lowenstein, D. (2002). Essential facts about the video and computer game industry. Website found at: http://www.idsa.com/pressroom.html

Lumbreras, M. & Sanchez, J. (2000). Usability and cognitive impact of the interaction with 3D virtual interactive acoustic environments by blind children. In Sharkey, P., Cesarani, A., Pugnetti, L. & Rizzo, A. (Eds.), Proceedings of the 3rd International Conference on Disability, Virtual Reality, and Associated Technology. Reading, UK: University of Reading, 129-136.

McGeorge, P., Phillips, L.H., Crawford, J.R., Garden, S.E., Della Sala, S., Milne, A.B., Hamilton, S. & Callander, J. (2001). Using virtual environments in the assessment of executive dysfunction. Presence-Teleoperators and Virtual Environments. 10, 375-383.

Mendozzi, L., Motta, A., Barbieri, E., Alpini, D. & Pugnetti, L. (1998). The application of virtual reality to document coping deficits after a stroke: Report of a case. Cyberpsychology & Behavior, 1, 79-91.

McComas, J., MacKay, M. & Pivak, J. (2002). Effectiveness of virtual reality for teaching pedestrian safety. CyberPsychology and Behavior, 5, 185-190.

McLaughlin, M.L., Hespanha, J.P. & Sukhatme, G. (2002). Touch in Virtual Environments. New Jersey: Prentice- Hall/IMSC Press.

Moore, K., Wiederhold, B.K., Wiederhold, M.D., Riva, G. (2002). Panic and Agoraphobia in a Virtual World. Cyberpsychology And Behavior, 5, 197-203.

Moncur, M. & Moncur, L. (2002). QuotationsPage.com. At: http://www.quotationspage.com/quotes/Plato/ Morrison, J.E. &

Meliza, L.L. (1999). Foundations of the after action review process. U.S. Army Research Institute for the Behavioral and Social Sciences Special Report #42.

Mowafty L. & Pollack, J. (1995). Train to travel. Ability, 15, 18-20.

Naveh, Y., Katz, N. & Weiss, P. (2000). The effect of interactive virtual environment training on independent safe street crossing of right CVA patients with unilateral spatial neglect. In Sharkey, P., Cesarani, A., Pugnetti, L. & Rizzo, A. (Eds.), Proceedings of the 3rd International Conference on Disability, Virtual Reality, and Associated Technology. Reading, UK: University of Reading, 243-248.

Neisser, U. (1978). Memory: what are the important questions? In M. M. Gruneberg, P. E. Morris, & R. N. Sykes (Eds.), Practical Aspects of Memory. London: Academic Press. 3-24.

North, M., North, S. & Coble, J. (2002). Virtual Reality Therapy: An effective treatment for psychological disorders. In Stanney, K. (Ed.) Handbook of Virtual Environments. New York: L.A. Earlbaum. 1065-1079.

Park, N.W. & Ingles, J.L. (2001). Effectiveness of attention rehabilitation after an acquired brain injury: a meta- analysis. Neuropsychology, 15, 199-210.

Pertaub, D., Slater, M. & Barker, C. (2002). An experiment on public speaking anxiety in response to three different types of virtual audience. Presence-Teleoperators and Virtual Environments. 11, 68-71.

Prensky, M. (2001). Digital Game-Based Learning. New York: McGraw-Hill.

Prigatano, G.P. (1997). Learning from our successes and failures: Reflections and comments on "Cognitive rehabilitation: How it is and how it might be". Journal of the International Neuropsychological Society, 3. 497 - 499.

Psotka, J. (1995). Immersive training systems: Virtual reality and education and training. Instructional Science, 23, 405-431.

Pugnetti, L., Mendozzi, L., Motta, A., Cattaneo, A., Barbieri, E., & Brancotti, S. (1995). Evaluation and retraining of adults' cognitive impairments: Which role for virtual reality technology? Computers in Biology and Medicine, 25, 213-227.

Pugnetti, L., Mendozzi, L., Attree, E., Barbieri, E., Brooks, B.M., Cazzullo, C.L., Motta, A. & Rose, F.D. (1998). Probing memory and executive functions with virtual reality: Past and present studies. CyberPsychology and Behavior,1, 151-162.

Rickel, J., Marsala, S., Gratch, J., Hill, R., Traum, R., & Swartout, W. (2002). Toward a new generation of virtual humans for interactive experiences. IEEE Intelligent Systems, 17, 32-38.

Rickel, J., & Johnson, W. L. (1999). Animated Agents for Procedural Training in Virtual Reality: Perception, Cognition, and Motor Control. Applied Artificial Intelligence , 13, 343-382.

Rizzo, A.A. (1994). Virtual Reality applications for the cognitive rehabilitation of persons with traumatic head injuries. In Murphy, H.J. (ed.), Proceedings of the 2nd International Conference on Virtual Reality And Persons With Disabilities. CSUN: Northridge. (available at: http://www.csun.edu/cod/conf/1994/proceedings/Table94.htm)

Rizzo, A.A., Buckwalter, J.G., Humphrey, L., van der Zaag, C., Bowerly, T., Chua, C., Neumann, U., Kyriakakis, C., van Rooyen, A. & Sisemore, D. (2000). The Virtual Classroom: A Virtual Environment for The Assessment And Rehabilitation Of Attention Deficits. CyberPsychology and Behavior. 3, 483-499.

Rizzo, A.A., Neumann, U., Enciso, R., Fidaleo, D. & Noh, J.Y. (2001a). Performance Driven Facial Animation: Basic research on human judgments of emotional state in facial avatars. CyberPsychology and Behavior, 4, 471- 487.

Rizzo, A.A., Neumann, U., Pintaric, T. and Norden, M. (2001b). Issues for Application Development Using Immersive HMD 360 Degree Panoramic Video Environments. In: Smith, M.J., Salvendy, G., Harris, D. & Koubek, R.J. (Eds.) Usability Evaluation and Interface Design. New York: L.A. Erlbaum. 792-796.

Rizzo, A.A., Buckwalter, J.G., Bowerly, T., McGee, J., van Rooyen, A., van der Zaag, C., Neumann, U., Thiebaux, M., Kim, L., Pair, J. & Chua, C. (2001c). Virtual Environments for Assessing and Rehabilitating Cognitive/Functional Performance: A Review of Project's at the USC Integrated Media Systems Center. Presence: Teleoperators and Virtual Environments, 10, 359-374.

Rizzo, A.A., Buckwalter, J.G. & van der Zaag, C. (2002a). Virtual Environment Applications for Neuropsychological Assessment and Rehabilitation. In Stanney, K. (Ed.) Handbook of Virtual Environments. New York: L.A. Earlbaum. 1027-1064.

Rizzo, A.A., Bowerly, T., Buckwalter, J.G., Schultheis, M.T., Matheis, R., Shahabi, C., Neumann, U. Kim, L., & Sharifzadeh, M. (2002b). Virtual Environments for the Assessment of Attention and Memory Processes: The Virtual Classroom and Office. In Sharkey, P., Lanyi, C.S. & Standen, P. (Eds.), Proceedings of the 4th International Conference on Disability, Virtual Reality, and Associated Technology. Reading, UK: University of Reading, 3-12.

Rizzo, A.A., Schultheis, M.T. & Rothbaum, B. (2002c). Ethical issues for the use of virtual reality in the psychological sciences. In S. Bush & M. Drexler (Eds.), Ethical Issues in Clinical Neuropsychology. Lisse, NL: Swets & Zeitlinger Publishers. 243-280.

Rizzo, M., Reinach, S., McGehee, D. & Dawson, J. (1997). Simulated car crashes and crash predictors in drivers with Alzheimer's disease. Archives of Neurology, 54, 545 - 551.

Riva, G., Bacchetta, M. Baruffi, M., Rinaldi, S., Molinari, E. (1999). Virtual reality based experiential cognitive treatment of anorexia nervosa. Journal of Behavior Therapy and Experimental Psychiatry, 30, 221-230.

Rose, F.D., Attree, E.A., & Johnson, D. A. (1996). Virtual reality: an assistive technology in neurological rehabilitation. Curr Opin Neurol., 9, 461-467.

Rose, D., Brooks, B.M. & Attree, E.A. (2000). Virtual reality in vocational training of people with learning disabilities. In Sharkey, P., Cesarani, A., Pugnetti, L. & Rizzo, A. (Eds.), Proceedings of the 3rd International Conference on Disability, Virtual Reality, and Associated Technology. Reading, UK: University of Reading. 129-136. Rose, F.D., Attree. E.A., Brooks, B.M. & Andrews, T.K. (2001). Learning and memory in virtual environments - a role in neurorehabilitation? Questions (and occasional answers) from UEL. Presence-Teleoperators and Virtual Environments. 10, 345-358.

Robertson, I. (1990). Does computerized cognitive rehabilitation work? A review. Aphasiology, 4, 381-405.

Sadowski, W. & Stanney, K. (2002). Presence in virtual environments. In Stanney, K. (Ed.) Handbook of Virtual Environments. New York: L.A. Earlbaum. 791-806.

Schultheis M.T. & Mourant, R.R. (2001). Virtual Reality and Driving: The Road to Better Assessment of Cognitively Impaired Populations. Presence: Teleoperators and Virtual Environments , 10, 436-444.

Schultheis, M.T. & Rizzo, A.A. (2002). The Virtual Office: Assessing & Re-training Vocationally Relevant Cognitive Skills. Paper presented at the 10th Annual Medicine Meets Virtual Reality Conference, Newport Beach, CA. January 26, 2002.

Shallice T, Burgess P. Higher order cognitive impairments and frontal lobe lesions in man. In: Levin H.S., Eisenberg, H.M. & Benton, A.L., Eds. Frontal Lobe Function and Dysfunction. Oxford University Press, 1991:125-138.

Skelton, RW, Bukach, CM, Laurance, HE, Thomas, KG, Jacobs, J. (2000) Humans with traumatic brain injuries show place learning deficits in computer generated virtual space. J Clin Exp Neuropscyhol. 22; 157-175.

Sohlberg, M.M. & Mateer, C.A. (1989). Introduction to Cognitive Rehabilitation. New York: Guilford Press.

Sohlberg, M.M. & Mateer, C.A. (2001). Cognitive Rehabilitation: An Integrative Neuropsychological Approach. New York: Guilford Press.

Stanton, D., Foreman, N. & Wilson, P. (1998). Uses of virtual reality in clinical training: Developing the spatial skills of children with mobility impairments. In Riva, G., Wiederhold, B. & Molinari, E. (Eds.), Virtual Reality in Clinical Psychology and Neuroscience. Amsterdam: IOS Press. 219-232.

Sternberg, R. J. (1997). Intelligence and lifelong learning: What's new and how can we use it? American Psychologist, 52, 1134-1139.

Strickland, D. (2001).Virtual Reality and Multimedia Applications for Mental Health Assessment and Treatment. Paper presented at the ACM SIGGRAPH 2001 Conference. August 16, 2001.

Wiederhold, B.K. & Wiederhold, M.D. (1998). A review of virtual reality as a psychotherapeutic tool. CyberPsychology and Behavior, 1, 45-52.

Weiss, P. & Jessel, A.S. (1998). Virtual Reality Applications to Work. Work, 11, 277-293.

Wilson, B.A., Alderman, N., Burgess, P.W., Emslie, H. & Evans, J.J. (1996). BADS:Behavioural Assessment of the Dysexecutive Syndrome. Bury St Edmunds, UK: Thames Valley Test Company.

Wilson, B.A. & Evans, J.J. (1996). Error-free learning in the rehabilitation of people with memory impairments. Journal of Head Trauma Rehabilitation, 11, 54-64.

Wilson, B.A. (1997). Cognitive rehabilitation: How it is and how it might be. Journal of the International Neuropsychological Society, 3. 487 - 496.

Ylvisaker, M. & Feeney, T. (1998). Everyday people as supports: Developing competencies through collaboration. In.

Ylvisaker, M. (ed.). Traumatic brain injury: Children and adolescents. Newton, MA: Butterwort-Heinemann. Zimand, E., Anderson, P., Gershon, G., Graap, K., Hodges, L., & Rothbaum, B. (2003). Virtual Reality Therapy: Innovative Treatment for Anxiety Disorders. Primary Psychiatry. 9(7), 51-54.