Link Search Menu Expand Document

Designing for Rich AR Experience

Chapter 3 of my thesis drew on a number of theoretical propositions, and put forward that AR has the potential to scaffold new modes of performance and expression in the arts and music, furthermore, that from an enactivist approach experience, this would consist in radically modulating the material, embodied, and spatial experience of participants. This is the starting point for ideating and designing an artistic AR experience in the present thesis. This pattern addresses the issue of the typicality of AR experience being simple interactions with visual overlay devices. It approaches experience ideation from a holistic and multisensory, or “modalities-encompassing” (Schraffenberger, 2018) perspective.

Furthermore, the ‘4Es’ of an enactivist approach can be considered as conditions for what could be described as immersive and “rich experience” (Bilbow et al., 2021). As highlighted in Section 3.3, enactivist principles have been offered as guidelines for the creation of interactive systems in the past; (Essl & O’Modhrain, 2006), (Armstrong, 2006), and (Hayes, 2019) suggest this approach in the design of new musical instruments.

The concept of rich experience also stands in stark contrast to the current direction of corporate XR technologies, where it is being developed to replace in-person interactions e.g. by facilitating in-headset ‘work from home’ virtual environments such as Meta Horizons. It also stands in contrast with the marketed push towards AR as a tool for driving commerce through targeted advertisements. How as artists and musicians can we avoid the corporate, commercial, ocularcentric, and overlay approach to AR? How can we offset the dystopian hell-scape, painted by designer and film-maker Keiichi Matusda in his various film shorts?

{The Pusher / The Entertainment} {Augmented (Hyper)Reality: Domestic Robocop} {HYPER-REALITY} {Merger}

Centre the experience on two or more sensory interactions

Whether it is Dewey’s concept of the “live creature”, or the contemporary enactivist’s framing of the importance of embodiment, the AR experience ought to be centred on two or more sensory interactions. It may include any combination of sensory interaction (display or sensing) types, i.e visual (vision), auditory (hearing), vestibular (movement and balance), olfactory (smell), gustatory (taste), and somatosensory (touch). This ensures grounding in the importance of the participants agency and sensorimotor structure. Here, the importance of considering the AR experience as an enaction, rather than an abstract internal representation that they are thinking and then acting upon is the important.

Invoke a meaningful relationship between the real and virtual

AR’s medium specificity, should be at the forefront of intentional design choices. If AR is unique because of its “invocation of relationships between real and virtual processes in the axes of spatial, thematic, material and ecological distance”, these relationships become a key handle by which artists and musicians can meaningfully steer experience to achieve aesthetic experiences. Consider the following:

  • Spatial: to what extent should the virtual and real be spatially aligned or even spatially related?
  • Thematic: to what extent should the virtual and real be thematically similar, what effect might this have on participants’ sense of sensory congruency if distant instead?
  • Material: to what extent should the virtual be materially similar to the real environment in which AR brings it into conversation with?
  • Ecological: to what extent should the virtual act as part of the real environment, how does this suit the overall narrative intention of the piece?

For the artist or musician, the interest and specificity of AR might lend itself to tending towards the revealing the differences rather than the similarities between the virtual and the real!

Implement an AR subform

In considering the embodied experience of participants, employ Schraffenberger’s taxonomy of AR subforms. As discussed in section 8.3 of my thesis, augmented embodiment is achieved through the fact that AR has the potential to radically modulate a participants sense of self, other, and environment. In my thesis has made a clear standpoint on the role of the military-industrial complex in biasing the development and conceptualisation of AR towards that of an overlay device or heads-up display. Artists and musicians engaging in AR will already be tending towards the altered and hybridised subforms of AR rather than the purely augmenting; but it is worth mentioning here that significant improvements in the effectiveness of AR in delivering aesthetic experience lies in considering the process (see subform) by which perception is mediated by AR.

Works Cited

  1. Armstrong, N. (2006). An Enactive Approach to Digital Musical Instrument Design [PhD thesis]. Princeton University.
  2. Bilbow, S., Kiefer, C., & Chevalier, C. (2021). The Value of Sound within a Multisensory Approach to AR in the Arts. Workshop Proceedings from Multisensory Augmented Reality, 8. https://drive.google.com/file/d/1OJBSo5_vJoHoFYHSr1cgDaEsm7iGW3y2/view?usp=sharing
  3. Essl, G., & O’Modhrain, S. (2006). An Enactive Approach to the Design of New Tangible Musical Instruments. Organised Sound, 11(3), 285–296. https://doi.org/10.1017/S135577180600152X
  4. Hayes, L. (2019). Beyond Skill Acquisition: Improvisation, Interdisciplinarity, and Enactive Music Cognition. Contemporary Music Review, 38(5), 446–462. https://doi.org/10.1080/07494467.2019.1684059
  5. Schraffenberger, H. (2018). Arguably Augmented Reality: Relationships between the Virtual and the Real [PhD thesis]. University of Leiden.