Revealing the Umwelt through Transformative Sites
4.s52 Affective Computing
Professor Takehiko Nagakura,
Guzden Varinlioglu
With Merve Akdoğan
The way environments are represented and experienced are highly humancentric, ignoring primary users of sites that are non-human. We look towards new forms of representation and immersion that takes on the concept of the umwelt. Point cloud models as a representational tool has the potential to engender new forms of awareness in both quantitative and qualitative terms about landscapes in peril. With immersive technologies like VR, we can gain the ability to experience other perspectives and design a multispecies world.
Research questions:
How does non-human player movement and navigation translate to EEG?
Does a background in immersive technologies and/or gaming have an influence on how comfortable a player is able to move around?
How do players make decisions on their path through drawing and moving with their bodies?
How big of a role does locomotion play in virtual exploration?
Does the idea of the umwelt come across through point cloud representation and non-human navigation?
Subjects:
This study is planned to include a sufficient number of both gamer and non-gamer participants to provide comparisons in a designed VR environment using EEG measurements. The only comparison between people are made based on this information.
Experiment Setup:
The first player enters the game and spends 2 minutes playing.
EEG starts on first player.
Particle traces of the first player are recorded in the Unreal Engine setup.
The first player exits the game.
The second player then enters the game and spends 2 minutes playing.
EEG recording starts on the second player.
The subject is notified when the time ends.
The subject is asked to complete a post-experiment questionnaire.
An informal discussion and sharing of insights takes place.
Spatial Sound Setup:
In the VR environment uses attenuation distances and audio recordings from the site to create a spatial sound design that layered various audio elements in a way that felt immersive and realistic, influencing the bodily movements of participants such as head tilt, flapping, and attention span.
Methodology
Subjects:
This study is planned to include a sufficient number of both gamer and non-gamer participants to provide comparisons in designed VR environment using EEG measurements. The only comparison between people are made based on this information.
Player Backgrounds:
Total number of participants: 10
Total number of successful participant data: 8
Number of advanced gamers: 4
Number of participants with a gaming/VR background: 2
Number of participants with no gaming background: 2
Non-human Locomotion:
Virtual flight allows for the player to experience the sight through an avion lens using active movement.
Comparison of Relative Brain Wave
Comparing relative brainwave data between participants is generally more meaningful than comparing absolute brainwave data, as absolute brainwave data can vary significantly between studies due to factors such as recording conditions and preprocessing techniques.
Relative brainwave data reflects the relative power or intensity of different brainwave frequency bands within a person's brain activity and can provide valuable insights into their mental state.
Advanced Gamers
Gaming Background
No Background
Channel 1
Delta waves on a graph of relative brain waves represents activity in the delta frequency range (0.5-4 Hz) and is typically associated with deep sleep and unconsciousness, providing information on an individual's sleep and arousal levels.
Insights on what triggers the dips and peaks: head tilts, extreme bodily movements.
Advanced Gamers
Gaming Background
No Background
Channel 2
Theta waves on a graph of relative brain waves represents activity in the theta frequency range (4-8 Hz) and is typically associated with relaxation, meditation, creativity, curiosity.
Advanced Gamers
Gaming Background
No Background
Channel 3
Alpha waves are a type of brain waves with a frequency of 8 to 12 Hz that are typically associated with relaxation and meditation.
Insights: Advanced gaming experience was found to be associated with higher alpha wave activity during non-human navigation tasks. This suggests that individuals with advanced gaming backgrounds may experience a greater sense of comfort and relaxation during such tasks compared to individuals without gaming experience.
Advanced Gamers
Gaming Background
No Background
Channel 4
Beta waves on a graph of relative brain waves represents activity in the beta frequency range (12-30 Hz) and is typically associated with alertness, concentration, and active thinking, providing information on an individual's cognitive activity and level of alertness.
Advanced Gamers
Gaming Background
No Background
Channel 5
Gamma waves on a graph of relative brain waves represents activity in the gamma frequency range (30-100 Hz) and is typically associated with high levels of consciousness and focus, providing information on an individual's level of consciousness and focus.
Advanced Gamers
Gaming Background
No Background
Site: Quincy Quarries
Location: Quincy, MA
Was a highly profitable granite quarry and where America’s large scale granite quarrying industry was born
The stone taken from here was famously used for the Bunker Hill Monument
After its closure it became a deadly site for cliff jumpers who would jump into the pits
Dirt from Boston’s “Big Dig” was placed here to prevent further accidents
Now home to a variety of artists and community members, such as graffiti artists, rock climbers, tight-ropers, performance artists, and of course diverse non-human species.
Point Cloud Models as Representational Tools
Point clouds have the ability not only to communicate qualitatively through aesthetic effect but quantitatively as well. They act as a replica immersive environments that provides additional information about how a particular species perceives or is impacted by its surroundings [umwelt]. They question the act of visually communicating landscape conditions endemic to the climate catastrophe, such as sites of extraction like Quincy Quarries, which can inform the development of new modes of representation that impact change. It also offers to expose the temporality of landscape through other forms of imaging, surveying, and data while exploring the ephemeral through non-human agency.
Questionnaire Analysis
Delta waves : individual's sleep and arousal levels.
Theta waves : relaxation, meditation, creativity, curiosity.
Alpha waves : relaxation and meditation.
Beta waves : cognitive activity and level of alertness.
Gamma waves : level of consciousness and focus.
Player 1: Advanced Gamer
“Those point clouds look like stars since a player can see through the ground, there is no feeling of ground, making me feel that I am always floating. It render the experience surreal.” [visualization]
“Weightlessness. Flying in a canyon.” [embodiment]
“Gave me a sense of speed and location” [navigation]
“It felt like flying a helicopter” [navigation]
“They leave room for imagination, they are ambiguous” [visualization]
“I would focus more on head tilting for navigation, and provide some kind of feedback” [navigation]
Player 7: No background
“Peace, discovery, open-mind” [visualization]
“The traces left behind their trail made me think about what their experience navigating this magical VR world must have been like” [visualization]
“The physicality of flapping my arms was quite challenging at first but gradually, even within 3 minutes, it started feeling almost natural.” [navigation]
“Ambiguous, yet informative” [visualization]
“Perhaps the seagull image can also be as abstracted as the surrounding environment” [visualization]
Player 4: Advanced Gamer
“Psychedelic.” [visualization]
“The arm flapping action made it quite a different experience - often a bit tiring, but definitely a very new perspective.” [embodiment]
“I didn't really engage with the other player.” [navigation]
“They referenced some geometric forms, and yet they left a lot to imagination.” [visualization]
“Point clouds make it a very surreal experience. Maybe use surfaces as well just as a comparison?” [visualization]
Player 8: No Background
“I was careful about the ghost path before me” [interaction]
“It was freeing and I loved free falling” [navigation]
“I wanted to make sure I didn’t overstep or overshadow the ghost players path before me [navigation]
“I think it allowed me more freedom to explore” [navigation]
“It provided a transparent structure as a baseline” [visualization]
Many players were attracted to the ghost path or their own path.
More advanced gamers tend to fly higher and faster while others stay a bit closer to the ground.
The landscape reveal also had an impact on the player’s chosen path.
Revealing the Umwelt
Using the VR along with point cloud models creates a new way of understanding the umwelt.
Seeing, learning, and navigating through the eyes of the animal allows for a new experience of site and environment that can enact change in the way we represent and design [rather than redemptive imaging or greenwashing].
Although it is incredibly difficult to actually understand and embody all the umwelten that may exist on a site, immersive technologies can get closer to this experience.
Using sensing technologies, such as EEG or GSR, can provide some insight into how other species may perceive and sense their own realities.
Conclusions
There is a lack of information about how mimicking non-human navigation through body gestures and movement affects brain activity as assessed by brainwave data.
Studies have shown that the usage of immersive technologies and/or gaming might affect a player's comfort in navigating the environment and choosing their next actions via bodily drawing and gestures.
The ability to move around in a virtual environment appears to play a critical element in exploration, since it allows players additional mobility and flexibility as they go around. Finally, point cloud representation and non-human navigation can embody the idea of the umwelt, which is the unique perceptual world of each individual, allowing gamers to explore the environment in a more meaningful and powerful way.