my role in this project
Interactive installation design, story telling testing, on-set installation assembling and design, coding and vision review
Sensing Nature, 2023
Goldsmiths computational arts MA final project.
Co-Creator: Tianyuan Zhang; Technical Support: Dingye Zhang
This page showcases the original version of Sensing Nature. The lastest version is not released.
“Sensing Nature” is designed to give you the feeling of awe that connecting with the natural world can bring. Touching and exploring the textiles and textures of the magic tree triggers a flow of layered interactions bringing together art and music to lift the spirits. Each individual’s sensory experience will be unique.
The background of this project centers around haptic sensing by collaborating multiple sensors. In today's digital age, people tend to immerse themselves in visual and auditory stimuli offered by screens and electronic devices. However, we must not underestimate the significance of touch as a means of emotional communication within the physical realm. Touch is very important way to communicate for emotion in a physical space. Research has shown that even a brief, gentle touch can have remarkable effects on an individual's well-being. It can reduce anxiety, alleviate stress, and mitigate symptoms of depression (Dueren al. 2023, p.1). However, there is a noticeable absence of haptic feedback in the realm of artistic exploration within virtual and augmented reality. Touch interactions are integral to countless human activities, yet among the various haptic technologies developed, very few are designed to enhance natural touch interactions. Furthermore, there is a dearth of technologies that can computationally augment free-hand interactions with real-world environments (Kawazoe, et al. 2021, p.835). Many of the haptic devices currently available are centered around controllers, instrumented surfaces, or hardware interfaces that necessitate manual operation by the hands. An interesting finding from a study by Gibbs( 2022, 157p) indicates that the incorporation of haptic feedback alone can significantly enhance the sense of presence in virtual environments when compared to relying solely on visual feedback. Consequently, 'Sensing Nature,' delves into the realm of touch to explore the growth of nature within virtual and augmented reality. Our primary focus is on replicating nature-inspired sensations, such as the feeling of wind and vibrations, through haptic feedback. By complementing these tactile sensations with visual and auditory cues, we aim to create a more immersive sense of existence. In pursuit of this objective, we have designed haptic feedback mechanisms that seamlessly integrate with free-hand interactions, enhancing the user's engagement with virtual and augmented environments. Our aim is to bridge the gap between the digital and physical worlds, creating a holistic and captivating user experience. The sketch is under (picture 1- 4).
Process
skectch
Technical Implementation
Incorporating Five Crucial Elements into the Project: Esp32, Conductive Thread, Haptic Feedback, OSC, and Unity VFX
1. Esp32 Huzzah Microcontroller:
The project's foundation lies in the utilization of the Esp32 Huzzah microcontroller, boasting integrated Wi-Fi capabilities and touch pins. This remarkable microcontroller serves as the bridge connecting physical devices, such as vibration motors and fans, with the Unity platform. The Esp32 plays a pivotal role in facilitating communication between these tangible components and Unity by transmitting touch pin data
2. Conductive Thread:
A key component in this endeavor is conductive thread, which serves as the linchpin for reading data when individuals interact with fabrics and textiles. This thread's unique property of being seamlessly sewn into textiles enables it to interface with the touch pins on the Esp32
3. Haptic Feedback Incorporating Fabrics, Vibration, and Wind:
The "Sensing Nature" experience is enriched through the incorporation of fabrics, vibrations, and wind as haptic feedback mechanisms. Firstly, fabrics were chosen for their ability to mimic various natural elements, thanks to their diverse textures. They offer tactile sensations reminiscent of petals, leaves, dandelions, and even the bark of a tree. Secondly, vibration motors breathe life into the tree's trunk, awakening it when touched. This simulation occasionally mirrors the rhythmic heartbeat of a tree as the motors operate intermittently. Lastly, the sensation of wind, a fundamental element of nature, breathes vitality into the project, mirroring its importance in sustaining plant life.
4. OSC (Open Sound Control):
The project seamlessly integrates OSC technology, a data transfer protocol that facilitates communication among various devices within the same network. To establish a network environment that connects Unity and Arduino, we implemented a Mango-based solution. Originally, we envisioned two versions of our project: one detecting touch through sensory inputs and another employing box collider interactions with hand tracking in Unity. Following multiple experiments, it became evident that OscJack was incompatible with Oculus hand tracking. Consequently, we opted to explore an alternative OSC solution, leading us to extOSC
5. Visualization and interaction in unity
The visualization is driven by the values received from the touch pins. The magic physical tree boasts a total of 18 sensors, each of which responds with visual and auditory feedback that corresponds to the material touched and the frequency of touch within a specified timeframe.
Reflection and Future Development
This project exceeded my expectations in terms of what I had initially set out to achieve. I successfully created a comprehensive interactive installation that incorporated physical interactive installation, stunning visualizations, and captivating audio. During the exhibition, the majority of users had an incredibly positive experience with my work. They were able to immerse themselves in the beauty of nature, free from any discomfort, and were encouraged to explore freely, unrestricted by any rules or guidelines.
This endeavor provided me with valuable insights into the art of communication through physical installations and the intricacies of coding interactive visualizations in unity. From my perspective, this project was an invaluable learning experience. I acquired new skills, such as using Osc to send and receive data from various microcontrollers, even within Unity. I also learned how to operate vibration motors and fans, improve touch pin data stability using averaging algorithms, trigger visual effects, and implement different timing mechanisms for calling other scripts.
However, I could make improvements is in stabilizing the touchpin data further. Some data fluctuations occurred when people were in contact with the installation, and this could potentially be addressed by replacing the touchboard. Additionally, finding a more streamlined method to map the virtual trunk to the physical trunk could save time, as I spent a significant amount of effort ensuring their alignment. Finally, the design of the trunk module proved to be crucial for display in various settings. This project marked my initial foray into creating XR (Extended Reality) pieces. Moving forward, I plan to expand my portfolio by crafting additional scenes, such as underwater and desert environments. Furthermore, I intend to develop more installations, incorporating elements like scent generators and temperature regulators to create a multisensory experience.
References
Dueren, A.L. et al. (2023) ‘Perspectives on interpersonal touch are related to subjective sleep quality’, Journal of sleep research, 32(3), p. e13766–n/a. doi:10.1111/jsr.13766.
Gibbs, J.K., Gillies, M. and Pan, X. (2022) ‘A comparison of the effects of haptic and visual feedback on presence in virtual reality’, International journal of human-computer studies, 157, p. 102717. doi:10.1016/j.ijhcs.2021.102717.
Kawazoe, A. et al. (2021) ‘Tactile Echoes: Multisensory Augmented Reality for the Hand’, IEEE transactions on haptics, 14(4), pp. 835–848. doi:10.1109/TOH.2021.3084117.
Picture
Adafruit DRV8833 DC/Stepper Motor Driver Breakout Board, 2023 by Pit Hut. https://thepihut.com/ [Online] https://thepihut.com/products/adafruit-drv8833-dc-stepper-motor-driver-breakout-board
Adafruit HUZZAH32 – ESP32 Feather Board, 2023 by Pit Hut. https://thepihut.com [Online] Accessed: https://thepihut.com/products/adafruit-huzzah32-esp32-feather-board-ada3405
GL.iNet GL-MT300N-V2 (Mango) Wireless Mini Portable VPN Travel Router, Mobile Hotspot in Pocket, WiFi Repeater Bridge, Range Extender, OpenVPN Client, 300Mbps High Performance, 128MB RAM, 2023. By Amazon. https://www.amazon.co.uk/ [Online] Accessed: https://www.amazon.co.uk/GL-iNet-GL-MT300N-V2-Converter-Pre-installed-Performance/dp/B073TSK26W/ref=sr_1_2?crid=1A88PJ7LZAV9P&keywords=mango+wifi&qid=1694238384&sprefix=mango+wifi%2Caps%2C60&sr=8-2
The pinout diagram, 2017 by tinymce. https://learn.adafruit.com/ https://thepihut.com/products/adafruit-huzzah32-esp32-feather-board-ada3405
Fluffy Feathers – White, 2023 by Corsage Creations, https://corsagecreations.co.uk/ [Online] Accessed: https://corsagecreations.co.uk/wedding-accessories/1293-fluffy-feathers-white-24cm-long-6pcs-per-pk-5060379313090.html
Code
OSC Jack, 2021, by Takahashi, K and Ibukuro, k. https://github.com [Online]. Available: https://github.com/keijiro/OscJack.
extOSC (Sigalkin, V et al. 2023). https://assetstore.unity.com/ [Online]. https://assetstore.unity.com/packages/tools/input-management/extosc-open-sound-control-72005