\“Alloplex BioAffective Worlds XR: Hydnum”

Extending Transmodal Worldmaking beyond Russel's Circumplex Model of Affect.

© transLAB, MAT, University of California, Santa Barbara / SBCAST 2022

research team: Iason Paterakis, Nefeli Manoudaki, Diarmid Flatley, Pau Roselló Diaz, Alan Macy

supervisor: Marcos Novak

“Hydnum” is an extended reality data-driven interactive installation that immerses the visitor in a transmodal continuum of scent, music, form, poetry, space, and light. This data-driven piece is based on research connecting virtual and physical elements to emotional affect and cross-sensory stimulus through human biodata. Hydnum extends James Russell's "Circumplex Model of Affect and Motivational State" into transmodal worldmaking in extended reality. 

Hydnum is a transmodal / symbiotic system consisting of a prototype medical scent delivery system, a cyber-physical artifact, an extended Reality experience, and a generative audio component. The biodata visualizations also extend into the surrounding area by using projection mapping. The interactive spatial system can transmit scents based on the user's real-time psychophysiological measures (heart rate and heart rate variability data).

A modular system of tubes, 3D printed outputs, and brackets emerges from the six scent modules of the BIOPAC modular scent delivery system and expands into the surrounding space, introducing a prominent tree-shaped structure. Our approach adopts the characteristics of a nonlinear network, usually referred to as "Rhizome" after the Ancient Greek word "ῥίζωμα", "rhízōma".

Hydnum is the result of the ongoing collaboration between the transLAB, BIOPAC systems, and SBCAST (Santa Barbara Center for Art, Science, and Technology). Hydnum is part of the <Alloplex: BioAffective Worlds XR Extending Transmodal Worldmaking beyond Russel's Circumplex Model of Affect> research initiative.

The installation has been presented twice during the End of the Year Show of the Media Arts and Technology Program, of the University of California, Santa Barbara.

The first event took place in Elings Hall on Dec 27, 2022, and the second event was held at the Santa Barbara Center of Arts and Technology (SBCAST) on June 2nd, 2022.

Contribution: Concept Design, VR Content Creation (Unreal Engine), Installation Design, Hardware Setup, Projection Mapping

Keywords: #Extended Reality, #Transmedia, #Scent Art, #Hydnum, #Spatial Design, #Transplex, #Digital #Futures, #Biodata, #Scentification, #Sonification

links:

Hydnum promo video

\Greece 2021 “Desire for Freedom”

© TUC TIE LAB 2021, “Greece 2021” Committee, ADAF

In June 2021, the most extensive projection mapping installation in Europe was simultaneously brought to life across 18 cities in Greece.

”On Saturday, June 12, and Sunday, June 13, 2021, the Greek Revolution "came to life" throughout Greece as a major event of the "Greece 2021" Committee. "DESIRE FOR FREEDOM" was screened on iconic buildings of 18 cities, using the technique of projection mapping. It is a visual narrative of the Struggle of 1821 through relevant works of art.

Athens (the Hellenic Parliament building), Alexandroupolis (Zarifeios Pedagogical Academy), Heraklion (the building of the Region of Crete), Thessaloniki (Royal Theatre), Ioannina (Judicial Hall), Kalamata (Visual Arts Workshop - Art Center), Corfu (Old Town Hall), Lamia (Municipal Theatre), Larissa (Club of Guard Officers), Messolonghi (Municipal Gallery), Naoussa (Town Hall), Nafplio (Town Hall), Piraeus (Municipal Theater of Piraeus), Rhodes (Town Hall), Syros (Apollo Theatre), Tripoli (Judicial Hall), Chania (Mikis Theodorakis Theatre), and Chios (Castle Wall - Porta Maggiore).

The narrative of the Revolution in a digital application, too.

About a month after the projections, a digital application began operating, which, through augmented reality, enables citizens and visitors of the 18 cities to watch the audiovisual narrative, which was shown on June 12 and 13 using their mobile phones or tablets.”

text source: https://greece2021.gr/en/provoles.html

links: https://www.youtube.com/watch?v=KZRjxncHSek

https://youtube.com/watch?t=804s&v=f4CWKtsJlxY

AR application on the App Store / Google Play

background video source: ©ADAF

“AI Worlds XR” is an extended reality (XR) installation that immerses the visitor in a transmodal continuum of AI, music, form, and space. The AI-generated forms express a continuous dialogue between the microcosms and the macrocosms. The piece is based on research connecting 2D AI images, biological forms (cells, and cosmic objects (Nebulae, star clusters, etc.). The biological forms are inspired by the German biologist Ernst Haeckel and his graphic work on biological forms.

VR, XR, and “Allotopes” with music and particle systems are formed inside a VR cosmos following a symbiotic principle. Rigid and familiar forms collide with natural and bioinspired structures to form an Allotope in Virtual Reality. The visualizations also extend into the surrounding area by using projection mapping. This research was expressed through VR and projection mapping in the transLAB during the end-of-the-quarter celebration of Tranvergence.

Hand drawings —generated with Procreate— were used along with text prompts to initiate the dialogues between AI Diffusion systems such as MidJourney and DALL-E. The 2D AI-generated images were then transformed into 3D forms using Blender’s shader nodes, and NVIDIA’s NERFs. A MAX MSP patch translated the images into spatial sound which was played back through the transLAB’s 16.2 sound system.

“AI Worlds” is based on the ongoing research initiative of <Alloplex: BioAffective Worlds XR Extending Transmodal Worldmaking beyond Russel's Circumplex Model of Affect>.

research team: Iason Paterakis, Nefeli Manoudaki

supervisor: Marcos Novak

Contribution: Concept Design, VR Content Creation (Unreal Engine 5.1), Installation Design, Projection Mapping, Sound Design

Software: Unreal Engine 5.1, Max MSP, Blender 3.3, Touchdesigner, Midjourney, Dall-E, NVIDIA NeRFs, MadMapper, Procreate

Keywords: #Extended Reality, #Transmedia, #Artificial Intelligence, #XR, #VR, #Allotopes, #Biological, #Digital #Futures, #Sonification

\"ETHO" (ἔθω) VR ”

an ML gallery in VR about the consequences of the war

© transLAB, MAT, University of California, Santa Barbara/ MIT XR Hack, Massachusetts Institute of Technology, Cambridge, Massachusetts

Etho refers to:

"... anyone afraid to talk about the war. To preserve history and remember the consequences and the impact of war. To raise awareness about the current hostile situation in Ukraine.

Etho, is the ancient Greek word that refers to the notion of a nation. Etho is offering an alternative reality that has been deprived of anything valuable and is composed of cultural ruins….”

“Etho” is a VR gallery related to the war in Ukraine. Aiming to introduce a strong feeling of immersion and detachment from our version of physical reality we used Virtual Reality technologies combined with physical interactive objects.

The artifacts are exhibited in a Virtual Apocalyptic Space surrounded by a fractal dome that contains life, culture, and the human. The user is immersed in the dark space containing remnants of a previous civilization. A high-level Machine Learning algorithm has processed the tangible remnants and projected them on the vertical spatial elements via virtual projection devices. The audio stimuli are granulated fragments from audio clips that refer to war and conflict.

More importantly, the piece is proposing a full embodiment of a fragmented cosmos. It is a balance between the safety of our western boundary and the remnants of a demolished civilization and culture.

research team: Iason Paterakis, Nefeli Manoudaki, Emma Zhao, Dillan Hoyos

Contribution: Concept Design, VR Content Creation (Unreal Engine 4.27)

Software: Unreal Engine 4.27, Max MSP, Blender 3.0, Procreate

Keywords: #Virtual Reality, #Transmedia, #Machine Learning, #XR, #VR, #Allotopes, #Virtualgallery, #Metaverse

\URBAN INTERACTIVE STUDIES

A series of Interactive Projection Mapping studies in urban environments.

UCSB, Media, Arts and Technology, Exhibitions @ SBCAST

© MAT, University of California, Santa Barbara / SBCAST 2022

The 2021-2022 End of the Year Show had three components, an Open Labs event, and two Exhibitions. The last exhibition occurred at the SBCAST with Studio F playing a prominent role in the event[1] [2]. This year’s show included the “Song Cycle: Live by Special Request” by Chris Kallmyer with members of the ambient collective called “superteam”, previously performing in the L.A. Philharmonic[3]. After the guest performance, the SBCAST was lit up with a large-scale projection mapping installation including an audio-visual performance by Professor Curtis Roads with his piece Always, an interactive perforated mixed reality installation with components of Virtual Reality, scent, and biodata, emotive robotics, 3D printed ceramics, light installations, etc. These performances and exhibitions go a long way in promoting an interface and exchange of ideas between the MAT/UCSB community, SBCAST, and the general public in Santa Barbara. Local Journalists are present at these events supporting and promoting our scope[4] [5] [6].

Photos of the UCSB MAT’s End of the Year Show, 1st Thursday Santa Barbara walk show, projections @ Santa Barbara Center for Art, Science and Technology (SBCAST), Santa Barbara, California, USA

works by Iason Paterakis, Nefeli Manoudaki, prof. Curtis Roads, Mert Toka, Jack Kilgore, Weidi Zhang, Pau Rosello Diaz projection mapping: Iason Paterakis, Nefeli Manoudaki contribution: Projection Mapping, Event co-organization, Content Creation (Blender, Max MSP, Touchdesigner)

Links:

[1] https://www.symades.net/

[2] https://www.independent.com/events/symades-2022-end-of-the-year-show-mat-ucsb/

[3] https://www.laphil.com/about/our-venues/venue-tours/chris-kallmyers-song-cycle

[4] https://www.montecitojournal.net/2022/06/28/mat-puts-on-a-show/

[5] https://www.montecitojournal.net/2022/05/31/media-arts-tech-grad-shows-on-display-may-27-june-2/

[6] https://www.youtube.com/watch?v=QDxzNnoAjt8&t=983s&ab_channel=LarryNimmer

Celestial Thoughts @ ADAF 2020

© Athens Digital Arts Festival 2020

Considering the World Wide Web as an infinite place composed of an infinite number of microscopic fragments- just like space. The  WWW represents an expanding -not to recognized scale- cosmos of digital thoughts, transmedia narratives. We are constantly seeking ways to express ourselves in this parallel virtual space.  Meanwhile, humans are looking for ways to explore the universe. To expand. To justify their existence. For, Space is vast and its darkness is deep. 

Humans neglect this fact by getting lost in their manufactured parallel space; the internet. Countless messages are flowing in the stream of the “internet”. A “milky way” of information,  and somewhere in there lies your thought, your message, your digital identity. 

project team: Iason Paterakis, Nefeli Manoudaki supervisors: Angelos Floros, M. Kaza contribution: Concept Design, Content Creation (C4D, Processing, TouchDesigner, AE)

TIE Lab Doodle

© TUC TIE lab 2020

TIE LAB DOODLE (real-time urban drawing + projection mapping)

© TUC TIE LAB 2020

TIE Lab Doodle is an application that allows any projection mapping façade to be turned into an interactive painting canvas. It uses a combination of a mobile and a desktop application in server‐client architecture. The desktop application (server) is running on the computer that controls the projection mapping software.

The mobile application (client) runs on an android phone or tablet and provides the functionality and tools to draw. With this combination of mobile application, desktop application, and projection mapping software the user is able to interactively draw on any surface/façade. This project has high interactivity and engagement levels.

research team: Marios Christoulakis, Iason Paterakis, Nefeli Manoudaki supervisors: Konstantinos- Alketas Oungrinis, Marianthi Liapi contribution: Concept Design, Projection Setup

URBAN PROJECTION MAPPING STUDIES

A series of Projection Mapping studies to measure the effects on Attention, Engagement, and Memory and introduce a correlation between the nature of the virtual content and the aforementioned Variables in URBAN ENVIRONMENTS.

research team : Iason Paterakis, Nefeli Manoudaki, Marios Christoulakis supervisors: Konstantinos- Alketas Oungrinis, Marianthi Liapi contribution: Concept Design, Content Creation (C4D, AE), Installation Design, Hardware Setup, Projection Mapping

PUBLICATIONS:

Iason PATERAKIS, Nefeli MANOUDAKI, Marios CHRISTOULAKIS, Marianthi LIAPI, Konstantinos – Alketas OUNGRINIS. VISUAL NARRATIVES. A methodology for layering and deconstructing digital projections. ISBN 978-3-9820-758-5-3/ S.Arch | In: Proceedings of the S.Arch 2020 / the 7th International Conference on Architecture and Built Environment with AWARDs | Tokyo, Japan / May 2020

Marios CHRISTOULAKIS, Marianthi LIAPI, Iason PATERAKIS, Nefeli MANOUDAKI, Konstantinos – Alketas OUNGRINIS. INTERACTIONS IN SPATIALLY AUGMENTED PLACES ISBN 978-3-9820-758-5-3/ In: Proceedings of the S.Arch 2020 / The 7th International Conference on Architecture and Built Environment with AWARDs | Tokyo, Japan / May 2020

Lorenz Attractor

Parameters:

σ = 10, ρ = 28, β = 8/3

System:

dx / dt = σ(-x + y)

dy / dt = -xz + ρx - y

dz / dt = xy - βz

Den Tsucs Attractor

Parameters:

a = 40, c = 0.833, d = 0.5, e = 0.65, f = 20

System:

dx / dt = α(y -x) + dxz

dy / dt = fy - xz

dz / dt = cz + xy - ex2

Chen - Lee Attractor

Parameters:

α = 5, β = -10, δ = -0.38

System:

dx / dt = αx - yz

dy / dt = βy + xz

dz / dt = δz + x

Allorenz Attractor

Parameters:

ρ = 3.94, σ = 8.79, β = 0.33

System:

dx / dt = yz

dy / dt = ρ(x-y)

dz / dt = (σ - β )xy - (1 - β) x2

The original thought came from the “Kinect imprint” of the user's figure, the composition concerns the existence of the body in a different virtual space. The captured figures appear in different virtual spaces - windows, through projection mapping technology. These interactive showcases-cells, symbolize the existential liberation from the physical world and its “entrapment” in a parallel digital environment.

The example of the kinectv2 library for processing was utilized as a basis for the experiment. Background videos were created in Cinema4d and were triggered when the user approached the sensor. Finally, the Processing program was integrated with Resolume via the Spout library for processing.

@7th Chania Film Festival

Project team: Iason Paterakis, Angelos Klothakis

Supervisor: Angelos Floros

Contribution: Concept Design, Content Creation (C4D, AE, Processing), Installation Design, Hardware Setup, Projection Mapping

\Interactive performances

© TUC TIE LAB 2019

FESTEN

Festen was a stage adaptation based on the homonymous Danish black comedy-drama. The play premiered in July 2019, at the Mikis Theodorakis theater in Chania. Compared to the previous theatrical performance study, the projection study in Festen had a radically different purpose. The visual content on this occasion was abstract and focused on exalting the atmosphere and ambiance of specific scenes.

Our evaluation process was conducted for six theatrical plays. The participants were over 18 years old, with the average age being close to 40 years old. The User experience questionnaires (UEQs) were administered to the attendants immediately after the play. The variables of Attention and Engagement have been studied.

Director: Theodoros Espiritou

research team: Iason Paterakis, Nefeli Manoudaki, Marios Christoulakis

supervisors: Konstantinos- Alketas Oungrinis, Marianthi Liapi

contribution: Concept Design, Content Creation (C4D, AE, PS, Ai), Hardware Setup, Projection Mapping

PROSPECTS FOR IMPLEMENTATION

EDUCATION
- Schools, Libraries, Auditoriums

CULTURE
- Museums, Exhibitions, Theaters, Concert halls

TOURISM
:: Edutainment (with cultural content)
- Airports, Ports, Train-stations, Public buildings, Hotels
:: Entertainment (with artistic content)
- Hotels

THE ACCOMPLICE

Designing visual content for Rupert Holmes’ play "Accomplice" theatrical performance proved to be challenging. The script involved four different scenes that had to be designed from scratch along with the producers and the scenographer of the play. TUC TIE Lab researchers produced the appropriate background scenes that complemented the main action. The scenes were both vibrant and literal, meaning that the composition of the structural and visual elements was rich and bold. However, exaggerated and salient visual elements should not undervalue the main action in this type of theatrical performance. The introduction of an enhanced peripheral atmosphere should elevate the perceived value of the whole theatrical play and further engage the audience. In the end, a balance between the main action and the background visuals was achieved with the appropriate stage-lighting setup, contributing to the overall atmosphere of the play.

Director: Christos Rachiotis

research team: Iason Paterakis, Nefeli Manoudaki, Marios Christoulakis

supervisors: Konstantinos- Alketas Oungrinis, Marianthi Liapi

contribution: Concept Design, Content Creation (C4D, AE, PS, Ai), Hardware Setup, Projection Mapping