← Back to Main

Semantic Interaction: Dimensional Transformation

2/8/2025, 5:14:40 PM | Jeanyoon Choi

Original Notes (Pre-LLM)

I talked about the importance of semantic interaction and how we should design it. The main challenge: people do not eager to participate with semantic interaction from the first point. When encountered with an empty text field, they don't know what to write. They just write random word (which is the case of my self) - Same for the voice interaction, word interaction, etc…. When talking about semantics, naturally 'language' based interaction comes out easily from the mind, but within exhibition setting, as an alien visitor, it is unlikely to interact/fulfill this linguistic requirement in a full manner. Audiences just usually murmur or skip through such interaction without deep semantic meaning…

Which is also the reason why lots of interactions within new media art just focuses on non-semantic interaction, especially movement based interaction. Which does not require active/semantic involvement from user side: Body tracking, face tracking, bodily movement, EEG, etc…. BUT WE CAN DO MORE THAN THIS. THESE interactions are non-semantic, and as result, usually just end up in abstractive audio-visual control without deeper meaning/connection to the society. BUT we need a way to carefully design semantic interaction and thus open a new field of expression… But how? How to make non-engaging/passive audience to involve within semantic interaction?

One of the possible methods: Dimensional Transformation of Non-semantic interaction into semantic output. Proceedingly, audiences create a semantic connection within the interaction by themselves naturally.

We keep on sticking with the old-school physical non-semantic interaction: Body tracking, face tracking, bodily movement, EEG… Which all converts the physical (and mostly non-semantic) movement into x,y (2D) coordinates or x,y,z (3D) coordinates. What we can do is use these x,y positional params into different dimension - dimensional transformation - of the output world/system. As a very naive example, x, y params can be transformed into t, and $ axis (temporal-monetary plane), accordingly.

Going one step further: Traditional web-based interfaces are good at this. When we're scrolling down through instagram/youtube/google, we sometimes just scroll down for the sake of scrolling down - for the sake of the act of its interaction itself (we're sort of addicted into it) – and then we confront with the reacting correspondingly appearing contents after we scroll through. We're doing a non-semantic interaction, which results in semantic output. Naturally my assumption - Practical, user-friendly (i.e. interface that user had seen a lot) interface presented → User do some interaction (scrolling down, button click, etc.) → good gateway to open up a semantic interaction, and more natural than just having an empty text field and requiring the user to fill out this. It's cognitively easier to click on the buttons/scroll through the existing interface from an input side, which will create semantic output within the surroundings. This interaction also still contains more semantics than just requiring user to dance, use their facial expressions, or connect with EEG sensors. All possible within multi-device web artwork context.

WITHIN THESE PROCESSES DOMAIN TRANSFORMATION (Unexpected domain transformation) might be important: This is completely about artistic imagination

English Version (LLM-Generated)

In our current exploration of art and technology, the conversation naturally turns to semantic interaction—an invitation for users to contribute meaning—but therein lies a paradox. How do we design an encounter where individuals willingly imbue digital spaces with nuance rather than defaulting to random, unplanned inputs? When confronted with a void—a blank text field or an unprompted microphone—users, myself included, often resort to scattered words or simple sounds rather than engaging deeply. It is as if we, like extraterrestrial visitors in an unknown realm, are faced with linguistic expectations that seem foreign in a gallery or interactive installation. Rather than constructing rich narratives or deliberate dialogues, audiences murmur or bypass these interfaces entirely.

This hesitancy explains a prevailing trend in new media art: a reliance on non-semantic, movement-based interactions. The body tracking, facial recognition, and even EEG interfaces that dominate many installations map physical gestures into abstract audiovisual expressions. Although such non-semantic interactions are intuitively reachable—requiring just a movement, a glance, or an electric spark—they often yield outputs that lack a deeper, societal connection.

Yet what if we could bridge that chasm? What if we could transform ephemeral, non-semantic data into layers of meaning—opening a portal to a whole new field of interactive expression? One provocative approach is the Dimensional Transformation of Non-semantic Interaction into Semantic Output. In this conceptual framework, every bodily gesture or subtle movement is not just a numerical coordinate (x, y on a flat plane or x, y, z in space), but a seed that, through a process of artistic alchemy, blossoms into a narrative or a concept. Imagine mapping those coordinates onto entirely different dimensions—a temporal axis, a symbolic monetary plane, or even an emotional gradient—where every gesture carries a potential story waiting to be decoded by the system and, in turn, by the audience.

Consider our everyday experience with traditional web interfaces. When scrolling through Instagram, YouTube, or Google, users become caught in a feedback loop: a non-semantic gesture (the scroll) effortlessly summons semantic content—a cascade of images, words, videos that articulate cultural trends, personal memories, or political commentaries. These familiar interactions, far more engaging than an empty prompt demanding textual devotion, provide an accessible entry point to deeper semantic engagement. They allow us to interact without the burden of creating meaning from scratch; instead, we are coaxed into a dialogue where the interface, steeped in cultural coding, mirrors back an enriched experience that is both intuitive and laden with metaphor.

Within these pathways, an essential mechanism emerges: Domain Transformation. Here, the interplay of unexpected shifts between domains—the quantitative to the qualitative, the physical to the narrative—reveals the vibrant potential of artistic imagination. It is as if the act of scrolling or moving is not merely a mechanistic operation, but a transformative ritual, a dance between the user and the digital canvas that redefines what interaction can be.

In sum, the challenge is not to abandon our trusted physical interfaces like body or face tracking, but to re-imagine their outputs. By carefully designing systems that transform raw, non-semantic data into rich, semantic landscapes, we open avenues for audiences to engage not only sensorially but intellectually and emotionally. This journey into the synthesis of art and technology is ultimately about creating a dynamic dialogue—a metamorphosis where every gesture holds the promise of meaning, and every interaction becomes a canvas for cultural expression.

Korean Version (LLM-Generated)

예술과 기술의 만남 속에서, 우리는 '의미적 상호작용'의 중요성을 재조명하게 된다. 본 글에서는 관람자와 작품 사이의 깊은 대화를 이끌어내는 상호작용의 설계 문제를 탐구하고자 한다.

첫 번째 도전은 참여의 문턱에 있다. 전통적인 인터페이스에서 빈 텍스트 필드 앞에 선 사람들은 무엇을 써야 할지 몰라 무작위로 단어를 던지곤 한다. 음성, 제스처, 단어 입력 등 언어를 매개로 한 상호작용은 자연스럽게 떠오르지만, 전시 공간이라는 이질적인 환경에서는 낯선 방문자가 온전히 언어적 소통을 시도하기 쉽지 않다. 관람자들은 종종 아무런 의미 없는 중얼거림이나, 빠르게 스쳐 지나가는 행동으로 그저 그 순간을 흘려보내곤 한다.

이러한 현실은 새로운 매체예술에서 비의미적, 주로 움직임 기반 상호작용에 집중하게 만든다. 몸의 움직임, 얼굴 인식, EEG 센서 등은 별다른 인지적 부담 없이 관람자의 물리적 반응을 포착한다. 그러나 이러한 비의미적 입력은 결과적으로 사회적 맥락이나 깊은 의미 연결고리 없이 단순한 시청각적 추상화를 이끈다.

우리는 한 걸음 더 나아가야 한다. 관람자가 스스로 의미를 구축할 수 있는 '의미적 상호작용'의 새로운 패러다임을 모색할 때다. 그 한 가지 방법은 바로, 비의미적 상호작용 요소를 차원의 변환을 통해 의미적으로 재해석하는 것이다. 예를 들어, 몸의 움직임이나 얼굴의 표정을 2차원 혹은 3차원 좌표로 기록한 후, 이 데이터를 시간(t)이나 경제적 가치(금전적 축)와 같은 전혀 다른 차원으로 변환함으로써, 관람자 스스로가 의미의 연결고리를 만들어가는 과정을 유도할 수 있다.

한 단계 발전된 예시로는, 웹 기반 인터페이스를 들 수 있다. 인스타그램, 유튜브, 구글 등 우리가 익숙한 스크롤 행위는 단순한 물리적 인터랙션에 불과해 보이지만, 그 속에서 예상치 못한 콘텐츠가 등장하며 관람자에게 심리적 만족감과 의미적 함의를 전달한다. 익숙한 버튼 클릭이나 스크롤 행동은 인지 부담이 비교적 적으며, 결과적으로 비어있던 텍스트 입력 대신 자연스럽게 의미가 생성되는 창구로 작용한다.

이와 같은 과정을 '도메인 변환', 즉 예상치 못한 영역으로의 전환으로 볼 수 있다. 이는 단순한 기술적 구현을 넘어, 예술적 상상력과 창조적 탐구를 통해 새로운 의미의 세계를 열어가는 행위이다. 기술과 예술이 손을 맞잡아 기존의 비의미적 상호작용을 넘어, 관람자와 작품이 함께 창조하는 심오한 대화의 장을 마련할 때, 우리는 진정한 의미의 혁신을 경험할 수 있을 것이다.