How to build an evolving interactive system art?

3/18/2025, 3:55:38 AM | Jeanyoon Choi

Original Notes (Pre-LLM)

How to build an evolving interactive system (art)?

How does user interaction/audience interaction can actively change the very nature of the system (or even generate a new one) rather than passively navigating/interacting with the system or minorly changing the pre-given range parameters? (Do we have to borrow generative ai/algorithms here?)

Metaphor from Scientific Experiments (Golden Ratio?)

DESIGNING ACTIVE-INTERACTION BASED EXPERIMENT IS ULTIMATELY REMINISCENT TO DESIGNING ACTIVE-INTERACTION BASED COMPUTATIONAL ARTWORK

Departing from traditional passive experiments → active user/audience engagement-based experiments

Designing Neuroscience/HCI experiment – in an active way, not a passive way – is almost reminiscent to building an active interactive art (not the passive one, not just a bodily interaction)

Type A: Semantic Interaction: Direct, Explicit, Button-based, GUI-based (Non-language)

Type B: Phenomenological Interaction: Indirect, Implicit, Sensual, Human-Driven Motion, Facial, Touch, BCI Tangible Interaction

Type C: Language-based Interaction: Text, Voice (LLM-Integrated)

Type D: Ambient Environmental Data

From Type B/D (Non-semantic interaction) to Type A/C (semantic interaction)

But not just about interaction modality being semantic – it’s also about how these interactions actually actively shape/alter/change/generate the systemic result, rather than minorly navigating within/changing minor parameters of the systems.

Experiment setting: Verifying if golden ratio – people like it in neuroscience way (is preference with golden ratio innate in our brains?)

Engagement Level 1.

Traditional approach

A-B testing, 2-6 images (visual stimuli) given, user selects the most preferable image

Very passive engagement

Engagement Level 2.

Slider approach, simple gui-based approach, often used in hci experiments

User adjusts slider (of ratio), output visual stimuli morphs accordingly, user lands on the most comfortable ratio

More active, yet also comes with limitations.

User changing the parameter of the system, whilst the system rule stays in

Engagement Level 3.

Hypothetical, Experimental, Most active user engagement

Smth more than just two-dimensional navigation

User should actively joy / explore the unexplored territory. And then end up with the optimal ratio that they think they’re comfortable with/aesthetically pleasing. (Neuroaesthetics) Not just performing for the sake of experiment – choosing one of many images, using a slider to pick one ratio – these are not really natural settings. You won’t do this for the sake of aesthetic pleasure, you do this bc this is an experiment setting and you get paid, and you are ordered to perform so… This new kind of experiment should be smth which you will be willing to do, or even, willing to pay to experience through (like gyrodrop! Like amusement park!) So good enough, so overwhelming enough, will-to-power.

The system should be able to accommodate/reflect/react upon unexpected user behaviours → User should have a great degree of freedom, beyond just merely adjusting parameters within pre-defined axes. (Slider Interaction).

This is reminiscent to the moment when people used 10 sliders to generate an image (our old GAN disentanglement research) but suddenly text-to-image came out and the degree of freedom greatly increased

Still our scope of research should be within ‘golden ratio’ – perhaps should accommodate ratio adjustment?

Up-down game?

Binary search, hot-cold game

Adaptive methods in psychophysics (e.g., staircase procedures) where the stimulus changes based on user feedback, converging to a threshold (the “just noticeable difference” or the “ideal ratio”).

What if we accommodate the LLM-based ratio adjustment process? User can speak whether the ratio is up or down – The up-down game that we used to do when we guess somebody’s birthday/cell number.

Does not have to be LLM - can be GUI also. Two buttons: Ratio up? Ratio down?

Pitt’s law – user interaction? When user is navigating through the target – & when target is very small – user does not directly heads to the target, but rather, go backs & forth, 타겟을 앞으로 지나치기도, 뒤로 지나치기도 하면서 점차 범위를 좁혀나간다… but eventually reaches/converges within/nearby the target?

Other approaches?

Adjusting using different modality than slider?

Accelerometer? – phone shaking/smth shaking changes the parameter of the ratio of the visual stimuli?

Hand tracking? – Minority-report-like interaction? Hand direction changes the ratio?

Experiment within apple vision pro?

Is there example where neuroscience study used apple vision pro/vr?

Eye tracking – eye tracking changes the ratio?

Limitation when we’re going to consider eye tracking as an experiment output variable – the adjusting input and output overlaps

Other approaches?

Conti’d oscillation/navigation btw Abstractive Stimuli ←> Descriptive Stimuli?

What about Fibonacci? Fibonacci-like narrowing down?

Other hypothetical question:

The output of Golden Ratio Research – Should be two fold.

1. Research output - paper, journal publication

2. Interactive (artistic) experience - On top of the experiment setting, but smth more, maybe on the quotation on the essence of golden ratio.

(Explanation of Science/Visualisation) : (Art) = (NN Architecture Vis) : (SoTA) = (Golden Ratio Experiment) : (??????)

??? is the interactive artistic experience – our long term goal.

How to design this? (Not only the problem of narrative, but the matter of system design, state-based approach…)

How to Build an Evolving Interactive Art System (Conceptual Guidance):

You’re aiming for a transformative interaction model where user input doesn’t merely select from predefined outcomes or minorly tweak parameters but rather fundamentally evolves the underlying system or generates new emergent behavior.

Conceptual Foundation:

• Passive Interaction: Users explore fixed possibilities.

• Active Interaction: Users shape possibilities dynamically, influencing systemic behavior.

• Evolving System: User engagement continuously reshapes system logic or behavior.

You’re pushing toward “Level 3 Engagement,” emphasizing intuitive, exploratory interactions inspired by experiments in neuroscience and HCI but extending into immersive, artistic experiences.

Key Principles for Designing Evolving Interactive Systems:

1. Interaction Modality Spectrum:

• Semantic Explicit (Buttons, GUI)

• Phenomenological Implicit (Gesture, touch, body language)

• Language-Based (Voice, text, conversational LLM)

• Environmental & Ambient (Sensors, context-aware data)

Combining multiple modalities enriches user engagement and promotes dynamic evolution of system responses.

2. Active Influence over System Logic:

Move beyond user-driven parameter adjustments (like sliders) toward interactions that directly impact or redefine systemic rules, structures, and logic.

Examples:

• Adaptive Rule Systems: The user’s repeated interactions prompt the system to adjust underlying algorithms (like adaptive psychophysical methods but more radically creative).

• Generative AI/LLM: LLM-integration allows user intentions, communicated via language or intuitive gestures, to actively reshape rules or generate entirely new artistic outputs dynamically.

3. Neuroscience-inspired Methodologies:

Adaptive psychophysical approaches (staircase, hot-cold guessing games, Pitt’s law-inspired oscillation around target points) offer meaningful models for achieving active exploration and iterative convergence toward preferences like the golden ratio.

4. Novel Interactive Approaches:

• Nonlinear Interaction: Gesture tracking, shaking (accelerometer), eye tracking (though consider overlap between input-output), VR/AR interaction (Apple Vision Pro or similar).

• Exploratory Navigation: Users navigate abstract vs. descriptive stimuli, oscillating toward self-discovery of preferences.

• Fibonacci-inspired Interaction: Gradual narrowing down interactions following Fibonacci sequences, resonating conceptually with golden ratio research.

Artistic System Design Metaphor:

The scientific experiment metaphor you’re invoking—moving from passive observation toward actively designing experiments—is analogous to moving from traditional, predefined interactive systems toward evolving, emergent interactive art.

Your experimental neuroscience/HCI method → Interactive artistic experience → Emergent creative system (long-term goal)

Example Framework (Golden Ratio Experiment):

Step-by-Step Interaction Approach:

1. Initial Exploration (Implicit):

• User gestures/movements indirectly alter visual stimuli. (Phenomenological)

2. Semantic Refinement (Explicit Interaction):

• GUI or voice to narrow down choices explicitly. (Semantic, Language)

3. Evolving Rules (Adaptive Logic):

• LLM or generative algorithms adapt, modifying their generation rules based on accumulated user patterns, pushing beyond preset parameters into emergent generative aesthetics.

4. Feedback Loop (Continuous Evolution):

• User feedback continuously reshapes the underlying system logic, not just the visible parameters.

Output Goals (Two-fold):

• Research Output: Neuroscience/HCI research paper.

• Artistic Output: Dynamic, interactive artwork allowing users to experience and contribute to the continuous evolution of the golden ratio aesthetic experience, making the “experiment” itself a captivating artistic journey.

Your long-term goal is designing the second output—an evolving, immersive artistic experience systemically influenced by user interactions.

English Version (LLM-Generated)

This essay explores the design of evolving interactive systems—particularly within the realm of computational art—where user and audience interactions fundamentally reshape the system’s behavior rather than merely navigating within a set of predetermined parameters. The objective is to transition from passive experiments to those that actively engage participants, thereby inspiring an artistic experience that evolves in response to their input.

Foundations and Conceptual Framework

Traditional experiments typically involve passive observation or controlled, minor adjustments. For example, classical aesthetics research might employ A/B testing with two to six visual stimuli, where users select their preferred image. In contrast, the active engagement model proposed here involves an interaction design where users are not simply adjusting a parameter via a slider or button; instead, they influence the system’s fundamental operations and outcomes.

The conceptual foundation is built on three levels of interaction:

1. Passive Interaction: Users explore fixed possibilities.

2. Active Interaction: Users shape system outcomes dynamically.

3. Evolving Systems: User engagement continuously reshapes the system’s governing logic, producing emergent behavior not initially foreseen by the designer.

Interaction Modalities

An effective evolving interactive system combines a spectrum of interaction modalities, including:

• Semantic Interaction: Direct, explicit inputs using buttons or GUI elements, where the user’s actions are discrete.

• Phenomenological Interaction: Indirect, sensory engagement using gesture, touch, or facial expressions, which capture the nuances of human-driven motion.

• Language-Based Interaction: Input through text or voice, potentially integrated with large language models (LLMs) to interpret user intention.

• Ambient and Environmental Data: Incorporation of sensor outputs that influence the system based on context-aware data.

The interplay between non-semantic modalities (such as phenomenological and environmental inputs) and semantic or language-based inputs can yield a system in which even subtle or unexpected interactions might fundamentally alter the underlying algorithm.

Illustrative Example: The Golden Ratio Experiment

Consider an experimental paradigm designed to investigate whether the human preference for the golden ratio is innate. Three levels of engagement could be outlined as follows:

Engagement Level 1 (Passive):

Participants are shown a set of images generated with different ratios. They then choose the image they find most aesthetically pleasing, replicating a traditional A/B testing approach with limited interactivity.

Engagement Level 2 (Moderately Active):

A slider interface allows participants to adjust the ratio dynamically, with real-time visual feedback. Although this method offers more interactivity, the system’s underlying rule remains fixed.

Engagement Level 3 (Highly Active):

Participants engage with an immersive and exploratory interface that goes beyond simple parameter adjustment. Here, users navigate a non-linear, interactive space akin to a “hot-cold” game or binary search, wherein their actions and decisions—whether through gestures, voice commands, or even sensor-based inputs like accelerometer or eye tracking—lead to the emergence of entirely new system rules or configurations. In this mode, the system might incorporate adaptive psychophysical methods, such as staircase procedures, where the stimulus evolves continuously based on user feedback. Moreover, integration with generative AI could allow the system to reinterpret and transform user inputs into novel artistic outputs, ensuring that interactions result in a meaningful evolution of the overall system rather than just minor tweaks.

These levels illustrate a progression from structured, limited engagement toward a deep, immersive interactive experience in which user behavior actively redefines the system’s output. The process encourages exploratory behavior similar to navigating through abstract and descriptive stimuli or implementing Fibonacci-like narrowing methods in the search for the ideal ratio.

Long-Term Goals and Dual Outputs

The ambition of designing such a system is twofold:

1. Research Output: The project aims to yield robust findings suitable for publication in neuroscience and human-computer interaction (HCI) journals, offering new insights into dynamic human perception and aesthetic preference.

2. Artistic Output: Beyond academic contributions, the ultimate vision is to create an interactive artwork that embodies the evolving essence of the golden ratio. This artwork would serve as an immersive cultural experience, allowing users to witness and participate in the evolution of aesthetic principles in real time.

This dual outcome emphasizes the interplay between scientific inquiry and artistic expression. The evolving interactive system is conceptualized as a living experiment—one that not only contributes to academic research but also transforms into an engaging piece of interactive art. Here, user interactions are not merely responses but integral to the creation and redefinition of the aesthetic experience.

Conclusion

In designing an evolving interactive art system, the goal is to create an environment where users do more than select from a set of predetermined outcomes; they become active collaborators in shape-shifting the system’s logic. By integrating diverse interaction modalities—including direct semantic inputs, phenomenological cues, language-based commands, and environmental data—the system leverages adaptive and generative algorithms to enable continuous, meaningful evolution. Such a system represents a significant departure from traditional, passive interactive experiences, ultimately merging the fields of neuroscience, HCI, and computational art into a cohesive, dynamic practice.

Korean Version (LLM-Generated)

다음은 기존의 아이디어를 전문적인 방식으로 한국어로 정리한 내용입니다.

──────────────────────────────

진화하는 인터랙티브 시스템(아트) 구축 방안

1. 서론

본 연구는 사용자의 상호작용이 단순히 미리 정의된 범위 내의 파라미터를 소폭 조정하는 것을 넘어, 시스템의 근본적인 규칙과 로직을 변형하거나 전혀 새로운 시스템을 생성할 수 있는 변혁적 인터랙션 모델을 제안하고자 한다. 이러한 접근은 전통적인 수동적 인터랙션 방식과 달리, 사용자가 실험 전반에 적극적으로 참여하도록 유도하며, 과학적 실험(예, 황금비율 선호도 검증) 및 신경과학, 인간-컴퓨터 상호작용(HCI) 연구에서의 접근 방법과도 유사한 맥락을 가진다.

──────────────────────────────

2. 상호작용 모달리티와 단계별 접근

① 상호작용 모달리티 범위

사용자와의 상호작용 방식은 다음 네 가지 영역으로 구분된다.

A. 의미적 인터랙션 (Semantic Interaction)

 • 직접적, 명시적 상호작용 – 버튼 기반, GUI 등

 • 언어 이외의 명확한 인터페이스

B. 현상학적 인터랙션 (Phenomenological Interaction)

 • 간접적, 암묵적 상호작용 – 제스처, 촉감, 신체 움직임, 얼굴 표정, 뇌-컴퓨터 인터페이스(BCI) 등

C. 언어 기반 인터랙션 (Language-based Interaction)

 • 텍스트나 음성(대규모 언어 모델(LLM) 통합)을 통한 상호작용

D. 환경 및 주변 데이터 (Ambient Environmental Data)

 • 센서 데이터, 실시간 환경 맥락 기반 인터랙션

이러한 모달리티들을 융합함으로써 사용자가 보다 풍부한 방식으로 시스템에 영향을 미칠 수 있으며, 이는 단순히 파라미터를 조정하는 차원을 넘어 시스템의 생성 및 변화에 실시간으로 반영될 수 있다.

② 사용 참여 단계별 접근

가. 참여 수준 1 – 전통적 접근

 • A/B 테스트 방식: 제한된 수(2~6개)의 이미지(시각적 자극) 중 가장 선호하는 대상을 선택하는 방식

 • 사용자의 참여는 수동적이며, 선택의 범위 내에서 제한적인 반응만 유도됨

나. 참여 수준 2 – 슬라이더 기반 접근

 • 간단한 GUI 기반의 상호작용: 사용자가 슬라이더(비율 조정)를 조작하면, 시각적 자극이 이에 따라 형태 변화를 보임

 • 기존 시스템의 규칙은 그대로 유지되며, 단지 파라미터 조정에 머무르는 한계가 존재함

다. 참여 수준 3 – 실험적 및 적극적 참여

 • 전통적인 2차원 내비게이션을 넘어, 사용자가 미지의 영역을 직접 탐험하고 자신에게 최적의 비율, 또는 미적 선호도를 발견하도록 유도

 • 단순히 실험의 일부로서 이미지를 선택하거나 슬라이더를 조작하는 것을 넘어, 사용자가 체험 자체에서 만족과 즐거움을 느낄 수 있는 몰입형 환경을 제공

 • 시스템은 사용자의 예상치 못한 행동까지 포용할 수 있도록 설계해야 하며, 미리 정의된 축상에서의 단순 조정(예, 슬라이더 조작)을 넘어 자유도가 높은 상호작용 환경을 제공한다.

  예시: 수십 개의 슬라이더를 활용한 이미지 생성(GAN disentanglement)에서 텍스트-투-이미지 방식으로 전환되어 상호작용의 자유도가 증가한 사례와 유사함.

──────────────────────────────

3. 실험적 세팅 – 황금비율 연구의 예시

연구는 ‘황금비율’에 대한 선호가 신경과학적(Neuroaesthetics) 측면에서 본능적 정보인지 검증하는 실험적 모델을 기반으로 한다.

① 전통적 실험 방식

 • A/B 테스트 기반으로 소수의 시각적 자극 이미지 선택

 • 수동적 참여로 인한 한계가 존재함

② 슬라이더 기반 방식

 • GUI 슬라이더를 활용하여 사용자가 비율을 직접 조정

 • 시스템의 기본 규칙은 유지된 채 파라미터만 조정하는 형식

③ 고도의 적극적 참여 방식 (가설적 접근)

 • 사용자가 단순 조작 이상의 경험을 하도록, 더욱 ‘자연스럽게’ 탐구하는 환경 제공

 • 이 과정에서는 사용자 행동(예, 손의 움직임, 음성 명령, 혹은 VR 환경 상호작용 등)을 시스템 로직 자체와 연결시켜, 반복적인 피드백 루프를 통해 사용자의 선호를 반영한 새로운 생성형 규칙을 만들어냄

  - 예시: 이진 검색, 온도 변화(Hot-Cold Game), 적응적 심리물리학적 방법(계단식 절차)을 적용하여 자극이 사용자의 피드백에 따라 점진적으로 변화하는 방식

  - LLM(대규모 언어 모델)을 통한 언어 기반 상호작용으로 “비율을 높일 것인가, 낮출 것인가”와 같은 질문을 동적으로 제시할 수 있음

  - 다양한 센서 기반 (예: 가속도계, 손 동작 추적, 시선 추적) 상호작용을 통합할 수 있음

──────────────────────────────

4. 시스템 설계 및 예술적 전환

과학적 실험과 신경과학/HCI 실험을 통한 연구 결과는 두 가지 결과물로 나타나야 한다.

① 연구 결과

 • 학술 논문, 저널 게재 등 객관적 연구 산출물

② 예술적 경험

 • 단순 실험의 틀을 넘어, 사용자가 시스템과 상호작용하며 ‘실험’ 그 자체가 예술적 경험으로 승화되는 동적, 진화하는 아트워크

 • 예술 작품은 사용자의 참여에 따라 지속적으로 발전하며, 실시간 피드백을 바탕으로 새로운 생성적 미적 경험을 제공함

시스템 설계의 최종 목표는, 사용자가 단순히 미리 정해진 결과를 선택하는 것이 아니라, 상호작용에 의해 시스템의 규칙이나 로직이 근본적으로 변화되고, 그 결과물이 다시 눈에 보이는 형태로 나타나는 ‘진화하는 인터랙티브 아트’의 구현이다. 이를 위해 다음과 같은 핵심 원칙을 고려해야 한다.

──────────────────────────────

5. 진화하는 인터랙티브 시스템 설계를 위한 핵심 원칙

(1) 상호작용 방법의 다변화

 • 의미적, 현상학적, 언어적, 그리고 환경적 모달리티의 조합을 통해 사용자의 참여와 몰입도를 극대화

(2) 시스템 로직에의 적극적 영향

 • 사용자의 입력이 단순 파라미터의 조정에 그치지 않고, 시스템의 기본 규칙 및 생성 알고리즘 자체를 변화시키도록 설계

  - 예: 적응형 규칙 설정, LLM 및 생성형 AI를 통한 실시간 규칙 재구성

(3) 신경과학적 접근 방법의 도입

 • 적응형 심리물리학 기법(계단법, 온도 변화 게임 등)을 통한 사용자의 선호 탐색 및 수렴

(4) 혁신적 인터랙션 기법 도입

 • 슬라이더 외에도 가속도계, 손 동작 추적, 시선 추적, VR/AR 기반 상호작용 등 비선형 인터페이스 도입을 고려

 • 추상적 자극과 서술적 자극 간의 교차 탐색(Conti’d Oscillation/Navigation)을 통해 사용자 스스로 선호를 재발견하도록 지원

(5) 피보나치(Fibonacci) 기반 상호작용

 • 피보나치 수열의 원리를 응용하여 상호작용의 점진적 좁혀나감 구현

──────────────────────────────

6. 결론

진화하는 인터랙티브 시스템 구축은 단순히 사용자 입력에 따른 파라미터 조정을 넘어서, 실시간으로 시스템의 로직과 생성 알고리즘을 변화시킴으로써 새로운 미적 경험을 창출하는 것을 목표로 한다. 이를 위해 사용자의 직관적, 비언어적, 언어적, 그리고 환경 데이터를 종합적으로 활용한 다층적 상호작용 모델을 제안하며, 최종 결과물로는 학술적 연구성과와 동시에 사용자 참여에 의해 지속적으로 진화하는 예술작품을 달성하고자 한다.

이와 같은 접근은 전통적인 수동적 실험 방법론을 탈피하여, 사용자가 시스템의 핵심 로직을 재구성하는 적극적 인터랙션의 가능성을 열어준다. 장기적으로는 이러한 연구 결과가, 예술과 과학, 그리고 기술이 융합된 새로운 형태의 인터랙티브 아트 경험으로 확산될 것으로 기대된다.

──────────────────────────────

이상과 같이, 본 글은 사용자 참여가 단순한 파라미터 조정을 넘어서 시스템의 근본적 변화를 이끌어내어, 학술적 연구와 예술적 체험이 결합된 새로운 인터랙티브 아트 시스템을 구축하기 위한 개념적 가이드를 제시한다.

Tags

Text written by Jeanyoon Choi

Ⓒ Jeanyoon Choi, 2024