System Art: Progressive Storytelling vs. State-Based Approach

2/17/2025, 10:41:02 AM | Jeanyoon Choi

Original Notes (Pre-LLM)

When designing multi-device web artwork, interactive artwork, and system artwork, we tend to depart from the moving images - videos - as this is the most widely accessible reference point in media art/aesthetics. This often forces MDWA to contaminate linear and/or progressive storytelling.

HOWEVER, interactive artwork, multi-device web artwork, and system artwork, does not have to necessarily have linear structure - where user lands on and when interacting, the state/stages moves progressively… This is actually awkward somehow (to force user to be in a certain state), and structurally limits the degree of freedom in user interaction - thus leading to very childish not really provocative user interaction and limited user's effect to the system dynamics (Take babel room vr artwork for example…) Mostly in this kind of linear storytelling, interaction is mostly used to progress user from one state to the next, and that's their only objective… User cannot be provocative and actively participate within the system to change and alter the system.

The ideal interaction: Is not user playing as a puppet in a pre-defined system and linear storyboard scenario, but more, user affecting the system as simultaneously the system affects the user. It's inter-action, not a one-way action. Ideal two-way user interaction: as user interacts the system, the user changes and alternates the core crucial system dynamics (both in short-term or long-term) rather than just changing mere visual effects and/or progressing towards pre-defined states. And thus the user changes the system, and this newly changed system iteratively changes the user. This creates a non-linear narrative and itself the system.

Easier said than done… Biggest caveat of these systems is that 1) It might overcomplicate user and confuse user - user might not interact with anything before understanding the complexity (or confronting with semantic interactions), 2) System design, 3) System unexpectability (If there is too much uncertainty within the system - both engineering-wise and conceptual-wise it might face trouble)

Therefore crucial to understand what I will here define 'state-based approach' to create system art, multi-device web artwork-based interactive systematic approach. Here I write about linear storytelling, progressive storytelling, and state-based approach.

Linear Storytelling: Linear states, no crucial difference with traditional moving images. Interaction is merely used for progressing to the next linear step and/or minor visual changes.

Progressive Storytelling: Bit of non-linearity but yet still progressive/pre-defined storytelling - There is a set of non-linear parallel states which can be adjusted through interaction, but the overall structure of the artwork itself is still defined on linear characteristic, and user's interaction is not powerful enough to change the overall system/state-based dynamics.

State-Based Approach: Different state and inter-state relationships define the interface that user confronts/interacts with. Each state is non-linearly combined, sometimes even generated. User takes a power to not only navigate through different states, but redefine inter-state dynamics and even create new states.

GPT's refined definition:

Below is a concise set of definitions for these three storytelling/interaction modes, derived directly from the essay:

Linear Storytelling: A single, pre-defined sequence of states or scenes, much like traditional moving images.

Interaction only advances the user from one fixed step to the next (e.g., clicking "next" in a slideshow). The user's role is passive or minor, with no real power to alter the overall flow—just to trigger the next segment.

Progressive Storytelling: A mostly pre-defined structure with branching or parallel states, allowing some non-linear navigation.

The user can choose which state to visit (e.g., selecting from parallel paths), but the overall shape of the narrative remains controlled by the designer. The user's role is a navigator, not a co-creator: interactions reveal or select from existing states rather than transform them.

State-Based Approach: A dynamic network of states, where inter-state relationships (and even new states) can emerge or be redefined through interaction.

The user wields genuine power to modify system dynamics, not just move between pre-set scenes. This yields non-linear, two-way interaction: the user changes the system, and the system changes the user in return—potentially altering the entire state-space over time.

HAHA

Linear storytelling sounds very clear. (Babel Room)

Progressive Storytelling: Good example is SoTA. SoTA - three big states (Idle, Frontend, Backend), defined by user interaction (no user, user active, user inactive) & sub-states in Frontend (118 different NN architectures) - and user scrolls through to define one of these 118 states. Non-linear, circulatory, double-layered state systems. YET, user chooses/navigates within these pre-defined states, acting as a mere navigator rather than an active state-changer / state-dynamics changer. User interaction does define the state, and user interaction is non-linear somehow, but it does not change the state or inter-state dynamics, nor its non-linearity is complex enough (it can be also interpreted as linear somehow: Idle - Frontend - Backend - Idle).

To construct good state-based approach, graph-based system analysis might be needed. Also should be lean approach of heading from progressive storytelling to state-based approach gradually.

HOLA

Linear/Progressive Narrative: Clear Start/Ending, mostly accommodate 1 audience

State-based approach: No clear start/ending, blurred start/ending, can accommodate N multiple audiences.

What about the audience 'controlling' the system?: Can be both progressive storytelling and state-based approach. But the state-based approach offers audience a change to be the PART of the system.

No media art is a perfectly state-based approach as of now: Rafael Lozano Hemmer has a bit of hint but it is too simple to be a state-based approach: It is more of mere use of data and projecting it simply (Cause-and-effect). AI Art? Don't even talk about it - it is very narrow and shallow.

Platformatic approach? Sort of might help. Should define the role on what data plays.

From my knowledge, it is almost impossible to reach the pure state-based approach as of now. But it is an attractive goal that a truly two-way interactive system art should head.

English Version (LLM-Generated)

When designing multi-device web artwork, interactive artwork, and system artwork, it is common practice to begin with the familiar reference of linear moving images—videos—due to their widespread accessibility and established role in media art aesthetics. However, this tendency can inadvertently impose a linear or progressive storytelling framework upon MDWA (multi-device web artwork), thereby limiting the potential for genuine interactive experiences.

Interactive, multi-device, and system art need not adhere to a fixed linear structure, in which a user’s journey is predetermined by sequential states. Forcing users to follow a rigid progression not only restricts their freedom to interact but can also result in simplistic, unengaging experiences where the user’s effect on the system is minimal. A case in point is the Babel Room VR artwork, where interaction is largely confined to advancing the narrative from one state to the next rather than allowing users to meaningfully influence the system’s dynamics.

The ideal interaction—in contrast—should embody a reciprocal relationship between the user and the system. Rather than acting as a passive participant moving through a pre-defined sequence, users should have the capacity to affect the core dynamics of the system as much as the system shapes their experience. In a truly interactive two-way system, user engagement does more than alter visual effects or move the narrative forward; it genuinely redefines system behavior. As users interact, they modify the system's state and, in turn, the evolving system influences the users, resulting in a non-linear, emergent narrative that is as much about the system as it is about the story.

While this ideal is conceptually appealing, it is not without challenges. First, there is the risk of overwhelming or confusing the user, who may hesitate to engage with a system whose complexity or semantic depth is not immediately clear. Second, the design and engineering of such systems demand careful planning to balance complexity with usability. Third, excessive uncertainty—whether technical or conceptual—might jeopardize the reliability of the system, detracting from the user experience.

To navigate these challenges, I propose adopting a “state-based approach” for creating interactive system art and multi-device web artwork. To clarify this concept, it is useful to contrast it with linear and progressive storytelling approaches:

• Linear Storytelling: This mode consists of a single, pre-defined sequence of states or scenes, analogous to traditional moving images. Here, interaction merely serves to advance the user from one fixed step to the next, with little capacity to alter the overall structure.

• Progressive Storytelling: While retaining an overall linear shape, this approach introduces elements of non-linearity through parallel or branching states. User interaction allows navigation between these states—for instance, selecting one of several predetermined options. However, the interaction remains limited to observation and selection rather than transformation, rendering the user’s role more of a navigator than an active co-creator.

• State-Based Approach: This paradigm redefines interactivity as a dynamic network of states interconnected in non-linear ways. In this system, users are empowered not only to navigate among states but also to modify inter-state relationships and even generate new states. The resulting interaction is reciprocal: the user’s input alters the system’s dynamics, and those changes in turn affect subsequent user experiences, fostering a continuously evolving narrative.

Using examples to illustrate these approaches, linear storytelling can be seen in works like Babel Room, where the progression is clear and direct. Progressive storytelling is exemplified by projects such as SoTA, which features three primary states—Idle, Frontend, and Backend—with an additional 118 sub-states within the Frontend. Although SoTA permits a degree of non-linear navigation, users ultimately remain within a framework that cycles through predetermined states (Idle → Frontend → Backend → Idle), functioning more as navigators than transformative participants.

Transitioning from progressive storytelling to a fully state-based approach may require advanced methodologies such as graph-based system analysis and a lean, iterative development process. In contrast to linear and progressive narratives, which generally offer a clear beginning and end and cater to a single audience, a state-based approach is characterized by blurred boundaries between start and finish and the capacity to engage multiple audiences simultaneously.

The role of the audience in controlling the system may apply to both progressive storytelling and state-based frameworks. However, state-based systems uniquely offer the audience an opportunity to become integral contributors to the system itself.

Although current media art has yet to fully realize a pure state-based approach—examples like Rafael Lozano-Hemmer’s work hint at its potential but often remain rooted in simple cause-and-effect dynamics, and many AI-driven artworks are still relatively narrow in scope—the concept remains an appealing long-term objective. A truly two-way interactive system art, where user and system continuously shape each other, represents an ambitious yet compelling direction for future media art endeavors.

Korean Version (LLM-Generated)

다중 장치 웹 아트워크, 인터랙티브 아트워크, 시스템 아트워크를 설계할 때, 우리는 흔히 대중적으로 접근성이 높은 미디어 아트의 기준점인 영상(무빙 이미지, 즉 비디오)에서 벗어나게 됩니다. 이로 인해 MDWA(다중 장치 웹 아트워크)는 종종 선형 혹은 점진적 스토리텔링 방식에 매몰되는 경향이 있습니다.

그러나 인터랙티브 아트워크, 다중 장치 웹 아트워크, 시스템 아트워크가 반드시 선형 구조를 취할 필요는 없습니다. 즉, 사용자가 특정 시점에 접속하고 인터랙션을 통해 순차적으로 진행되는 상태 또는 단계를 강제로 경험할 필요는 없습니다. 오히려 이러한 방식은 사용자를 억지로 특정 상태에 머물게 하며, 결과적으로 사용자 인터랙션의 자유도를 제한하여 다소 유치하고 도발적이지 않은 인터랙션을 유발할 수 있습니다. 예를 들어 Babel Room VR 아트워크를 들 수 있듯, 선형 스토리텔링에서는 인터랙션이 오직 사용자 상태를 다음 단계로 진행시키기 위한 수단으로만 활용되며, 사용자가 시스템 내부에서 능동적으로 변화를 이끌어내기 어렵습니다.

이상적인 인터랙션은 미리 정해진 시스템과 선형 스토리보드 내에서 단순히 꼭두각시처럼 움직이는 것이 아니라, 사용자가 시스템에 영향을 미치는 동시에 시스템 또한 사용자에게 영향을 주는 상호작용을 의미합니다. 이는 일방적인 작용이 아니라 상호 작용(inter-action)입니다. 즉, 사용자가 시스템과 상호작용할 때, 단순히 시각적인 효과를 변화시키거나 정해진 상태로 진행하는 것을 넘어서, 핵심 시스템 역학(단기적 또는 장기적으로 모두)에 영향을 주어 이를 변화시킬 수 있어야 합니다. 결과적으로 사용자가 시스템을 변화시키며, 새롭게 변화된 시스템 역시 사용자에게 지속적으로 영향을 미치게 됩니다. 이 과정은 비선형 내러티브를 창출하며, 그 자체가 하나의 시스템이 됩니다.

물론 이상적인 상호작용 구현은 결코 간단하지 않습니다. 이러한 시스템 설계 시 고려해야 할 주요 문제점은 다음과 같습니다.

1) 사용자가 시스템의 복잡성을 이해하기 전에 과도한 인터랙션 요소로 인해 혼란을 야기할 수 있음.

2) 시스템 디자인 자체의 복잡성.

3) 시스템의 예측 불가능성(엔지니어링 및 개념적 측면에서 지나친 불확실성이 문제로 작용할 수 있음).

따라서 여기서 정의하는 '상태 기반 접근(state-based approach)'을 통해 시스템 아트, 다중 장치 웹 아트워크 기반의 인터랙티브 시스템을 창작하는 방법론을 이해하는 것이 중요합니다. 이에 대해 선형 스토리텔링, 점진적 스토리텔링, 그리고 상태 기반 접근의 차이를 다음과 같이 서술할 수 있습니다.

──────────────────────────────

선형 스토리텔링(Linear Storytelling):

전통적인 무빙 이미지와 유사하게, 미리 정의된 단일 상태 또는 장면의 연속으로 구성됩니다.

사용자의 인터랙션은 단순히 다음 단계로의 진행(예: 슬라이드쇼에서 “다음” 버튼 클릭)이나 미세한 시각적 변화에 국한됩니다. 사용자는 수동적 역할을 하게 되며, 내러티브 전체 흐름에 영향을 미칠 수 없습니다.

점진적 스토리텔링(Progressive Storytelling):

부분적인 비선형성을 내포하고 있지만, 전체적으로는 여전히 미리 정의된 스토리구조를 따릅니다.

상호작용을 통해 사용자는 여러 평행 상태 중 하나를 선택할 수 있으나, 내러티브의 전반적인 구조는 디자이너에 의해 통제됩니다. 즉, 사용자는 내비게이터의 역할만 수행하며, 기존 상태를 변형하거나 새로운 상태를 창출하는 능력은 제한적입니다.

──────────────────────────────

상태 기반 접근(State-Based Approach):

상호 상태 관계에 의해 정의되는 동적 네트워크로 구성되어 있으며, 사용자의 인터랙션을 통해 새로운 상태가 등장하거나 기존의 상태 간 관계가 재정의될 수 있습니다.

사용자는 단순히 미리 설정된 시나리오 내를 이동하는 것이 아니라, 시스템의 핵심 역학을 적극적으로 변화시킬 수 있는 권한을 가집니다. 이로써 상호 양방향 인터랙션이 형성되며, 사용자가 시스템을 변화시키고 변경된 시스템이 다시 사용자를 변화시키는 비선형 내러티브를 실현할 수 있습니다.

──────────────────────────────

예를 들어,

• 선형 스토리텔링은 Babel Room과 같이 명확한 시작과 끝이 존재하며, 단일 청중을 대상으로 합니다.

• 점진적 스토리텔링의 좋은 예는 SoTA로, Idle, Frontend, Backend와 같이 세 개의 큰 상태와 Frontend 내의 118가지 신경망 아키텍처의 하위 상태를 포함하면서, 사용자가 스크롤을 통해 상태를 선택하게 합니다. 이 경우, 사용자의 선택이 비선형적이긴 하나, 여전히 미리 정의된 상태 내비게이션에 머무르며, 사용자가 시스템 역학 자체를 변화시키는 데는 한계가 있습니다.

• 상태 기반 접근은 시작과 끝이 모호하며, 다양한 청중이 동시에 참여할 수 있는 구조를 지닙니다. 사용자가 시스템에 영향을 주어 시스템의 상태 간 관계를 재정의하거나 새로운 상태를 창출할 수 있다는 점에서, 단순히 시스템 외부의 관찰자가 아니라 시스템 자체의 일부로 기능합니다.

물론, 관객이 시스템을 '제어'하는 방식은 점진적 스토리텔링과 상태 기반 접근 모두에서 발견될 수 있으나, 진정한 상태 기반 접근은 관객에게 시스템의 일원으로 참여할 수 있는 기회를 제공합니다.

현재까지 순수한 상태 기반 접근 방식을 구현한 미디어 아트는 거의 없으며, Rafael Lozano Hemmer의 작품이 일부 시사점을 제공하지만, 이는 데이터의 단순 사용과 투사에 머무르고 있습니다. AI 아트 또한 좁고 피상적인 수준에 머물러 있는 것이 현실입니다. 플랫폼 기반 접근은 일정 부분 도움을 줄 수 있으나, 데이터가 어떤 역할을 해야 하는지 명확히 정의할 필요가 있습니다.

제 경험에 따르면, 순수한 상태 기반 접근에 도달하는 것은 현재로서는 거의 불가능에 가깝습니다. 그럼에도 불구하고, 진정한 양방향 인터랙티브 시스템 아트가 나아가야 할 매력적인 목표로 볼 수 있습니다.

Tags

Text written by Jeanyoon Choi

Ⓒ Jeanyoon Choi, 2024