De Stijl
A human-robot interaction game to translate 2D image into 3D space.
My Role
UX Designer
Collaboration
Lisa Fan - Industrial Design
Xiaoye Ren - Design Research
Timeline
3 months, 2024 fall
What I did
Storytelling, Game Design, Coding,
User Test Data Analysis
Context
De Stijl architects masterfully translated the spatial organization of 2D art into 3D by designing spaces that emphasize on architectural transparency—where space not only recedes but fluctuates in a continuous activity.
Problem
However, visualizing how 3D space emerges from 2D projections can be challenging for the human eye. Additionally, the 3D structures derived from these projections are not predetermined but rather open-ended, inviting exploration and interaction as the spaces evolve.
Solution
We seeks to explore this De Stijl 2D to 3d translation process by integrating computer vision and robotic arms with human-responsive input, ultimately providing an analysis of the spatial qualities inherent in the 3D spaces created through De Stijl principles.
Outcome
A multi-interfaces Human - Robot Interactive game
A 2D to 3D translation and visualization algorithm
An Computer Vision analysis of user test on the validity of the game
Design Process
Computer Vision
We break transparency down into four perceptual components: Original point of view (POV), which establishes the baseline visual field; Shadow, which helps reveal spatial depth; Figure, which describes the spaces beyond immediate focus; and
Forms, which define the overall spatial parameters and layering.
We evaluate transparency in three modes—2D transparency, 3D point-of-view transparency, and 3D isometric transparency—each using a different calculation method to measure how much spatial information is visually accessible.
Robotic Kinematics
Through computation, the robotic arm achieved a level of intelligence that mirrors human’s decision-making process—it could recognize spatial depth, assess structural stability, and make decisions between collaborating with and competing against human players.
Game Interface Integration
The game spans across multiple interfaces, linking 2D input images, the real 3D physical environment, and layers of real-time digital analysis. These different modes of information—visual, spatial, and computational—are fused through a multimodal system that synchronizes the robot’s actions, the participant’s moves, and the live point-of-view visualization. Together, this integration allows players to interact simultaneously with physical bricks, on-screen representations, and analytical feedback, creating a cohesive hybrid game environment.
User Tests and Analysis
We conducted a user study followed by brief interviews, and these are a few of the documented results. For each session, we measured several quantitative indicators—the total number of bricks placed, the number of ancillary pieces, enclosed volume count, enclosed volume rate, and the final overall transparency score.
When we compared these metrics with participants’ qualitative feedback, we found a strong alignment between the analytical results and users’ own understanding of spatial transparency.
