How to Understand Experience Through Expressive Data

This documentation was created as part of the course MAS.S60 Critical Matter: Emotive Design from Fashion to Urban Scale, taught by Dr. Behnaz Farahi at the MIT Media Lab, Fall 2025. Team members: Zach Deocadiz, Ami Mehta, Cyan D'Anjou, and Yuxiang Cheng.

Conceptual Framing

With the rise of LLMs we gain the ability to process information beyond traditional numeric data points. Images, voice, poetry, and anything that can be described in natural language can now to "understood" through a computational system. Drawing from this, our team wanted to imagine a world where the kind of data most highly represented and valued is data collected through creative expression.

The core question our work addresses is How can “expressive data” help us understand ourselves? With new ways of processing information, how can we encourage ourselves to think beyond traditional metrics of productivity, success, and optimization by defining a new way of understanding experience through mark-making.

On Expressive Data

We introduce Expressive Data as a form chosen and expressed by the data collector, based on their own subjective experience. It is tracked in a visual format that feels right to them, and aims to allow them to share information that is important to them, yet often remains untracked.

The Approach

On custom-designed sheets, we challenged our audience members to track an aspect of their lives that a machine usually does not have access to, and something that they might ordinarily not track.

We would then process this information in our interactive installation, which also serves as a performance for embodying the step-by-step systems of data processing. The output is another expressive form of data which helps us understand whether we can resonate equally with the machine interpretation as with a human interpretation given that the input is something deeply personal and creative.

Custom-Designed Tracking Sheets

Data Display Wall/Screen

1. 20 pcs 1in x 2in x 6ft wood slats
2. Nails
3. Paper clips
4. Metal to wood epoxy

Data Collection/Input Station

1. Standing desk
2. Webcam
3. Overhead camera mount
4. Standard desk
5. PC
6. Custom software + UI interface

Output System via Printer

1. Plain white paper
2. Printer

Performance Area via microphone + stand

1. Performance microphone
2. Long XRL cable
3. Microphone stand
4. USB Audio interface
5. Bluetooth speaker

This step covers the setup of the physical space and pre-distribution of our custom materials for the final experience.

Custom Tracking Sheet Bundle

1. Prepare the Expressive Data Tracking Sheets and pair them with a dedicated marker and a pen.
2. Ensure the packet contains clear instructions to encourage tracking non-traditional data (like an emotion, pattern, or symptom)
3. Include examples of possible forms of mark-making on the grid (varying thickness, roundness, shapes, density of scribbliness, etc.) to help participants translate abstract observations into visual data
4. Importantly, the data sheets should include data terms of use for participants to fill in to determine how their data is used after submission for our work
5. Distribute the data sheets to participants in advance of the artwork performance to give ample time to fill them out and bring them back day-of

Building the Installation Space

1. Source tables for both the participant station and the data processing station
2. Construct the Data Display Wall in a grid form factor to mirror the data sheets using the wood slats and nails
3. Using epoxy, attach thick paper clips to the tops of each square for participants' easy access to clip their data sheets after use
4. Make sure the display wall is sanded and that any hardware is safely finished

Using the distributed Tracking Sheet packet, this is the initial creation of the expressive data by the participant (data collector).

1. Choose the marker or pen that feel right to you from the provided packet.
2. Track something (such as an emotion, pattern, symptom, etc.) that isn't usually tracked by you or your devices. This can be done from memory. In the prototyping stage, our team members have tracked our anxiety levels, actions that were moved around our to-do lists, daydreams, etc.
3. Using your chosen marker and pen, express the observations you've made on the grid in a way that makes sense for you, utilizing the guide if needed or with full creative autonomy.
4. Fill in the terms of usage form underneath the grid to ensure your data is being used according to your ideal conditions and comfort.

At the installation, the participant places their data underneath the webcam in the Data Collection/Input Station. Equipped with computer vision, the sheet is automatically detected and the expressive data is now processed through two simultaneous systems: the human sensor and the LLM.

The LLM Processor uses Computer Vision to interpret the expressive visual data and terms to generate a mantra.

Through the custom UI active at the The Processing System desk on the opposite side of the wooden screen, a Human Processor (performer) simultaneously interprets the expressive data and is guided through the steps to generate a second mantra.

The two generated mantras are combined in a randomized order onto a sheet, producing the initial output for the Data Collector participant to perform in the next step. This is the machine's and human's interpretation of the subjective input.

Once the data processing step is complete, the mantra sheet is printed at the Data Collection/Input Station printer in a seamless flow. The participant is instructed to move towards the microphone stand and speak the mantra out loud into the microphone however many and in whatever way makes sense for them.

The participant is then asked to annotate the mantra in similar expressive form to the original data collection to indicate what parts of this mantra resonated the most with you. The newly annotated mantra is placed on the scanner to complete the processing cycle.

The final annotated output is a tangible artifact of the full data interpretation cycle. The subjective mark-making is returned back through a personal annotation exercise when processed through the combined human and machine intelligence systems.

The core of the experiment lies in evaluating the experience of daring to self-extend through data and the value of the machine's interpretation in comparison with that of the other human being.

Does this machine-processed output resonate with the original personal experience put onto paper? Through this experience, we encourage a neutral but critical view of how computational systems aim to "understand" and give value to human experience. Through the steps in our project, we hope to ultimately shift the focus from purely quantitative metrics to open up the potential for expressive data to provide profound insight on human experience.