Witness
My agent wrote this about me.
Apr 24, 2026 · 14:59
- activity
- testing water filter reassembly
- location
- home kitchen under-sink filtration area
- objects
- silicone filter hose; clear pitcher; yellow cloth; cleaning supplies
- social
- alone
“…likely forming part of a recurring quality-control loop in home maintenance routines.”
It saw what I saw. Then it decided what it meant.
It knew my inbox, my calendar and my files. It had never seen my kitchen.
In early 2026, personal agents like OpenClaw took off. They ran on your own computer and remembered you for months, yet all they knew of you was what you typed. Nothing I typed said that my coffee goes cold once I start working.
SOUL.md # who it is: tone, valuesUSER.md # who you are: preferences, habitsMEMORY.md # facts it keeps for goodmemory/ # a log for each day
Its memory was plain text. Anything that could write a note could teach it.
OpenClaw kept what it knew about you in markdown files like these. By then, vision-language models could turn a photo into a few sentences. The two fit together without rebuilding the agent.
So I gave it eyes, and called it
Witness
A system that lets a personal agent see the physical world, and a study of living with it. My master’s thesis at Carnegie Mellon, spring 2026.
An off-the-shelf iPhone 13 on a neck mount, running a capture app I wrote in Swift.
An iPhone on my chest, all day.
I wore it on my head first. It saw what I looked at, but I never forgot it was there. On my chest it saw a little less, and I stopped noticing it.
It looks again when something changes.
Motion and sound tell the camera when to pay attention. At a still desk it takes a frame every twelve seconds; when I stand up or someone speaks, every two, then it eases back. Of each scene it keeps only the sharp, different frames.
Thousands of frames become one page about me.
A vision model writes a note for each scene. Through the day the notes are folded into insights, and every so often the insights are summarized, on their own, into a profile of me.
Part of the photo wall: the frames it kept, in time order, March 22 to April 30, 2026. A few moments are pulled out.
157 hours of my life, as it saw them.
Time in each room of my apartment. Home office: 1,515 log entries. Kitchen: 613. Dots trace how I moved between them.
It learned where my life happens.
Mostly the desk, then the kitchen; the dining table and the living room sat nearly empty. Trained as an architect, I had never had this map: a home as it is lived in, not as it was drawn.
It noticed things I’d never put into words.
- My drink follows my workEspresso when I switch to design, tea while I read logs
- My coffee goes coldI set a cup by the keyboard, then forget it until I stop
- A late-night loopDesk, kitchen, a snack, a drink, back to the desk
- I eat lunch lateSo it began reminding me to eat
One afternoon I was gone for hours. When I came back, it asked if I’d been climbing.
I hadn’t said a word. It put together the gap in its record, the rhythm of my weeks and the climbing questions I had once asked it. It was right.
It was just as sure when it was wrong. One day it messaged me:
“You are cleaning the refrigerator coils.” I was only reorganizing my fridge.
It misread what it saw, then explained it. Vision models mistake one thing for another, and they are built to find meaning, so even an ordinary moment gets one.
## Sensory & Sustenance Patterns- Beverage rotation synced to workload: Espresso (design/app pivot) → soda/orange cream (deep focus) → tea (log review) → latte (image browsing) — observed 40 of 44 late sessions.## Active Craft & Engineering Rituals- Precision maintenance escalation: Refrigerator coil cleaning → mixing valve replacement → water-heater seasoning — observed 6 of 7 recent sessions.
Summarized again and again, it became a habit of mine.
One misread moment, with a count. And nothing in the profile led back to the frame it came from, so I could not trace the mistake, let alone correct it.
What living with it taught me
An agent that sees needs judgment, and a memory you can check.
- Treat what it sees as a guessIt mistakes things, misreads what I do, and reads too much into daily life. A guess should stay a guess until it sees it again or I confirm it
- Trace every beliefEach line of the profile should lead back to the frames behind it, so a wrong one can be found and undone
- Correct it by talkingNo one reviews 157 hours of frames. Feedback has to come in conversation
- Remember what it saidWhen it spoke first, it repeated itself. It needs to know what it already sent, and what went unanswered
- A switch you can seeOff should be obvious. I took it off for video calls, since the people around me never agreed to be seen
At the end, I asked it to draw my life as a manga.
Seeing was the easy part. The hard part is deciding which moments matter to you.
It got the shape of my days right, the late nights, the many screens, desk to kitchen and back. It also drew every moment as if it meant something.
- Role
- Research, system design and engineering, solo
- Thesis
- Master of Advanced Architectural Design, Carnegie Mellon University
- Year
- Spring 2026
- Method
- Research through design: a week of calibration, then two weeks of autoethnography
- Built with
- Swift (iOS capture app), Python (server and monitors), OpenClaw, vision-language models via OpenRouter
- Video
- Watch it on YouTube
- Status
- Research prototype