Four question types
- Direct — predict behavior from the inferred rule.
- Counterfactual — reason about altered conditions.
- Noisy — hold up under distracting evidence.
- Inverse — run the rule backwards.
Understanding humans over long horizons requires agents to infer not only what people need in the moment, but also how routines form, why they repeat, and when they change. We introduce SimLife, a scalable platform for simulating long-term household life with rich visual observations, ground-truth action logs, and synthetic dialogues with audio. Built on SimLife, SimLife-BP evaluates long-context pattern understanding: the ability to infer latent behavioral rules from weeks or months of everyday observations. The benchmark contains 106 episodes averaging 15.49 hours and 38.57 in-game days, and 1,439 question-answer pairs. Each task probes direct, counterfactual, noisy, and inverse reasoning under different levels of rule hints. Evaluating frontier models and architectures, we find that current models often achieve surface-level prediction without comprehensive rule understanding, rely on frequency-based heuristics rather than if-then reasoning over evidence, and struggle to adapt when behavioral patterns change. These findings suggest that long-context pattern understanding remains a major bottleneck for future embodied agents, while SimLife opens a broader space for studying memory, personalization, adaptation, and long-horizon planning in everyday human-AI interaction.
A single SimLife video spans a full household day. Below is one 24-minute recording of simulated day #1457 — four Sims and a household robot, from 6 AM to 10 PM. You can read across everyone at any moment. Click any block to jump the video there; as it plays, the in-game clock and every character’s current action update below.
Five characters share one 6 AM–10 PM axis, so blocks line up in time. Repeated actions are merged (×N). Click any block to jump the video there.
We tried our best to augment and improve the log quality, while you may still notice that some of them are not perfectly accurate.
Each SimLife-BP episode is a long chain of days governed by a hidden behavioral rule. Every day is colour-coded by its day-condition set, and every task is a group of question types across three hint levels. The episode below holds one fixed rule for all 30 days: the model must infer it purely from what it observes.
Real routines do not stay fixed. In this episode two rules govern the month, and an evolution event in the middle switches from one to the other. The catch: the event is never shown — it is revealed only in dialogue. Afterwards the target activity sometimes happens off-screen too, mentioned only in conversation.
On those two days the clip is captioned with the transcript (scroll it) and a Hear it button plays the real audio. Tasks anchored to the past are marked History — and history tasks never ask a direct prediction.
SimLife-BP asks whether a model can infer the latent rules behind everyday behavior — not just recall what happened. Two axes structure every task.
Evidence is dispersed across weeks; a rule like “coffee on weekdays, or after a late night” links seemingly unrelated events. Models must integrate scattered cues, rule out partial explanations, and adapt when patterns shift.
@misc{peng2026simlife,
title = {SimLife: Pattern Understanding for Long-Horizon Human-Agent Partnership},
author = {Peng, Run and Nie, Zinnia and Ding, Jing and Dai, Yinpei and Zhang, Yichi and
Wu, Zengqing and Fu, Yao and Ma, Ziqiao and Mao, Jiayuan and Chai, Joyce},
year = {2026}
}