Understanding humans over long horizons requires agents to infer not only what people need in the moment, but also how routines form, why they repeat, and when they change. We introduce SimLife, a scalable platform for simulating long-term household life with rich visual observations, ground-truth action logs, and synthetic dialogues with audio. Built on SimLife, SimLife-BP evaluates long-context pattern understanding: the ability to infer latent behavioral rules from weeks or months of everyday observations. The benchmark contains 106 episodes averaging 15.49 hours and 38.57 in-game days, and 1,439 question-answer pairs. Each task probes direct, counterfactual, noisy, and inverse reasoning under different levels of rule hints. Evaluating frontier models and architectures, we find that current models often achieve surface-level prediction without comprehensive rule understanding, rely on frequency-based heuristics rather than if-then reasoning over evidence, and struggle to adapt when behavioral patterns change. These findings suggest that long-context pattern understanding remains a major bottleneck for future embodied agents, while SimLife opens a broader space for studying memory, personalization, adaptation, and long-horizon planning in everyday human-AI interaction.
A single SimLife video spans a full household day. Below is one 24-minute recording of simulated day #1457 — four Sims and a household robot, from 6 AM to 10 PM. You can read across everyone at any moment. Click any block to jump the video there; as it plays, the in-game clock and every character’s current action update below.
Five characters share one 6 AM–10 PM axis, so blocks line up in time. Repeated actions are merged (×N). Click any block to jump the video there.
We tried our best to augment and improve the log quality, while you may still notice that some of them are not perfectly accurate. The green bars on the right of blocks indicate the visibility of the actions. All green means the action is always visible from the beginning to the end.
Each SimLife-BP episode is a long chain of days governed by a hidden behavioral rule. Every day is colour-coded by its day-condition set, and every task is a group of question types across three hint levels. The episode below holds one fixed rule for all 30 days: the model must infer it purely from what it observes.
Real routines do not stay fixed. In this episode two rules govern the month, and an evolution event in the middle switches from one to the other. The catch: the event is never shown — it is revealed only in dialogue. Afterwards the target activity sometimes happens off-screen too, mentioned only in conversation.
On those two days the clip is captioned with the transcript (scroll it) and a Hear it button plays the real audio. Tasks anchored to the past are marked History — and history tasks never ask a direct prediction.
@misc{peng2026simlifepatternunderstandinglonghorizon,
title={SIMLIFE: Pattern Understanding for Long-Horizon Human-Agent Partnership},
author={Run Peng and Zinnia Nie and Jing Ding and Yinpei Dai and Yichi Zhang and Zengqing Wu and Yao Fu and Ziqiao Ma and Jiayuan Mao and Joyce Chai},
year={2026},
eprint={2609.19610},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2609.19610},
}