EGO-CENTRIC INTELLIGENCE / ONLINE

PAPER / EMNLP 2026

EgoTools: Towards Tool-Centric Reasoning in Real-World Egocentric Videos

Beyond seeing.Understand the tool.

EgoTools is a 100-hour dataset, a diagnostic benchmark, and an 8B reference model for reasoning about tools in real-world first-person video.

Shulin Tian^1,2,*Junsu Kim^1,3,*Shuai Liu^1,*Hao Li^1,*,★Yujiao Shen¹Sihan Li¹Zhe Yang¹Yeongon Kim¹Feiyu Li⁴Jialin Wu⁵Yichi Zhang¹Wenhui Wang⁶Runmao Yao¹Yuhao Dong¹Zhaoxi Chen^1,★Fangzhou Hong^1,★Antonino Furnari⁷Jingkang Yang¹Hongyuan Zhu²Ziwei Liu^1,†,★

¹S-Lab, Nanyang Technological University
²A*STAR
³KAIST
⁴PKU
⁵FDU
⁶School of Biological Sciences, Nanyang Technological University
⁷University of Catania

* Equal contribution† Corresponding author Ropedia Author

Try the benchmark

REAL CASE ROTATION / 01

GROUND → TRACK → INFER

KitchenSpatula

Transfer without piercing

01
GroundLocate the tool
02
TrackFollow its use
03
InferExplain the choice

01 / DATA02 / BENCH03 / MODEL

FILM

THE FULL STORY

Watch tools become evidence.

The complete EgoTools project film—from real-world collection to tool-centric narration, grounding, evaluation, and learning.

01 / PROJECT FILMEgoTools in 100 hours of real-world tool use.VIDEO / SOUND AVAILABLE

ACADEMIC OVERVIEW

Recognition names the object.Reasoning explains the action.

Real work unfolds over minutes, not frames. A model must know why one tool is chosen, how it changes the target, and which earlier step made the current action possible.

DATA / HOURS100.37

hours of real-world egocentric tool use

VIDEO / EPISODES646

long-form first-person videos

CAPTION / HIERARCHY361K

361,332 hierarchical captions across multiple timescales

NARRATION / INTENT6,519

tool-centric narrations grounded in action

THE FIELD LAB

Seven domains. Zero studio shortcuts.

Everyday spaces become a proving ground for physical reasoning: messy, occluded, long-horizon, and irreducibly real.

Open 3D PlaygroundExperimental

First-person kitchen scene with a knife and cutting board — D-01Kitchen

First-person laboratory scene using a multichannel pipette — D-02Research Lab

First-person repair scene using pliers — D-03Repair Workshop

First-person classroom balloon rocket experiment — D-04Classroom

First-person craft scene sewing a plush object — D-05Craft

First-person office scene working at a desk — D-06Office

First-person household scene preparing coffee — D-07Household

INTENT / ANNOTATIONWhy tools are chosen

EgoTools records the decision around a tool interaction—not just a list of low-level hand motions. Tool-centric narrations connect the selected tool and its properties to the task, the intended use, and the reason it was chosen over an alternative.

That structure preserves the task-state changes needed to study affordance, causality, and procedure across long, real-world episodes.

01 / INTENT

READ THE DECISION

Not just what happened.Why this tool, for this task, now.

A tool interaction becomes useful supervision when action is connected to purpose.

Narration interface and examples connecting tool choice to task intent — FIG / NARRATIONFrom visible action to tool choice and task intent.ANNOTATION / TOOL + PURPOSE + REASON

THE BENCHMARK

One thousand questions. No frame lookup.

Every question asks for evidence-grounded reasoning across tool choice, perception, procedure, or spatial relations.

40.34H RESERVED8-WAY MCQ4 TRACKS

Affordance & Causality

Why this tool, and what changes because of it?

363

Perception & Grounding

Which object, state, property, or evidence moment?

236

Procedural Dynamics

How do steps, transitions, and outcomes connect?

222

Spatial Reasoning

How do the hand, tool, and target align in 3D?

179

TOOL IDENTIFICATIONDIFFICULTY / EASY

What is the most accessible and convenient tool used to move the asparagus from the pan to the plate?

THE SIGNAL

Training on real tool use moves the needle.

The paper reports that EgoTools supervision lifts the 8B backbone from 50.0 to 60.9, a gain of 10.9 percentage points.

PAIRED COMPARISON / ACCURACY (%)Same backbone. Matched 64-frame protocol.

Affordance & Causality (AC)
+9.3
Perception & Grounding (PG)
+13.8
Procedural Dynamics (PD)
+11.0
Spatial Reasoning (SR)
−11.0
Overall (ALL)
+10.9

12.5% chance level · 8-way multiple choice

READ THE WHOLE RESULT

Better overall.Not everywhere.

AC +9.3 · PG +13.8 · PD +11.0 · SR −11.0Spatial reasoning remains the open problem.

FULL BENCHMARK SNAPSHOT

Every model. Every track.

All results reported in the current arXiv manuscript, now in one sortable ranking. Choose any track to compare every model.

Have a new run? Send the checkpoint, input protocol, predictions, and reproduction notes.

Submit to EgoTools-Bench

Unified ranking · Accuracy (%) · Sorted by Overall
Rank	Model	Category	Eval. frames	Audio
01	Human Expert	Human reference	Free-view	✓	82.1	85.2	83.3	82.7	83.2
02	Gemini-3.1-Pro	Proprietary	1 fps	✓	69.7	51.7	72.1	74.9	66.9
03	Gemini-3-Flash	Proprietary	1 fps	✓	64.5	50.0	73.0	72.6	64.4
04	EgoTools-8B	Ours · Fine-tuned	64	—	55.4	59.7	68.1	44.4	60.9
05	Gemini-3.1-Flash-Lite	Proprietary	1 fps	✓	58.1	44.5	59.9	58.7	55.4
06	GLM4.1V-Thinking	Open · Reasoning	64	—	46.1	50.0	57.1	53.3	50.7
07	Qwen3-VL-8B-Instruct	Open · Instruct	64	—	46.1	45.9	57.1	55.4	50.0
08	InternVL3.5-8B-Instruct	Open · Instruct	64	—	45.8	45.9	53.4	53.3	48.7
09	InternVL3-8B-Instruct	Open · Instruct	64	—	44.7	44.9	51.8	48.9	47.1
10	Qwen2.5-Omni-7B-Instruct	Open · Instruct	64	✓	44.7	45.9	51.8	46.7	47.1
11	Qwen3-VL-4B-Instruct	Open · Instruct	64	—	41.7	45.4	50.2	50.0	45.7
12	Qwen3-VL-8B-Thinking	Open · Reasoning	512	—	43.6	41.8	48.2	55.4	45.7
13	MiMo-VL-7B-RL	Open · Reasoning	64	—	43.1	45.9	50.6	39.1	45.3
14	Qwen2.5-VL-7B-Instruct	Open · Instruct	64	—	40.1	43.3	50.2	47.8	44.3
15	MiMo-VL-7B-SFT	Open · Instruct	64	—	41.4	44.9	49.4	40.2	44.2
16	Qwen2-VL-7B-Instruct	Open · Instruct	64	—	46.6	35.6	46.2	47.8	44.2
17	LLaVA-Video-7B-Qwen2	Open · Instruct	64	—	38.2	39.7	43.3	37.0	39.8
18	LLaVA-OneVision-7B	Open · Instruct	64	—	37.6	37.1	42.9	37.0	38.9
19	Qwen3-VL-4B-Thinking	Open · Reasoning	512	—	37.6	36.6	36.4	41.3	37.4

Human Expert is shown as a free-viewing ceiling; model protocols remain visible through category, frame budget, and audio columns. Gemini models use 1 fps video.

A check means synchronized non-narration audio is provided. Narrations, dense captions, and annotation metadata are never used as evaluation inputs.

THE SYSTEM

From experience to intelligence.

A single traceable pipeline connects real-world collection to a model that can explain what it sees.

04 / TRACE

TEMPORAL EVIDENCE

A frame shows the tool.A trace explains the choice.

Ground the evidence, verify it through time, then lift the path into space.

EgoTools visualization grounding a kitchen tool in a first-person frame — 01 / GROUNDResolve the referenced tool in the first-person frame.2D GROUNDING / EVIDENCE

EgoTools visualization tracking a tool through kitchen footage — 02 / TRACKVerify tool identity as the evidence moves through time.TEMPORAL TRACK / CONSISTENCY

EgoTools visualization reconstructing a tool trajectory in 3D — 03 / LIFTLift the trace into depth, pose, and a 4D trajectory.3D + 4D / RECONSTRUCTION

01Collect
Real tool-use episodes across seven domains
02Rectify
Canonical first-person video at 1024 × 1024
03Narrate
Hierarchical captions and tool-centric intent
04Ground
2D/3D tool trajectories through time
05Evaluate
1,000 evidence-grounded 8-way questions
06Train
A reference model with benchmark-window exclusions

RELEASE PACKAGE / COMING SOON

Build models that understand the work.

EgoTools: Towards Tool-Centric Reasoning in Real-World Egocentric Videos. Accepted at EMNLP 2026 Main Conference.

PAPERCOMING SOON

CODECOMING SOON

DATACOMING SOON

MODELCOMING SOON