Dataset + Benchmark + Reference Model

EgoTools

Tool-Centric Reasoning in Real-World Egocentric Videos

Accepted at EMNLP 2026 Main Conference

Shulin Tian^1,2,*, Junsu Kim^1,3,*, Shuai Liu^1,*, Hao Li^1,*,★, Yujiao Shen¹, Sihan Li¹, Zhe Yang¹, Yeongon Kim¹, Feiyu Li⁴, Jialin Wu⁵, Yichi Zhang¹, Wenhui Wang⁶, Runmao Yao¹, Yuhao Dong¹, Zhaoxi Chen^1,★, Fangzhou Hong^1,★, Antonino Furnari⁷, Jingkang Yang¹, Hongyuan Zhu², Ziwei Liu^1,†,★

¹S-Lab, Nanyang Technological University ²A*STAR ³KAIST ⁴PKU ⁵FDU ⁶School of Biological Sciences, Nanyang Technological University ⁷University of Catania

^*Equal contribution ^†Corresponding author ^★Ropedia Author

Paper PDF arXiv Coming soon Code Coming soon HuggingFace Enter Interactive Demo

Tool-use montage

EgoTools evaluates whether VLMs can reason about tool choice, physical state changes, and long-horizon activity dependencies in real-world first-person video.

TL;DR

From real tool use to measurable reasoning

EgoTools pairs dense real-world egocentric supervision with a diagnostic benchmark and a reference model that demonstrates the value of training on tool-centric data.

100hEgocentric video

1,000QA pairs

7Tool-use domains

+10.9 ppOver the backbone

What makes EgoTools different?

Why EgoTools?

Tool-use reasoning goes beyond recognizing objects in a frame: models must infer why a tool is selected, what it changes, and how earlier actions shape later outcomes.

Tool-choice reasoning

EgoTools asks models to understand why a person uses one tool rather than another, grounded in object properties, task goals, and the surrounding context.

Physical state and outcome reasoning

The benchmark requires tracking how objects change after tool interactions, such as cutting, mixing, fastening, cleaning, measuring, or assembling.

Long-horizon dependency

Many questions require reasoning over minutes-long activities, where the relevant evidence may appear before or after the visible tool interaction.

Egocentric ambiguity

First-person videos include occlusion, hand-object interaction, motion, and viewpoint uncertainty, making simple frame lookup insufficient.

Explore EgoTools data

Explore the EgoTools World

Walk through tool-use environments, select an object, and inspect the egocentric videos and reasoning examples associated with it.

New interactive explorer

Enter the three-room demo

Move through Kitchen, Research Lab, and Repair Workshop, select a tool, and open its release-approved EgoTools clip, narration, and reasoning example.

Enter Interactive Demo Browse Examples without 3D

Benchmark examples

Questions that depend on tools, time, and outcomes

EgoTools questions are multiple-choice, visually grounded, and tied to evidence moments spread across a first-person clip rather than a single object label. Select an answer to reveal the timestamped evidence and reasoning behind the benchmark label.

Demo progress 0 / 3 examples explored

Dataset & benchmark at a glance

Scale, domains, and reasoning taxonomy

A 40.34-hour benchmark subset covers seven real-world domains and four complementary reasoning tracks, with 900 human-crafted and 100 spatial questions.

Dataset coverage

Seven real-world domains with synchronized video, narration, and QA.

Kitchen Classroom Research Lab Repair Workshop Craft Office Household

40.34h quality-controlled benchmark subset

QA construction 1,000 total

Human-crafted QAs 900

Spatial QAs 100

Benchmark subset 40.34h

Benchmark tracks 4 tracks

Affordance & Causality 363 Perception & Grounding 236 Procedural Dynamics 222 Spatial Reasoning 179

Main result · EgoTools-Bench

EgoTools training raises overall accuracy from
50.0% to 60.9%

EgoTools supervision substantially improves tool-centric reasoning under strict source-video separation on the full quality-controlled 1,000-question EgoTools-Bench.

Per-track comparison

Gains across complementary reasoning skills

Exact values

Reference model trained with EgoTools-Data

Model	Overall	AC	PG	PD	SR
Qwen3-VL-8B-Instruct	50.0	46.1	45.9	57.1	55.4, best SR result
ReferenceEgoTools-8B	60.9, best Overall result	55.4, best AC result	59.7, best PG result	68.1, best PD result	44.4

AC: Affordance & Causality
PG: Perception & Grounding
PD: Procedural Dynamics
SR: Spatial Reasoning

Single-stage full supervised fine-tuning improves Overall by 10.9 points, with gains of 9.3 on AC, 13.8 on PG, and 11.0 on PD. SR decreases by 11.0 points, indicating an important direction for stronger spatial supervision.

Training recipe and data details

Single-stage training

Full supervised fine-tuning on EgoTools-Data

184,679 EgoTools-derived instruction examples
69,760 tool-centric reasoning examples (37.8%)
69,554 episode-level multiple-choice examples (37.7%)
31,281 general video-QA examples (16.9%)
9,084 caption and narration examples (4.9%)
5,000 single-image open-ended examples (2.7%)
Visual encoder and multimodal aligner frozen

Analysis · Not the primary claim

Cross-benchmark Generalization Analysis

Transfer to general egocentric benchmarks is reported transparently as a secondary analysis alongside the primary EgoTools-Bench result.

Model	EgoSchema	EgoPlan	EgoThink
Qwen3-VL-8B-Instruct	69.0	42.3	61.0
EgoTools-8B	67.6	35.2	64.0

EgoTools-8B remains within 1.4 points of the backbone on EgoSchema and improves EgoThink by 3.0 points, while EgoPlan decreases by 7.1 points. These secondary results are answer-format dependent; the primary claim remains improved tool-centric reasoning on EgoTools-Bench rather than uniform transfer across all egocentric benchmarks.

Construction pipeline

From first-person collection to a reference model

A reproducible six-stage path connects raw collection, rectification, annotation, benchmark construction, evaluation, and source-video-disjoint training.

1Egocentric Collection
Collect real tool-use activities across seven domains.
2Rectification
Prepare synchronized rectified clips for consistent inspection.
3Tool-Centric Narration
Describe tools, objects, actions, and task progress.
4Benchmark QA
Build grounded multiple-choice reasoning questions.
5VLM Evaluation
Evaluate tool choice, outcomes, time, and grounding.
6Reference Model
Train using source-video-disjoint EgoTools supervision.

Resources & release status

What is available now and what is planned

Unreleased items are shown as status information rather than active links.

Available now

Paper and approved project-page samples

The current paper PDF and the representative clips on this page can be viewed now.

Paper PDFAvailable

Coming with the official release

Release package

Dataset accessComing soon
Benchmark filesComing soon
Official splitsComing soon
Evaluation codeComing soon
Reference modelComing soon
License informationPlanned

FAQ

Common questions

Is EgoTools just object recognition?

No. It targets tool choice, physical state changes, spatial coordination, and temporally extended actions.

Are all videos publicly downloadable now?

No. Release-approved samples are shown here; full access details will accompany the official release package.

What should models output?

EgoTools-Bench uses multiple-choice QA grounded in timestamped visual evidence and reasoning explanations.

Citation

BibTeX

@article{egotools2026,
  title   = {EgoTools: Tool-Centric Reasoning in Real-World Egocentric Videos},
  author  = {Tian, Shulin and Kim, Junsu and Liu, Shuai and Li, Hao and Shen, Yujiao and Li, Sihan and Yang, Zhe and Kim, Yeongon and Yao, Runmao and Dong, Yuhao and Hong, Fangzhou and Furnari, Antonino and Yang, Jingkang and Zhu, Hongyuan and Liu, Ziwei},
  journal = {arXiv preprint},
  year    = {2026}
}

Acknowledgements

Thanks

We thank the data collection, annotation, engineering, and evaluation contributors.