DepthJev

Turning Depth into Text for Embodied Navigation

DepthJev converts every RGB frame into metric depth and target detections, then writes them as short text facts.

0%
success on all 300 episodes
0 s
per step on one H100
+0
points over text-only GPT-4o
0
episodes, five subsets
Method

Perceive. Describe. Act.

Monocular metric depth and open-vocabulary detection become a compact, metre-level description of the scene. Jev reasons over that description and returns one of eight navigation actions.

DepthJev pipeline: RGB observation, task and history β†’ metric depth and target detection β†’ text facts β†’ Jev navigation action β†’ next observation
1
πŸ‘οΈ

Perceive

Depth Anything 3 estimates metric depth; OWLv2 finds the target in the frame.

2
πŸ“

Describe

Free space in five sectors, a 0.25 m collision check, target distance and recent actions become text facts.

3
🎯

Act

Jev reads the facts, picks one action, and the loop repeats with the next observation.

Example

β€œI need a vessel to boil pasta”

A common_sense episode. DepthJev resolves the instruction to Pot and reaches it in 9 steps. Step through what the agent sees and what Jev reads.

Observation at the current stepstep 0
0 / 9
Free space ahead
far Lleftcenterrightfar R
Jev decision
move_ahead
2.70 mpot distance
centerpot sector
1.00probability
1,491input tokens
facts Β· step 0

    
Results

Beats vision models, without vision

Success rate on all 300 EB-Navigation episodes. Every other agent passes the image to its decision model; DepthJev passes only text facts.

GPT-4oimage + text
57.7
Gemini-2.0-flashimage + text
48.7
DepthJev (ours)text facts only
46.7
Claude-3.5-Sonnetimage + text
44.7
InternVL2.5-78Bimage + text
30.7
Qwen2-VL-72Bimage + text
21.2
GPT-4otext only
17.4

Baselines from Table 3 of the EmbodiedBench paper.

Per subset success rate

base
32/60
common_sense
31/60
complex_instr.
30/60
visual_appear.
22/60
long_horizon
25/60

One step, 0.76 s mean on one H100

DA3
OWLv2
Jev
Depth Anything 30.15 s
OWLv2 detection0.22 s
Jev decision0.34 s
Rest of the step0.05 s