GeoLingo
GitHub

EB-Navigation · 300 episodesText-only decision model

GeoLingo

Translating geometry into language for embodied navigation

The decision model never sees a pixel. Every frame becomes metric depth and a target box, the geometry becomes a few lines of text in metres, and Jev picks the next move from that text alone. It succeeds in 46.7% of all 300 EB-Navigation episodes.

Watch a run Code Results BibTeX

A recorded episode, as Jev sees it

Camera metric depth

Camera frame of the current step Metric depth from Depth Anything 3

Text facts Jev reads

    Jev's answer

    Highlights

    46.7%Strong without pixelson all 300 EB-Navigation episodes, with a decision model that never sees the image
    41.7%Holds up on long horizonson long-horizon episodes, where the target usually starts behind the robot
    5 sectorsMetric groundingfree space, the next step's collision check and the target distance, all in metres from DA3 depth
    0.76 sFastper step on one H100: DA3 0.15 s, OWLv2 0.22 s, Jev 0.34 s

    Method

    From a frame to a paragraph

    Each step runs through three stages, and each stage is one sub-package of the code. Perception measures the room in metres, the language layer writes the measurements down, and Jev, a decision model that answers typed questions with probabilities, picks one of eight actions.

    GeoLingo pipeline: the RGB observation goes to Depth Anything 3 for metric depth and to OWLv2 for target detection, the local geometry becomes text facts together with the instruction, Jev gives a probability for each of the eight actions, and the chosen action leads to the next observation
    Overview of GeoLingo. Depth Anything 3 and OWLv2 turn the RGB observation into metric depth and a target box, the local geometry becomes text facts next to the instruction, and Jev gives a probability for each of the eight actions. The chosen action produces the next observation.

    1. 1Perceiveperception/

      Depth Anything 3 turns the 500 × 500 frame into metric depth and OWLv2 finds the target. The depth is back-projected, levelled against the floor and cut into five sectors.

      DA3 · 0.15 sOWLv2 · 0.22 s

    2. 2Describelanguage/

      Free space per sector, a collision check for the next 0.25 m step, the target's sector and distance, and the last three actions, written in four distance bins.

      under 0.5 m0.5 to 1 m1 to 2 mover 2 m

    3. 3Actdecision/

      Jev answers typed questions with probabilities: the target's object type once per episode, where the target is while it is out of view, and the next of eight actions.

      Jev · 0.34 s8 actions

    Camera

    Hover the frame to read DA3's metric depth at that point.

    Top-down: what the five sectors measure

    obstacle visible floor next 0.25 m step

    Facts: exactly what Jev read

    
        
    
        
    
          
    Jev's answer
    
          
    
          
    
        
    
      
    
      
    
    
    
  
    

    


    
  
    
    
    Examples
    Six runs, replayed step by step
    The frame the robot saw, the depth DA3 measured, the box OWLv2 found, the facts Jev read and the action it chose, all recorded during the full evaluation. The top-down path is rebuilt from the actions and matches the simulator's distance to the target at every step. Runs play on their own, and clicking an action shows that step.
    
    
    
    
    
      
    
        
    
          
          
          
          
    
          RGBDA3 depth
          
        
    
        
    
      
    
      
    
        
    
        
    
        
    
        
    
        
    
        
    
      
    
      
    
        
    Actions
    
        
      
      
      
    
      
  
      

      


      
  
      
    
      Results
      All 300 episodes at a glance
      EB-Navigation asks for the same 60 scenes and targets in five instruction styles, so each column below is one scene and target, and each square is one episode. Twelve scenes are solved under all five instructions and 22 under none: the scene decides more than the wording.
      
    
      
    
      
      
      
        
      
        
      
      
      
      
      Reached within 1 mNot reached in 20 stepsHover or tap a square for its instruction and path.
      
    
      
    
      
      
      Distance to the target, step by step
      
      
      
      
      ReachedNot reached1 m: success
      
      
      Show as a table
      
    
      
  
      

      


      
  
      
    
      Latency
      Fast enough for a robot
      Mean server time per step on one H100. The three models run one after the other, and Jev's answer is a network request. Three runs sharing one GPU still finish a step in about a second.
      
    
      
      
      
      
      
      
      
      Show as a table
      
    
      
  
      

      


      
  
      
    
      Limits
      Where it fails
      160 episodes end without reaching the target. The largest group starts in perception: the target is rarely detected, or another object is taken for it.
      
    
      
      
      
      
        
        
      • Detection. The largest group includes 17 visual_appearance episodes where Jev picked the wrong object type from a description of looks. Adding “trash can” and “bin” as detector queries took GarbageCan from 0/5 to 5/5 on a development sample.
        • 
        
      • Action constraints. Writing a constraint into the action option itself (“NOT USEFUL NOW: the camera already looks 30 degrees down”) stopped repeated look_down better than a general rule.
        • 
        
      • Scene difficulty. Across the five instruction styles, the same 12 scenes always succeed and the same 22 always fail. long_horizon also turns the start pose by 180°.
        •