T-WLA

Tactile World–Language–Action

T-WLABridging Slow Vision
and Fast Touch

Visual memory. Tactile context. Fast action correction.

A visual policy generates an action chunk and history-aware visual memory. Between RGB updates, a lightweight tactile editor uses that cached memory and recent touch to correct the remaining actions.

Explore the demos
39.7 / 119.1 HzReported visual / tactile update rates
30%Lower finger-joint MPJPE on EgoTouch
50.0%Average success across four robot tasks

Reported in the paper. The 30% reduction is relative to the strongest reported EgoTouch baseline.

Paper overview

Abstract

Contact-rich manipulation requires fast tactile corrections grounded in visual information and past observations. T-WLA connects visual action generation and tactile correction through shared visual memory. At each RGB update, the visual branch combines the current observation and task instruction with retrieved visual history, producing an updated visual memory and a base action chunk.

Between RGB updates, a lightweight editor combines cached visual memory, the original base action chunk, elapsed time, and fresh tactile context encoded from recent touch. It corrects only valid, unexecuted actions without rerunning vision. Future-image and future-pressure prediction provide training-only supervision; both prediction heads are removed at inference. We evaluate future hand-motion prediction on EgoTouch and Tachin, and closed-loop control on four real-world manipulation tasks.

Method

Slow vision. Fast touch.

Visual memory retains task and scene information. Tactile context captures evolving contact. The editor uses both to revise an ongoing action chunk without rerunning vision.

T-WLA Figure 2: learned queries retrieve visual history from a memory bank. At RGB updates, the vision–language model produces visual memory and a base action chunk. At tactile updates, encoded tactile context modulates a residual editor conditioned on the cached visual memory and base chunk; only remaining actions are corrected.
Each RGB update refreshes visual memory and generates a base action chunk. Both are cached between RGB updates. Fresh tactile context drives corrections to valid, unexecuted actions, always relative to the original base chunk.
01

Visual memory

A memory bank stores past visual memories. Learned queries retrieve history, which the visual model combines with the current inputs. The resulting visual memory guides both action generation and tactile correction.

02

Tactile context

Recent tactile history is encoded at each touch update. Through FiLM, it modulates action features conditioned on cached visual memory, the original base chunk, and elapsed time.

03

Training-only prediction

Future-image and future-pressure prediction shape features for control. Future observations are supervision, never controller inputs. Both prediction heads are removed at inference.

Real robot demonstrations

Contact-rich manipulation, in motion.

Choose a task, then switch methods. RGB, tactile feedback, and recorded 3D motion share one playback timeline.

Each demo is an individual recorded sequence with waiting periods shortened, not an aggregate success result. The 3D view replays recorded joint motion; it is not a separate simulation rollout. Nonempty recording notes are shown within the player.

Latest video demo · Human hand-motion prediction

Tactile-guided future hand-motion prediction.

The latest demo compares T-WLA with SmolVLA + tactile on the same recorded sequence. Oblique and side views show left-hand meshes and fingertips with a shared wrist pose.

T-WLA vs. SmolVLA + tactileRGB · Hand mesh · Fingertips
Example instruction: “Pick the pink and blue Oreo cookie box from the shelf.” Ground-truth meshes are green and predicted meshes are gray. Green circles mark ground-truth fingertips, gray crosses mark predictions, and red lines show fingertip errors. The clip plays at 0.25× speed. Wrist-local MPJPE values shown in the video describe this example, not dataset averages.

Reported results

From hand motion to real-world control.

Dataset-level errors and closed-loop robot success, as reported in the paper. These results are separate from the individual video examples above.

Future hand-motion prediction

Wrist-relative finger-joint MPJPE (cm). Lower is better.

MethodEgoTouchTachin
VITRA + tactile0.500.63
SmolVLA + tactile0.501.28
MemoryVLA + tactile1.341.87
T-Rex1.141.08
T-WLA0.350.46
Relative reduction30.00%26.98%

EgoTouch: macro-average across five environments. Tachin: complete held-out test set. Reductions are relative to the best baseline for finger-joint MPJPE on each dataset.

Closed-loop manipulation

Task success (%). Higher is better.

ControllerEggSheetCupBagAvg.
SmolVLA02803916.75
SmolVLA + tactile03013717.00
T-Rex43226323.00
T-WLA w/o tactile1872178247.25
T-WLA2178178450.00

Unweighted average over egg transfer, single-sheet pickup, paper-cup nesting, and paper-bag dragging. Robot adaptation uses 30 teleoperated demonstrations per task.