DIRECT PERCEPTION CONTROL · SYMBIOSIS RESEARCH

Direct Perception Control Model

August 2026·SYMBIOSIS RESEARCH

Scaling perception helps robots understand the world, but understanding a task does not automatically make action reliable. We want to move the boundary of an intelligent model from generating a kinematic goal that must be tracked to generating physical action that depends on the robot’s current bodily state and can be executed directly.

TABLE OF CONTENTS

01 / PRELIMINARY

Revisiting Hierarchical Perception Control Paradigm

The limits of hierarchical perception control appear at three points along the execution path, in the same order shown in Figure 01 and examined below. First, an insufficient motion interface can discard task-relevant control information when System 1 compresses the complete context into Z_t. Second, under separate training, coupled inference, System 1 and System 0 are optimized by different objectives even though they must act as one chain at deployment. Third, a low-level action boundary confines executable action to the control distribution learned by the frozen System 0.³⁴

Figure 01. The two-layer architecture maps complete task context through a Latent Policy and motion interface into a Whole-Body Tracker. Its three limits—an insufficient motion interface, separate training, coupled inference, and a low-level action boundary—correspond directly to Problems 1–3 below.

Problem 1: Insufficient Motion Interface

A motion goal specifies what the body should do; physical action must also determine how the current body should realize it. If distinct contexts X_t collapse into nearby Z_t values but require different Joint Targets, a tracker reading only Z_t cannot recover the discarded control information. This gap is especially acute in loco-manipulation, where contact, posture, and support are tightly coupled.

Figure 02 tracks the same samples from Latent Space to Joint Target Space. In Compare mode, latent-near pairs are matched by sample ID, and the distance panel measures how far their required actions separate. The larger separation under loco-manipulation shows that similar coarse motion intent can hide different contact, balance, and upper–lower-body coordination requirements. If neighborhood relations do not survive this mapping, Z_t is not a sufficient action interface.

Figure 02. Pair Explorer tracks the same real samples from Latent Space to Joint Target Space, revealing how similar motion intents can correspond to different physical actions.

Problem 2: Separate Training, Coupled Inference

A hierarchical architecture appears as one continuous path during inference, but this continuity does not exist in its training objective. System 1's Latent Policy learns the motion interface, while System 0 independently learns tracking, stability, or control reward. They are connected through Z_t only at deployment, so execution error and visual consequences cannot pass through the frozen interface to co-optimize System 1.

Figure 03. The Latent Policy and Whole-Body Tracker receive latent-level and control-level supervision separately during training, then are chained through motion interface Z_t at inference; the complete execution path therefore has no unified end-to-end action objective.

Problem 3: Low-Level Action Boundary

Final action is constrained not only by decoder form but also by the motion distribution covered by System 0's training data. If the low-level prior is learned primarily from locomotion-centric behavior, new loco-manipulation tasks at the high level must still project back into that execution distribution; joint-level actions outside it do not become reachable simply through stronger perception or reasoning.

Figure 04 should be read as a distribution rather than a trajectory. The horizontal Locomotion and Manipulation axes locate motion activation, while height represents relative sample density; overlap marks the shared motion prior. The locomotion-centric distribution remains concentrated around locomotion-heavy behavior, whereas loco-manipulation extends into mixed-activation regions. These non-overlapping regions expose actions required by new tasks but absent from System 0's learned support.

Figure 04. Locomotion-centric data covers only a limited whole-body execution region, while loco-manipulation extends into areas where locomotion and manipulation are tightly coupled, exposing the distributional boundary of a frozen low-level prior.

HIERARCHICAL PIPELINESemantic intent is compressed, then handed to a frozen executor

01Perception + LanguageScene · Task

S1Latent PolicyHigh-level Latent

↘Compressed InterfaceToken / Trajectory

S0Whole-Body TrackerFrozen Motion Prior

05Physical ExecutionJoint Target · PD

Information flowThree limits: representation · joint training · action space

These limits answer three distinct questions: whether task-control information survives the intermediate representation, which action representation can be co-trained with future visual dynamics, and which PD targets the final policy can express. Dimensionality alone is not the criterion; the decisive factors are representation semantics, supervision, and the frozen decoder’s action image.

X_tVision + taskH_tState historyZ_tMotion tokenA_tJoint target

KINEMATIC REPRESENTATION & SUPERVISION

Kinematic Representation and Supervision Bottleneck

Task and visual context X_t together with state history H_t must first be compressed into the SONIC motion representation Z_t. Yet Z_t is supervised as a desired motion or reference pose: it answers “what motion should the body exhibit,” not “which joint target should be issued at this control step.” Task-control semantics discarded by this kinematic abstraction cannot be recovered downstream.

REPRESENTATION + SUPERVISION

01 / SUFFICIENCY GAP

R_Z^* = R_X^* + 𝔼 [ ‖ 𝔼[A^* | X, H] − 𝔼[A^* | Z, H] ‖² ] ≥ R_X^*

02 / LOSS GEOMETRY

L_token = δz^T I₆₄ δz ; L_action ≈ δz^T J_h^T J_h δz ; rank(J_h^TJ_h) ≤ 29 < 64

Under Z = f_SONIC(X, H) and squared loss; J_h = ∂D_frozen(z, h) / ∂z

REPRESENTATION BOTTLENECKIf two task contexts collapse to the same Z under the same H but require different A^*, no policy reading only (Z, H) can separate them. The risk gap is precisely task-relevant control information removed by the kinematic representation.

SUPERVISION BOTTLENECKThe issue is not that “64D is too small.” Since J_h ∈ ℝ^29×64, the rank of J_h^TJ_h is at most 29, so it cannot be globally equivalent to the full-rank I₆₄: some token errors barely alter action, while others are amplified by the decoder. Direct-joint places supervision in the final control space.

DYNAMICS GROUNDING UNDER A FROZEN DECODER

Dynamics-Grounding Gap under a Frozen Decoder

Both paths observe X_t and H_t. The key difference is what supervises the hidden representation attended by the future visual branch. In the old path, E_Z,t is explicitly required only to decode a kinematics-level SONIC target; the state-conditioned joint action appears only after the frozen decoder. Direct-joint instead lets future visual tokens attend E_A,t, a representation supervised by the real joint-control target.

WHAT FUTURE VISUAL ATTENDS

FROZEN SONIC PATH

Y_future ← E_Z,t → Z_t ; A_t = D_frozen(Z_t, H_t)

DIRECT-JOINT PATH

Y_future ← E_A,t → A_t → S_t+1

SHARED GRADIENT

∂L_total / ∂E_A,t = ∂L_action / ∂E_A,t + λ_visual ∂L_future / ∂E_A,t

JOINT-TRAINING GAPThe frozen decoder may use proprioceptive feedback, but its state-feedback law lies outside the visual-action model, so ∂L_future / ∂θ_D = 0. Future prediction cannot reshape the mapping from motion goal and current state to joint control.

DYNAMICS GROUNDINGA joint action is not the full dynamics, but together with H_t it is the control variable entering the short-horizon transition. Action loss makes A_t decodable from E_A,t, while future loss learns its consequence from the same representation—more direct grounding than a representation guaranteed only to decode the SONIC target.

FROZEN ACTION-MANIFOLD CONSTRAINT

Frozen Action-Manifold Constraint

For a fixed state history h, a latent-based policy—regardless of upstream model strength—must output an action inside the frozen decoder’s image of valid SONIC latents, M_h. Direct-joint removes this learned decoder constraint: it retains the same 29D PD-target interface without requiring each task action to first lie in M_h.

EXPRESSIVITY AT THE SAME PD INTERFACE

FROZEN DECODER IMAGE

M_h = { D_frozen(z, h) : z ∈ Z_valid } ⊆ ℝ²⁹

UNREACHABLE-TARGET ERROR

infz ∈ Z_valid ‖D_frozen(z, h) − a^*‖² = dist(a^*, M_h)²

DIRECT-JOINT PD TARGET

A_joint = q_default + C_scale A_raw ; det(C_scale) ≠ 0

The fixed affine map is invertible for nonzero scales. The comparison concerns a learned action manifold—not the removal of joint limits, torque saturation, or safety clipping.

STRICT EXPRESSIVITY FLOORIf the task-required action a^* lies outside M_h, the latent path has a positive irreducible action error. Better perception or planning cannot cross the expressivity boundary of a frozen System 0.

FREEDOM AND ITS COSTA 64D → 29D map is not automatically restrictive; the relevant object is the actual image of valid latents. Direct-joint enlarges the hypothesis class within the same PD-target interface, but also relinquishes decoder priors for balance, smoothness, and whole-body coordination—capabilities that must be relearned from data and closed-loop training.

FROM GOAL TO CONTROL

The three limits occur sequentially along the old path: task semantics are filtered by a kinematic representation, state-conditioned action remains outside joint training, and final output is constrained by the decoder image. Direct-joint relaxes these layers through action-level supervision, dynamics-grounded co-training, and a freer joint-space output—gaining reach while assuming responsibility for relearning stable control priors.

02 / METHOD

DIRECT PERCEPTION CONTROL MODEL

Our method answers the three limits in Preliminary one for one. For the Insufficient Motion Interface, the Direct Perception Control Model removes the intermediate motion representation entirely: instead of producing a reference for a separate Whole-Body Tracker, one model maps vision, language, body state, action history, and execution feedback directly to executable joint and hand targets. For Separate Training, Coupled Inference, Symbiotic Attention lets perceptual and control representations attend to each other under a shared action objective, so task understanding and body dynamics co-adapt rather than meet only at deployment.⁵⁶

Figure 05. The Direct Perception Control Model maps perception and live body context directly to joint and hand targets without an intermediate motion interface. At its center, Symbiotic Attention couples perceptual and control representations during training, while asynchronous vision updates and continuous proprioceptive feedback sustain closed-loop execution.

For the Low-Level Action Boundary, DriftDistill expands the Student's closed-loop training distribution beyond the fixed support inherited from demonstrations and the previous controller. Offline demonstrations cover ideal trajectories, but closed-loop errors push the Student into states outside that support. DriftDistill makes this drift trainable: Offline BC initializes the semantic-action prior, the Student rolls out in the real loop, and a Frozen Teacher supplies recovery targets only for states the policy actually reaches. Distilling each correction back into the Student turns the next rollout into a broader, more recovery-capable policy. Together, the three designs remove the interface, couple learning, and expand executable support.⁷⁸

01 / GROUND

Offline demonstrations

02 / INITIALIZE

Unified Policy

03 / VISIT

Policy-visited states

ON-POLICY RECOVERY LOOPVisit → Correct → AbsorbEach rollout expands the training distribution

05 / DISTILL

L_rec(aₜ, aₜ^*)MINIMIZE

Recovery loss

04 / CORRECT

qₜ→π_T→aₜ^*

Frozen Teacher

Figure 06. DriftDistill addresses the Low-Level Action Boundary through a Visit–Correct–Absorb loop: the Student visits drifted states, the Frozen Teacher provides recovery targets, and those corrections are absorbed into the deployed Student policy.

DriftDistill is therefore not a fixed two-stage recipe. Its training distribution unfolds with the Student: each rollout exposes a new frontier of drift and recovery states, the Teacher converts that frontier into supervision, and the Student absorbs it before the next rollout.

03 / DATA

Boosting Loco-Manipulation Data

Direct physical-action learning requires supervision in the Joint Target Space where execution occurs. Existing human and robot corpora cannot be combined directly because their embodiments, action representations, and control interfaces differ. We therefore convert every source into G1-executable, time-aligned joint trajectories, then scale this common supervision to 15,010 hours.

Unifying Heterogeneous Motion Evidence

Conversion uses the most reliable motion signal available from each source. Teleoperated humanoid data are standardized across coordinate frames, joint definitions, and control rates. Head-mounted recordings are reconstructed and retargeted from recoverable whole-body motion. Egocentric video is lifted from hand, wrist, and end-effector trajectories into humanoid whole-body motion. All three paths produce the same joint-target supervision.

Figure 07. The heterogeneous data conversion pipeline uses the reliable motion information preserved in each source to produce unified, G1-executable, time-aligned joint trajectories.

Scaling the Unified Corpus

The resulting corpus contains 15,010 hours: 6,781 hours of Human Ego, 4,024 hours of Armed Robot, 3,660 hours of Wheeled Humanoid, and 545 hours of Bipedal Humanoid data. These sources span human and robot embodiments with different mobility and interaction capabilities; after conversion, all of them enter one execution space without tying training to a single platform.

Figure 08. The 15,010-hour embodied training corpus spans Human Ego data and multiple robot embodiments, providing a scalable foundation for whole-body loco-manipulation.

Figure 09. Representative converted samples synchronize original data from different sources with the corresponding humanoid joint motion, showing how heterogeneous data is unified into the same whole-body execution space.

04 / EXPERIMENTS

What the Experiments Must Answer

QUESTION 01

Does joint optimization reduce cascading error?

Task-level success measures the accumulated effect of errors across perception, action generation, and closed-loop execution. Figure 10 compares that outcome under the same four evaluation tasks.

Success Rate (%) ↑

Figure 10. Our method achieves a higher task success rate than the Baseline across all four evaluation tasks, with the largest gains on Task A and Task B.

QUESTION 02

Are semantics and control better aligned?

This question tests whether the task instruction remains consistent with whole-body behavior as contact, posture, and scene state evolve during execution.

Mobile pick-and-place

The robot approaches the cone, bends and squats to grasp it with one hand, turns right, walks forward two steps, and places it down. The sequence couples precise reaching, balance under load, turning, and object transport.

Hand–eye–foot coordination

The right foot controls the accelerator while the hands rapidly steer right near the corner exit. Success requires precise throttle control, visual timing, and semantic understanding of the driving objective.

Figure 11. Mobile pick-and-place and hand–eye–foot driving connect semantic task intent to closed-loop whole-body execution.

QUESTION 03

Is execution still bounded by System 0?

Execution quality should be inspected at the physical-action level rather than inferred from task success alone.

Constrained whole-body loco-manipulation

A constrained posture and tight clearance push beyond locomotion-centric support: the torso and both arms must reconfigure while stable support and precise object contact are preserved.

Figure 12. This sequence provides a representative case for inspecting closed-loop physical execution beyond task success alone.

05 / FUTURE WORK

Making Intelligence Physically Executable

Scaling perception has helped robots read the world and understand tasks. True Whole-Body Intelligence must also understand the task objective, bodily state, dynamic constraints, and the consequences of action. Direct Perception Control aims to shorten this Embodiment Gap. We are not training a stronger Motion Planner; we are training a system that learns continuously from understanding the world to using its own body to complete a task.¹³¹⁴

The next frontier of robot intelligence is not only better perception or better planning. It is learning how intelligence becomes physically executable action.

Citation

@article{symbiosis2026directperceptioncontrol,
  author = {SYMBIOSIS Research},
  title = {Direct Perception Control Model},
  journal = {SYMBIOSIS Research Blog},
  year = {2026}
}