Direct Perception Control Model

August 2026·SYMBIOSIS RESEARCH

Scaling perception helps robots understand the world, but understanding a task does not automatically make action reliable. We want to move the boundary of an intelligent model from generating a kinematic goal that must be tracked to generating physical action that depends on the robot’s current bodily state and can be executed directly.

TABLE OF CONTENTS

01 / PRELIMINARY

Revisiting Hierarchical Perception Control Paradigm

Most whole-body systems still use a modular architecture: System 1 understands the task and generates a high-level motion goal, while System 0 tracks that goal from the current body state.¹² This makes existing controllers reusable, but introduces three problems across the complete Perception-to-Control execution path. First, an insufficient motion interface can discard task-relevant control information when System 1 compresses the complete context into Z_t. Second, under separate training, coupled inference, System 1 and System 0 are optimized by different objectives even though they must act as one chain at deployment. Third, a low-level action boundary confines executable action to the control distribution learned by the frozen System 0.

Figure 01. The two-layer architecture maps complete task context through a Latent Policy and motion interface into a Whole-Body Tracker. Representation compression, fragmented optimization, and the low-level motion prior produce an insufficient motion interface, separate training with coupled inference, and a low-level action boundary.

Problem 1: Insufficient Motion Interface

A motion goal specifies what the body should do, while the tracker combines Z_t with the current robot state to produce state-dependent Joint Targets. The limitation is not that the tracker lacks bodily state. It is that scaling only the goal generator improves task understanding and goal expression without expanding the tracker's physical action mapping across contact, balance, and dynamic conditions. Additional perceptual and reasoning capacity therefore still stops at the motion interface.

Figure 02 tracks the same samples from Latent Space to Joint Target Space. In Compare mode, latent-near pairs are matched by sample ID, and the distance panel measures their separation after state-conditioned action mapping. The larger separation in whole-body loco-manipulation is not a tracker error; it shows that decisive choices about contact, balance, and upper–lower-body coordination occur after the interface. Scaling only the upper-level goal generator leaves this mapping—and its physical capability boundary—unchanged.³⁴

Figure 02. Pair Explorer tracks the same real samples from Latent Space to Joint Target Space, revealing how similar motion intents can correspond to different physical actions.

Problem 2: Separate Training, Coupled Inference

A hierarchical architecture appears as one continuous path during inference, but this continuity does not exist in its training objective. System 1's Latent Policy learns the motion interface, while System 0 independently learns tracking, stability, or control reward. They are connected through Z_t only at deployment, so execution error and visual consequences cannot pass through the frozen interface to co-optimize System 1.⁵

Figure 03. The Latent Policy and Whole-Body Tracker receive latent-level and control-level supervision separately during training, then are chained through motion interface Z_t at inference; the complete execution path therefore has no unified end-to-end action objective.

Problem 3: Low-Level Action Boundary

Final action is constrained not only by decoder form but also by the motion distribution covered by System 0's training data. If the low-level prior is learned primarily from locomotion-centric behavior, new loco-manipulation tasks at the high level must still project back into that execution distribution; joint-level actions outside it do not become reachable simply through stronger perception or reasoning.

Figure 04 uses Locomotion and Manipulation activation as its horizontal coordinates and relative sample density as height, forming a landscape of the two motion distributions. The locomotion-centric distribution remains concentrated around locomotion-heavy behavior, whereas loco-manipulation extends into mixed-activation regions. Their overlap marks the shared motion prior; the non-overlapping regions expose actions required by new tasks but absent from System 0's learned support.

Figure 04. Whole-body locomotion-centric data covers only a limited whole-body execution region, while whole-body loco-manipulation extends into areas where locomotion and manipulation are tightly coupled, exposing the distributional boundary of a frozen low-level prior.

HIERARCHICAL PIPELINESemantic intent is compressed, then handed to a frozen executor

01Perception + LanguageScene · Task

S1Latent PolicyHigh-level Latent

↘Compressed InterfaceToken / Trajectory

S0Whole-Body TrackerFrozen Motion Prior

05Physical ExecutionJoint Target · PD

Information flowThree limits: representation · joint training · action space

These limits answer three distinct questions: whether task-control information survives the intermediate representation, which action representation can be co-trained with future visual dynamics, and which PD targets the final policy can express. Dimensionality alone is not the criterion; the decisive factors are representation semantics, supervision, and the frozen decoder’s action image.

X_tVision + taskH_tState historyZ_tMotion tokenA_tJoint target

KINEMATIC REPRESENTATION & SUPERVISION

Kinematic Representation and Supervision Bottleneck

Task and visual context X_t together with state history H_t must first be compressed into the SONIC motion representation Z_t. Yet Z_t is supervised as a desired motion or reference pose: it answers “what motion should the body exhibit,” not “which joint target should be issued at this control step.” Task-control semantics discarded by this kinematic abstraction cannot be recovered downstream.

REPRESENTATION + SUPERVISION

01 / SUFFICIENCY GAP

R_Z^* = R_X^* + 𝔼 [ ‖ 𝔼[A^* | X, H] − 𝔼[A^* | Z, H] ‖² ] ≥ R_X^*

02 / LOSS GEOMETRY

L_token = δz^T I₆₄ δz ; L_action ≈ δz^T J_h^T J_h δz ; rank(J_h^TJ_h) ≤ 29 < 64

Under Z = f_SONIC(X, H) and squared loss; J_h = ∂D_frozen(z, h) / ∂z

REPRESENTATION BOTTLENECKIf two task contexts collapse to the same Z under the same H but require different A^*, no policy reading only (Z, H) can separate them. The risk gap is precisely task-relevant control information removed by the kinematic representation.

SUPERVISION BOTTLENECKThe issue is not that “64D is too small.” Since J_h ∈ ℝ^29×64, the rank of J_h^TJ_h is at most 29, so it cannot be globally equivalent to the full-rank I₆₄: some token errors barely alter action, while others are amplified by the decoder. Direct-joint places supervision in the final control space.

DYNAMICS GROUNDING UNDER A FROZEN DECODER

Dynamics-Grounding Gap under a Frozen Decoder

Both paths observe X_t and H_t. The key difference is what supervises the hidden representation attended by the future visual branch. In the old path, E_Z,t is explicitly required only to decode a kinematics-level SONIC target; the state-conditioned joint action appears only after the frozen decoder. Direct-joint instead lets future visual tokens attend E_A,t, a representation supervised by the real joint-control target.

WHAT FUTURE VISUAL ATTENDS

FROZEN SONIC PATH

Y_future ← E_Z,t → Z_t ; A_t = D_frozen(Z_t, H_t)

DIRECT-JOINT PATH

Y_future ← E_A,t → A_t → S_t+1

SHARED GRADIENT

∂L_total / ∂E_A,t = ∂L_action / ∂E_A,t + λ_visual ∂L_future / ∂E_A,t

JOINT-TRAINING GAPThe frozen decoder may use proprioceptive feedback, but its state-feedback law lies outside the visual-action model, so ∂L_future / ∂θ_D = 0. Future prediction cannot reshape the mapping from motion goal and current state to joint control.

DYNAMICS GROUNDINGA joint action is not the full dynamics, but together with H_t it is the control variable entering the short-horizon transition. Action loss makes A_t decodable from E_A,t, while future loss learns its consequence from the same representation—more direct grounding than a representation guaranteed only to decode the SONIC target.

FROZEN ACTION-MANIFOLD CONSTRAINT

Frozen Action-Manifold Constraint

For a fixed state history h, a latent-based policy—regardless of upstream model strength—must output an action inside the frozen decoder’s image of valid SONIC latents, M_h. Direct-joint removes this learned decoder constraint: it retains the same 29D PD-target interface without requiring each task action to first lie in M_h.

EXPRESSIVITY AT THE SAME PD INTERFACE

FROZEN DECODER IMAGE

M_h = { D_frozen(z, h) : z ∈ Z_valid } ⊆ ℝ²⁹

UNREACHABLE-TARGET ERROR

infz ∈ Z_valid ‖D_frozen(z, h) − a^*‖² = dist(a^*, M_h)²

DIRECT-JOINT PD TARGET

A_joint = q_default + C_scale A_raw ; det(C_scale) ≠ 0

The fixed affine map is invertible for nonzero scales. The comparison concerns a learned action manifold—not the removal of joint limits, torque saturation, or safety clipping.

STRICT EXPRESSIVITY FLOORIf the task-required action a^* lies outside M_h, the latent path has a positive irreducible action error. Better perception or planning cannot cross the expressivity boundary of a frozen System 0.

FREEDOM AND ITS COSTA 64D → 29D map is not automatically restrictive; the relevant object is the actual image of valid latents. Direct-joint enlarges the hypothesis class within the same PD-target interface, but also relinquishes decoder priors for balance, smoothness, and whole-body coordination—capabilities that must be relearned from data and closed-loop training.

FROM GOAL TO CONTROL

The three limits occur sequentially along the old path: task semantics are filtered by a kinematic representation, state-conditioned action remains outside joint training, and final output is constrained by the decoder image. Direct-joint relaxes these layers through action-level supervision, dynamics-grounded co-training, and a freer joint-space output—gaining reach while assuming responsibility for relearning stable control priors.

02 / METHOD

DIRECT PERCEPTION CONTROL MODEL

The Direct Perception Control Model redesigns the execution path at the action interface, representation coupling, and closed-loop training. It removes the intermediate motion representation entirely: instead of producing a reference for a separate Whole-Body Tracker, one model maps vision, language, body state, action history, and execution feedback directly to executable joint and hand targets. Symbiotic Attention lets perceptual and control representations attend to each other under a shared action objective, so task understanding is constrained by action feasibility while action generation continuously draws on semantic context.

Figure 05. The Direct Perception Control Model maps perception and live body context directly to joint and hand targets without an intermediate motion interface. At its center, Symbiotic Attention couples perceptual and control representations during training, while asynchronous vision updates and continuous proprioceptive feedback sustain closed-loop execution.

For the Low-Level Action Boundary, DriftDistill exposes the Student to drifted states during closed-loop rollouts and uses Frozen Teacher corrections to learn recovery. Offline BC provides initialization; Stage 2 combines online rollout states with offline demonstrations, using repeated correction and distillation to progressively cover the states and recovery paths encountered at deployment. The intermediate interface is removed, perception and control are learned jointly, and the executable distribution expands through closed-loop training.

01 / GROUND

Offline demonstrations

02 / INITIALIZE

Unified Policy

03 / VISIT

Policy-visited states

ON-POLICY RECOVERY LOOPVisit → Correct → AbsorbEach rollout expands the training distribution

05 / DISTILL

L_rec(aₜ, aₜ^*)MINIMIZE

Recovery loss

04 / CORRECT

qₜ→π_T→aₜ^*

Frozen Teacher

Figure 06. DriftDistill first initializes the Student with Offline BC. Stage 2 then jointly trains on drifted states visited by online rollouts and offline demonstrations, with the Frozen Teacher providing recovery targets so the Student can handle state shifts and recovery paths encountered at deployment.

03 / DATA

Boosting Loco-Manipulation Data

Direct physical-action learning requires supervision in the Joint Target Space where execution occurs. Existing human and robot corpora cannot be combined directly because their embodiments, action representations, and control interfaces differ.⁶⁷ We therefore convert every source into G1-executable, time-aligned joint trajectories, then scale this common supervision to 15,010 hours.

Unifying Heterogeneous Motion Evidence

Conversion uses the most reliable motion signal available from each source. Teleoperated humanoid data are standardized across coordinate frames, joint definitions, and control rates. Head-mounted recordings are reconstructed and retargeted from recoverable whole-body motion. Egocentric video is lifted from hand, wrist, and end-effector trajectories into humanoid whole-body motion. All three paths produce the same joint-target supervision.

Figure 07. The heterogeneous data conversion pipeline uses the reliable motion information preserved in each source to produce unified, G1-executable, time-aligned joint trajectories.

Scaling the Unified Corpus

The resulting corpus contains 15,010 hours: 6,781 hours of Human Ego, 4,024 hours of Armed Robot, 3,660 hours of Wheeled Humanoid, and 545 hours of Bipedal Humanoid data. These sources span human and robot embodiments with different mobility and interaction capabilities⁸⁹¹⁰; after conversion, all of them enter one execution space without tying training to a single platform.

Figure 08. The 15,010-hour embodied training corpus spans Human Ego data and multiple robot embodiments, providing a scalable foundation for whole-body loco-manipulation.

04 / PERFORMANCE

Performance

Mobile pick-and-place

The robot approaches the cone, bends and squats to grasp it with one hand, turns, walks two steps, and places it down—coupling grasping, loaded balance, turning, and transport.

Constrained whole-body loco-manipulation

Within a constrained posture and tight clearance, the torso and arms reconfigure while preserving stable support and precise object contact.

Hand–eye–foot coordination

The right foot controls the accelerator while the hands steer near the corner exit, combining visual timing, precise throttle control, and driving semantics.

05 / OUR VISION

Making Intelligence Physically Executable

Scaling perception has helped robots read the world and understand tasks. True Whole-Body Intelligence must also understand the task objective, bodily state, dynamic constraints, and the consequences of action. Direct Perception Control aims to shorten this Embodiment Gap. We are not training a stronger Motion Planner; we are training a system that learns continuously from understanding the world to using its own body to complete a task.

The next frontier of robot intelligence is not only better perception or better planning. It is learning how intelligence becomes physically executable action.

Citation

@article{
  symbiosis2026directperceptioncontrol,
  author = {Symbiosis Robotics Team},
  title = {Direct Perception Control Model},
  journal = {Symbiosis Robotics Blog},
  month = {August},
  year = {2026},
  url = {https://symbiosis-robotics.com/research/dpc},
}