Chapter 10 — A Real Brain on a Real Box

The mock proved the plumbing (Chapter 9). Now we put a real 3-billion-parameter humanoid model on a real GPU box and drive the Unity G1 with it — and meet the first thing the mock could not warn us about. It assumes you know the embodiment idea, the observation contents, and URDF joint order.

Swapping the mock for a mind

On paper, going from mock to real is exactly the swap Chapter 5 promised. The real model — NVIDIA’s GR00T N1.7, a humanoid foundation model — runs on the owned GPU box (the DGX Spark) reached over a private network. In the Unity editor, you flip the endpoint to ws://chaotic-spark:8766, press Play, and the same humanoid scene that drove the mock now drives the real model. No Unity code changes. That part genuinely is a flag.

But between “press Play” and “the robot moves correctly” sits the friction the mock hid. The mock accepted whatever observation Unity sent and returned a plausible chunk. The real model is far pickier about what an observation is — and here the first surprise lands.

The observation is not a flat array

The mock treated the observation’s state as a flat list of joint angles. A real GR00T embodiment refuses that outright. Its observation is a nested structure, organized by what kind of information each piece is, keyed by an embodiment profile:

{
  "video":    { "ego_view": "<a stack of recent camera frames>" },
  "state":    { "left_arm": [...], "right_arm": [...], "waist": [...],
                "left_wrist_eef_9d": [...], "right_wrist_eef_9d": [...], ... },
  "language": { "annotation.human.task_description": [["push the objects off the table"]] }
}

Every real GR00T embodiment expects this shape: a video group, a state group split into named body parts, and a language group. The old, simpler idea — hand the model one flat array called state — does not exist for any real N1.7 embodiment. It was tried; nothing accepted it.

Insight: the embodiment profile is where the abstraction stops being free. Up to here, “send the observation” was a clean, uniform act. The profile is where the specific model’s specific training shows through: this model was trained on cameras named ego_view, on a state split into left_arm / right_arm / waist / wrist-pose parts, with the instruction under a key called annotation.human.task_description. Get any name or any dimension wrong and the model receives garbage in a slot it trusts. This is the first real friction point of the whole project, and it is unavoidable — the model’s input shape is a fact about how it was trained, not a choice you get to make.

The job of the humanoid bridge (the Python groot_server.py) is precisely to translate: Unity sends its simple flat observation, and the bridge repackages it into the nested, profile-keyed structure the model demands, then forwards it to the actual model process. On the Unity side, a single flag — useStructuredObs = ON — tells the scene to build and expect the structured form, and a small builder assembles the 49-number state in the exact block order the profile wants.

What REAL_G1 wants

The specific profile for the base-checkpoint G1 is called real_g1, and its state is a 49-number vector with a very particular layout:

Slot	Part	Size	Meaning
0–8	`left_wrist_eef_9d`	9	left wrist pose: position (3) + orientation (6)
9–17	`right_wrist_eef_9d`	9	right wrist pose
18–24	`left_hand`	7	left hand joints (zeros — the plain G1 has no fingers)
25–31	`right_hand`	7	right hand joints (zeros)
32–38	`left_arm`	7	left arm joint angles
39–45	`right_arm`	7	right arm joint angles
46–48	`waist`	3	waist joints

Two things about this layout are worth noticing. First, it carries hand slots even though the plain G1 has no hands — they are filled with zeros. That is not waste; it is what lets the same profile drive a G1 that does have hands (the Dex3-equipped version) by simply filling those slots. Second, the state includes wrist poses in end-effector form (the “9d” — three for position, six for a compact orientation encoding), not just joint angles. The model wants to know where the wrists are in space, not only how the arm is bent.

The action that comes back is 53 numbers — the same body parts plus a couple of whole-body command channels — and Unity’s mapping code writes the arm and waist targets into exactly the joint slots Chapter 7 laid out (waist at 12, left arm at 15, right arm at 22). One calibrated detail, learned from the first live rollout: the model returns absolute joint targets, not deltas, so Unity applies them directly (the flag relativeArms is left OFF).

The gotchas that greet you

Two sharp edges show up the moment you send a real observation, and both are the profile being unforgiving:

The video needs at least two frames. The video slot is not a single image but a short stack of the most recent frames — the model reasons about motion, so it wants a small window of time. The bridge keeps a rolling buffer of recent frames and, at the very start when only one exists, pads the window by repeating the oldest. Send a single frame where the model expects a window and it errors.
Zeros in a wrist-pose slot crash the model. The wrist-pose “9d” encoding includes an orientation, and the model converts that orientation into a rotation matrix internally. If the slot is all zeros — a degenerate, meaningless orientation — that conversion fails with a cryptic “SVD did not converge.” The bridge guards against it by seeding any all-zero wrist-pose slot with an identity orientation ([0,0,0, 1,0,0, 0,1,0]), which is meaningless-but-valid and keeps the math happy.

Neither of these is a bug in the model; both are the model telling you, tersely, that you handed it something outside what it was trained to accept. The bridge exists to absorb exactly this kind of friction so Unity never has to know about it.

Zero-shot: reasonable, not good

So you align the profile, seed the wrist poses, feed a proper video window, and the real GR00T drives the Unity G1’s arms in Play mode — a genuine end-to-end loop, verified. And the honest result is exactly what Chapter 4 warned: the motion is reasonable, not good. The arms move in the right spirit for the instruction, but this is a generalist that has never seen your exact table, your exact camera, your exact task. It is a plausible first draft, not a finished skill.

That is not a disappointment; it is the whole reason the second half of the book exists. The base model gets you a working loop and a believable starting point. Turning “reasonable” into “good at my task” is what your demonstrations and fine-tuning are for — and that is Parts V and VI.

Insight: a working loop with mediocre behavior is a milestone, not a failure. It is easy to look at a hesitant zero-shot rollout and conclude something is broken. Nothing is. Every seam is proven, the real model is running, and the robot is doing a rough version of the right thing. That is the launch pad. From here, improvement comes not from a better download but from your data — which is only collectable because the loop already works.

What you now understand

Going from mock to real GR00T is, in Unity, a change of address — but the real model’s observation is a nested structure keyed by an embodiment profile (video / state-by-body-part / language), not a flat array. The bridge translates Unity’s simple observation into that structure.
The profile is the first real friction point: the model’s input names and dimensions are facts about its training. The real_g1 profile wants a 49-number state (wrist poses, hand slots — zeros for the plain G1, arms, waist) and returns a 53-number absolute-target action.
Two gotchas greet a real observation: the video slot needs a ≥2-frame window (padded at the start), and all-zero wrist-pose slots crash the model (“SVD did not converge”) unless seeded with an identity orientation.
Zero-shot behavior is reasonable, not good — a working loop with a mediocre policy, which is a milestone. Improving it is what the rest of the book is for.

You now have a real model driving a real robot. The next chapter is what happened when we served a fine-tuned model and one number in the reply was wrong — the most instructive failure in the book.

Continue to Chapter 11 — The Cadence Bug.

The real humanoid path is groot_server.py --backend zmq --embodiment real_g1, which wraps NVIDIA’s Isaac-GR00T PolicyServer (GR00T is not pip-installable and runs in its own environment on the GPU box). The nested-observation builder and all six embodiment profiles live in groot_server.py; the Unity structured-observation builder is GrootG1ObsBuilder.cs. The real_g1 profile is the only one that runs zero-shot on the base GR00T-N1.7-3B checkpoint.