Teaching a Robot to Take Instructions — a zero-background workbook

Show a robot a camera image and a sentence — push the objects off the table, pick up the red block — and it answers with motion. Nobody wrote the joint angles. The instruction is plain English, the picture is whatever the robot can see, and the reply is a stream of numbers that drive its arms. That is a vision-language-action model, and this book is about wiring one into a simulated robot you can drive on a laptop — then teaching it your way, by hand, until your own demonstrations become a policy that drives the robot back.

This is a twenty-chapter workbook that takes you through that whole loop from the beginning, with zero background in robotics or machine learning assumed. Every term is defined the first time it appears. Every design decision is shown as a real message on a real wire. The failures stay failures — the chapter where a robot’s motion ran ten times too fast teaches as much as any success.

One sentence runs underneath all twenty chapters, and by the end it will feel obvious:

The model is the easy part. The pipeline is the product — and one shared action format is what makes the pipeline possible. A mock, a human, and a billion-parameter model all speak it, so every way of driving the robot produces the same training data, and every demonstration is one conversion away from a served policy.

Start here

Chapter 1 — The Big Picture is the on-ramp. Read the chapters in order: each one builds on the concepts the previous one introduced, and the series is designed as one cohesive arc, not a collection of standalone notes.

The eight parts

Part I — What this is

Chapters 1–2

The two questions a newcomer brings before anything else can land: what kind of project is this, and what are the dozen words you need before you can read further?

Start: Chapter 1 — The Big Picture

Part II — The two halves and the wire

Chapters 3–5

The system is a body and a brain with a wire between them. Three chapters on each: Unity as the simulated body you can crash for free, the vision-language-action model as the brain that turns a picture and a sentence into motion, and the WebSocket protocol that lets the two talk — the same protocol whether the brain is a scripted mock or a real model on a GPU.

Start: Chapter 3 — Unity as the Body

Part III — Speaking the robot’s language

Chapters 6–8

The unglamorous conventions where sim2real actually lives: why “left” means something different to the robot and to Unity, why the order of a robot’s joints is load-bearing, and the difference between telling an arm to move a little and handing a humanoid forty poses at once.

Start: Chapter 6 — Left, Right, and Why They Disagree

Part IV — Driving the robot

Chapters 9–12

The system running for real: prove the entire loop on a laptop with a scripted mock, then put a real model on a real GPU box and watch a humanoid move — then the honest failure where the motion played ten times too fast, and the tool that lets you see the robot think before you trust it.

Start: Chapter 9 — First Light

Part V — Teaching it your way

Chapters 13–14

The heart of the thesis. One recorder captures a demonstration and does not care who — or what — is driving: a policy, your keyboard, or your own hands tracked through a webcam. Instead of writing the robot’s behavior, you show it, and the recording comes out in exactly the format a model trains on.

Start: Chapter 13 — The Recorder That Doesn’t Care Who’s Driving

Part VI — Closing the loop

Chapters 15–16

Take the demonstrations you recorded, convert them into a training dataset, and fine-tune the model on your own data — then serve the result back to the same Unity robot. The first fine-tune in this project, run end to end on an owned GPU box, with the honest number for how well it worked.

Start: Chapter 15 — From Demonstrations to a Dataset

Part VII — Beyond imitation

Chapters 17–19

Three tracks that surround the humanoid spine: a second and third VLA model for robot arms and other bodies, a reinforcement-learning track for the one thing imitation cannot buy you — balance — and a classical motion planner that is a reminder that not everything should be learned.

Start: Chapter 17 — A Second Opinion

Part VIII — Synthesis

Chapter 20

The capstone names the single principle every previous chapter was circling and shows why the pipeline, not the model, is the thing you actually build.

Start: Chapter 20 — The Pipeline Is the Product

Appendix

A Chaotic Curiosity project. The servers, the Unity project, the teleoperation prototypes, and the training scripts behind this book all live in a private repository (available on request).