Skip to main content

import UrduTranslateButton from '@site/src/components/UrduTranslateButton';

Preface

Why Physical AI Matters Now​

We are witnessing a pivotal moment in robotics. For decades, robots have been confined to structured environments—factory floors with precise positioning, warehouses with known layouts, surgical suites with controlled conditions. But the dream of robots operating in unstructured, human-centric environments—homes, offices, outdoor spaces—has remained largely out of reach.

That's changing. Fast.

Three technological convergences are making physical AI a reality:

  1. Robotic Middleware (ROS 2): Mature, real-time capable frameworks for distributed robot control
  2. GPU-Accelerated AI: Deep learning models running at 60+ FPS on edge devices
  3. Foundation Models: Vision-language models (GPT-4 Vision, RT-2) that transfer web-scale knowledge to physical tasks

This textbook exists because these technologies, when combined, enable robots to:

  • Perceive their environment in real-time with superhuman accuracy
  • Understand natural language instructions
  • Reason about complex tasks using world knowledge
  • Act safely and efficiently in unstructured spaces

Who I Wrote This For​

The Problem I'm Solving​

When I started teaching physical AI and humanoid robotics, I couldn't find a single textbook that:

  • Covered the entire stack from ROS 2 to vision-language-action models
  • Used modern tools (ROS 2 Humble, Isaac Sim, OpenVLA—not outdated frameworks)
  • Provided reproducible examples (not just theory or pseudocode)
  • Addressed real-world deployment (edge devices, latency, sim-to-real gap)

Students were piecing together knowledge from scattered tutorials, outdated textbooks, and research papers. They'd master ROS 1 (when the industry moved to ROS 2), learn Gazebo Classic (when Gazebo Harmonic was released), or miss entirely the revolution happening with VLA models.

This textbook solves that problem.

My Assumptions About You​

I assume you:

  • Can program in Python or C++ (not an expert, but comfortable)
  • Understand basic calculus and linear algebra (kinematics, transforms)
  • Know Linux command-line basics (cd, ls, apt install)
  • Are motivated to learn by building (not just reading theory)

I don't assume you:

  • Have prior ROS experience (we start from scratch)
  • Know deep learning (we explain DNNs, TensorRT, training)
  • Own a humanoid robot (simulations work fine)
  • Have a PhD in robotics (this is accessible to MS students and engineers)

How This Book Is Different​

1. End-to-End Systems, Not Isolated Topics​

Most textbooks teach perception or control or planning. This book teaches how to integrate them into working systems. By Week 13, you build a robot that:

  • Listens to voice commands ("Pick up the red cup")
  • Plans tasks with an LLM (GPT-4 breaks task into steps)
  • Perceives with GPU-accelerated vision (Isaac ROS detects objects)
  • Executes actions (ROS 2 controls the robot)

2. Modern Stack (2024-2025)​

We use:

  • ROS 2 Humble (LTS, released 2022)—not ROS 1 Noetic
  • Gazebo Harmonic or Isaac Sim—not Gazebo Classic
  • RT-2, OpenVLA—not outdated imitation learning methods
  • TensorRT, Isaac ROS—not CPU-only inference

If you learn these tools, you're ready for industry and cutting-edge research today.

3. Hands-On, Test-Driven Learning​

Every chapter has:

  • Verified code examples (tested on Ubuntu 22.04, ROS 2 Humble)
  • Step-by-step tutorials (with screenshots and expected outputs)
  • Exercises (easy → medium → hard → challenge)
  • Troubleshooting sections (common errors and fixes)

You learn by doing, not just reading.

4. Research-Backed, Industry-Relevant​

This book bridges research and practice:

  • Research: We cover RT-1/RT-2 (Google DeepMind), ORB-SLAM3, Isaac Gym (NVIDIA)
  • Industry: We address deployment (Jetson edge devices), cloud GPUs, latency, costs

You'll understand why technologies exist (research motivation) and how to use them (practical deployment).

How to Use This Book​

For Students (13-Week Course)​

Follow the course sequentially:

  • Weeks 1-4: Master ROS 2 (Module 1)
  • Weeks 5-8: Build digital twins (Module 2)
  • Weeks 9-11: Implement AI perception (Module 3)
  • Weeks 12-13: Integrate VLA models (Module 4)

Do every lab. Robotics is learned through practice, not passive reading.

For Self-Learners​

You have flexibility:

  • Fast Track (4 weeks): Read selectively, focus on code examples, skip exercises
  • Standard Track (3 months): Follow 13-week course at your own pace
  • Deep Dive (6 months): Do all exercises, extensions, challenge problems

For Instructors​

This textbook is designed for a graduate-level course (MS/PhD):

  • Lectures: 2 hours/week (theory, concepts, examples)
  • Labs: 1 hour/week (hands-on implementation)
  • Assessments: Weekly labs (40%), midterm (20%), final project (40%)

Instructor resources (solutions manual, additional exercises, grading rubrics) are available separately.

For Researchers​

Use as a reference:

  • Module 1: Quickly spin up ROS 2 experiments
  • Module 2: Prototype in simulation before deploying to hardware
  • Module 3: Leverage GPU acceleration for perception
  • Module 4: Integrate latest VLA models into your research

What You'll Build​

By Week 4 (Module 1 Complete)​

  • Multi-node ROS 2 system with publishers, subscribers, services, actions
  • Custom ROS 2 package (C++ and Python)
  • Launch files for configurable robot systems

By Week 8 (Module 2 Complete)​

  • Humanoid robot simulated in Gazebo or Unity
  • Digital twin synchronized with ROS 2
  • Visual SLAM for autonomous navigation
  • Sim-to-real transfer techniques

By Week 11 (Module 3 Complete)​

  • GPU-accelerated object detection (YOLO with TensorRT)
  • Autonomous navigation with Isaac ROS + Nav2
  • Motion planning with cuMotion
  • (Optional) RL policy trained in Isaac Gym

By Week 13 (Module 4 Complete)​

  • Final Project: Complete VLA system
    • Voice input ("Pick up the red cup and place it on the table")
    • Whisper transcription → GPT-4 task planning
    • Isaac perception → object detection and pose estimation
    • ROS 2 motion control → execution
    • Unity/Gazebo visualization

This is a production-ready system, not a toy demo.

Technical Philosophy​

Accuracy Over Simplicity​

I don't "dumb down" content. Robotics is complex. I explain concepts clearly, but I don't hide the complexity. You'll learn about DDS middleware, QoS policies, TensorRT optimization, and domain randomization—because you need them for real systems.

Open Source, Open Access​

This textbook relies on open-source tools:

  • ROS 2 (Apache 2.0)
  • Gazebo (Apache 2.0)
  • Isaac Sim (free for individuals)
  • OpenVLA (Apache 2.0)

Commercial alternatives (Isaac SDK, Unity Pro) are discussed but not required.

Reproducibility as a Core Value​

Every code example includes:

  • Full source code (no snippets with "...")
  • System requirements (Ubuntu version, ROS version, dependencies)
  • Installation commands (tested, step-by-step)
  • Expected outputs (screenshots, terminal outputs, behaviors)

If something doesn't work, it's a bug in the textbook—not your fault.

Acknowledgments​

This textbook wouldn't exist without:

The ROS Community: For building the software infrastructure that powers modern robotics. Special thanks to Open Robotics, Geoffrey Biggs, and the thousands of contributors to ROS 2.

NVIDIA's Isaac Team: For making GPU-accelerated robotics accessible. Isaac Sim, Isaac ROS, and Isaac Gym are transformative technologies.

Google DeepMind Robotics: For pioneering vision-language-action models (RT-1, RT-2) and showing that foundation models can control robots.

Open-Source Contributors: Especially the developers of Gazebo, Unity Robotics Hub, ORB-SLAM3, Whisper, and countless other tools used in this textbook.

Early Reviewers and Students: Whose feedback shaped this textbook's structure, examples, and exercises.

A Note on AI-Generated Content​

This textbook was written with assistance from AI tools (Claude Code, Spec-Kit Plus) to:

  • Generate boilerplate code examples
  • Format technical documentation
  • Suggest exercise ideas
  • Check consistency

However, all technical content was:

  • Verified against official documentation
  • Tested on real systems (Ubuntu 22.04, ROS 2 Humble)
  • Reviewed by human experts
  • Curated for accuracy and pedagogy

AI accelerated writing, but humans ensured correctness.

Continuous Improvement​

Robotics evolves fast. This textbook will be updated regularly:

  • Bug fixes: Corrections for errors (see errata)
  • Software updates: New ROS 2 distributions, Isaac versions
  • New research: Latest VLA models, SLAM algorithms

Feedback is welcome: If you find errors, have suggestions, or want to contribute, please reach out via the textbook's GitHub repository.

Let's Begin​

Robotics is one of humanity's grand challenges. We're building machines that interact with the physical world—not just virtual bits and bytes, but atoms, forces, sensors, and uncertainty.

It's hard. It's rewarding. And you're about to learn how.

Welcome to the journey.

Ready? Proceed to How to Use This Book or jump straight to Module 1.

Written with passion for robotics, respect for learners, and hope for the future of physical AI.

— The Authors, December 2025