How AI Learns to Understand Walking

Watch someone walk across a room. One foot accepts weight while the other advances. The body moves over the supporting leg, the trailing foot pushes off, and the roles reverse. What looks effortless is a changing relationship between joint motion, balance, and contact with the ground.

Now imagine trying to reconstruct that movement using a few small sensors on the body and pressure-sensing insoles. Could a model recover the pose of the whole person? Could it recover how each foot loads the ground throughout a stride? Could the same learned representation work for someone taller, a different device, or a movement it has never encountered?

These questions connect human pose estimation to a larger ambition: a human motion foundation model, or HMFM. The aim is a reusable model of movement that learns across people, tasks, and measurement systems, then adapts to specific reconstruction problems.

Understanding a walk brings these ideas together: reconstructing the body, tracking its path, and estimating its interaction with the ground. Research in graphics, computer vision, biomechanics, and robotics offers complementary pieces of this puzzle.

Two ways to read: follow the illustrations and opening paragraphs for the conceptual story. Open the technical boxes for equations and network details. The paper map near the end provides a route into the original research.

Explore: What we reconstruct · What sensors measure · How pose networks work · Forces and pressure · Foundation models · Putting the pieces together · How to evaluate · Reading map

1. Movement has both geometry and mechanics

A pose describes the arrangement of body parts at one instant. Motion adds change over time. Kinematics describes that movement: joint rotations, positions, velocities, and accelerations. Kinetics concerns the forces and moments associated with it.

During walking, a knee-angle curve tells us how the knee moves; a ground-reaction force tells us how the foot loads the ground. Carrying a bag, changing speed, or walking uphill can alter the mechanical demands. Recovering a convincing walking animation does not, by itself, establish accurate loading.

In ordinary walking, support alternates between one foot and brief periods with both feet on the ground. The model must follow that transition without confusing a moving foot with a planted one. This is why contact information matters to both trajectory reconstruction and force estimation.

A full-body estimator usually needs two levels of geometry. The root locates and orients the body in the world. Local joint rotations describe how one segment turns relative to its parent. A person can have the same relative joint configuration while facing another direction or standing elsewhere in the room.

A body model connects these rotations to positions. It also introduces assumptions about segment lengths and joint definitions. That is why an error measured on one skeleton is not automatically comparable to the same-looking number on another.

2. Sensors observe fragments of the answer

An inertial measurement unit, or IMU, usually contains an accelerometer and gyroscope. The gyroscope measures angular velocity. The accelerometer measures specific force: its reading depends on both the sensor's acceleration and gravity expressed in the sensor's coordinates. Some devices also provide a magnetometer or a fused orientation estimate.

A wrist sensor does not directly report the elbow angle. A foot sensor does not directly measure the body's position in a room. Pressure insoles add information about loading under the foot, but their cells do not directly measure every component of a ground-reaction wrench.

There are also mundane but consequential issues: a strap rotates, one device has a delayed clock, a sensor saturates, or a packet disappears. If a model interprets these as human movement, a sophisticated network can make a confidently wrong prediction.

Sparse sensing makes the inverse problem ambiguous. Several elbow or torso configurations can produce similar measurements at the observed locations. Learning helps because human movements have structure: joints do not move independently, and the recent past provides clues about the present.

That structure is a prior—knowledge of what movements are likely. A prior can resolve ambiguity, but it can also favor a familiar movement when the person is doing something unusual.

3. How a model learns to reconstruct a walk

Consider a short window of synchronized measurements. An encoder turns each available sensor stream into numerical features. Fusion combines information across sensors; a temporal model relates successive frames. An output head converts the resulting representation into a particular quantity, such as joint rotations or a foot's force. Different papers divide these steps differently.

During supervised training, predictions are compared with reference targets through a loss function. An optimizer adjusts the network's weights to reduce that loss over training examples. During inference, those learned weights process a new recording using the declared inputs and initialization. During evaluation, reference targets score the predictions. Those references may come from direct measurements or fitted body models, and may not be independent of the inputs. A deployable system must obtain every required input without access to evaluation-only references.

The literature suggests several ways to make that reconstruction easier.

Deep Inertial Poser (DIP, SIGGRAPH Asia 2018) established a useful temporal approach: a bidirectional recurrent network learns pose from six IMUs, with synthetic sensor training and real-data fine-tuning. Context helps distinguish movements that are difficult to identify from one frame. Its online sliding-window mode still uses future context; “online” does not mean zero latency. It also does not directly solve global translation. Huang et al., DIP.

TransPose (SIGGRAPH 2021) divides reconstruction into stages. It first estimates positions of sensor-bearing extremities, then completes the joint positions, and then estimates rotations. A separate translation path combines learned velocity with an inferred supporting foot. The intuition is simple: a foot believed to be planted can temporarily anchor the body. A sliding foot or incorrect contact prediction weakens that anchor. Yi et al., TransPose.

DynaIP (CVPR 2024) adds intermediate velocity prediction and divides learning across body regions. It also tackles an important data problem: motion datasets use different skeleton conventions. Its additional-data experiments include negative transfer: improving coverage of one source can worsen performance on another. This motivates matched source-addition tests. Zhang et al., DynaIP.

Fast Inertial Poser (FIP, Nature Communications 2024) explicitly uses body measurements. Shared recurrent processing estimates sensor-bearing segment displacement, body-region regressors complete positions, and a learned inverse-kinematics stage predicts rotations. This connects anatomy to computation: the same angular movement produces different linear motion at the end of a longer limb. For walking, its output scope matters: ankle/toe rotations and intrinsic root-translation recovery are excluded. Speed and overall pose accuracy therefore do not establish complete gait reconstruction. Xiao et al., FIP.

The shared lesson is to expose useful structure: time, anatomical geometry, or an easier intermediate target. Their mixed-activity benchmarks do not establish walking-specific accuracy for every joint or clinical population; those questions need their own evaluation.

4. Physics and calibration change what the signals mean

A pose network can produce a smooth sequence whose feet slide or penetrate the ground. Physical Inertial Poser (PIP, CVPR 2022) combines neural predictions with a dynamics optimization layer. Mechanics helps select motions a simulated body can plausibly execute. PIP assumes flat ground, which limits transfer to stairs and uneven terrain. Its simulated contact forces support motion reconstruction; they are not independent measurements of human kinetics. Yi et al., PIP.

There is an earlier issue, even before optimization: which frame is moving? Imagine describing a passenger from inside a turning bus. The bus-relative description needs additional acceleration terms. Physical Non-inertial Poser (PNP, SIGGRAPH 2024) studies this problem in a body-relative frame. Its treatment motivates careful motion-coordinate modelling, while its sensor-synthesis pipeline remains a separate training choice. Yi et al., PNP.

Transformer IMU Calibrator (TIC, SIGGRAPH 2025) addresses mounting offsets and orientation drift using temporal sensor windows and a motion-diversity trigger. Its correction is defined in an ego-yaw frame; it does not solve global heading drift. Calibration needs informative movement: insufficient orientation diversity leaves some errors hard to identify. Device changes therefore require testing the calibration stage as well as the pose network. Zuo et al., TIC.

5. A correct pose does not uniquely determine the forces

During a walking stride, the centre of mass accelerates, support transfers between feet, and each foot moves between contact and swing. The total ground force therefore changes over time; it is not a constant body-weight number that we merely divide in two.

Even with a good estimate of whole-body motion, identifying each foot's contribution remains a separate problem. Double support is particularly informative: both feet touch the ground, and a total force alone does not specify their individual forces, centres of pressure, or moments. Pressure observations can reduce this ambiguity, while the remaining wrench components still require modelling and validation.

An external wrench contains three force components and three moment components. For each foot, that means more than a single vertical-force curve. Shear forces act along the ground; moments describe rotational loading around a stated origin.

Early deep-learning work on multidimensional ground-reaction forces and moments by Johnson et al. (IEEE TBME 2021) demonstrates richer kinetics prediction from accelerations, principally in running and sidestepping. It also exposes a deployment detail worth looking for in every paper: stance segmentation used laboratory force events. A wearable model that needs those events in preprocessing has a dependency beyond its advertised sensor channels. The study also separates force and moment performance rather than treating them as interchangeable. Paper.

What pressure adds to the reconstruction

An insole observes how normal load is distributed over its cells. With suitable calibration and cell areas, pressure can estimate normal load and a centre of pressure. It does not automatically reveal shear or every rotational component.

GRIP (CVPR 2026) combines wrist/foot IMUs and pressure with two stages: KinematicsNet estimates movement, while DynamicsNet controls a simulated humanoid using state differences. It offers a concrete example of combining observation and physical realization. Its results also show that improvement in one global-motion measure need not mean improvement in every pose or force measure. Its terrain map and reference-based initialization are additional assumptions to account for in wearable-only comparisons. Hori et al., GRIP.

Scherpereel et al. (IEEE TBME 2024) test added EMG and simulated insoles for joint-moment estimation using motion-capture kinematics and subject-dependent models. Benefits depend on task: common hip-moment tasks, including walking, did not show significant gains from the added modalities in that study. It motivates task-specific comparisons, rather than a blanket claim that pressure improves walking estimates. Paper.

Biological joint moments and ground-reaction moments are different quantities. The former describe loading at joints; the latter belong to external contact wrenches. They require different reference quantities and evaluation.

6. What would make a motion model a foundation model?

A model does not become foundational merely because it is large or trained on several datasets. The useful ambition is a representation that transfers: something learned in one setting should help another task or condition without rebuilding everything from the beginning.

For human movement, the transferable structure could include temporal coordination, relationships between body segments, contact patterns, and the connection between movement and external loads. Three research directions are especially useful here.

Learn from recordings without complete labels

In masked self-supervised learning, we hide part of a signal and ask the network to reconstruct it. The model has to learn relationships across time and channels before it receives the downstream target labels.

Tan et al. (IEEE TBME 2024) use masked inertial reconstruction before force estimation. Their transformer uses 128-sample, 48-channel windows and six attention blocks. Results support improved data efficiency in the studied walking settings, with less uniform benefit for landing. This evidence concerns right-leg three-axis force with contact-phase scoring, not continuous bilateral wrenches. This is a concrete reason to investigate pretraining, not a guarantee that any unlabelled recording will help any kinetic task. Paper.

Tell the model where each sensor is

IMUCoCo (UIST 2025) learns placement-conditioned representations using body-surface coordinates. Its sensor features are routed into a joint representation for downstream tasks. This is more specific than saying that a transformer will automatically understand any arrangement of sensors. Its experiments support flexible placement, while its fixed-six-IMU TotalCapture comparison favors PNP on global angular error. Flexibility and accuracy are separate questions. Zhou et al., IMUCoCo.

For a wearable foundation model, a sensor vocabulary might contain many possible locations while a particular recording supplies only a few. The vocabulary describes possible inputs; it does not mean every user must wear every sensor. Missing sensors still remove information, however. A mask tells the model what is absent—it cannot recreate the observation.

Learn a shared distribution of movement and loading

GaitDynamics (Nature Biomedical Engineering 2026) is a particularly relevant example. A diffusion model learns joint kinematics together with ground-reaction quantities, supports different conditioning patterns, and uses a separate force-refinement network. Its evaluation includes held-out people and studies. Crucially, it is conditioned on kinematic representations, not arbitrary raw IMU streams. Its reference skeleton omits arms, and its walking/running focus limits transfer claims to other movements or clinical populations. Tan et al., GaitDynamics.

These examples teach different forms of reuse: transfer from unlabelled signals, transfer across sensor placement, and transfer across conditioning tasks. A strong HMFM needs to state which forms it actually demonstrates.

7. The hardest data problem is making the labels mean the same thing

One dataset measures inertial signals and estimates pose through an inertial suit. Another fits an articulated body model to optical markers. A third provides force-plate measurements but no native pressure cells. These can all be useful, but their target fields are not interchangeable.

The AddBiomechanics Dataset (ECCV 2024) illustrates the value of harmonization. Its first version combines optical and force-plate studies using standardized biomechanical representations and derived dynamics. That makes cross-study learning more practical, while preserving a distinction between raw measurements, fitted states, and inverse-dynamics quantities. It does not turn a motion-only recording into measured IMU data. Werling et al., dataset paper.

Source compatibility has three parts: what was physically measured, how targets were constructed, and which recordings share an underlying person or trial. Units, coordinate frames, clocks, calibration, and licences must remain traceable.

A concrete example comes from GaitDynamics: its authors avoid force-based tuning of kinematics so that force targets do not leak into kinematic inputs. A field called “pose” is not automatically an independent input for evaluating force prediction.

Splits must happen before windows. Fit population-level normalizers on training data, and evaluate with only deployment-available calibration. If overlapping pieces of one recording enter both training and testing, a model can appear to generalize while seeing almost the same event twice. The PRISM author documentation explicitly describes a chunk-level split that can share takes across roles and synthesized wrist data in some training chunks. A chunk-level split therefore answers a different question from testing on entirely unseen people or recording sessions. Measured and synthesized channels also need to remain distinguishable.

Synthetic sensor data offer another route to training: motion recordings can be converted into virtual inertial measurements. DIP and several later methods use this approach. Its usefulness depends on how well the generated signals represent real sensor noise, mounting, and motion; evaluation on independently recorded signals is essential.

8. Putting the pieces together: a wearable motion model

A flexible model needs to know both what a sensor measures and where it is attached. Training recordings can have different sensor layouts, provided their signals, anatomical locations, timing, and calibration are represented consistently. A pelvis sensor and an upper-back sensor remain different measurements.

Consider an illustrative configuration with eight IMUs and pressure-sensing insoles. Body measurements define the geometry; calibration relates each sensor to its segment. The model could estimate root translation and orientation, local joint rotations, and a six-component ground-reaction wrench for each foot. These are different prediction tasks sharing a representation of the same movement.

Contact and gait events can then be estimated from the reconstructed motion and loading. Joint moments and powers require additional biomechanical calculations and assumptions, rather than simply reading another component of the ground-reaction wrench. All wrench outputs need declared axes and moment origins; for example, a common world frame and a trial-fixed origin allow consistent comparison over time.

One possible architecture uses a shared IMU encoder with location and side information, a separate pressure encoder, fusion across available sensors, and a temporal model. Separate output heads reconstruct motion and contact wrenches. This combines the earlier ideas into a design pattern; no single cited paper establishes that this combination is optimal.

Two choices determine how such a network can be used. A causal temporal model uses only past and present measurements; a model that uses future frames may provide additional context but introduces delay or requires offline processing. Similarly, sharing features between pose and force estimation can be useful, but an improvement in one output does not guarantee an improvement in the other. Comparing otherwise matched models reveals which components actually help.

9. Uncertainty and evaluation determine whether the result deserves trust

The question is not only whether a model produces an answer. It is whether we understand where that answer is reliable.

ProbIP (ICCV 2025) models rotations with matrix-Fisher distributions and a Mamba-based sequence architecture. It illustrates a useful shift from one pose estimate to a distribution over plausible orientations. The concentration of a predicted distribution still needs empirical checking: a narrow distribution can be confidently wrong. Kim et al., ProbIP.

A useful evaluation report answers at least these questions:

• Pose: which joints, which local or global rotations, and which reference body model?
• Trajectory: is root translation measured directly, aligned away, or evaluated for accumulated drift?
• Kinetics: which foot, force axis, moment axis, origin, and normalization? Report whole-stride performance alongside stance/swing breakdowns; contact-only scores answer a narrower question.
• Generalization: were participants, sessions, activities, or sites truly held out from the complete training and selection process?
• Robustness: what happens after sensor movement, dropout, clock error, or a device change?
• Uncertainty: are confidence claims calibrated on the population where they will be used?

Correlation is not enough: a prediction twice the true force can have perfect correlation while having the wrong scale. Similarly, an aligned pose score can remove a translation error that matters in an actual application.

The official supplement to task-agnostic exoskeleton control (Nature 2024) gives a concrete counterexample: on an unfamiliar layup task, predicted joint moments can track the reference curve while getting its amplitude wrong. Although the application is wearable assistance, the measurement lesson applies more broadly: correlation and magnitude error answer different questions. Molinaro et al..

10. What a useful walking model must deliver

A useful model follows the body through space and reconstructs each foot's changing interaction with the ground. Temporal structure, anatomy, contact observations, and shared representations offer complementary ways to do that. Which combination works best must be decided through matched experiments.

The promise of a human motion foundation model is that these learned relationships become reusable: across people, sensor arrangements, and prediction tasks. Realizing that promise means more than producing a convincing animation. It means recovering movement and loading accurately, recognizing uncertainty, and remaining useful when the person or recording conditions change.

Reading map

These papers offer complementary entry points into sparse sensing, motion reconstruction, biomechanics, and transferable representations. The table pairs each central idea with a condition that matters when applying it.

Visual credits. The hero and original explanatory illustrations were generated with AI. They depict concepts, not experimental results or measured pressure maps. Reproduced paper figures retain their source, version, and Creative Commons credits beside each image. Click figures to inspect their full resolution.