agentleFS
mirzaaghazadehmirzaaghazadeh/unity-meta-quest-skills/skills/community/xr-perceptual-constraints-displays-and-tracking/SKILL.md

xr-perceptual-constraints-displays-and-tracking

Skill1 starsChanged 32 days agoMITView on GitHub

Use when reasoning about why an XR experience feels wrong: motion-to-photon latency and its budget, cybersickness and what actually causes it, the vergence-accommodation conflict, and presence; displays and optics including panels, lenses, field of view and distortion correction; and tracking — inside-out and outside-in, degrees of freedom, drift and prediction. Includes the router for the whole vr-ar-development reference.

1 other project uses a version of this.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

1 rule. About 0 are common, 1 is distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    ⚠️ Never take control of the camera: No cutscene head movement, no forced rotation, no screen shake.

    line 105

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

VR and AR Development: The Perceptual Constraints, Displays and Optics, and Tracking

Part 1 of 5 of the VR and AR Development reference (plugin vr-ar-development), covering §0–§3. Sibling skills: xr-rendering-input-and-spatial-understanding (§4–§6), xr-platforms-audio-and-comfort (§7–§9), xr-ux-design-and-performance-budgeting (§10–§11), xr-reference (§12–§17). Section numbers are shared across the set; a reference written as §N → skill points into that sibling skill.

Currency: The perceptual constraints are human physiology and permanent; the platform and hardware landscape moves fast. See §14 → xr-reference for the current picture.

Scope. Assumes a graphics reference for the rendering fundamentals (rasterization, PBR, the rendering equation) and a computer-vision reference for SLAM and multi-view geometry. This is the XR-specific layer built on top of both.

⚠️ GOTCHA boxes mark what makes users physically ill or breaks presence.

The three facts that make XR its own discipline:

  1. ⚠️ You are rendering to a nervous system, not a screen. The frame budget is set by human perception, and missing it doesn't drop quality — it makes people nauseated (§1). No other software has this property.
  2. ⚠️ Latency is the master constraint and it is not a performance metric. Motion-to- photon under ~20 ms is the consensus threshold, and it governs your entire architecture (§1.1).
  3. ⚠️ Presence is fragile and asymmetric. It takes sustained correctness to build and one bad frame, one wrong-scale object, one mistracked hand to destroy. Everything in §9 → xr-platforms-audio-and-comfort and §10 → xr-ux-design-and-performance-budgeting is about protecting it.

§0. Routing

You want...Go to
The perceptual constraints§1
Displays and optics§2
Tracking§3
The XR rendering pipeline§4 → xr-rendering-input-and-spatial-understanding
Input and interaction§5 → xr-rendering-input-and-spatial-understanding
Locomotion§5.4 → xr-rendering-input-and-spatial-understanding
AR spatial understanding§6 → xr-rendering-input-and-spatial-understanding
Platforms and APIs§7 → xr-platforms-audio-and-comfort
Spatial audio§8 → xr-platforms-audio-and-comfort
Comfort, safety, accessibility§9 → xr-platforms-audio-and-comfort
UX design for 3D§10 → xr-ux-design-and-performance-budgeting
Performance budgeting§11 → xr-ux-design-and-performance-budgeting
Anti-patterns§12 → xr-reference
Numbers§13 → xr-reference
The platform landscape§14 → xr-reference
Resources§15 → xr-reference
Quick reference§16 → xr-reference

§1. The Perceptual Constraints

⚠️ Read this section before writing any XR code. Everything else is downstream of it.

1.1 Motion-to-photon latency

The time from a head movement to the corresponding photons reaching the eye.

sense IMU → fuse → predict → simulate → render → composite → scanout → photons

⚠️ Consensus is the "20 millisecond rule": end-to-end MTP should stay below ~20 ms to be generally imperceptible and give a comfortable experience. Above it, the world feels attached to your head with elastic, and ⚠️ latency is a documented cause of cybersickness and a documented reducer of presence.

⚠️ You cannot hit 20 ms honestly by rendering fast alone — the pipeline is too long. The trick that makes XR viable is prediction plus reprojection (§4.2 → xr-rendering-input-and-spatial-understanding): predict where the head will be at scanout, and re-warp the finished frame against the newest pose immediately before display. ⚠️ This means your effective latency is decoupled from your frame rate — which is why a dropped frame is survivable and a broken reprojection is not.

⚠️ Note the distinction for eye tracking: gaze-contingent foveated rendering has its own "eye-motion-to-photon" budget, and it is more forgiving than head latency — measured maximum tolerable system latency for foveated rendering lands in the 42–91 ms range depending on foveal region size and the degradation applied, with artifact detection reportedly unchanged below ~60 ms. ⚠️ Do not confuse the two budgets; they differ by roughly 3×.

1.2 Cybersickness

⚠️ The dominant explanation is sensory conflict: your eyes report motion your vestibular system does not. The documented primary contributors are latency, field of view, vergence-accommodation mismatch, and unnatural locomotion.

⚠️ GOTCHA — susceptibility varies enormously between people, and developers are the worst possible test population. You acclimate. ⚠️ Something that feels fine to you after six months of daily use can make a first-timer ill in ninety seconds. Test with fresh users, and take the first ten minutes seriously.

Mitigations that have evidence behind them:

  • ⚠️ Hit frame rate consistently. The single biggest lever.
  • Avoid imposed acceleration — ⚠️ the worst offender is camera motion the user did not initiate. Constant velocity is much better than accelerating; instant is better still.
  • Dynamic FOV restriction (vignetting) during locomotion — reduces peripheral optic flow, which is what drives vection.
  • ⚠️ Rest frames — a static cockpit, nose reference, or grid gives the vestibular system something consistent. Peripheral teleportation and similar rest-frame designs are an active research direction.
  • Foveated depth-of-field blur and peripheral degradation — ⚠️ reduces peripheral motion information, and studies indicate a measurable reduction in simulator sickness.
  • ⚠️ Never take control of the camera. No cutscene head movement, no forced rotation, no screen shake.

1.3 Vergence-accommodation conflict

⚠️ The unsolved optical problem of every mainstream headset. In natural vision, vergence (the eyes rotating to converge on a point) and accommodation (the lens focusing) are coupled and specify the same depth. In an HMD, the eyes accommodate to a fixed screen distance while continuing to converge freely on virtual objects at varying depths. ⚠️ The documented consequences: discomfort and fatigue, cybersickness, longer time-to-focus, distorted distance perception, and impaired motor planning and control — ⚠️ which matters enormously for surgical training and any task where a few centimetres of error is serious. Mitigations: varifocal and multi-focal displays (⚠️ prototypes and research; multi-focal AR prototypes add their own MTP burden), light fields, and the practical one available today — keep interactive content in a comfortable depth range and avoid forcing rapid depth changes.

1.4 Presence

The perceptual illusion of being there. Built by consistent, low-latency, plausible sensory input; ⚠️ destroyed instantly by tracking loss, a hand passing through a solid object, wrong-scale environments, or latency spikes. Presence is the product, and §9 → xr-platforms-audio-and-comfort and §10 → xr-ux-design-and-performance-budgeting exist to protect it.


§2. Displays and Optics

The metrics that matter, and ⚠️ note that FOV and PPD trade against each other for a fixed panel:

FOV                horizontal/vertical, per-eye and combined
⚠️ PPD (pixels per degree)   THE resolution metric — human acuity is ~60 PPD
Refresh rate       72 / 90 / 120 Hz
Persistence        ⚠️ low-persistence illumination (~2 ms) prevents smearing during
                   head motion — a critical and under-appreciated property
IPD                interpupillary distance ⚠️ — wrong IPD causes eye strain and
                   distorts perceived scale

Panel technologies: LCD, OLED, micro-OLED / OLED-on-silicon (⚠️ the current high-end standard), and MicroLED in development for brightness and efficiency.

Optics: Fresnel (⚠️ god rays and glare), pancake lenses (⚠️ thinner and sharper, at a significant light-efficiency cost — which is why pancake headsets need brighter panels), and for AR: waveguides (diffractive/reflective — ⚠️ the only practical path to glasses form factor, and they cost you FOV, brightness and colour uniformity) vs birdbath (better image, bulkier).

⚠️ Optical distortion correction is mandatory and it is not free: lenses distort, so you render and then apply an inverse barrel distortion, with per-channel correction for chromatic aberration. ⚠️ This is why you never render at exactly display resolution — distortion resampling means the source must be supersampled to avoid losing detail in the centre.

AR display modes: optical see-through (⚠️ you cannot render black — additive only, so occlusion of real objects is physically impossible) vs video passthrough (⚠️ full control and true occlusion, but the passthrough camera's own latency now sits in the user's view of the real world).


§3. Tracking

3DoF (rotation only) vs 6DoF (rotation + position). ⚠️ 3DoF for seated video; anything interactive needs 6DoF.

Approaches: outside-in (external base stations — ⚠️ highly accurate, fixed volume, setup burden) vs inside-out (on-headset cameras running SLAM — ⚠️ the consumer standard; no setup, but it fails in the conditions SLAM fails in).

⚠️ The sensor fusion is the whole trick: an IMU gives you angular rate and acceleration at ~1000 Hz with very low latency but unbounded drift (double-integrating accelerometer noise diverges fast). Cameras give you drift-free absolute pose at 30–60 Hz with high latency. Fusing them — typically an EKF or a factor graph — gives you both, and this is exactly the VIO problem from a computer-vision reference.

⚠️ Prediction is mandatory (§1.1): you must render for where the head will be. Prediction error shows up as swimming or overshoot on rapid head turns — and ⚠️ it's worse the longer your pipeline, which is another reason to keep it short.

⚠️ Where inside-out tracking actually fails — design for these, don't hope: featureless white walls, ⚠️ low light, highly repetitive texture (patterned carpet, brick), mirrors and large glass, ⚠️ moving environments — a train, a car, a boat, where the visual world and the vestibular world genuinely disagree, and rapid motion causing blur. Detect tracking degradation and degrade gracefully rather than letting the world lurch.

Controller tracking: constellation IR + IMU, or ⚠️ camera-visible controllers, which lose tracking behind the body or out of the camera frustum — hence the IMU dead reckoning that briefly covers the gap.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.