Skip to content

Malcolmnixon/GodotXRHandPoseDetector

Repository files navigation

Godot XR Hand Pose Detector

GitHub forks GitHub Repo stars GitHub contributors GitHub

This repository contains a hand pose detector for Godot that detects poses on XRHandTracker sources.

Versions

Official releases are tagged and can be found here.

The following branches are in active development:

Branch Description Godot version
master Current development branch Godot 4.3-beta1+

Overview

Godot XRHandTracker data is generated by some XR systems such as OpenXR. This project contains assets capable of detecting standard and user-defined hand poses and firing signals when the user poses their hands in those configurations.

Data Flow

Hand Pose Pipeline

This system takes XRHandTracker information provided by the XR System and feeds it to a Pose Detector. The pose detector feeds into a Pose Controller which generates an XRControllerTracker to drive an XRController3D. Game functions can then be bound to the controller.

Usage

The following steps show how to add the Godot XR Hand Pose Detector to a project.

XR Hand Tracking Project

Ensure the existing project is configured with XR hand tracking. The demo project and main scene shows how to do this for OpenXR.

OpenXR Hand Tracking Scene

Install Addon

The addon files need to be copied to the /addons/hand_pose_detector folder of the Godot project.

Add Hand Pose Controllers

Add Hand Pose Controller nodes into the scene - one for each hand. These will detect hand poses and drive XRController3D nodes.

Add Hand Pose Controller

Add XRController3D nodes for the virtual hand-driven controllers - one for each hand.

Add XRController3D

Configure the Hand Pose Controller nodes with:

  • The XRControllerTracker name for the virtual controller
  • The type of pose to drive (Aim is most widely supported)
  • The hand pose action map for actions to trigger on the virtual controllers
  • The XRHandTracker name for the hand
  • The set of hand-poses to detect

Hand Pose Controller Settings

Configure the XRController3D Virtual Controller nodes with the name of the XRControllerTracker for each hand.

Virtual Controller Settings

If needed, connect the hand pose detector signals. The preferred approach is to generate actions using the hand pose action map, and then detect those actions as if generated by a standing XR controller.

Hand Pose Detector Signals

Custom Hand Poses

This section describes the process of creatnig custom hand poses. Additionally the Creating Custom Hand Poses video walks through the process.

New hand poses can be made by creating new Hand Pose Resource instances.

Hand Pose Resource

Hand Pose Resources consist of:

  • A Pose Name (reported in the pose detector signals)
  • A Threshold (a minimal fitness threshold to report the pose)
  • A Hold Time (a debounce time necessary to register the pose)
  • A Release Time (a debounce time necessary to release the pose)
  • A set of fitness functions to apply to each pose component

Pose Components

Type Description
Flexion The angle (in degrees) of a fingers proximal joint curving into the palm to make a fist.
Curl The curl (in degrees) of a finger from the proximal to the distal joints.
Abduction The spread (in degrees) between two selected fingers.
Tip Distance The distance (in millimeters) between the tips of two selected fingers.

Fitness Function

The fitness function converts a measurement (degrees or milimeters) into a fitness in the range 0..1 with 0 being a bad match, and 1 being a perfect match. Two types of fitness function are supported:

  • Smoothstep
  • Range

The fitness of a Hand Pose is the product of the fitness of all the components.

Smooth-Step Function

The Smooth-Step function transitions from 0 to 1 over the specified range. The paramerters may be reversed to reverse the function.

SmoothStep Positive SmoothStep Negative

Range Function

The Range function provides non-zero values in a finite range.

Fitness Transform

Designing and Tuning

The inspect scene provided in the demo project can be used to inspect the flexion, curl, abduction, and tip-distance of a hand, and can also inspect a selected hand pose to diagnose the fitness of each component.

Inspect Scene

Licensing

Code in this repository is licensed under the MIT license.

About this repository

This repository was created by Malcolm Nixon

It is primarily maintained by:

For further contributors please see CONTRIBUTORS.md