3D Reconstruction of Hand Postures by Measuring Skin Deformation on Back Hand, ICAT-EGVE 2017

1. 3D Reconstruction of Hand Postures by
Measuring Skin Deformation on Back Hand
*Wakaba Kuno, Yuta Sugiura, Nao Asano,
Wataru Kawai and Maki Sugimoto

2. Hand Interaction in Virtual Environment
24th Nov 2017 1
Hand interaction depending on finger posture is important for intuitive
interaction in virtual environments.
ICAT-EGVE 3D Reconstruction of Hand Postures
[1] https://www.leapmotion.com/ (2017/9/21)
[2] https://www.youtube.com/watch?v=B9tF7_nK4lI (2017/9/21)
[3] https://www.youtube.com/watch?v=4LVVpl9tCNE (2017/9/21)

3. Camera-based Method
24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 2
Estimate finger postures by processing images capturing hands
Merit: Markerless finger tracking
Limit: Space for capturing hands / Hand occlusion
[4] Jonathan Taylor et al. Siggraph 2016.

4. Glove-type Method
24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 3
Various sensors (inertial, magnetic, etc) on gloves measure finger postures.
Merit: High accuracy without occlusion
Limit: Weight and mechanism may inhibit natural finger movements.
[5] Tommaso Lisini Baldi et al. IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS. 2017

5. Gesture Recognition Measuring Wrist / Forearm
24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 4
Myoelectric or pressure sensors measure wrist or forearm.
Recognize hand gestures without measuring finger movement directly
[6] Myo. https://www.myo.com (2017/9/21) [7] Dementyev A. and Paradiso A. J.. UIST. 161-166. 2014.

6. Our Previous Research : Behind The Palm
24th Nov 2017 5
A prototype device with thirteen photo-reflective sensors in a straight array
Measure skin deformation on back of hand to recognize finger gestures
ICAT-EGVE 3D Reconstruction of Hand Postures
[8] Yuta Sugiura et al. SICE Annual Conference. 2017

7. Principle of Measuring Skin Deformation
24th Nov 2017 6
Skin on back of hand deforms when fingers move.
Measure skin deformation on back of hand with photo-reflective sensors
ICAT-EGVE 3D Reconstruction of Hand Postures

8. Proposal : Finger Posture Estimation from Back Hand
24th Nov 2017 7ICAT-EGVE 3D Reconstruction of Hand Postures

9. Principle of Finger Posture Estimation
24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 8
Obtain relationship between skin deformation on back of hand and
finger posture by using a multivariate regression model

10. System Configuration
24th Nov 2017 9
Sensor values are transmitted wirelessly through Microcontroller.
Desktop PC learns a model and estimates finger posture.
ICAT-EGVE 3D Reconstruction of Hand Postures

11. Estimation of Finger Posture
24th Nov 2017 10
Use Principal Component Analysis (PCA) to reduce the dimensions of data
Use Random Forest Regressor (RFR) model to estimate finger posture
ICAT-EGVE 3D Reconstruction of Hand Postures

12. Finger Posture Representation
24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 11
Some finger joints work in conjunction.
-> Measure relative positions of ten parts from hand center
five fingertips, one interphalangeal (IP) joint and four proximal IP (PIP) joints
Reference

13. Evaluation1 – Static-state Finger Posture
24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 12
Participants 7 (Male) + 2 (Female)
Data set per participant 2500 frames (= 50fps x 10 seconds x 5 trials)
Finger posture 5 postures
Evaluation 10-fold cross validation
Estimation error 3-dimensional Euclidean distance between the finger postures
We evaluated the estimation accuracy of finger posture in static state.

14. Mean Estimation Error (Static-state)
24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 13
Average estimation error for all parts: 3.34 mm.
Mean errors for each part: 1.5 – 2.8 mm (middle, ring and little finger)
Large variances of the estimation errors

15. Time Transition of Small Error Sequence (Static-state)
24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 14
Temporal transition of estimation error (small error)
Fingers are about 10 - 15 mm thick for adults.
-> Our method estimates postures except thumb and index finger sufficiently for
interaction in a virtual environment.
10
2.5
7.5
5
10
2.5
7.5
5
Posture in the sequence

16. Time Transition of Large Error Sequence (Static-state)
24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 15
Temporal transition of estimation error (large error)
Large errors in some frames and similar time transitions of errors
-> Cause of high variances of estimation errors
Posture in the sequence

17. Discussion (Static-state)
24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 16
Relative positions depend on finger posture and hand rotation.
-> Relative position can not be uniquely decided without hand rotation.

18. Evaluation2 – Dynamic-state Finger Posture
24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 17
Participants 7 (Male) + 2 (Female)
Data set per participant 2000 frames (= 50fps x 5 seconds x 8 trials)
Finger posture 5 postures
Evaluation 4-fold cross validation
Estimation error 3-dimensional Euclidean distance between the finger postures
We evaluated the estimation accuracy of finger posture in dynamic state.

19. Mean Estimation Error (Dynamic-state)
24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 18
Average estimation error for all parts: 11.7 mm.
Mean errors for each part: 6.5 – 10.9 mm (middle, ring and little finger)

20. Time Transition of Error Sequence (Dynamic-state)
24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 19
Large estimation errors for thumb and index fingers in some frames.
-> Cause of large mean errors for thumb and index finger

21. Discussion (Dynamic-state)
24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 20
Our device cannot distinguish the skin deformation caused by thumb
movement and index finger movement.

22. Applications
24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 21
Hand manipulation in VR space
Without occlusion and
inhibiting natural motion
Puppet manipulation metaphor

23. Limitations
• Need of User dependent training
- Each user has a different distance between the sensors and skin of back of hand.
• Affected by re-wearing the device
- Different mounting position due to re-attaching the device
24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 22

24. Future Work
• Improvement of device measuring skin deformation
- Measure the whole skin deformation on the back of the hand
- Measure skin deformation on back of hand and wrist or forearm simultaneously
• Improvement of algorithm Estimating finger postures
- Data normalization
- Other dimension reduction method of data
- Other representation of finger postures
24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 23

25. Conclusion
• We proposed a method for estimating finger postures from skin
deformation on back of hand.
• A regression model provides relationship between skin deformation of
back of hand and finger postures.
24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 24
Keio University, The University of Tokyo
Wakaba Kuno, Yuta Sugiura, Nao Asano, Wataru Kawai and Maki Sugimoto