VR / AR 의료 응용 (2018.01.25 엡슨 컨퍼런스 EPIC 발표자료)

1. VR / AR for Medicine
2018.01.25
한국과학기술연구원 의공학연구소
김영준

2. 2

3. Contents
▪ 연구소개
▪ VR/AR 의료응용 사례
▪ VR/AR 의료응용 분야 및 전망
3

4. 4

5. 3D 의료영상 처리 S/W
5
▶ 3D 가상 수술
▶ 3D 환자모델링
▶ 3D 형상 모델 처리기술

6. 6
3D Patient Modeling
Automatic Orbital Wall Modeling
Existing method: manual operation on >100 images
 Proposed method: ~1 min. (1/40 less time)
Avr. 31m 12s
Avr. 43.7s
Comparison test of 3D modeling time
(manual vs. proposed method)
Semi-automatic 3D modeling of orbital wall
Resulted 3D orbital wall model

7. 3D Surgical Planning
7

8. 3D Modeling of Surgical Guide
8

9. 9

10. AI기반 3D 의료용 S/W 기술
10
임상 빅데이터
환자 3D 데이터
AI 진단
Classification
AI 치료 계획
치료방법 추천
AI 치료 분석
수술결과 분석

11. 3D CNN 기반 어깨 관절 진단
11
▪ 학습
 Rotator Cuff Tear 진단
 Voxception-ResNet 네트워크 사용, epoch 110
 Train set 3573개 (None-RCT : 1749, RCT : 1824) , Test set 200
 >95% 정확도
학습 진행에 따른 정확도 및 activation map 변화
Voxception-ResNet

12. 3D CNN 기반 어깨 관절 (Rotator Cuff Tear) 진단
12

13. 13

14. 14https://github.com/vtk-book/example 14

15. 3D simulation, 3D printing, AI, VR/AR
 Medical application

16. Contents
▪ 연구소개
▪ VR/AR 의료응용 사례
▪ VR/AR 의료응용 분야 및 전망
16

17. Epson MOVERIO
17
https://epson.com/moverio-augmented-reality

18. Epson MOVERIO
18

19. Evena Eye-On Glasses
19
https://evenamed.com/

20. Microsoft HoloLens
20

21. Microsoft HoloLens
21

22. 3D4Medical
22
http://3d4medical.com/

23. MediSIM
23
http://3d4medical.com/

24. The Body VR: Anatomy Viewer
24
http://TheBodyVR.com/

25. 3D Organon VR Anatomy
25
https://www.oculus.com/experiences/rift/872418872856459/

26. NEOFECT
26
http://www.neofect.com/

27. Motek
27
https://www.motekforcelink.com/

28. Motek
28
https://www.motekforcelink.com/

29. Dental Simulation - 3D Smile Design (CEREC)
29

30. NeuroVR
• 신경외과 수술 훈련용 시뮬레이터
• 두개골 개폐, 뇌 내시경 수술 술기훈련

31. Osso VR
31

32. Help Me See – Cataract Surgery (SOFA, InSimo)
32

33. SOFA (Simulation Open FrameworkArchitecture)
33

34. Virtual Training Simulation
34
Arthroscopic surgery trainingLaparoscopic surgery simulation
Dental simulation Intravenous injection simulator

35. http://www.intuitivesurgical.com/products/skills_simulator/

36. Surgical simulator
▪ http://simbionix.com/
U/S Mentor
VirtaMed HystSim
PELVIC Mentor
ANGIO Mentor

37. Laparoscopic surgery simulator
SimSurgery / Norway
Surgical-Science / Sweden
Simbionix / USA
Mimic Technologies
/ USA
Mentice / Sweden
Haptica / USA
Immersion / USA

38. Laparoscopic Surgical Training Simulation
Deformable body modeling
3D visualization / haptic rendering / collision detection
Gallbladder removal simulation
38
Gallbladder removal simulation
Haptic device for laparoscopic
surgery simulation

39. Laparoscopic training simulator
39

40. ARTHRO MENTOR™
http://simbionix.com/simulators/arthro-
mentor/
관절경 수술
Diagnostic and therapeutic
knee / shoulder arthroscopy training
in orthopedic surgery
Arthroscopic surgery simulator

41. Arthroscopic surgery simulator
3D human body modeling (bone, muscle, ligament, …)
Realistic visualization : normal mapping (texture) + lighting
Arthroscopic camera simulation
Arthroscopic camera simulation
3D human shoulder modeling

42. Arthroscopic surgery simulator
Deburring Navigation
Task:
Basic navigation skill, feeling of burr Navigation, tool manipulation and orientation
skills

43. Simulation for Craniopagus Surgery
43

44. Contents
▪ 연구소개
▪ VR/AR 의료응용 사례
▪ VR/AR 의료응용 분야 및 전망
44

45. 혼합현실 기반 수술 계획 및 가이드
Pisa Univ. (이탈리아)
• 악안면 수술을 위한 AR 기반 네비게이션 시스템
• 수술계획대로 악안면 골 재배치 AR 수술계획 정보 제공
Tokyo Univ. (일본)
• 악안면 수술 환자 3차원 해부모델 AR
• 환자 치아형태 3D 매칭, makerless 실시간 트래킹

46. 혼합현실 기반 원격 치료
Proximie (미국)
• AR 기반 원격진료 솔루션
• 아이폰/아이패드로 연결된 두 지점의
전문가 수술보조 서비스 제공
Google, University of Alabama at Birmingham (미국)
• AR 기반 원격수술 가이드 솔루션
• 구글글래스 이용
• Teaching surgeon의 술기를 AR로 보면서 수술진행

47. Nadine Hachach-Haram(TEDWomen 2017)
How augmented reality could change the future of surgery
47
https://www.ted.com/talks/nadine_hachach_haram_how_augmented_reality_could_
change_the_future_of_surgery

48. Visible Korean (한국과학기술정보원, 2000)
Visible Human Project (NLM, 1986)
Anatomage Table Medical education(MS Hololense, 2016)
Medical Education
KIST 김래현

49. Medical Realities, UK
Live Surgery
KIST 김래현

50. Surgical Simulation
Arthroscopy surgery (VirtaMed)
Eye surgery (HelpMeSee) Dental training (KIST)
Laparoscopic surgery (CAE, KIST)
Intravenous injection (KIST)
KIST 김래현

51. Surgical Planning & Guide
Neuro surgery planning (UCLA)
Knee & Maxillofacial surgery planning (KIST)
AR-based surgical guide
(Fraunhofer MEVIS Institute)
KIST 김래현

52. AR Navigated Spine Surgery: Scopis
52
https://navigation.scopis.com/news/scopis-introduces-mixed-reality-surgical-holographic-navigation-platform

53. 53

54. 54

55. LINKFLOW
55
http://linkflow.co.kr/

56. Vive Pro (HTC)
56
해상도: 2880 x 1600
전방 카메라 2대, 마이크 2대
가상공간: 10m x 10m
무선 어댑터 출시 예정

57. Future of VR
Hi resolution(8K), wide FOV(120), fast updated(120 fps)
Multimodal stimuli (tactile, smell,..)
Seamless and precise tracking ( <10 ms latency, sub-mm)
HTC VIVE (2K resolution, 110 degree FOV, 90 fps, 2 controller,
Tracking: sub-millimeter precision and < ~11ms latency)
KIST 김래현

58. Future of VR
Mobile VR
Multi-view 360 video
Social VR(Facebook)
360 Panoramic video (YouTube, Facebook)
Social VR
KIST 김래현

59. Future of AR
Lighter, Cableless, Brighter H/W
Wide FOV (Field of View)
Easy Calibration
MS HoloLens 3rd generation (2019)
Accurate Registration
AI for Improved Function
Magic Leap One (2018 spring)
MAXST

60. 60

61. Youngjun Kim
E-mail: junekim@kist.re.kr
3D printing, 3D simulation, AI, VR/AR
 Medical application

62. Journal Paper Publication (2017)
1. J. Charton, M. Laurentjoye, Y. Kim, "3D Boolean operations in virtual surgical planning", International Journal of Computer
Assisted Radiology and Surgery, Vol. 12, No. 10, pp. 1697-1709, Oct. 2017
2. J. Charton, L. Kim, Y. Kim, "Boolean operations by a robust, exact, and simple method between two colliding shells".
Journal of Advanced Mechanical Design, Systems, and Manufacturing, 11(4), Sep. 2017
3. S. Yoon, S. Shin, H. Cho, Y. Kim, D. Lee, L. Kim, G. Noh, "Enhanced markerless surgical robotic guidance system for
keyhole neurosurgery", Journal of Advanced Mechanical Design, Systems, and Manufacturing, Vol. 11, No. 4,
p.JAMDSM0046, Sep. 2017
4. S.W. Chung, K.S. Oh, S.G. Moon, N.R. Kim, J.W. Lee, E. Shim, S. Park, Y. Kim, "Serial Changes in 3-Dimensional
Supraspinatus Muscle Volume following Rotator Cuff Repair", The American Journal of Sports Medicine, Vol. 45, No. 10,
Aug. 2017
5. Y.D. Choi*, Y. Kim*, E. Park, “Patient-Specific Augmentation Rhinoplasty Using a Three-Dimensional Simulation Program
and Three-Dimensional Printing”, Aesthetic Surgery Journal, DOI: 10.1093/asj/sjx046, May, 2017
6. S. Kim, D. Lee, S. Park, K. Oh, S.W. Chung, Y. Kim, "Automatic segmentation of supraspinatus from MRI by internal shape
fitting and autocorrection", Computer Methods and Programs in Biomedicine, Vol. 140, pp. 165-174, Mar. 2017
7. Y. Kim, B.H. Lee, K. Mekuria, H. Cho, S. Park, J.H. Wang, D. Lee, "Registration accuracy enhancement of a surgical
navigation system for anterior cruciate ligament reconstruction: A phantom and cadaveric study", The Knee, S0968-
0160(16)30248-4, Feb. 2017
8. C. Kyu Lee, Y. Kim, N. Lee, B. Kim, D.Y. Kim, S. Yi, "Feasibility study of utilization of action camera, GoPro Hero 4, Google
glass and Panasonic HX-A100 in Spine surgery", Spine, Vol. 42, No. 4, pp. 275-280, Feb. 2017
9. S.H. Park, S.W. Moon, B.H. Lee, S. Park, Y. Kim, D. Lee, S. Lim, J.H. Wang, "Arthroscopically blind anatomical anterior
cruciate ligament reconstruction using only navigation guidance: a cadaveric study", The Knee, DOI:
10.1016/j.knee.2016.02.020, Feb. 2017
10. Q.C. Nguyen, Y. Kim, H. Kwon, "Optimization of layout and path planning of surgical robotic system", Int’l Journal of
Control, Automation and Systems, Vol. 15, No. 1, pp. 375-384, Jan. 2017
11. E. Shim, Y. Kim, D. Lee, B.H. Lee, S. Woo, K. Lee, “2D-3D registration for 3D analysis of lower limb alignment in a weight-
bearing condition”, Applied Mathematics – Journal of Chinese Universities (Accepted)
62