It is a slide presented by ETRI IDX tech seminar. This presentation introduces standardization trends for artificial intelligence in Medicine.

1. Standardization for
Artificial Intelligence in Medicine
http://www.etri.re.kr
Jonghong Jeon
PEC(protocol engineering center)
ETRI
Email: hollobit@etri.re.kr

2. Agenda
• Standard & standardization
• International Standardization
– ITU-T (with WHO)
– ISO/IEC JTC1
• De facto standardization
– ONNX
– Khronos Group
– RSNA
• KoSAIM
• Conclusions
2

3. STANDARDIZATION
3

4. Standard?
4
Source: https://xkcd.com/927/

5. Too many standards ?

6. 표준 및 표준화의 정의
• 표준(standards)
– 표준은 합의에 의해 작성되고 인정된 기관에 의해 승인 되었으며,
주어진 범위 내에서 최적 수준을 성취할 목적으로 공통적이고
반복적인 사용을 위한 규정 (KS A ISO/IEC:Guide2)
– “개별 표준화 노력의 성과로서 공인된 단체에 의해 승인된 것”
• 표준화(standardization)란
– “일반적으로 사물에 합리적인 기준 또는 표준(standard)을
설정하고 다수의 사람들이 어떤 사물을 그 기준표준에 맞추는 것
즉 표준화란 표준을 설정하고 이것을 활용하는 조직적 행위”
• 표준화(standardization)란
– “실제적이거나 잠재적인 문제들에 대하여 주어진 범위내에서 최적
수준의 질서 확립을 목적으로 공통적이고 반복적인 사용을 위한
규정을 만드는 활동” ISO/IEC Guide 2(2004)

7. 표준의 분류

8. 표준의 분류
분류 기준 종류 표준화 조직 (예)
표준화 주체 및
참여 범위
국제표준
지역표준
국가표준
단체표준
사내표준
ITU, ISO, IEC
ESTI, ASTAP
ANSI, MIC, MPT
TTA, TTC, 협단체
KT, SKT, samsung
표준화
진행 단계
• ISO, IEC: 아이디어(PWI),신규
제안(NP), 작성초안(WD), 초
안(CD), 표준안(DIS), 국제표
준(IS)
• ITU: NP, WD, CD, REC
표준 적용 방법 강제표준, 권고표준

9. Three pillars of Standard
9
Relationships (Membership, Liaisons with other SDOs, Open Community…)
Openness (IPR policy, Open source)
Relationships (Membership, Liaisons with other SDOs, Open Community…)
Openness (IPR policy, Open source)

10. STANDARDIZATION FOR AI
(ITU-T)
10

11. ITU-T FG-AI4H
• The ITU-T Focus Group on Artificial Intelligence for Health (AI4H) was
established by ITU-T Study Group 16 at its meeting in Ljubljana, Slovenia, 9-20
July 2018.
• partnership with the World Health Organization (WHO) to establish a
standardized assessment framework for the evaluation of AI-based methods
for health, diagnosis, triage or treatment decisions.
• Chairman:
– Thomas Wiegand (Fraunhofer HHI, Germany)
• Vice-Chairmen
– Min Dong (CAICT, China)
– Stephen Ibaraki (ACM and REDDS Capital, USA)
– Ramesh Krishnamurthy (WHO/Health Metrics and Measurement Cluster)
– Sameer Pujari (Be Healthy Be Mobile Initiative and WHO/Non-communicable
Diseases Cluster)
– Marcel Salathé (EPFL, Switzerland)
• Working Groups
– Operations (WG-O)
• Chair: Markus Wenzel (Fraunhofer HHI, Germany)
– Health Requirements (WG-HR)
• Co-Chairs: Laragh Gollogly (WHO) and Ramesh Krishnamurthy (WHO)
• benchmarking of AI-algorithms through a standardized process 11

12. ITU-T FG-AI4H
• first meeting (A) at the World Health
Organizations headquarters in
Geneva, Switzerland on 26-27
September 2018
– Updated FG-AI4H Whitepaper
– A-102: Draft Call for Proposals: use
cases, benchmarking, and data
– A-103: Draft criteria for data to be
accepted by the Focus Group
– A-104: Draft thematic classification
scheme
12

13. ITU-WHO Workshop (1st)
• WHO, Geneva, Switzerland, 25 September 2018
– https://www.itu.int/en/ITU-T/Workshops-and-
Seminars/20180925/Pages/default.aspx
• Session 1: Focus Group on AI for Health
– "Consideration on the Development of AI", Min Dong
– "Framing the AI for Health to advance national health-related Sustainable
Development Goals and targets", Ramesh Krishnamurthy
– "On Benchmarking", Marcel Salathé
• Session 2: Applications and Use Cases for AI in Health
– "AI in cancer diagnostics", Frederick Klauschen, Charité Berlin
– “International Epidemiology Databases to Evaluate AIDS”. Julia Bohlius, Cancer
Epidemiologist, University of Bern
– "FastMRI", Nafissa Yakubova, Visiting Researcher, Facebook AI Research
– "First Derm", Alexander Börve, CEO and Founder of First Derm
– "AI Technology Development and Applications in HealthCare by Tencent", Yen-
Hsiang Wang, PhD, Tencent AI Lab
– "Child Growth Monitor", Jochen Moninger, Director of Innovation,
Welthungerhilfe
13

14. ITU-WHO Workshop (1st)
• Session 3: Data Availability and Benchmarking
– "Building Personalized Medicine in Switzerland", Christian Lovis, Chairman, Division of Medical
Information Sciences, University Hospitals of Geneva (HUG)
– "Clinical terms and ICD 11", Robert Jakob, Clinical terms and ICD 11, WHO
– "Ada Health- Our Appraoch to Assess Ada's Diagnostic Performance" Henry Hoffmann, Chief Scientist,
ADA
– "Skapa- Swedish Quality Registry for Caries and Periodontal Disease", Hans Östholm, Deputy Registry
Manager, Skapa Initiator
– "Chinese Practice of Benchmarking and Algorithm Evaluation", Shan Xu, Smart Health Department,
CAICT
– "Transparent and Trustworthy AI for medical applications", Wojciech Samek, Head of Machine
Learning Group, Fraunhofer HHI
• Session 4: Regional and funding perspectives of AI for Health
– "Country Priorities for Use of AI for Health-Croatian Experience", Antonija Balenović, Director Health
Center Zagreb, Croatia
– "Impact of AI on Healthcare", Severence MacLaughlin, Ph.D, Global Head AI & Data Sciences, AI
Learning Institute, New York
– "Change Makers Lab", Vasili Sofiadellis, CEO, Visions2Ventures and Founder, Changemakers Lab
– "Having a transformational impact on health", Fareed Mirza, Head of Research, Novartis Foundation
– "Fondation Botnar-The Future Now; For Children Worldwide", Siddhartha Jha, AI/Digital Program
Manager, Botnar Foundation
– "Digital Technologies for Global Health-Innovator Awards" Alexandre Cuenat, Expert in
Residence, Wellcome Trust
14

15. ITU Workshop (2nd)
• Columbia University, New York City, United States, 14 November 2018
– https://www.itu.int/en/ITU-T/Workshops-and-Seminars/20181114/Pages/default.aspx
• Session 1: Focus Group on AI for Health
– https://youtu.be/pE6WHuo_n2c
• Session 2: Applications and Use Cases for AI in Health
– https://youtu.be/9elIZcW_AuM
– Fostering a Strong Ecosystem for AI in Medical Imaging (Geraldine McGinty, Weill Cornell
Medicine)
– AI to Prevent Vision Loss for Millions Globally (Arun Shroff, Medindia)
– Rehabilitating the Mind: Using AI to Track and Treat Mental Illness (Paul Sajda, Columbia University)
– Disease Prediction Using Machine Learning and Electronic Health Records (Narges Razavian, New
York University)
• Session 3: Data Availability and Benchmarking
– https://youtu.be/3J3cAfT6a_4
– Data Availability and Benchmarking in Observational Health Data Sciences and Informatics (OHDSI)
(George Hripcsak, Columbia University)
– Secure and Privacy-Preserving Benchmarking for Artificial Intelligence in Health (Jean-Louis Raisaro,
Lausanne University Hospital)
– Computational Psychiatry and Neuroscience: Scientific Gaps and Funding Opportunities (Michele
Ferrante, National Institute of Mental Health)
– Evaluation of Medical Algorithms (Naomi Lee, The Lancet)
•
15

16. ITU Workshop (2nd)
• Session 4: Country Priorities for Use of AI for Health
– https://youtu.be/ErpmOZaoXF0
– Artificial Intelligence- Anticipating the Opportunities and the Challenges
for Therapeutic Development (M. Khair Elzarrad, Center for Drug
Evaluation and Research (CDER), U.S. FDA)
– AI in Drug Development: Opportunities and Challenges (Archana Reddy,
MS Regulatory Affairs)
– AI & Healthcare In The Low and Middle Income Countries (Mehul Mehta,
Harvard Medical School)
• Session 5: Funding of AI for Health
– https://youtu.be/MWWN5wzvGTA
– Third Wave of AI in Healthcare (Paul Lee, Founder & CEO, Mind AI)
– Mark Minevich, Strategic Advisor and Digital Fellow, IPsoft, United States
[ Biography ]
16

17. ITU Workshop (3rd)
• EPFL SwissTech Convention Center, Ecublens (Lausanne), Switzerland, 22 Jan.
2019
– https://www.itu.int/en/ITU-T/Workshops-and-Seminars/ai4h/20190122/Pages/programme.aspx
• Session 1: Focus Group on AI for Health
• Session 2: Use Cases and Data Availability
– Yanwu Xu, Chief Scientist at Department of Artificial Intelligence Innovation Business, Baidu Inc
– Weihong Huang, Mobile Health Ministry of Education­ China Mobile Joint Laboratory, Xiangya
Hospital, Central South University
– Indra Joshi, Digital Health & AI Critical Lead, NHS England
– Saurabh Gupta, Assistant Professor in the Department of Cardiology, All India Institute of Medical Sciences
(AIIMS)
• Session 3: Benchmarking and Security
– Christian Djeffal & Alina Wernick, Lawyers/Researchers at Alexander-von-Humboldt Institute for Internet
and Society, Germany
– Giovanni Cherubin, Postdoc Researcher, EPFL
– Jean-Pierre Hubaux, Full Professor, EPFL
• Session 4: Regulations and Country Priorities for use of AI for Health
– Shashank Rai, Head of Strategic Consulting, United Nations International Computing Centre (UNICC)
– Leena Sankla, Co-founder and Director of Public Health & Lifestyle Services & Keshav Sankla, AI Programme
Lead, Solutions 4 Health
– Spiros Dimolitsas, Senior Vice President for Research & Chief Technology Officer, Georgetown University
• Session 5: Funding of AI for Health
– Karin Jestin, Strategic Philanthropy Advisor, PHI (Philanthropic & Humanitarian Initiatives)
17

18. Future meetings
• D: Shanghai, China, 2-5 April
2019
• E: Geneva, Switzerland 29-31 May
2019 [TBC]
• F: September 2019
18

19. STANDARDIZATION FOR AI
(ISO/IEC JTC1)
19

20. SC 42
• Scope : Standardization in the
area of Artificial Intelligence
– Serve as the focus and
proponent for JTC 1's
standardization program on
Artificial Intelligence
– Provide guidance to JTC 1, IEC,
and ISO committees
developing Artificial
Intelligence applications
20
https://www.iso.org/committee/6794475.html
èWG3
èWG4

21. 1st plenary meeting
• 2018.4.18 ~ 4.20 (17개국 100명)
21

22. 2nd plenary meeting
• 2018. 10. 8 ~ 2018. 10. 12, 서니베일, 미국
• 참석자 : 19개국 131명
• 주요 결과
– JTC 1/SC 42 인공지능 산하 조직 변경
• WG 1(인공지능 기반표준): Foundational Standards (기존)
• WG 2(빅데이터): Big Data (신규)
• WG 3(인공지능 신뢰성): Trustworthiness (신규, ç SG2 전환)
• WG 4(인공지능 사례 및 응용): Use Cases and Applications (신규,çSG3 전환)
• SG 1(인공지능 시스템): Computational Approaches and Characteristics of Artificial
Intelligence Systems
– 5건 신규 문서 개발 착수
• WG3 (기존 SG2)
– (IS) Information technology - Artificial Intelligence ­ Risk Management
– (TR) Information technology - Artificial Intelligence (AI) - Bias in AI systems and AI aided
decision making
– (TR) Information technology - Artificial Intelligence (AI) ­ Assessment of the robustness of
neural networks, part 1: Overview
– (TR) Information technology - Artificial Intelligence (AI) ­ Overview of Trustworthiness in
Artificial Intelligence
• WG4 (기존 SG3)
– (TR) Information technology - Artificial Intelligence (AI) ­Use cases
– 기존 문서 수정 작업
• ISO/IEC 22989 Artificial Intelligence Concepts and Terminology (Wei Wei, 독일)
• ISO/IEC 23053 Framework for Artificial Intelligence Systems Using Machine Learning
(Milan Patel, 영국)
22

23. N202 - Risk Management
23

24. Other project in JTC1
• ISO/IEC JTC1 WG12 (3D printing & scanning)
– PART X-X : Optimization of orbit segmentation
from medical images for 3D printing
24

25. DE FACTO
25

26. ONNX
Open Neural Network Exchange Format
https://onnx.ai/
https://github.com/onnx
Source: ONNX

27. Why ONNX ?
27Source: Microsoft, ONNX

28. Open Neural Network Exchange
28Source: Microsoft, ONNX

29. 29Source: Khronos group

30. NNEF - Solving Neural Net Fragmentation
• Fragmentation issue
30Source: Khronos group

31. Platform Neural Network Stacks
31Source: Khronos group

32. NNEF Ecosystem
32Source: Khronos group
(‘18.8)

33. Neural Network Workflow
33Source: Khronos group

34. RSNA 2018 June

35. RSNA MLDS
• Machine Learning Data Standards Subcommittee
•
Responsible for publishling and fostering adoption of
standards for image data and associated information to be
used for training and evaluation of machine learning
algorithms.
George L. Shih, MD, MS, Member
New York, NY
Carol C. Wu, MD, Member
Bellaire, TX
Curtis P. Langlotz, MD, PhD, Board Liaison
Menlo Park, CA
Steve Drew , Staff Contact
Oak Brook, IL
Safwan Halabi, MD, Chairman
Stanford, CA
Katherine P. Andriole, PhD, Member
Dedham, MA
Falgun H. Chokshi, MD, Member
Atlanta, GA
Adam E. Flanders, MD, Member
Philadelphia, PA
Matthew P. Lungren, MD, Member
Palo Alto, CA
Luciano M. Prevedello, MD, MPH, Member
Columbus, OH

36. K-FDA guideline (‘17.11)
36

37. K-FDA guideline (‘17.12)
37

38. K-FDA roadmap (‘18.8)
38

39. China’s benchmark dataset
39Shan Xu, CAICR, Chinese practice of benchmarking and algorithm evaluation

40. 대한의료인공지능학회 (‘18.10)
• http://www.kosaim.org/
• 비전
– 의료 인공지능 융합 기술을 이용한 지능형
의료기기 개발 및 임상적용
– 의료 인공지능 기술 연구개발을 위한
산/학/연/병/정 융합의 장
– 유용하고 안전한 의료인공지능 기술
임상시험 촉진
– 융합 교육을 통한 의료 인공지능 인재 육성
– 의료 인공지능 관련 산업 활성화
– 의료 인공지능 관련 국가 정책 개발/규제
조정을 위한 소통의 장
• 표준화 분과를 통한 협력/협력
40

41. 대한의료인공지능학회 (‘18.10)
41
임원 이름 소속 분야 직위
학회장 서준범 서울아산병원 영상의학 교수
부회장
최병욱 연세대학교 영상의학 교수
예종철 KAIST 바이오및뇌공학 교수
윤형진 서울대학교 신장내과/의공학 교수
총무이사 김남국 서울아산병원 융합의학 교수
총무간사 이현나 서울아산병원 빅데이터센터 교수
재무이사 고범석 서울아산병원 유방외과 교수
학술이사
한보형 서울대학교 전기정보공학 교수
김광준 연세대학교 내분비/노년내과 교수
교육이사
이준구 서울아산병원 의공학연구소 교수
홍헬렌 서울여대 멀티미디어학과 교수
민정준 전남대 핵의학 교수
장진희 서울성모병원 영상의학과 교수
기획이사
윤성로 서울대학교 전기정보공학 교수
양동현 서울아산병원 영상의학과 교수
서성욱 성균관대학교 정형외과 교수
최윤섭 DHP 융합 대표파트너
의무이사
장동경 성균관대학교 소화기,삼성병원 CIO 교수
박창민 서울대병원 영상의학과 교수
김영보 가천 길병원 신경외과 교수
김동억 동국대일산병원 신경과 교수
백정흠 가천대 길병원 일반외과 교수
법제이사
신수용 성균관대학교 의료정보 교수
유소영 서울아산병원 빅데이터센터 교수
김현철 보건산업진흥원 R&D 기획단 단장
산업이사
정규환 뷰노 산업체 CTO
백승욱 루닛 산업체 CEO
홍보이사
김영준 KIST 의공학 책임
손장욱 고려대 감염내과 교수
최병관 부산대 신경외과 교수
표준화이사 전종홍 ETRI 표준연구본부 책임연구원
정보이사 김휘영 연세대학교 의공학 교수
감사이사 이학종 분당서울대학교병원 영상의학 교수

42. 2019 KoSAIM calendar
• 회원 정례 세미나: 2019년 02월 12일
• 회원 정례 세미나: 2019년 04월 09일
• 제1회 KOSAIM 정기학술대회
– 019년 5월 18일(토) 분당서울대병원 헬스케어 혁신파크
• 회원 정례 세미나: 2019년 06월 11일
• 회원 정례 세미나: 2019년 08월 13일
• KOSAIM Summer School:
– 2019년 8월 22일-24일, 제주도 예정
• 제 2회 KOSAIM 추계 심포지움:
– 2019년 9월 초 / 코엑스 예정
• 회원 정례 세미나: 2019년 10월 8일
• 회원 정례 세미나 및 송년회: 2019년 12월 10일
42

43. Conclusions
• Rapidly growth of standardization demands,
but still initial stage
– ITU-T, JTC1, Khronos, ONNX, IEEE …
– assessment, risk management, data, exchange
format
• RSNA organized a SC for DL
• K-FDA published the guidelines
– FDA preparing
• New standards are essential for growth of AI
in medicine
• Standardization with KoSAIM
– Strategic planning & collaboration
– for leading the global standardization
43

44. Conclusions
44
Wojciech Samek: Transparent & Trustworthy AI for Medical Applications

45. JongHong Jeon (hollobit@etri.re.kr)
+82-42-860-5333
https://www.linkedin.com/in/hollobit
http://twitter.com/hollobit