Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
基於CNN對易混淆中藥的手機辨識系統
Recognition	of	Easily-confused	TCM	Herbs	
Using Convolutional	Neural	Network
On	The	Smartphone
Kun-chan...
A	little	about	me
• Background in sensor network (aka. IoT)
•2011: experienced TCM
•2013: started doing research on TCM
• ...
TCM	101
• Based on thousands of years of clinical experiences
• Data -> model (similar to DNN?)
• Treat by symptom 症(perso...
Chinese	Herbal	Medicine		
• Traditional	Chinese	medicine	(TCM)	originated	in	China	and	has	
evolved	over	5000 years. TCM i...
Easily-confused	herbs	
5
山藥 木薯
黃耆 紅耆
人參 西洋參
川木通 關木通
川母貝(松貝) 平母貝
黃芩 綠黃芩
GTC	Taiwan	2017
黃耆 vs.	紅耆
•Some TCM herbs have similar shape and
color but different utilities and cost.
6
GTC	Taiwan	2017
Smartphone	to	the	rescue?	
Information
Illustrated	handbooks Smartphones
7
GTC	Taiwan	2017
Internet
A	simple	client-server	framework	
Pre-trained
Clustering
Model
Pre-trained
Classification	
Model
CHM
Info.
Image
...
Prior	work	on	TCM	herb	recognition
Tao	et	al.	 Liu	et	al.	 Sun	et	al. Ours
Category 18 8 95 24
Confused Herbs	Pair	 1 0 2 ...
What	we	did	(	a	demo)
• 山藥 vs. 木
• 黃耆 vs. 紅耆
• GTC-demo_video.wmv
10
Test1 Test2 Test3 Test4 Test5 Avg.
Xiaomi 3.083 2.535...
Why	Deep	Learning?
• With	traditional	hand-crafted	methods,	It	is	not	easy	to	find	
representative	features	for	easily-con...
CNN-CaffeNet
24
12
GTC	Taiwan	2017
Dataset	(中藥飲片)
• CHM dataset collected by iPhone6 camera.
• 2400 images of 24 CHMs
• 1440 images for training
• 960 images...
Experimental	Environment
• INTEL	i7-4790 CPU	&	16GB	RAM
• NVIDIA	GTX	1060
• Python
• Caffe
14
GTC	Taiwan	2017
Result
Naïve	CNN	Method
15
Training Phase
Testing Phase
Input	Image
Pre-trained
CNN	Model
CNN
Model
Feature	Extraction
Tra...
16
80%
84%
88%
92%
96%
100%
A1 A2 B1 B2 C1 C2 D1 D2 E1 E2 F1 F2 G1 G2 H1 H2 I1 I2 J1 J2 K1 K2 L1 L2 Avg
CNN HCNN	by	AP	alg...
Hierarchical	Clustering	CNN
17
GTC	Taiwan	2017
Result
Hierarchical	Clustering	CNN	Method
18
Training Phase
Testing Phase
Input	Image
Second-layer
CNN-based
Classificatio...
Clustering:	Affinity	Propagation
Training	Images
…
Affinity	Propagation
algorithm
Feature	
Extraction
Each	kind	of	herbs	
...
current	results
20GTC	Taiwan	2017
Usefulness	of	CNN?
• Hand-Crafted Method
• SIFT
• HOG
• LBP
• SVM(classifier)
CNN method
• CaffeNet model
(5 conv layers)
...
Effect	of	Fine-tune
• Fine-tune by pre-trained CaffeNet model (based on ImageNet
data)
22
0.0%
10.0%
20.0%
30.0%
40.0%
50....
CNN	vs.	HCNN	
80%
84%
88%
92%
96%
100%
A1 A2 B1 B2 C1 C2 D1 D2 E1 E2 F1 F2 G1 G2 H1 H2 I1 I2 J1 J2 K1 K2 L1 L2 Avg
CNN HCN...
Effect	of	Smartphones?
iPhone Xiaomi Samsung Asus
24
GTC	Taiwan	2017
Effect	of	Smartphones
A1 A2 B1 B2 C1 C2 D1 D2 E1 E2 F1 F2 G1 G2 H1 H2 I1 I2 J1 J2 K1 K2 L1 L2 Avg.
iPhone 100.00% 95.00% 8...
Not	enough	data	=>	data	augmentation?
Zoom	In
Zoom	OutClockwise	Rotation
Counter-clockwise Rotation Darken
Brighten
26
GTC...
Data	Augmentation
(1).	iPhone	camera	(The	original	training	data)
iPhone
1440	images
(2).	iPhone	camera	+	AUG*2(Rotation)
...
data	augmentation	vs.	adding	more	phone	data
70.00%
75.00%
80.00%
85.00%
90.00%
95.00%
100.00%
iPhone Xiaomi Samsung Asus ...
Conclusions
• Automatic	recognition	of	24	easily-confused	CHMs	on	the	smartphone.
• Compared	to	traditional	hand-crafted	m...
Future	work
• Short term
• Collect data for all 300+ TCM
herbs
• Try with more different phones
under different lighting c...
Prochain SlideShare
Chargement dans…5
×

GTC Taiwan 2017 基於 CNN 對易混淆中藥的手機辨識系統

國立成功大學 藍崑展

  • Identifiez-vous pour voir les commentaires

GTC Taiwan 2017 基於 CNN 對易混淆中藥的手機辨識系統

  1. 1. 基於CNN對易混淆中藥的手機辨識系統 Recognition of Easily-confused TCM Herbs Using Convolutional Neural Network On The Smartphone Kun-chan Lan (藍崑展) National Cheng Kung University Joint work with Min-Chun Hu and Juei-Chun Weng 1GTC Taiwan 2017
  2. 2. A little about me • Background in sensor network (aka. IoT) •2011: experienced TCM •2013: started doing research on TCM • smartphone APPs for TCM • Tongue diagnosis (https://lens.csie.ncku.edu.tw/~john/) • AR-based acupoint localization (https://www.youtube.com/watch?time_continue=1&v=RyzKMuo3Gjo) • TCM Herb recognition •2015: studying TCM at China Medical University (中國醫藥⼤學) 2 GTC Taiwan 2017
  3. 3. TCM 101 • Based on thousands of years of clinical experiences • Data -> model (similar to DNN?) • Treat by symptom 症(personalized treatment) • Considering individual constitution and the interaction with the environment • Western Medicine : Treat by disease 病(same treatment for same disease) • Four diagnoses (四診) : collect biometrics using sensors on the human body • Inspection (望) • Listen and smell (聞) • Inquiry (問) • Palpation (切) 3 GTC Taiwan 2017
  4. 4. Chinese Herbal Medicine • Traditional Chinese medicine (TCM) originated in China and has evolved over 5000 years. TCM is one of Complementary Medicines (互補醫學) recognized by World Health Organization (WHO) • Chinese Herbal Medicine (CHM) is one of the important therapies in TCM (⼀針, ⼆灸, 三湯藥) 4 GTC Taiwan 2017
  5. 5. Easily-confused herbs 5 山藥 木薯 黃耆 紅耆 人參 西洋參 川木通 關木通 川母貝(松貝) 平母貝 黃芩 綠黃芩 GTC Taiwan 2017
  6. 6. 黃耆 vs. 紅耆 •Some TCM herbs have similar shape and color but different utilities and cost. 6 GTC Taiwan 2017
  7. 7. Smartphone to the rescue? Information Illustrated handbooks Smartphones 7 GTC Taiwan 2017
  8. 8. Internet A simple client-server framework Pre-trained Clustering Model Pre-trained Classification Model CHM Info. Image Preprocessing Predict Result server 8 GTC Taiwan 2017
  9. 9. Prior work on TCM herb recognition Tao et al. Liu et al. Sun et al. Ours Category 18 8 95 24 Confused Herbs Pair 1 0 2 10 Method Hand-Crafted Method Hand-Crafted Method CNN Hierarchical Clustering CNN Implemented on smartphone No No No Yes 9 GTC Taiwan 2017
  10. 10. What we did ( a demo) • 山藥 vs. 木 • 黃耆 vs. 紅耆 • GTC-demo_video.wmv 10 Test1 Test2 Test3 Test4 Test5 Avg. Xiaomi 3.083 2.535 2.755 2.856 2.508 2.7474(s) Asus 2.594 2.907 3.133 2.294 2.820 2.7496(s) Smartphones recognition time GTC Taiwan 2017
  11. 11. Why Deep Learning? • With traditional hand-crafted methods, It is not easy to find representative features for easily-confused TCM herbs. • Deep learning can automatically learn about the features. Color? Shape? Texture? 11 GTC Taiwan 2017
  12. 12. CNN-CaffeNet 24 12 GTC Taiwan 2017
  13. 13. Dataset (中藥飲片) • CHM dataset collected by iPhone6 camera. • 2400 images of 24 CHMs • 1440 images for training • 960 images for testing 山藥(A1) 木薯(A2) 黃耆(B1) 紅耆(B2) 人參(C1) 西洋參(C2) 綠衣枳實(F1) 枳實(F2) 川木通(D1) 關木通(D2) 川母貝(松貝) (E1) 平母貝(E2) 川烏(G1) 草烏(G2) 黃芩(H1) 綠黃芩(H2) 半夏(I1) 水半夏(I2) 石蓮子(J1) 苦石蓮(J2) 川牛膝(K1) 味牛膝(K2) 北板藍根(L1) 南板藍根(L2) 13 GTC Taiwan 2017
  14. 14. Experimental Environment • INTEL i7-4790 CPU & 16GB RAM • NVIDIA GTX 1060 • Python • Caffe 14 GTC Taiwan 2017
  15. 15. Result Naïve CNN Method 15 Training Phase Testing Phase Input Image Pre-trained CNN Model CNN Model Feature Extraction Training Images … GTC Taiwan 2017
  16. 16. 16 80% 84% 88% 92% 96% 100% A1 A2 B1 B2 C1 C2 D1 D2 E1 E2 F1 F2 G1 G2 H1 H2 I1 I2 J1 J2 K1 K2 L1 L2 Avg CNN HCNN by AP algorithm (Average) HCNN by illustrated handbook poor results for some herbs (green bars) GTC Taiwan 2017
  17. 17. Hierarchical Clustering CNN 17 GTC Taiwan 2017
  18. 18. Result Hierarchical Clustering CNN Method 18 Training Phase Testing Phase Input Image Second-layer CNN-based Classification-1 ModelFirst-layer CNN-based Clustering Model First-layer Pre-trained Clustering Model Second-layer Pre-trained Classification Model Training Images … … CNN-based Classification-n Model Second-layer If there are more than one category in the group Data clustering GTC Taiwan 2017
  19. 19. Clustering: Affinity Propagation Training Images … Affinity Propagation algorithm Feature Extraction Each kind of herbs randomly samples images. Each kind of herbs decides an final exemplar. If the exemplar of two kind of herbs are the same, we cluster two herbs into a group. 1 2 3 19 . "Clustering by passing messages between data points". Science. 315 (5814): GTC Taiwan 2017
  20. 20. current results 20GTC Taiwan 2017
  21. 21. Usefulness of CNN? • Hand-Crafted Method • SIFT • HOG • LBP • SVM(classifier) CNN method • CaffeNet model (5 conv layers) • VGG16 model (13 conv layers) Test time(s) 1.93684 5.95769 Using five-fold cross validation to calculate accuracy 21 Method Accuracy LBP+SVM 86.85% HOG+SVM 75.31% SIFT+SVM 70.83% Method Accuracy CNN[CaffeNet] 95.69% CNN[VGG16] 95.63% GTC Taiwan 2017
  22. 22. Effect of Fine-tune • Fine-tune by pre-trained CaffeNet model (based on ImageNet data) 22 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0% 0 1 2 4 6 8 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190 200 210 220 230 240 250 Accuracy Iterations Fine-tune Re-train GTC Taiwan 2017
  23. 23. CNN vs. HCNN 80% 84% 88% 92% 96% 100% A1 A2 B1 B2 C1 C2 D1 D2 E1 E2 F1 F2 G1 G2 H1 H2 I1 I2 J1 J2 K1 K2 L1 L2 Avg CNN HCNN by AP algorithm (Average) HCNN by illustrated handbook Using five-fold cross validation to calculate accuracy 95.63% 97.54% 97.85% 88.80% 93.55% 94.5% 80% 84% 88% 92% 96% 100% A1 A2 B1 B2 C1 C2 D1 D2 E1 E2 F1 F2 G1 G2 H1 H2 I1 I2 J1 J2 K1 K2 L1 L2 Avg CNN HCNN by AP algorithm (Average) HCNN by illustrated handbook 1 2 3 4 5 6 7 8 9 10 97.65% 97.48% 97.85% 97.08% 97.85% 97.88% 97.48% 97.23% 97.65% 97.23% 23 GTC Taiwan 2017
  24. 24. Effect of Smartphones? iPhone Xiaomi Samsung Asus 24 GTC Taiwan 2017
  25. 25. Effect of Smartphones A1 A2 B1 B2 C1 C2 D1 D2 E1 E2 F1 F2 G1 G2 H1 H2 I1 I2 J1 J2 K1 K2 L1 L2 Avg. iPhone 100.00% 95.00% 85.00% 80.00% 92.50% 77.50% 95.00% 97.50% 97.50% 97.50% 90.00% 95.00% 92.50% 97.50% 92.50% 85.00% 80.00% 92.50% 100.00% 100.00% 100.00% 92.50% 87.50% 100.00% 92.60% Xiaomi 100.00% 75.00% 95.00% 70.00% 62.50% 60.00% 82.50% 100.00% 87.50% 62.50% 85.00% 77.50% 100.00% 70.00% 65.00% 87.50% 100.00% 60.00% 90.00% 95.00% 87.50% 92.50% 92.50% 90.00% 82.81% Samsung 100.00% 95.00% 85.00% 72.50% 100.00% 60.00% 100.00% 100.00% 67.50% 97.50% 100.00% 100.00% 95.00% 97.50% 100.00% 47.50% 97.50% 80.00% 100.00% 100.00% 100.00% 100.00% 87.50% 80.00% 90.10% Asus 100.00% 70.00% 90.00% 62.50% 70.00% 35.00% 87.50% 100.00% 100.00% 85.00% 97.50% 55.00% 100.00% 87.50% 50.00% 95.00% 82.50% 67.50% 77.50% 100.00% 90.00% 90.00% 87.50% 82.50% 81.77% Avg 100.00% 83.75% 88.75% 71.25% 81.25% 58.13% 91.25% 99.38% 88.13% 85.63% 93.13% 81.88% 96.88% 88.13% 76.88% 78.75% 90.00% 75.00% 91.88% 98.75% 94.38% 93.75% 88.75% 88.13% 86.82% 25 iPhone iPhone Xiaomi Samsung Asus value value The number of pixels The number of pixels GTC Taiwan 2017
  26. 26. Not enough data => data augmentation? Zoom In Zoom OutClockwise Rotation Counter-clockwise Rotation Darken Brighten 26 GTC Taiwan 2017
  27. 27. Data Augmentation (1). iPhone camera (The original training data) iPhone 1440 images (2). iPhone camera + AUG*2(Rotation) (3). iPhone camera + AUG*4(Rotation+Size) (4). iPhone camera + AUG*6(Rotation+Size+Brightness) iPhone 1440 images iPhone 10080 images iPhone 7200 images iPhone 4320 images (6). 4 smartphones camera + AUG*6 iPhone 1440 images Xiaomi 1440 images Samsung 1440 images ASUS 1440 images iPhone 10080 images Xiaomi 10080 images Samsung 10080 images ASUS 10080 images (5). 4 smartphones camera iPhone 1440 images Xiaomi 1440 images Samsung 1440 images ASUS 1440 images Model trained by 6 different training data 27 GTC Taiwan 2017
  28. 28. data augmentation vs. adding more phone data 70.00% 75.00% 80.00% 85.00% 90.00% 95.00% 100.00% iPhone Xiaomi Samsung Asus Average 1 2 3 4 5 6 Training Data 1. iPhone camera 2. iPhone camera + AUG*2(Rotation) 3. iPhone camera + AUG*4(Rotation + Size) 4. iPhone camera + AUG*6(Rotation + Size + Brightness) 5. 4 smartphones camera 6. 4 smartphones camera + AUG*6 iPhone Xiaomi Samsung Asus Average 1 92.60% 82.81% 90.10% 81.77% 86.82% 2 92.81% 84.48% 88.13% 84.27% 87.42% 3 94.27% 85.21% 88.96% 84.58% 88.26% 4* 94.48% 88.96% 91.02% 90.52% 91.24% 5* 94.06% 93.02% 95.31% 93.85% 94.06% 6 96.04% 95.83% 96.25% 94.90% 95.76% 28 GTC Taiwan 2017
  29. 29. Conclusions • Automatic recognition of 24 easily-confused CHMs on the smartphone. • Compared to traditional hand-crafted method, CNN works better! • We propose a hierarchical CNN method which automatically clusters the herbs using AP algorithm. This brings an accuracy improvement up to 5% for some TCM herbs • Differences between phones need to be considered when designing image recognition Apps on the phone 29 GTC Taiwan 2017
  30. 30. Future work • Short term • Collect data for all 300+ TCM herbs • Try with more different phones under different lighting conditions • Long term • A TCM robot assistant 30 GTC Taiwan 2017

×