1. Q-Metric Based
Support Vector Machines
Dimension2
dp=infinity = 1 dλ=-1 = 1
de = dp=2 = 1 y=(y1,y2)
--- advances in kernel based machine learning systems ---
Dimension1
x=(x1,x2)
dλε(-1,0) = 1
dt = dp=1 = 1 dλ=0 = 1
Graph of d(x,y)=1 in 2-D Space
Method for Constructing Q-Metric Based
Support Vector Classification and Regression Machines
2009:01:24 Magdi A. Mohamed 1/23
2. Metrics
“distance indicators”
λ=-1
Max
P=INF
Euclidian
Q-Metrics P-Metrics P=2
Euclidian
Mainstream
Manhattan
Approaches
λ=0 P=1
ge
Probability
era
Av
Measures
Plausibility
ion
Believe Probability
λ=INF
ect
“confidence quantifiers”
ers
Int
ge
Q-Aggregates
er a
λ=0
Av
ion
Un
λ=-1
−1<λ<0 λ>0
λ=0
Aggregates Q-Measures
“connective operators”
2009:01:24 Magdi A. Mohamed 2/23
3. Overview
frontiers on nonlinear modeling techniques
Nonlinear
Models
Motivating Object Image Pattern
Applications Tracking Processing Recognition
Fundamental Probability Robust Neural
Theories Measure Estimation Networks
1940 Weiner Filter (MIT) Feed Forward Networks
1962 Hough Transform Filter
1960 Extended Kalman Filter (NASA) 1965 Morphological Filters (ECOLE) Shared Weight Network (IBM)
Approaches Hidden Markov Model 1979 Alternating Sequential Filter (ERIM) Back Propagation Through Time
Gaussian Sum Filter Weighted Median Filter Self Organizing Maps
Condensation Filter (MS) Order Statistics Filters Dynamic Programming Networks
1993 Particle Filter Stack Filters 1997 Support Vector Machines (ATT)
Motivating Automatic Computer Data Analysis &
Applications Control Vision Financial Predictions
Non-Additive Measures,
Fundamental Non-Linear Integrals,
Theories and Random Sets
1954 Choquet Capacities/Integral (ADIF)
1975 Sugeno Measure/Integral (TIT)
Approaches Order Weighted Average
2000 Generalized Hidden Markov Model (UMC)
2003 Q-Filters (MOT)
2005 Q-Machines (MOT)
2009:01:24 Magdi A. Mohamed 3/23
4. Q-Metrics Modeling (QMM)
Computational Intelligence Applications & Impact on State of the Art
Supervised Learning Unsupervised Learning (NP-Hard)
Supervised Learning Objective : Unsupervised Learning Objective :
Find the set of centers, Q ⊂ P, that minimizes objective criterion
Find the form, f, that minimizes objective criterion
Φ(f) = ∑ distance ( f(p), t ) Ψ(Q) = ∑ min distance ( p, q )
q∈Q
p∈P p∈P
Applications Applications
• linear/nonlinear optimization • vector quantization & cluster analysis
• sequence analysis
• automatic feature extraction
• decision making
• visualization & dimensionality reduction
• compression (lossy & loss-free)
Impact on Existing Machine Learning Paradigms
1. Feed-Forward Artificial Neural Nets • automated data labeling & data cleaning tools
2. Genetic Computing • data mining & knowledge discovery
3. Tree Classifiers • continuous adaptation (automatic tuning &
4. Dynamic Programming & Reinforcement Learning customization)
5. Hidden Markov Models
Impact on Existing Machine Learning Paradigms
6. Nearest Prototype Classification
1. Crisp and Soft Clustering Algorithms
7. Crisp and Soft K-Nearest Neighbor Algorithms
2. Self Organizing Maps
8. Discriminant Analysis
3. Adaptive Resonance Theory
9. Support Vector Machines
2009:01:24 Magdi A. Mohamed 4/23
5. Support Vector Machines
Prior Art and Problem Statement
Linear Partitioning Nonlinear Partitioning Several Kernel Functions
• Original Theory developed by Vapnik & Chervonenkis (VC Dimension) in 1974
• Boser, Guyon & Vapnik (AT&T) issued first patent (US5649068(A)) in July 15, 1997
• Several Kernel functions K(x,x’) exist (linear and nonlinear)
• Kernel functions are defined using weighted Euclidean Distance (P-Metrics, P=2)
• Fixing P=2, and other parameters (such as γ) causes critical limitations
2009:01:24 Magdi A. Mohamed 5/23
6. The Idea
Systematic Application of Q-Metrics Modeling to Support Vector Machines
• A Q-Metric is defined for computing distances in a Q-Metric Based Support
Vector Machine (QMB-SVM) network using a variable parameter λ, that is
bounded between the real values -1 and 0 resulting in an efficient distance
function covering feasible range of potential metrics. The Q-Metric is
constructed by computing a polynomial in the variable parameter λ. The
parameter λ can then be automatically optimized to discover the ideal
functionalities of the Q-Metric, based on the data to be analyzed.
• The mathematical programming (training) task is formulated as an
optimization problem where the QMB-SVM network parameters are adjusted
to minimize an overall risk criterion quantified using Q-Metrics Modeling.
2009:01:24 Magdi A. Mohamed 6/23
7. Metrics
“distance indicators”
λ=-1
Max
P=INF
Euclidian
Q-Metrics P-Metrics P=2
Q-Metrics
QMB-SVM
Manhattan
Space
λ=0 P=1
Measures
Plausibility
ion
Believe Probability
λ=INF
ect
“confidence quantifiers”
ers
Int
ge
ge
Probability
era
Q-Aggregates
er a
λ=0
Av
Av
ion
Un
λ=-1
−1<λ<0 λ>0
λ=0
Aggregates Q-Measures
“connective operators”
2009:01:24 Magdi A. Mohamed 7/23
8. Implementation
Java Applet
2009:01:24 Magdi A. Mohamed 8/23
11. 4-Dimensional XOR Data Set
Nonlinear Classification Case
2009:01:24 Magdi A. Mohamed 11/23
12. More Experimental Results
Testing Kernel Types Using X-DATA Set
Type=0 Type=1 Type=2 Type=3 Type=4
B P/ B P/ B P/ B P/ B P/
B 21 15 B 36 00 B 36 00 B 07 29 B 36 00
P/ 21 15 P/ 13 23 P/ 08 28 P/ 27 09 P/ 00 36
Acc 50.0% Acc 81.9% Acc 88.9% Acc 22.2% Acc 100%
Conventional Conventional Conventional Conventional Novel
Linear Polynomial RBF Sigmoid QMB-RBF
2009:01:24 Magdi A. Mohamed 12/23
13. More Experimental Results
Testing Over Fitting Using X-DATA Set
Novel QMB-SVC
λ=-1.00 λ=-0.75 λ=-0.50 λ=-0.25 λ=0.00
Conventional RBF-SVC
γ=0.5 γ=11 γ=111 γ=1111 γ=11111
2009:01:24 Magdi A. Mohamed 13/23
14. Advantages of QMB-SVM
Characteristics and Promises
1. computational efficiency
2. numerical stability
3. per unit calculations simplify implementations (software and hardware)
4. suitability for massive parallel implementations
5. automatic discovery of multiple metric spaces
6. consistent handling of “curse of dimensionality” concerns
7. improvement over existing kernel functions
8. usability for both classification and regression applications
9. ease of use
“A hypothesis or theory is clear, decisive, and positive, but it is believed by no one
but the man who create it. Experimental findings, on the other hand, are messy,
inexact things, which are believed by everyone except the man who did the work.”
- Harlow Shapley
2009:01:24 Magdi A. Mohamed 14/23
15. Potential Applications
one vision for suites of techniques that work together
INPUTS
EVENTS
CLASSIFIER
SENSOR
SIGNALS
SIGNALS
ACTION CODES
FEATURES
SIGNAL DATA SIGNAL
SENSOR
SENSOR PRE- PROCESSING/ POST-
FUSION
PROCESSING ANALYSIS PROCESSING
ACTIONS
SENSOR
SIGNALS
ACTION CODES
DECISIONS
OUTPUTS
EVENTS
INPUTS
FEATURES
SIGNALS
SENSOR
CLASSIFIER
SIGNAL DATA SIGNAL
SENSOR CLASSIFER DECISION
SENSOR PRE- PROCESSING/ POST- DISPLAY
FUSION FUSION CONTROL
PROCESSING ANALYSIS PROCESSING
SENSOR
SIGNALS
SIGNALS
ACTION CODES
EVENTS
INPUTS
FEATURES
SENSOR CLASSIFIER
SIGNAL DATA SIGNAL
SENSOR
SENSOR PRE- PROCESSING/ POST-
FUSION
PROCESSING ANALYSIS PROCESSING
SENSOR
Q-AGGREGATES Q-FILTERS Q-METRICS Q-FILTERS Q-AGGREGATES
Q-METRICS
2009:01:24 Magdi A. Mohamed 15/23
16. Novel
QMB-SVC
2009:01:24 Magdi A. Mohamed 16/23
17. Conventional
RBF-SVC
2009:01:24 Magdi A. Mohamed 17/23
18. Novel
QMB-SVR
2009:01:24 Magdi A. Mohamed 18/23
19. Conventional
RBF-SVR
2009:01:24 Magdi A. Mohamed 19/23
20. Novel
QMB-SVC
2009:01:24 Magdi A. Mohamed 20/23
21. Conventional
RBF-SVC
2009:01:24 Magdi A. Mohamed 21/23
22. Novel
QMB-SVR
2009:01:24 Magdi A. Mohamed 22/23
23. Conventional
RBF-SVR
2009:01:24 Magdi A. Mohamed 23/23
