This document discusses the author's approach to classifying radio signal modulations using deep neural networks in the 2018 Army Signal Classification Challenge. It summarizes the author's 6th place solution using two deep learning models: a ResNet architecture and a CLDNN architecture. Key details provided include the dataset and evaluation metrics, model architectures and implementations, training procedures, performance on validation and test sets, and challenges faced in the competition.
Radio Signal Classification with Deep Neural Networks
1. Radio Signal Classification with Deep
Neural Networks
2018 Army Signal Classification Challenge
6th place solution
Kachi Odoemene
29 Aug 2018
2. Competition
• Army Rapid Capabilities Office (RCO)
– April 30th – August 13th 2018
– $150,000 in prizes
• Radio signal modulation classification
– Automatically identify modulation type of
received radio signal
– 24 modulation classes, including noise
class
5. Quadrature Signals
• I/Q: In-phase and Quadrature components
• 90° shift between identical periodic signals
– e.g. sine and cosine wave
Q
I
6. I/Q Modulation
• Summation of I/Q pairs results in any modulation
(frequency, amplitude, phase, etc)
• Transmitted & received radio signal represented
as I/Q components
Modulated
RF Signal
I channel, I(t)
Q channel, Q(t)
Inputs
(Modulating waves)
Carrier
wave
7. I/Q Time Domain Examples – high SNR
PSK: phase shift keying
FSK: frequency shift keying
ASK: amplitude shift keying
MSK: Minimum (frequency) shift keying
QAM: quadrature amplitude modulation
CP: continuous phase
OQ: offset quadrature
SNR: Signal to Noise
9. Dataset
• > 30 GB
– 4.32 million I/Q instances
• Dimension: 2 x 1024
– Short time window, real world conditions
– 24 modulation classes
– White noise added to signals
• 6 signal-to-noise (SNR) levels
• Test datasets (public)
– 2 sets of 100000 I/Q instances (unlabeled)
11. Goal
• Automatically identify the modulation type
of the received radio signal
– What modulation format was employed?
– Eg. Military applications:
• Spectrum surveillance, electronic warfare, and
threat analysis
– Identify modulation type of intercepted enemy
communication
12. Traditional approach
• Hand-crafted features (feature engineering)
– Higher order statistics
– Autocorrelation and spectral correlation functions
– Measures derived from instantaneous frequency,
amplitude, phase
• Mean, standard deviation, kurtosis, etc
– Small number of features (28-32)
• SVM, decision trees, ensemble, neural
networks
13. Deep Neural Networks for Radio Modulation
Recognition
(2016)
Convolutional networks outperform
expert feature-based classifiers
14. • Proposed 3 additional architectures for
modulation recognition:
– Inception
– ResNet
– Hybrid of Convolutional, Long short term memory
(LSTM), and Fully Connected (FC) Deep Neural
Network (CLDNN)
• No source code provided
• Sparse details on architecture hyperparameters
(2017)
15. Residual Unit
Layer Input
Model 1: ResNet (original)
• Image classification
• CNNs with skip (residual or
shortcut) connections
– Feed previous representations
(activations) into downstream layers
– Prevents information loss
• Enables training of deeper
networks
– 100s to 1000s of layers
He et al Deep Residual Learning for Image Recognition (ArXiv 2015)
Skipconnection
16. Input
(2x1024)
Conv
(1x128)
D-Conv
(2x1)
Avg. Pool
“ResBlock” x4
Global Avg
Pool
FC (128) x2
Softmax (24)
(Temporal)
(Spatial)
Model 1: ResNet (modified)
• Temporal convolution on each
IQ channel separately
• Depth-wise convolution (D-
Conv)
• Batch normalization after
convolutional and fully
connected (FC) layers
• Multiple residual units within
ResBlock
• # Parameters: 255,944
He et al Deep Residual Learning for Image Recognition (ArXiv 2015)
17. Model 2: CLDNN (original)
• Speech recognition
• Unified model: CNN, LSTM, FC
• CNN: reduce spectral variations of
input data
• LSTM: learn temporal structure
• FC: transform LSTM features into
output easy to classify
Sainath et al Convolutional, Long Short-Term Memory, fully
connected Deep Neural Networks (IEEE, 2015)
Convolutional
layers
Linear
layer
LSTM
layers
Fully
Connected
layers
Output targets
(1)
(2)
C
C
L
L
D
D
Dim.
red.Xt
[Xt-l,..,Xt,…,Xt+r]
18. Model 2: CLDNN (modified)
• Temporal convolution on each IQ
channel separately
• Depth-wise convolution (D-Conv)
• Batch normalization after each
convolutional and fully connected
(FC) layers
• “ConvBlock”: Conv + BatchNorm +
ReLU
• Dropout between FC layers
• # Parameters: 147,480
Sainath et al Convolutional, Long Short-Term Memory, fully
connected Deep Neural Networks (IEEE, 2015)
Input
(2x1024)
Conv
(1x128)
D-Conv
(2x1)
Avg. Pool
ConvBlock x2
Concatenate
LSTM (48)
x2FC (128)
Softmax (24)
(Temporal)
(Spatial)
19. Data preparation
• Raw data, no preprocessing
• Trained from scratch
• Data split
– Train: 80%, Valid: 13.33%, Holdout: 6.67%
20. Implementation details
• Hyper-parameter selection
– Temporal filter kernel size
– Number of FC units
• Maximum of 25 epochs, early stopping
• Adam optimizer
• Learning rate: 1e-3
• Keras (Tensorflow)
• Hardware (Personal)
– GTX 1080Ti (11 GB) GPU
– 16 GB RAM
30. Challenges
• Structural (competition organization)
– Multiple changes to scoring procedure and test set,
submission site shutdown, etc
• Time constraint
– 1 submission/day (10am) & leaderboard update
(5pm)
– On Kaggle: up to 5 submissions/day, immediate LB
update
• Technical
– Hardware failure final week of competition
• Backup: Google Colab
31. Future Efforts
• Incorporate additional features:
– Amplitude
– Phase difference
– Magnitude of Fourier transform
– Spectrogram
• Explore other architectures & ensembling methods
– Inception-like architecture: process and combine multiple
frequency scales
• Model interpretability
– DeepLIFT (Deep Learning Important FeaTures)
– LIME (Local Interpretable Model-agnostic Explanations)
– Ablation studies
Radio Signal Modulation Recognition
Radio Signal Classification with Neural Networks
Blind signal classification- little to no prior knowledge of signal being detected
Waves consists of phase, amplitude, and frequeny
Input signal is the data/information you wish to transmit. This information is mixed with a carrier wave.
Analog modulation: input signal varies continuously
Digital modulation: input signal are discrete values of 1’s and 0’s. analog data such as speech (sampled at some rate) is first compressed and then converted into bit stream (1s and 0s). This is used as the input signal.
Also include phase modulation and combinations of all three
Image from https://www.taitradioacademy.com/topic/how-does-modulation-work-1-1/
Radio signal represented as I/Q component
Modulating inputs are the data to be sent
RF carrier waves are in quadrature, 90deg shift
----- Meeting Notes (8/28/18 04:45) -----
Right down equation of received signal from OShea paper
Explain that this is transmitted signal
Panel of images (n=24)
Received I/Q samples sampled from a desired carrier frequency
Minimize logloss
Probability assigned to each class
Heavily penalizes confident(high probability) and wrong predictions more than it rewards confident (high probability) and correct predictions
the sum of log loss values for each class prediction in the observation
Nuanced view of model performance, takes into account uncertainty of predictions.
Traditional approach to automatic modulation classification
Expensive and tedious to develop analytically, also not flexible
Digital modulations e.g. HOS (moments and cumulants)
and Analog modulations for measures derived from instantaneous phase, amplitude, frequency,
General example of expert feature: compute the m-th order statistic on n-th power of the instantaneous or time delayed received signal
max value of spectral power density of normalized instantaneous amplitude, standard deviation of absolute value of nonlinear component of instantaneous phase of non
----- Meeting Notes (8/28/18 04:45) -----
extract 28-32 features
Fortunately the hard work of figuring out whether neural networks can be successfully applied to the problem of modulation recognition had been worked out by Tim O’Shea and colleagues
Harder these days to find a problem where deep learning has not already been applied
Learn features from the data directly
----- Meeting Notes (8/28/18 18:35) -----
Replace with deep neural networks
Proposal based on well performing architectures in computer vision
Like most papers posted on Arxiv, the paper did not provide source code for the models and was very light on details, hence I had to reverse engineer and run test experiments
Settled on two models
Done with 8 to 11 modulation classes, not clear whether it would scale to 24 modulations
I made modifications to original ResNet architecture, namely adding the temporal convolution
Skip connection or shortcut, which feed activations from one layer into the next layer
Allows training of deep networks
Plain networks without skip connections tend to struggle when layers are too deep, unable to choose parameters, leading to overfitting.
Residual networks on the other hand, learn the identity function because of the skip connections, which leads to better training
----- Meeting Notes (8/28/18 18:35) -----
fix skip connection
Parameters less than the number of samples, helps to regularize the model
CLDNN model first developed by Google for speech recognition applications
leverage advantages of convolution and rnn layer
CNNs to learn spectral features and reduce spectral variations in the input
LSTMs to learn temporal variations
FC to transform LSTM features into easily classifiable format
LSTMs on their own need better input features, and easily classifiable outputs, hence need help from
CLDNN model first developed by Google for speech recognition applications
leverage advantages of convolution and rnn layer
One LSTM layer
Hold out set for local evaluation
----- Meeting Notes (8/29/18 05:41) -----
no augmentation
Multiple changes to judging criteria and testset
Submission site failure
1 submission per day, compared to multiple (up to 5 on Kaggle)
----- Meeting Notes (8/28/18 18:35) -----
update 5pm
Things that I didn’t get to try
Magnitude of the square of the
----- Meeting Notes (8/28/18 04:45) -----
Magnitude of the frequency spectrum
----- Meeting Notes (8/28/18 18:35) -----
add ablation!
for organizing and hosting the Army Signal Classification Challenge
Powerful application to real world problems
Moving beyond classifiers