A novel quantile Fourier neural network for time series based probabilistic forecasting.

1. Multiple Quantile Fourier Neural Network:
Time Series Based Probabilistic Forecasting
By Kostas Hatalis
hatalis@gmail.com
Dept. of Electrical & Computer Engineering
Lehigh University, Bethlehem, PA
2019
Kostas Hatalis Fourier Neural Network 2019 1 / 19

2. Motivation
Study nonparametric probabilistic Forecasting for univariate time
series (very rare).
Study regression based extrapolation:
Kostas Hatalis Fourier Neural Network 2019 2 / 19

3. Goals
1 Show how to estimate composite quantiles using a new Fourier neural
network approach.
2 Demonstrate ability to model periodic and non-periodic components
of nonstationary time series.
3 Demonstrate how to conduct multi-step probabilistic forecasts.
4 Design experiments to validate our approach for direct probabilistic
forecasting.
Kostas Hatalis Fourier Neural Network 2019 3 / 19

4. Quantile Fourier Neural Network (QFNN)
QFNN is defined by
qτ
t = f (t) + b[2]
τ +
N
k=1
aτ,k · cos (ωkt + φk) + b
[1]
k
where given time t as the input, it attempts to predict the τ-level quantile.
QFNN is loosely modeling a time series as a partial Fourier cosine series
x(t) = A0 +
N
n=1
An cos(nω0t + φn)
where ω0 = 2π
T , T is the period of the signal x(t), and A0, An, and φn are
real numbers. Same optimization function as SPNN:
E =
1
NM
N
t=1
M
m=1
τm(yt − ˆq
(τm)
t ) + α log 1 + exp −
yt − ˆq
(τm)
t
α
.
(1)
Kostas Hatalis Fourier Neural Network 2019 4 / 19

5. Quantile Fourier Neural Network (QFNN)
𝑡
cos
cos
f
…
…
𝑞
𝑞
𝑞
𝑏[ ]
𝑊[ ]
𝑊[ ]
𝑏[ ]
𝑏[ ]
𝑏[ ]
𝑏[ ]
𝑏[ ]
cos
𝑏[ ]
Kostas Hatalis Fourier Neural Network 2019 5 / 19

6. Quantile Fourier Neural Network (QFNN)
Time Series
𝑌 > 10?𝑌 > 10?
𝑌 ∈ [0,10]
Normalize
𝑌 ∈ [0,10]
Yes
No
Partition Generator
Normalize Training and Testing Times
Parameter
Initialization
Forward
Propagation
Backward
Propagation
Update
Parameters
Max epochsMax epochs
reached?
Training Phase
Trained QFNN
Model
Test Set
Predictions
Testing Phase
Multiplicative
seasonality?
Yes, apply log filter.
No
Testing set. Training set.
Training set.Testing set.
No
Yes
Reverse
preprocessing if
needed.
Kostas Hatalis Fourier Neural Network 2019 6 / 19

7. Dropout Regularization
This can significantly reduce overfitting and gives major improvements
over other regularization methods.
For any connection, occurrence of dropout has a Bernoulli distribution
with probability rate p of being 1 (not being being dropped).
Kostas Hatalis Fourier Neural Network 2019 7 / 19

8. Case Studies
Table: Datasets used in the experiments.
Case Study Target Samples Granularity
1 Air Passengers 144 Month
2 Sunspots 318 Year
3 Real-Time Load Demand 744 Hour
4 Internet Traffic Data (in bits) 686 Hour
5 Apple Closing Stock Price 1581 Day
6 Solar Power 760 Hour
7 Wind Power 744 Hour
8 Ocean Wave Elevation 400 Second
Kostas Hatalis Fourier Neural Network 2019 8 / 19

9. Case Studies
Estimate 100 (equally spaced) quantiles.
For comparison plot median quantile.
Benchmark Methods:
Uniform Method (UM)
Persistence Method (PM)
Support Vector Quantile Regression (SVQR)
Quantile Regression Neural Network (QRNN/SPNN)
Exponential Smoothing with Trend and Seasonality (ETS)
ARIMA
SARIMA
Kostas Hatalis Fourier Neural Network 2019 9 / 19

10. Grid Search Dropout Rate
1 2 3 4 5 6 7 8
Data Set
5
10
15
20
25
30
35
40
45
50
55
60
DropoutRate(%)
0.14 0.64 -0.78 0.65 -0.09 -1.37 -1.22 -0.68
0.06 0.54 1.23 0.85 -1.22 -0.96 -1.11 -0.58
-0.10 0.26 -0.16 0.27 -0.14 -1.11 -1.04 -0.57
2.04 0.90 0.36 1.12 -0.70 -0.69 -0.94 -0.51
0.98 0.59 -1.30 -0.13 -1.05 -0.48 -0.32 -0.46
0.79 -0.09 -0.49 -0.51 0.80 -0.21 -0.03 -0.36
-0.51 -0.05 0.65 0.51 0.14 0.15 -0.12 -0.36
0.11 0.75 0.99 0.63 0.04 0.25 0.25 -0.28
-2.02 -0.23 -1.70 -0.42 0.82 0.56 0.52 -0.24
-0.69 0.65 -0.62 -1.20 0.48 1.15 1.07 -0.18
-0.63 -1.67 1.38 0.60 -1.26 0.72 1.05 2.04
-0.15 -2.29 0.43 -2.37 2.17 1.99 1.90 2.19
1.6
0.8
0.0
0.8
1.6
Heat-map of the grid search results for dropout rate in the QFNN model on
estimating median values. QS measures have been standardized to show
comparison between datasets.
Kostas Hatalis Fourier Neural Network 2019 10 / 19

11. Case Study: Air Passengers
1949 1951 1953 1955 1957 1959
Time (years)
0
100
200
300
400
500
600
AirPassengers
Training Samples Testing Samples
QRNN
QFNN
SARIMA
ETS
Kostas Hatalis Fourier Neural Network 2019 11 / 19

12. Case Study: Air Passenger (PIs)
1949 1951 1953 1955 1957 1959
Time (years)
100
200
300
400
500
600
700
800
AirPassengers
Training Samples Testing Samples
Kostas Hatalis Fourier Neural Network 2019 12 / 19

13. Case Study: Load
2017-01-03 2017-01-06 2017-01-09 2017-01-12 2017-01-15 2017-01-18 2017-01-21 2017-01-24 2017-01-27 2017-01-30
Time (hours)
0
5000
10000
15000
20000
25000
30000
LoadDemand(MW)
Training Samples Testing Samples
QRNN
QFNN
SARIMA
ETS
Kostas Hatalis Fourier Neural Network 2019 13 / 19

14. Case Study: Load
2017-01-03 2017-01-06 2017-01-09 2017-01-12 2017-01-15 2017-01-18 2017-01-21 2017-01-24 2017-01-27 2017-01-30
Time (hours)
6000
8000
10000
12000
14000
16000
18000
20000
22000
24000
LoadDemand(MW)
Training Samples Testing Samples
Kostas Hatalis Fourier Neural Network 2019 14 / 19

15. Case Study: Sunspots
1703 1743 1783 1823 1863 1903 1943 1983
Time (years)
0
50
100
150
200
250
Sunspots
Training Samples Testing Samples
QRNN
QFNN
SARIMA
ETS
Kostas Hatalis Fourier Neural Network 2019 15 / 19

16. Case Study: Wave
00:00:10 00:00:40 00:01:10 00:01:40 00:02:10 00:02:40 00:03:10
Time (seconds)
4
2
0
2
4
WaveElevation(meters)
Training Samples Testing Samples
QRNN
QFNN
SARIMA
ETS
Kostas Hatalis Fourier Neural Network 2019 16 / 19

17. Case Study: Stock Price
2013 2014 2015 2016 2017 2018
Time (days)
0
50
100
150
200
250
300
Price($)
Training Samples Testing Samples
QRNN
QFNN
SARIMA
ETS
Kostas Hatalis Fourier Neural Network 2019 17 / 19

18. Conclusion
First time studying Fourier Neural Networks in the context of probabilistic
forecasting.
Advantages
Can estimate trend, cycles, and seasonality almost perfectly.
Excellent reliability and sharpness.
Can conduct indefinite multi-step forecasts.
Disadvantages (Future work could address these.)
Performs poor if components change over time (eg. if frequency or
period size change).
Unable to capture chaotic patterns such as wind or stock prices.
Kostas Hatalis Fourier Neural Network 2019 18 / 19

19. Work Done
Probabilistic Forecasting for Time Series
[1] Kostas Hatalis and Shalinee Kishore. A Composite Quantile Fourier
Neural Network for Multi-Step Probabilistic Forecasting of Nonstationary
Univariate Time. In-Submission.
Kostas Hatalis Fourier Neural Network 2019 19 / 19