8. Activation Functions for DNNs
Input
x
Activation
function
Weight
Output
y
Conv.
Activation
function
Input
x
Output
y
Activation functions
Sigmoid tanh ReLU
𝜎𝜎 𝑥𝑥 =
1
1 + 𝑒𝑒−𝑥𝑥
max(𝑥𝑥, 0)
10. WiG: Weighted Sigmoid Gate (Proposed)
Existing activation functions are
element-wise function.
Sigmoid Gated Network can be
used as activation function.
Weight
Activation
function
Weight
Activation
networkunit
Proposed WiG (Weighted sigmoid gate unit)
W ×
Wg
WiG activation unit
It is compatible to existing activation functions.
It includes the ReLU.
Sigmoid
W
Wg
×
My recommendation is:
You can improve the network performance just by
replacing the ReLU by the proposed WiG.