Fig. 1. The Functionally Separate Auto-encoder
The activation function sigmoid is
[1] (3)
The inverse function is
(4)
The function of leaky relu is
[2] (5)
The inverse function is
(6)
Why we use the activation function leaky relu and sigmoid? Because the
inverse function of relu function will lose some value when x is
negative and this will cause generation loss. When you face multi-class
problems, you can use softmax function. It is
(7)
Because the inverse function of it is difficult to derive, so we can
useinstead.
Take,as example, the derivation is as follows.
(8)
(9)
(10)