Fig. 1. The Functionally Separate Auto-encoder
The activation function sigmoid is
[1] (3)
The inverse function is
(4)
The function of leaky relu is
[2] (5)
The inverse function is
(6)
Why we use the activation function leaky relu and sigmoid? Because the inverse function of relu function will lose some value when x is negative and this will cause generation loss. When you face multi-class problems, you can use softmax function. It is
(7)
Because the inverse function of it is difficult to derive, so we can useinstead.
Take,as example, the derivation is as follows.
(8)
(9)
(10)