Using Sigmoid Functions

The sigmoid function is defined as,

\[ S(x) = \frac{1}{1 + \exp(-kx)} \,,\]

and creates an "S" shape when plotted which ranges from \(0\) to \(1\). As we vary \(k\), we see that the slope increases.

\(k\) is sometimes called the gain.

Therefore \(S(x)\) is often used to normalize values in \(\mathbb{R}\) to \(\mathbb{R}^{[0, 1]}\). A variation of \(S(x)\) is where we can scale the sigmoid to \(\mathbb{R}^{[0, \mu]}\),

\[ S_\mu(x) = \frac{\mu}{1 + \exp(-kx)}\,.\]

Let \(\mu = 10\) and \(k = 1\), then \(S_\mu(x)\) yields the plot below.

You likely already see the power of this function as it has a domain over the reals from \(-\infty\) to \(\infty\) and is bijective with a codomain over the reals from \(0\) to \(1\). Furthermore, as we have seen, varying \(k\) will impact the slope of the function.

In a finite environment, \(S(x)\) may appear to be surjective, but it indeed is bijective and has an inverse – the Logit function,

\[ S^{-1}(x) = \frac{1}{k}\log\Big( \frac{x}{1-x} \Big) = \frac{1}{k}\Big( \log(x) - \log(1 - x)\Big)\]