It is well-known that the expressivity of a neural network depends on its architecture, with deeper networks expressing more complex functions. For ReLU networks, which are piecewise linear, the number of distinct linear regions is a natural measure of expressivity. It is possible to construct networks for which the number of linear regions grows exponentially with depth. However, we show that the expressivity of networks is in practice far below the theoretical maximum. At initialization, we prove that the average number of regions along any one-dimensional subspace grows only linearly, instead of exponentially, in the total number of neurons. More generally, the average number of regions in a k-dimensional subspace is upper bounded by the kth power of the number of neurons, irrespective of network architecture. Our theory and empirical results suggest that this behavior persists during training. We conclude that inductive bias may play a more significant role than expressivity in the success of deep networks. Joint work with Boris Hanin.
Informational Appetites + (un)Natural Statistics = “Screen Addiction”
It is a truth not yet universally acknowledged that a self-regulating system which is stable in one environment can become unstable when the environment changes. This truth is called homeostatic fragility. Mathematically, the key mechanism is sign-reversal, which converts a negative-feedback loop into a positive-feedback loop. Sign-reversal explains all sorts of self-regulatory malfunctions in biological systems: energy and salt balance, opioid analgesia, chemical dependencies, behavioral addictions like gambling, and now “screen addiction” and its brethren. A second truth is that an active learning or self-calibrating system mathematically requires “informational appetites” (Sommer) for rare but useful inputs, inputs the system thus finds interesting. Humans, uniquely among species, can artificially copy or synthesize such interesting inputs, e.g. bright colors, shiny things, pictures, or news from afar. Unfortunately, because of homeostatic fragility, over-exposure to formerly rare inputs often converts a stable and functional information-foraging instinct into compulsive over-use, for example by compelling people already made lonely and depressed by screen-based socializing to seek solace online rather than in person. While this vicious circle is rooted in the abstract mathematics of information theory, the results are all too real: worldwide, the use of artificial entertainment and communication channels, most especially interactive digital ones, is tightly correlated with rising mental misery, self-harm, and violence. It is not clear whether or how humans can learn to avoid these enticingly decalibrating channels, but it is clear that human cognition and social function will collapse otherwise. (Based on work with Criscillia Benford, e.g. Sensory Metrics of Neuromechanical Trust, Neural Computation 29, 2293–2351, 2017).
We conclude that we humans are the victims of our own success, our hands so skilled they fill the world with captivating things, our eyes so innocent they follow eagerly.