Eur. Phys. J. B 10, 583-588 (1999)
https://doi.org/10.1007/s100510050889

Statistical physics and practical training of soft-committee machines

¹ Institut für Theoretische Physik, Julius-Maximilians-Universität Würzburg, Am Hubland, 97074 Würzburg, Germany,
² Neural Computing Research Group, Aston University, Aston Triangle, Birmingham B4 7ET, UK

Corresponding author: ^a ahr@physik.uni-wuerzburg.de

Received: 16 December 1998
Published online: 15 August 1999

Abstract

Equilibrium states of large layered neural networks with differentiable activation function and a single, linear output unit are investigated using the replica formalism. The quenched free energy of a student network with a very large number of hidden units learning a rule of perfectly matching complexity is calculated analytically. The system undergoes a first order phase transition from unspecialized to specialized student configurations at a critical size of the training set. Computer simulations of learning by stochastic gradient descent from a fixed training set demonstrate that the equilibrium results describe quantitatively the plateau states which occur in practical training procedures at sufficiently small but finite learning rates.