NeuroCOLT workshop
on
Generalisation Bounds Less than 0.5
Windsor, 29 April - 2 May 2002
Cumberland Lodge

"PAC-Bayes for Bayesian Architectures"

Matthias Seeger, Edinburgh University

Approximate Bayesian Gaussian process (GP) classification techniques are powerful and easy-to-use nonparametric learning methods. Based on proper probabilistic process models, they can easily be embedded in probabilistic frameworks for model selection, feature selection, etc. In this paper, by applying McAllester's PAC-Bayesian theorem \cite{mcallester:99}, we prove PAC generalization error bounds for a wide range of approximate Bayesian GP classification techniques. As is shown in experiments on a real-world task, the bounds can be very tight for moderate training sample sizes. We show that our results apply also to a class of sparse Bayesian GPC approximations and thereby provide very tight generalization error guarantees for learning methods of high practical significance. Our results give a strong learning-theoretical motivation for Bayesian GP techniques.