Comments on: The Everything Ensemble EdgeMachine learning and learning theory researchBy: jl- November 28, 2005 That's a good point. The first level of analysis here should probable be unprobabilistic. One analysis along these lines is lemma 4 of this paper. It appears that lemma 4 generalizes to something like the following: Let b = the expected proportion of bad pairs = Prx,y D, i,j 1,...,n (hi y and hj y) Let t be any threshold on the number of binary errors required to induce an overall error. (For example, this is 12 the number of voters in a strict majority vote.) The the following...http://hunch.net/?p=141#comment-12928 By: David Corfield- November 28, 2005 Tighter bounds for ensembles can only come for more information than the individual error rates of their members. If you have a library of classifiers which you know to be 95% accurate, if the world conspires against you, you could end up with 90% voting accuracy. On the other hand, if you knew that a number of classifiers in the library err independently of one another, you could form a very accurate voting classifier. So how does you integrate covariance of error information into a PAC resulthttp://hunch.net/?p=141#comment-12926 By: jl- November 26, 2005 Everything you say makes sense, but it does not address the theorypractice gap I see. The difficulty with thinking "margin theory tells us to use ensembles" is that every margin bound I have studied in detail has the property that I can extract a better non-ensemble bound in any particular application. This holds for the original margin bound, an improvement on it, and the PAC-Bayes margin bound. In each case, you can construct a stochastic classifier which will have a smaller bound than the.http://hunch.net/?p=141#comment-12803 |