Comments on: The Everything Ensemble EdgeMachine learning and learning theory researchBy: jl- December 5, 2005 Roughly speaking, this is a nice observation which can explain ensemble system performance mathematically. Before writing up we should ask the question: "Can we construct better ensembles" From this viewpoint, exactly how we measure the error rates is irrelevant. Given that we have measured them well, how would we use the information We have a set of n predictors. We know for all predictors, ei = the probability that predictor errs, and for all pairs of predictors, eij = the probability...http://hunch.net/?p=141#comment-13407 By: David Corfield- December 4, 2005 And there's no need to stop at pairs. The voting error would be bounded by 8 X prob. that 3 random hypotheses err simultaneously. And so on. I wonder how good these bounds would be for the GPs Seeger goes on to treat. The K-L distance doesn't seem to grow inordinately. On the other hand, perhaps this approach would work better with classifiers coming from different algorithms, although VC or PAC-Bayesian treatment might be messier. Would it be worth writing this up Another thought: in the...http://hunch.net/?p=141#comment-13380 By: jl- December 3, 2005 I understand: you want to bound b. And the answer is "yes", this can be done after making an additional IID data assumption. Typically, we measure the error rate of classifiers, but we could in general measure any parameterized event. This means we could treat hypothesis pairs as hypotheses and then apply the standard bounds (VC, PAC-Bayes, etc...) on the hypothesis pair space. The PAC-Bayes bound is particularly compelling because it becomes very tight when the average is over a...http://hunch.net/?p=141#comment-13272 By: David Corfield- December 2, 2005 Having bounded the voting classifier error by 4 x the probability that 2 random classifier both err, I was wondering if the latter could be bounded along the lines of the PAC-Bayesian results, e.g., theorem 3.1 of Seeger's thesis, http:www.kyb.tuebingen.mpg.debspeopleseegerpapersthesis.html, so that it is within such and such of the empirical error rate for a pair of classifiers. One would expect the gap to involve the Kullback-Leibler distance between the prior over W xW and the posterior.http://hunch.net/?p=141#comment-13242 By: jl- December 2, 2005 Sorry, I'm not quite following you. What is the precise theorem you have in mind (With the definitions of the variables)http://hunch.net/?p=141#comment-13239 By: David Corfield- December 2, 2005 I'll start a new thread as comments won't nest further, but referring to your Nov 28, is there any reason a PAC-Bayesian bound won't go through We have voting error less than or equal to 4 X double error rate for 2 hypotheses Gibbs sampled. With the usual notation, e.g. Seeger's thesis, the latter is E_(w_i,w_j)Q2 (E_(x,y)I_intersectionsign(uw_i, x) + b) not = y, where Q is the posterior distribution over W. With finitely many hypotheses one might exclude the diagonal. Of course, one...http://hunch.net/?p=141#comment-13117 |