A couple of points:
1) First off, Kane did not include all of these schools in his matched study, only the ones that are oversubscribed. Second, there is a huge discrepency between the high perfoming schools and the low performing schools, as Pallas acknowledges. I think it's better to flesh out the picture, as I've done below. My guess is that Klein was using data from the study without reflecting on the fact that all charter schools in Boston were not included.
2) Pallas argues that what makes a high quality school is the percentage of kids who score advanced. As we can see in the chart above, only one charter school (Excel) in Boston matched Brookline's 44% in math last year. In ELA, a number of schools equalled or surpassed Brookline's 24%. As a teacher, I can see the validity in that point. I always want all of my students achieving at the highest level. But what about the argument that the goal of a good school is to get all of their students to be proficient or better? In that case, Brookline is matched by a number of charter schools. This is how the Boston Globe compares schools each year. I think both methods are important.
3) Note that 44% advanced on 8th grade math is 10th highest in the state. The highest percentage was 55% (Wayland). Brookline's ranking on ELA was not quite as high.
I think charter schools need to worry about performance at a number of levels:
1) Are we scoring higher than Boston? - lowest bar
2) Are we scoring higher than Massachusetts?
3) Are we one of the top schools statewide in A +P?
4) Are we pushing as many kids into Advanced as possible without slipping in #3?
Update: Thanks to Aaron Pallas for catching an error with Roxbury Prep's data on the ELA chart. The new chart should be correct.