Hi Jennie. Thanks for reading! Yes as I said there are a lot of limitations to consider with these studies. I don't agree with everything in Ben's piece but he raises many important cautions. This is why I keep calling for a rigorous ongoing assessment of what the body of rigorous evidence says. No one study or meta analysis will ever be perfect and some are far more imperfect than others. And many folks in this field weighing in on such studies hold bias one way or another. I think what's needed at this point is something my center has done many times to sort through evidence: a consensus panel of recognized experts in research methodology to weigh in on what the body of evidence says and how it is evolving. Here is one example of what we've done in the past: https://crpe.org/key-issues-in-studying-charter-schools-and-achievement-a-review-and-suggestions-for-national-guidelines/
Thanks for writing this newsletter! I find it really interesting and informative.
I’m curious what you think about some of the criticism two studies cited here have generated. See here for criticism of the meta analysis and world bank study: https://buildcognitiveresonance.substack.com/p/something-rotten-in-ai-research?
Hi Jennie. Thanks for reading! Yes as I said there are a lot of limitations to consider with these studies. I don't agree with everything in Ben's piece but he raises many important cautions. This is why I keep calling for a rigorous ongoing assessment of what the body of rigorous evidence says. No one study or meta analysis will ever be perfect and some are far more imperfect than others. And many folks in this field weighing in on such studies hold bias one way or another. I think what's needed at this point is something my center has done many times to sort through evidence: a consensus panel of recognized experts in research methodology to weigh in on what the body of evidence says and how it is evolving. Here is one example of what we've done in the past: https://crpe.org/key-issues-in-studying-charter-schools-and-achievement-a-review-and-suggestions-for-national-guidelines/
Thanks for the thoughtful response! I appreciate it.