Causative Assessments

There’s a great amount of debate about what the SAT measures. Does it measure income? Intelligence? Preparation? Or some combination thereof? But for all the buzz about what the SAT measures, no one ever asks what the SAT changes.

When you’re measuring the height of a plant, the plant could care less. At least, until you start selecting for or against height. But testing students will affect what you are measuring, because people are sentient. So no test can be a truly accurate measurement of achievement, because the very existence of said test distorts whatever measure of achievement you are using. It’s like Heisenberg’s uncertainty principle; by observing it, it changes.¹

If the existence of a test affects how you learn, we should demand that it has a positive effect on learning. Especially if you still have things to learn down the road. And there is always something more to learn.

Preparing for the tests we take is also incredibly time consuming. Let’s say that in junior year, you spend two hours a week preparing for the SAT, from August to March. That comes out to be about 60 hours. If a kid wasted that much time on video games his parents would be mad. But when it comes down to it video games teach you more than the SAT.

Sixty hours is a lot of time. It was enough for me to learn Linux and become fairly proficient in it. I took less time to write and ship a functional first version of mapm — library, command-line interface, and documentation. Not every chunk of sixty hours will be this valuable. But multiply a lot of tests to study for and the entire population of the United States, and you get a sense of how much productivity has been lost.

It’s true that whatever high school students do mostly amounts to chump change. But just as we give kids allowances to promote responsible spending, we also have to promote responsible learning.

Some tests have been able to do this. I think any major STEM contest (especially the AMCs) does this well. You can’t just “study to the test”, or more accurately, the only effective way to study is to genuinely learn. For the AMCs, you don’t agonize over the specific format or types of questions,² you just work on problem-solving and learning new things.

In all fairness you can’t reliably “hack” the SATs anyway. Evidence of a significant benefit from prep programs is limited at best. But the problem is, if the SAT is an IQ test that also determines your future, two things are true: you have extremely high incentive to study for the SATs, and you can’t get better at the SATs.

So what should a good test look like? Pretty much the opposite of the SATs. It should not be too high-stakes, because even an initially positive distortive effect taken to an extreme becomes harmful, it should not be something worth specifically studying for, because it should measure things that align with skills people find worth developing, and it should also be possible (though not guaranteed!) to improve significantly with hard work. And perhaps most importantly, it should be fun.