A British exam board plans to grade written responses to English tests with computers. It’s the first time machines have been used to assess language rather than merely mark multiple choice tests.
At the moment, the marking will only be used for a test designed to check if students have the English skills necessary for going into university. It’s not part of the mainstream educational system and is mainly aimed at adult learners and foreign students.
The computers will “scan” test responses, presumably in the form of analyzing text a student has typed in rather than attempting optical character recognition of handwriting. (If that had been used, I’d have failed every exam going.) The machines will then search for text which matches a range of possible correct solutions and award marks where it finds matches.
The system is being billed as a way of avoiding the unreliability of human marking and may well have been prompted by severe delays in marking national tests for younger schoolchildren last year. But critics rightly point out that the system is flawed because there are so many linguistically correct ways of expressing the same facts or concepts. The big fear is that students will wind up trying to figure out the correct trigger phrases to get marks, effectively limiting rather than expanding their range of communication.
Automated marking of writing will always be troublesome simply because human language often doesn’t follow logical rules, particularly mathematical ones. Just look at the controversial singular “they” (for example, a sign telling children that “If anyone misbehaves they will be punished.”)
From a communications standpoint, “they” in this context is by far the best solution, avoiding the need to either guess the gender (“he will be punished”), use unwieldy and unnatural language (“he or she will be punished”) or make up ugly expressions (“s/he will be punished”.) But to a computer, or to the type of grammar purist who insists on logical rules, this simply can’t work because “they” is plural and “anyone” is singular.
As a writer, I’m relieved that computers can’t yet master human language. Like most wordsmiths, I still shudder at reading Roald Dahl’s 1954 short story The Great Automatic Grammatizator in which a computer is trained to automatically write novels in any style, leaving human writers unable to feed their families. (For a more detailed explanation of the story and a look at how it matches up to today’s technology, check out this piece from Durham University.)
In case you were wondering, the staff behind British exam boards are not all shining models of linguistic clarity. A research director at the company which owns one board is quoted as saying “It’s extremely unlikely that automated systems will not be deployed extensively in educational assessment.”
I don’t know what the computer would say, but my elementary school teacher Mrs Dobney would have gone haywire at a double negative such as that one.