I've pondered this for a while. One solution would be, imo: 1. Maintain a large set of questions that are mechanically markable (e.g. no room for bias/teacher

I've pondered this for a while. One solution would be, imo: 1. Maintain a large set of questions that are mechanically markable (e.g. no room for bias/teacher choice in marking). You can expand this set by a small percentage each year. 2. Randomly assemble each student's exam every year by sampling questions without replacement. 3. Use trueskill to simultaneously estimate question difficulty and student skill. 4. Report skills as the primary metric. I think this would work, although of course writing many questions is challenging.