9 ms·

Codeup CEO here. We actually did some brief testing with classical IQ tests and saw two problems: 1) IQ tests are really long and expensive, so difficult to im

Codeup CEO here. We actually did some brief testing with classical IQ tests and saw two problems: 1) IQ tests are really long and expensive, so difficult to implement 2) They didn't correlate with performance as well as algebra.

Paul_S 12y ago
I've read your answer about your methodology of gauging "programming aptitude" and I'm worried it might be susceptible to your judges' biases - it being purely subjective. I'll put more faith in your point number 2 if you come up with a more objective measure.
- mgirdley 12y ago
  Agreed that an objective measurement would be ideal and this is second best. However, we've had a hard time creating or locating an objective measurement of general programming skill. As demonstrated by the industry's non-use of standardized tests for programming, I don't think anyone else has either. If you have a lead on something, would love to see it.
  - strict9 12y ago
    Trying to create an accurate measurement of programming skill is far more difficult than looking for correlation on tests compared with some other subjective measure. This misguided attempt at instrumenting "skill" could prevent highly capable individuals from entering the field who don't have a solid math foundation, for societal or other reasons.
    - sdenton4 12y ago
      OTOH, identifying a decent mathematical thinking background as a predictor of CS potential is a pretty big win at a societal level. Teaching math is relatively cheap; you don't even have to buy computers...
murbard2 12y ago
How do you know they didn't correlate with performance as well as algebra if they were too difficult to implement?
- rhiever 12y ago
  Because they tried them, then decided to not continue trying them because they didn't correlate as well and were difficult and expensive to implement.
  - murbard2 12y ago
    There main test has 60 datapoints... how many tries do you think they made exactly before deciding they didn't correlate as well? Yes, IQ tests are expensive, but if you buy one test, the marginal cost of giving to an extra student is basically 0. I suspect it wasn't tested at all.
    - mzl 12y ago
      A real IQ test takes at least 2 hours of 1-on-1 time with a psycholigist to administer, so in the event that someone actually wants to do real iq testing the marginal cost is quite high.
      - murbard2 12y ago
        That's what psychometricians want you too believe. There is no scientific reason it needs to be administered by a human or be so long.
      - barry-cotter 12y ago
        The ASVAB, GRE, LSAT, MCAT, and SAT all have correlations with psychologist administered IQ tests of around 0.9. WORDSUM, which is a ten word vocabulary test has a correlation of 0.71[1]. [1]Every time I use the WORDSUM variable from the GSS people will complain that a score on a 10-question vocabulary test is not a good measure of intelligence. The reality is that “good” is too imprecise a term. The correlation between adult IQ and WORDSUM = 0.71. The source for this number is a 1980 paper, The Enduring Effects of Education on Verbal Skills. I’ve reproduced the relevant table… The Enduring Effect of Education on Verbal Skills http://www.jstor.org/discover/10.2307/2112492?uid=3737800&uid=2&uid=4&sid=21104842098621 http://www.jstor.org/discover/10.2307/2112492?uid=3737800&ui...
fredophile 12y ago
I remember reading about a similar but different test that had good results. I can't find the source so this is just from what I remember. Before starting a course the students were shown short snippets of code and asked simple questions about it. They found that consistent answers were a predictor of who would do well. The theory was that people who are able to construct a mental model for the problems ended up being good at programming tasks, even if the models they had made didn't match what code actually does. Have you heard of or tried this approach?
- brians 12y ago
  I think you're remembering some papers from Dehnadi and Bornat, starting with http://www.eis.mdx.ac.uk/research/PhDArea/saeed/paper1.pdf http://www.eis.mdx.ac.uk/research/PhDArea/saeed/paper1.pdf
  - fredophile 12y ago
    That looks like the paper I was thinking of. Thanks for including te link.
    - rebelshrug 12y ago
      http://retractionwatch.com/2014/07/18/the-camel-doesnt-have-two-humps-programming-aptitude-test-canned-for-overzealous-conclusion/ http://retractionwatch.com/2014/07/18/the-camel-doesnt-have-...

Paul_S 12y ago

I've read your answer about your methodology of gauging "programming aptitude" and I'm worried it might be susceptible to your judges' biases - it being purely subjective. I'll put more faith in your point number 2 if you come up with a more objective measure.

mgirdley 12y ago
Agreed that an objective measurement would be ideal and this is second best. However, we've had a hard time creating or locating an objective measurement of general programming skill. As demonstrated by the industry's non-use of standardized tests for programming, I don't think anyone else has either. If you have a lead on something, would love to see it.
- strict9 12y ago
  Trying to create an accurate measurement of programming skill is far more difficult than looking for correlation on tests compared with some other subjective measure. This misguided attempt at instrumenting "skill" could prevent highly capable individuals from entering the field who don't have a solid math foundation, for societal or other reasons.
  - sdenton4 12y ago
    OTOH, identifying a decent mathematical thinking background as a predictor of CS potential is a pretty big win at a societal level. Teaching math is relatively cheap; you don't even have to buy computers...

murbard2 12y ago

How do you know they didn't correlate with performance as well as algebra if they were too difficult to implement?

rhiever 12y ago
Because they tried them, then decided to not continue trying them because they didn't correlate as well and were difficult and expensive to implement.
- murbard2 12y ago
  There main test has 60 datapoints... how many tries do you think they made exactly before deciding they didn't correlate as well? Yes, IQ tests are expensive, but if you buy one test, the marginal cost of giving to an extra student is basically 0. I suspect it wasn't tested at all.
  - mzl 12y ago
    A real IQ test takes at least 2 hours of 1-on-1 time with a psycholigist to administer, so in the event that someone actually wants to do real iq testing the marginal cost is quite high.
    - murbard2 12y ago
      That's what psychometricians want you too believe. There is no scientific reason it needs to be administered by a human or be so long.
    - barry-cotter 12y ago
      The ASVAB, GRE, LSAT, MCAT, and SAT all have correlations with psychologist administered IQ tests of around 0.9. WORDSUM, which is a ten word vocabulary test has a correlation of 0.71[1]. [1]Every time I use the WORDSUM variable from the GSS people will complain that a score on a 10-question vocabulary test is not a good measure of intelligence. The reality is that “good” is too imprecise a term. The correlation between adult IQ and WORDSUM = 0.71. The source for this number is a 1980 paper, The Enduring Effects of Education on Verbal Skills. I’ve reproduced the relevant table… The Enduring Effect of Education on Verbal Skills http://www.jstor.org/discover/10.2307/2112492?uid=3737800&uid=2&uid=4&sid=21104842098621 http://www.jstor.org/discover/10.2307/2112492?uid=3737800&ui...

fredophile 12y ago

I remember reading about a similar but different test that had good results. I can't find the source so this is just from what I remember. Before starting a course the students were shown short snippets of code and asked simple questions about it. They found that consistent answers were a predictor of who would do well. The theory was that people who are able to construct a mental model for the problems ended up being good at programming tasks, even if the models they had made didn't match what code actually does. Have you heard of or tried this approach?

brians 12y ago
I think you're remembering some papers from Dehnadi and Bornat, starting with http://www.eis.mdx.ac.uk/research/PhDArea/saeed/paper1.pdf http://www.eis.mdx.ac.uk/research/PhDArea/saeed/paper1.pdf
- fredophile 12y ago
  That looks like the paper I was thinking of. Thanks for including te link.
  - rebelshrug 12y ago
    http://retractionwatch.com/2014/07/18/the-camel-doesnt-have-two-humps-programming-aptitude-test-canned-for-overzealous-conclusion/ http://retractionwatch.com/2014/07/18/the-camel-doesnt-have-...