8 ms·

I foresee a dystopian education outcome: 1. Classifiers like this are used to flag possible AI-generated text 2. Non-technical users (teachers) treat this lik

I foresee a dystopian education outcome: 1. Classifiers like this are used to flag possible AI-generated text 2. Non-technical users (teachers) treat this like a 100% certainty 3. Students pay the price. Especially with a true positive rate of only 26% and a false positive rate of 9%, this seems next to useless.

cjbgkagh 4y ago
> false positive rate of 9% Yeah, that is useless. You couldn't punish based on that alone and students will quickly figure out to never confess.
TheRealPomax 4y ago
So, status quo then? This is already the case for educational software that's used to detect plagiarism. People get wrongly flagged, and then you'll have to plead your case. But the times software like this finds actual problems vastly outnumbers of times it doesn't, and when you choice is between "passing kids/undergrads who cheat the system" and "the occasional arbitration", you go with the latter. Schools don't pay teachers anywhere near enough to not use these tools.
- michaericalribo 4y ago
  Given the published true and false positive rates, it's clear that the true positives do not "vastly outnumber" false positives.
- notahacker 4y ago
  > This is already the case for educational software that's used to detect plagiarism. People get wrongly flagged, and then you'll have to plead your case. How often is that the case though? A while since I've had to worry about it, but I thought plagiarism detection generally worked on the principle of looking for the majority of the content being literal matches with existing material out there with only a few small edits, which - unlike using some "AIish" turns of phrase a bot wrongly attributes to humans 9% of the time and correctly attributes to AI with a not much better success rate - is pretty hard to do accidentally.
  - i_have_an_idea 4y ago
    A long time ago when I was a student, I would run my papers through Turnitin before submitting. The tool would sometimes mark my (completely original) work as high as mid 20% similarity. As a result, I have taken out quotes and citations to appease it and not have to deal with the hassle. I expect modern day students will resort to similar measures.
    - notahacker 4y ago
      IIRC the marker got the same visualization that you used to take out quotes and citations that highlighted that the similar bits were in fact quotes and citations! Maybe high school is a different matter, but I'm pretty sure even the most technophobic academic knows that jargon, terse definitions and the odd citation overlapping with stuff other people have written is going to make a similarity of at least 10% pretty much inevitable, especially when the purpose of the exercise is to show you understand the core material well enough to cite and paraphrase and compare it, not to generate novel academic insight or show you understood the field so well you didn't need to refer back to the source material. The people they were actually after were the ones that downloaded something off essaybank, removed a couple of paragraphs and rewrote the intro to match the given title and ended up with 80%+ similarity
- PeterisP 4y ago
  Currently the false positive rate is far lower. E.g. I get 500-ish submissions over a school year then a 1% false positive rate would mean I'd falsely accuse 5 innocent students annually, which isn't acceptable at all - and a 9% FP rate is so high that's even not worth investigating; do you know of any grader who has the spare time to begin formal proceedings/extra reviews/investigation for 9% of their homework? For plagiarism suspicions at least the verification is simple and quick (just take a look at the identified likely source, you can get a reasonable impression in minutes) - I can't even imagine what work would be required to properly verify ones flagged by this classifier..
  - TheRealPomax 4y ago
    > I can't even imagine what work would be required to properly verify ones flagged by this classifier. Yet.
    - flatline 4y ago
      At the same time the classifier is improving, the generative models are improving. It’s a classic arms race and this equilibrium is not likely to shift much either way. We are talking about models that approximate human behavior with a high degree of accuracy, I think the goal would be to make them indistinguishable in any meaningful way.
    - PeterisP 4y ago
      Can you elaborate? I don't think that this is something that can change through tech advances for the classifiers - in all cases the classifier is just flagging for investigation, it's not sufficient for any action. For plagiarism, appropriate evidence comes from a person comparing the submission with the possible source of plagiarism. For this one, the proper evidence would require getting confirmation that the student actually generated that data - e.g. identifying the exact tool and prompt that was used, or logs from the students' computer showing that this was done, or logs from the text generation service provider. All of those are quite tricky to get and perhaps even not possible.
  - Fomite 4y ago
dougmwne 4y ago
The cheating students who know how to use the classifier will be the big winners.
LarryMullins 4y ago
> 2. Non-technical users (teachers) treat this like a 100% certainty This is the part that needs to be addressed the most. Teachers can't offload their critical reasoning to the computer. They should ask their students to write things in class and get a feeling for what those individual students are capable of. Then those that turn in essays written at 10x their normal writing level will be obvious, without the use of any automated cheat detectors. I was once accused of cheating by a computer; my friend and I both turned in assignments that used do-while loops, which the computer thought was so statistically unlikely that we surely must have worked together on the assignment. But the explanation was straight forward; I had been evangelizing the aesthetic virtue of do-while loops to anybody that would listen to me, and my friend had been persuaded. Thankfully the professor understood this once he compared the two submissions himself and realized we didn't even use the do-while loop in the same part of the program. There was almost no similarity between the two submissions besides the statistically unlikely but completely innocuous use of do-while loops. It's a good thing my professor used common sense instead of blindly trusting the computer.
- Gigachad 4y ago
  I once got called up because my work was flagged as 100% copied. I had uploaded it, made a mistake so I deleted it and uploaded a new file. Second file was flagged as copied. Was able to explain it by pointing at the screen that was claiming I plagiarized my own name.
- londons_explore 4y ago
  > blindly trusting the computer. Professors blindly trust the computer not out of laziness, but to protect themselves from accusations of unfairness... "The work was detected as plagiarism, but the professor overrode it for the pretty girl in class, but not for me"
  - mitchdoogle 4y ago
    Seems like something like this should only be used as a first-level filter. If the writing doesn't pass, it warrants more investigation. If no proof of plagiarism is found, then there's nothing else to do and professor must pass the student
thewataccount 4y ago
Hopefully they just flag relevant sections. Essay/Plagiarism checkers already exist, although in my experience professors were reasonable. For example I had a paragraph or two get flagged as being very similar to another paper - but both papers were about a fairly niche topic (involving therapy animals) and we had both used the relevant quotes from the study conclusions from one of only a few decent sources at the time - so of course they were going to be very similar. Given that most essays are about roughly the same set of topics, and there are literally hundreds of thousands of students writing these - I wonder how many variations are even possible for humans to write as I would expect us to converge on similar essays?
- michaericalribo 4y ago
  Plagiarism is easier to verify, because you can directly compare with the plagiarized source material
  - thewataccount 4y ago
    Absolutely. I think it may have to end up more as a statistics thing with behaviour. For example: "Tom had a single paragraph flag as possibly generated" vs "Every single paper Tom writes has paragraphs flag" Basically we might have to move to detecting statistical outliers as cheating. Now whether the tools/teachers will understand/actually do that - we can only hope....
    - blueblimp 4y ago
      That's a good point: the effectiveness at detecting AI generation is probably going to depend strongly on the length of the text.
amelius 4y ago
Solution: just write your texts with a bit less confidence than gpt3 would.
janalsncm 4y ago
I urge anyone with time to write to tech journalists explaining why this is so bad. Given previous coverage of GPTZero they don’t seem to be asking the right questions.
claytonjy 4y ago
Is there a longer-form paper on this yet? TPR (P(T|AI)) and FPR (P(T|H)) are useful, but what I really want is the probability that a piece flagged as AI-generated is indeed AI-generated, i.e. P(AI|T). Per Bayes rule I'm missing P(AI), the portion of the challenger set that was produced by AI. If we assume the challenger set is evenly split 50-50, that means P(AI|T) = P(T|AI)P(AI)/P(T) = (0.26)(0.5)/(0.26+0.09) ~ 37% So slightly better than a 1/3 chance of the flagged text actually being AI-generated. They say the web-app uses a confidence threshold to keep the FPR low, so maybe these numbers get a bit better, but very far from being used as a detector anywhere it matters.
- drc500free 4y ago
  Precision is impossible to calculate without knowing P(AI), which is use-case specific. Source: Spent 10 years trying to explain this to government people who insisted that someone tell them Precision based purely on the classifier accuracy without considering usage.
- TchoBeer 4y ago
  >Per Bayes rule I'm missing P(AI), the portion of the challenger set that was produced by AI This will obviously depend on your circumstances.
Kiro 4y ago
Funny how everyone praised GPTZero that has even worse rates but starts being skeptical when it's OpenAI, the new bad guy.
- deleted 4y ago
  [deleted]
- dns_snek 4y ago
  "Everyone" didn't. In fact, the 5 top comments in that thread[1] all called it useless or pointed out serious flaws. [1] https://news.ycombinator.com/item?id=34556681 https://news.ycombinator.com/item?id=34556681
kmkemp 4y ago
Any solution here is just an arms race. The better AI's get at generating text, the more impossible the job of identifying if an AI was responsible for writing a given text sample.
- e_i_pi_2 4y ago
  You could even just set up a GAN to make the AI better at not being detected as something written by an AI, I don't see a good general solution to this, but I also see it as a non-issue - if students have better tools they should be able to use them, just like a calculator on a test - that's allowed on tests because you still need to understand the concepts to put it to use
e_i_pi_2 4y ago
I can't remember the keyword to look it up, but there's a problem of statistics you run into with stuff like terrorism detection algorithms If we have 300M people in the US and only 1k terrorists, then you need 99.9999% accuracy before you start getting more true positives than false positives. If you use this in a classroom where no one is actually using AI you'll get false positives, and in a class where the usage is average you'll still get more false positives than true ones, which makes the test do more harm than good unless it's just a reason to look into it more - and the teacher is presumably already reading the text so if that doesn't help than this surely won't
- xmddmx 4y ago
  It's the False Positive Paradox: https://en.wikipedia.org/wiki/Base_rate_fallacy#False_positive_paradox https://en.wikipedia.org/wiki/Base_rate_fallacy#False_positi...
adamsmith143 4y ago
Or we realize that essays aren't that important and technical skills will become more highly valued. Either way, ChatGPT can't do your exams for you so the truth will come out anyway.
- mitchdoogle 4y ago
  Writing is very important for understanding a topic and long-term recall. I still remember topics from papers I did 15 years ago because I spent 10s of hours researching and writing and forming ideas about each topic. Instead of being overzealous about catching cheaters, teachers should learn to express the importance of writing and why it is done. Convince the students that they should do it to be a smarter person, not just to get a grade, and they will care more about doing it honestly.
- 8note 4y ago
  Writing is itself a technical skill With ai taking over technical skills, it seems clear to me that they will be values less. Instead, the soft skills will be the valued ones
ibejoeb 4y ago
I think there is a more dystopian near future: 1. There will be commercial products to tune per-student writing models. 2. Those models will be used to evaluate progress and contribute directly to scores, grades, and rankings. They may also serve to detect collaboration. 3. The models persist indefinitely and will be sold to industry for all sorts of purposes, like hiring. 4. Thy will certainly be sold to the state for law enforcement and identity cataloging.
tremon 4y ago
I dare hope for a less dystopian outcome: - teachers will assign less mind-numbing essay homework assignments and focus more on oral interviews.
- Fomite 4y ago
  That heavily favors a particular learning style, which isn't necessarily a desirable outcome.
  - juve1996 4y ago
    You can't make everyone happy.
    - Fomite 4y ago
      Generally speaking, education (when done correctly) tries to avoid "...and devil take the hindmost" as a guiding philosophy.
      - juve1996 4y ago
        Mass education is like mass transit. It gets the majority of the population somewhere. Not everyone gets to take the ferrari. Someone will always be left out and we shouldn't let perfect be the enemy of good enough.
        tremon 4y ago
        ...but if we are forced to choose, it's better to spend our effort on the ones who don't own a Ferrari.
        juve1996 4y ago
        That's exactly the point - mass education. Busses. Not ferraris - even for people who might need that level of support. It isn't economical or a net benefit for society to make sure everyone's needs are met to the best possible point.
sometimeshuman 4y ago
Sorry for the tangent but a surprising number the general public doesn't know the meaning of percent[1]. So even if a teacher is told those percentages many wouldn't know what to conclude. [1] Me, giving young adults that worked for me a commission rate. Then asking if their commission rate is 15% and they sell $100 of goods what is their payment. Many failed to provide an answer.
nonrandomstring 4y ago
A more likely outcome is that teachers will pay the price [1]. [1] https://www.timeshighereducation.com/opinion/ai-will-replace-academics-unless-our-teaching-challenges-students-again https://www.timeshighereducation.com/opinion/ai-will-replace... (turn off js to jump signup-wall)
mitchdoogle 4y ago
4. Parents sue schools 5. Admins eliminate all writing requirements
Verdex 4y ago
I wonder if I should help my kids setup a server + webcam + screen capture tool so they can document 100% of their essay writing experience. That way if they ever get hit with a false positive they can just respond with hundreds of hours of video evidence that shows them as the unique author of every essay they've ever written.
- anotherjesse 4y ago
  You will certainly have a lot of training video to create a "essay writing video generator" ml product
- causalmodels 4y ago
  You could always teach them how to use git and have them commit frequently. Seems like it would be less intrusive than a webcam.
  - Verdex 4y ago
    Source control would certainly help establish a history of incrementally performing school work by someone when viewed by a highly technical examiner and when periodically stored someplace where a trusted 3rd party can confirm it wasn't all generated the night after a supposed false positive. However, hundreds of hours of video is compelling to non-technical audiences and even more importantly is a preponderance of evidence that's going to be particularly damning if played in front of a PTA meeting. With a git history it's going to come down to who can spin the better story. The video is the story and everyone recognizes it, so I expect fewer people would bother even challenging its authenticity.
    - causalmodels 4y ago
      I guess that's fair. I just personally don't think the additional gain is worth taking away your child's privacy.
      - Verdex 4y ago
anonobviously 4y ago
This is extremely concerning. The co-author on this is includes Professor Scott Aaronson. Reading his blog Shtetl-Optimized and reading his [sad/unfortunate/debate-able/correct?/factual?/biased?] views on adverse/collateral harm to Palestinians civilians makes me question whether this model would fully consider collateral damage and harm to innocent civilians, whomever that subgroup might be. What if his model works well, except for some minority groups' languages which might reflect OpenAI speak? Does it matter if the model is 99.9% accurate if the 0.1% is always one particular minority group that has a specific dialect or phrasing style? Who monitors it? Who guards these guards?
jameshart 4y ago
We can’t release the essay writing language model. Lazy children will use it to write their essays for them! We can’t release the ai-generated text detection model. Lazy teachers will use it to falsely accuse children of cheating! The problem here appears to be lazy people. Can we train an AI to detect lazy people? I promise not to lazily rely on it without thinking.
ren_engineer 4y ago
>false positive rate of 9% bringing the Roman decimation to the classroom based on AI, this is the future
la64710 4y ago
Exactly IMHO it is irresponsible to release such classifier with a title that touts the desired feature and totally do not spell its limitations. At least precede such title with experimental or something.
tshaddox 4y ago
It's almost as if you need to give exams in person and watch the students if you don't want them to cheat. This is fundamentally no different than cheating by writing notes on your hand in an exam or paying someone to write a take-home essay for you. It's cheaper than the latter, but that just means the lazy curriculum finally needs to be updated.
kilgnad 4y ago
This isn't that dystopian. The dystopian outcome is when there's a classifier that rates the quality of the text and that this classifier becomes indistinguishable from the AI-generated classifier because AI generated text is beginning to be superior to human generated text.
flandish 4y ago
In the same way deepfake video should not be allowed as evidence, thereby ensuring no video is allowed… we can apply that to text as well. We’re entering an uncanny valley before a period of “reset” with self taught (to stay on subject here) people re-learning for the sake of learning. In 30 years we will be in an educational renaissance of people learning “like the old masters did in the 1900’s.”
- EGreg 4y ago
  Nah. In 30 years it will be as useless to learn most subjects as it is right now to learn crocheing and knitting, or learning times tables or using an abacus. People are wayyyy too optimistic, just like in the 1900s they thought people would have flying cars but not the Internet, or how Star Trek’s android Data is so limited and lame. Bots will be doing most of the work AND have the best lines to say, AND make the best arguments in court etc. You don’t even need to look to AI for that. The best algorithms are simply uploaded to all the bots and they are able to do 800 things, in superhuman ways, and have access to the internet for whatever extra info they need. When they swarm, they’ll easily outcompete any group of humans. For example they can enter this HN thread and overwhelm it with arguments. No, the old masters were needed. Studying will not be. The Eloi and Morlocks is closer to what we can expect.
  - flandish 4y ago
    As someone who’s known how to crochet and knit since he as 6… I disagree.
  - tokai 4y ago
    Apparently knitwear is forecasted to have a CAGR of 12% the rest of the decade. With hand knitted garments commanding the high prices. It's definitely not the worst cottage industry one can chose.
    - EGreg 4y ago
      https://xkcd.com/1102/ https://xkcd.com/1102/
jupp0r 4y ago
This is worse than useless, if taking base rate fallacy into account.
saltysnowball 4y ago
This is already an issue, I'm a student in college right now and even technical professors are operating with full confidence in systems like turnitin which try their hand at plagiarism detection (with often much higher false negative/false positive rates). The problem was even more prevalent in high school where teachers would treat it as a 100% certainty. Thus, I think that OpenAI making atleast a slightly better classification algorithm won't make the state of affairs any worse.
screye 4y ago
Hilariously, this has already happened with music composition. Especially drumming. Since the advent of drum machines, a lot of younger players have started playing with the sort of precision that drum machines enable. eg: The complete absence of swing, and clean high-tempo blasts/rides. So you'd get accusations of drummers not being able to play their own songs, because traditional drummers think such technically complex and 'soulless' performances couldn't possibly be human. Only to then be proven wrong, when it turns out that younger players can in fact do it. The machine conditions man.
mr_toad 4y ago
Maybe they should start asking questions that AI can’t answer, instead of having students regurgitate what they’ve memorised.
headsoup 4y ago
Ah but as with AI generally before now, 'you can't stop progress.' It'll end up being used and falling into an arms race of better AI vs better detection, all the while losing the point of why it is there at all in the first place.
blueblimp 4y ago
> true positive rate of only 26% and a false positive rate of 9% That's uninformative enough that I'm surprised they launched this publicly at all.
deltree7 4y ago
Also, there will exist Prompt => AIGen (White Hat) => Obfuscate(Black Hat) => Final Text
Fomite 4y ago
I think the much more proximate threat is that fear of ChatGPT kills a lot of progress that's been made in making exam material more accessible (take home tests, etc.) to a broader audience of students.
deleted 4y ago
[deleted]
rvba 4y ago
I guess students will get recorded writing their homework, say on a tablet. Then of course the AI can whisper the student what to write to your ear. So perhaps homework will have to be done at school? School that checks its students with a metal detector when they enter. (Some schools use them already to check for guns?) On a side note Im very shocked how lax is everything in those proffessional chess tournaments. It feels there are many ways to cheat and they dont try to do anything against cheating. They should use metal detectctors (to detect computers inside stomach or tooth), they should host everything inside a bunker (so no radio), without audience (who can do various tricks) and in a secured environment (all cameras chcecked to be sure they are legit). Those chess tours look like cheating galore for me, although I dont play chess.
p-e-w 4y ago
Let me soothe your fear: This isn't a novel cheating technology, it's a technology that will make humans obsolete. Neither teachers nor students are going to matter in the future. Most or all of the population is going to be enslaved for all practical purposes, either to an all-powerful super-elite, or to AI itself. Any worries about how mundane things like education are going to be impacted by petty cat-and-mouse games are going to become irrelevant, because education itself is going to be irrelevant, along with everything else that once defined our world.
bilater 4y ago
yup - you can beat em using simple things like mixing up words to throw off the word distribution. GPT-Minus1 is an exmaple. https://gptminus1.vercel.app/ https://gptminus1.vercel.app/
dirtyid 4y ago
I imagining future that involves programs monitors students writing in proctored setting to establish some sort of individual finger print and use that to match against future writing assignments, again persistently monitored for authenticity. Clippy going to pop up in the corner to warn you when you've been behaving too artificial. Whatever that means.

cjbgkagh 4y ago

> false positive rate of 9% Yeah, that is useless. You couldn't punish based on that alone and students will quickly figure out to never confess.

TheRealPomax 4y ago

So, status quo then? This is already the case for educational software that's used to detect plagiarism. People get wrongly flagged, and then you'll have to plead your case. But the times software like this finds actual problems vastly outnumbers of times it doesn't, and when you choice is between "passing kids/undergrads who cheat the system" and "the occasional arbitration", you go with the latter. Schools don't pay teachers anywhere near enough to not use these tools.

michaericalribo 4y ago
Given the published true and false positive rates, it's clear that the true positives do not "vastly outnumber" false positives.
notahacker 4y ago
> This is already the case for educational software that's used to detect plagiarism. People get wrongly flagged, and then you'll have to plead your case. How often is that the case though? A while since I've had to worry about it, but I thought plagiarism detection generally worked on the principle of looking for the majority of the content being literal matches with existing material out there with only a few small edits, which - unlike using some "AIish" turns of phrase a bot wrongly attributes to humans 9% of the time and correctly attributes to AI with a not much better success rate - is pretty hard to do accidentally.
- i_have_an_idea 4y ago
  A long time ago when I was a student, I would run my papers through Turnitin before submitting. The tool would sometimes mark my (completely original) work as high as mid 20% similarity. As a result, I have taken out quotes and citations to appease it and not have to deal with the hassle. I expect modern day students will resort to similar measures.
  - notahacker 4y ago
    IIRC the marker got the same visualization that you used to take out quotes and citations that highlighted that the similar bits were in fact quotes and citations! Maybe high school is a different matter, but I'm pretty sure even the most technophobic academic knows that jargon, terse definitions and the odd citation overlapping with stuff other people have written is going to make a similarity of at least 10% pretty much inevitable, especially when the purpose of the exercise is to show you understand the core material well enough to cite and paraphrase and compare it, not to generate novel academic insight or show you understood the field so well you didn't need to refer back to the source material. The people they were actually after were the ones that downloaded something off essaybank, removed a couple of paragraphs and rewrote the intro to match the given title and ended up with 80%+ similarity
PeterisP 4y ago
Currently the false positive rate is far lower. E.g. I get 500-ish submissions over a school year then a 1% false positive rate would mean I'd falsely accuse 5 innocent students annually, which isn't acceptable at all - and a 9% FP rate is so high that's even not worth investigating; do you know of any grader who has the spare time to begin formal proceedings/extra reviews/investigation for 9% of their homework? For plagiarism suspicions at least the verification is simple and quick (just take a look at the identified likely source, you can get a reasonable impression in minutes) - I can't even imagine what work would be required to properly verify ones flagged by this classifier..
- TheRealPomax 4y ago
  > I can't even imagine what work would be required to properly verify ones flagged by this classifier. Yet.
  - flatline 4y ago
    At the same time the classifier is improving, the generative models are improving. It’s a classic arms race and this equilibrium is not likely to shift much either way. We are talking about models that approximate human behavior with a high degree of accuracy, I think the goal would be to make them indistinguishable in any meaningful way.
  - PeterisP 4y ago
    Can you elaborate? I don't think that this is something that can change through tech advances for the classifiers - in all cases the classifier is just flagging for investigation, it's not sufficient for any action. For plagiarism, appropriate evidence comes from a person comparing the submission with the possible source of plagiarism. For this one, the proper evidence would require getting confirmation that the student actually generated that data - e.g. identifying the exact tool and prompt that was used, or logs from the students' computer showing that this was done, or logs from the text generation service provider. All of those are quite tricky to get and perhaps even not possible.
- Fomite 4y ago

dougmwne 4y ago

The cheating students who know how to use the classifier will be the big winners.

LarryMullins 4y ago

> 2. Non-technical users (teachers) treat this like a 100% certainty This is the part that needs to be addressed the most. Teachers can't offload their critical reasoning to the computer. They should ask their students to write things in class and get a feeling for what those individual students are capable of. Then those that turn in essays written at 10x their normal writing level will be obvious, without the use of any automated cheat detectors. I was once accused of cheating by a computer; my friend and I both turned in assignments that used do-while loops, which the computer thought was so statistically unlikely that we surely must have worked together on the assignment. But the explanation was straight forward; I had been evangelizing the aesthetic virtue of do-while loops to anybody that would listen to me, and my friend had been persuaded. Thankfully the professor understood this once he compared the two submissions himself and realized we didn't even use the do-while loop in the same part of the program. There was almost no similarity between the two submissions besides the statistically unlikely but completely innocuous use of do-while loops. It's a good thing my professor used common sense instead of blindly trusting the computer.

Gigachad 4y ago
I once got called up because my work was flagged as 100% copied. I had uploaded it, made a mistake so I deleted it and uploaded a new file. Second file was flagged as copied. Was able to explain it by pointing at the screen that was claiming I plagiarized my own name.
londons_explore 4y ago
> blindly trusting the computer. Professors blindly trust the computer not out of laziness, but to protect themselves from accusations of unfairness... "The work was detected as plagiarism, but the professor overrode it for the pretty girl in class, but not for me"
- mitchdoogle 4y ago
  Seems like something like this should only be used as a first-level filter. If the writing doesn't pass, it warrants more investigation. If no proof of plagiarism is found, then there's nothing else to do and professor must pass the student

thewataccount 4y ago

Hopefully they just flag relevant sections. Essay/Plagiarism checkers already exist, although in my experience professors were reasonable. For example I had a paragraph or two get flagged as being very similar to another paper - but both papers were about a fairly niche topic (involving therapy animals) and we had both used the relevant quotes from the study conclusions from one of only a few decent sources at the time - so of course they were going to be very similar. Given that most essays are about roughly the same set of topics, and there are literally hundreds of thousands of students writing these - I wonder how many variations are even possible for humans to write as I would expect us to converge on similar essays?

michaericalribo 4y ago
Plagiarism is easier to verify, because you can directly compare with the plagiarized source material
- thewataccount 4y ago
  Absolutely. I think it may have to end up more as a statistics thing with behaviour. For example: "Tom had a single paragraph flag as possibly generated" vs "Every single paper Tom writes has paragraphs flag" Basically we might have to move to detecting statistical outliers as cheating. Now whether the tools/teachers will understand/actually do that - we can only hope....
  - blueblimp 4y ago
    That's a good point: the effectiveness at detecting AI generation is probably going to depend strongly on the length of the text.

amelius 4y ago

Solution: just write your texts with a bit less confidence than gpt3 would.

janalsncm 4y ago

I urge anyone with time to write to tech journalists explaining why this is so bad. Given previous coverage of GPTZero they don’t seem to be asking the right questions.

claytonjy 4y ago

Is there a longer-form paper on this yet? TPR (P(T|AI)) and FPR (P(T|H)) are useful, but what I really want is the probability that a piece flagged as AI-generated is indeed AI-generated, i.e. P(AI|T). Per Bayes rule I'm missing P(AI), the portion of the challenger set that was produced by AI. If we assume the challenger set is evenly split 50-50, that means P(AI|T) = P(T|AI)P(AI)/P(T) = (0.26)(0.5)/(0.26+0.09) ~ 37% So slightly better than a 1/3 chance of the flagged text actually being AI-generated. They say the web-app uses a confidence threshold to keep the FPR low, so maybe these numbers get a bit better, but very far from being used as a detector anywhere it matters.

drc500free 4y ago
Precision is impossible to calculate without knowing P(AI), which is use-case specific. Source: Spent 10 years trying to explain this to government people who insisted that someone tell them Precision based purely on the classifier accuracy without considering usage.
TchoBeer 4y ago
>Per Bayes rule I'm missing P(AI), the portion of the challenger set that was produced by AI This will obviously depend on your circumstances.

Kiro 4y ago

Funny how everyone praised GPTZero that has even worse rates but starts being skeptical when it's OpenAI, the new bad guy.

deleted 4y ago
[deleted]
dns_snek 4y ago
"Everyone" didn't. In fact, the 5 top comments in that thread[1] all called it useless or pointed out serious flaws. [1] https://news.ycombinator.com/item?id=34556681 https://news.ycombinator.com/item?id=34556681

kmkemp 4y ago

Any solution here is just an arms race. The better AI's get at generating text, the more impossible the job of identifying if an AI was responsible for writing a given text sample.

e_i_pi_2 4y ago
You could even just set up a GAN to make the AI better at not being detected as something written by an AI, I don't see a good general solution to this, but I also see it as a non-issue - if students have better tools they should be able to use them, just like a calculator on a test - that's allowed on tests because you still need to understand the concepts to put it to use

e_i_pi_2 4y ago

I can't remember the keyword to look it up, but there's a problem of statistics you run into with stuff like terrorism detection algorithms If we have 300M people in the US and only 1k terrorists, then you need 99.9999% accuracy before you start getting more true positives than false positives. If you use this in a classroom where no one is actually using AI you'll get false positives, and in a class where the usage is average you'll still get more false positives than true ones, which makes the test do more harm than good unless it's just a reason to look into it more - and the teacher is presumably already reading the text so if that doesn't help than this surely won't

xmddmx 4y ago
It's the False Positive Paradox: https://en.wikipedia.org/wiki/Base_rate_fallacy#False_positive_paradox https://en.wikipedia.org/wiki/Base_rate_fallacy#False_positi...

adamsmith143 4y ago

Or we realize that essays aren't that important and technical skills will become more highly valued. Either way, ChatGPT can't do your exams for you so the truth will come out anyway.

mitchdoogle 4y ago
Writing is very important for understanding a topic and long-term recall. I still remember topics from papers I did 15 years ago because I spent 10s of hours researching and writing and forming ideas about each topic. Instead of being overzealous about catching cheaters, teachers should learn to express the importance of writing and why it is done. Convince the students that they should do it to be a smarter person, not just to get a grade, and they will care more about doing it honestly.
8note 4y ago
Writing is itself a technical skill With ai taking over technical skills, it seems clear to me that they will be values less. Instead, the soft skills will be the valued ones

ibejoeb 4y ago

I think there is a more dystopian near future: 1. There will be commercial products to tune per-student writing models. 2. Those models will be used to evaluate progress and contribute directly to scores, grades, and rankings. They may also serve to detect collaboration. 3. The models persist indefinitely and will be sold to industry for all sorts of purposes, like hiring. 4. Thy will certainly be sold to the state for law enforcement and identity cataloging.

tremon 4y ago

I dare hope for a less dystopian outcome: - teachers will assign less mind-numbing essay homework assignments and focus more on oral interviews.

Fomite 4y ago
That heavily favors a particular learning style, which isn't necessarily a desirable outcome.
- juve1996 4y ago
  You can't make everyone happy.
  - Fomite 4y ago
    Generally speaking, education (when done correctly) tries to avoid "...and devil take the hindmost" as a guiding philosophy.
    - juve1996 4y ago
      Mass education is like mass transit. It gets the majority of the population somewhere. Not everyone gets to take the ferrari. Someone will always be left out and we shouldn't let perfect be the enemy of good enough.
      - tremon 4y ago
        ...but if we are forced to choose, it's better to spend our effort on the ones who don't own a Ferrari.
        juve1996 4y ago
        That's exactly the point - mass education. Busses. Not ferraris - even for people who might need that level of support. It isn't economical or a net benefit for society to make sure everyone's needs are met to the best possible point.

sometimeshuman 4y ago

Sorry for the tangent but a surprising number the general public doesn't know the meaning of percent[1]. So even if a teacher is told those percentages many wouldn't know what to conclude. [1] Me, giving young adults that worked for me a commission rate. Then asking if their commission rate is 15% and they sell $100 of goods what is their payment. Many failed to provide an answer.

nonrandomstring 4y ago

A more likely outcome is that teachers will pay the price [1]. [1] https://www.timeshighereducation.com/opinion/ai-will-replace-academics-unless-our-teaching-challenges-students-again https://www.timeshighereducation.com/opinion/ai-will-replace... (turn off js to jump signup-wall)

mitchdoogle 4y ago

4. Parents sue schools 5. Admins eliminate all writing requirements

Verdex 4y ago

I wonder if I should help my kids setup a server + webcam + screen capture tool so they can document 100% of their essay writing experience. That way if they ever get hit with a false positive they can just respond with hundreds of hours of video evidence that shows them as the unique author of every essay they've ever written.

anotherjesse 4y ago
You will certainly have a lot of training video to create a "essay writing video generator" ml product
causalmodels 4y ago
You could always teach them how to use git and have them commit frequently. Seems like it would be less intrusive than a webcam.
- Verdex 4y ago
  Source control would certainly help establish a history of incrementally performing school work by someone when viewed by a highly technical examiner and when periodically stored someplace where a trusted 3rd party can confirm it wasn't all generated the night after a supposed false positive. However, hundreds of hours of video is compelling to non-technical audiences and even more importantly is a preponderance of evidence that's going to be particularly damning if played in front of a PTA meeting. With a git history it's going to come down to who can spin the better story. The video is the story and everyone recognizes it, so I expect fewer people would bother even challenging its authenticity.
  - causalmodels 4y ago
    I guess that's fair. I just personally don't think the additional gain is worth taking away your child's privacy.
    - Verdex 4y ago

anonobviously 4y ago

This is extremely concerning. The co-author on this is includes Professor Scott Aaronson. Reading his blog Shtetl-Optimized and reading his [sad/unfortunate/debate-able/correct?/factual?/biased?] views on adverse/collateral harm to Palestinians civilians makes me question whether this model would fully consider collateral damage and harm to innocent civilians, whomever that subgroup might be. What if his model works well, except for some minority groups' languages which might reflect OpenAI speak? Does it matter if the model is 99.9% accurate if the 0.1% is always one particular minority group that has a specific dialect or phrasing style? Who monitors it? Who guards these guards?

jameshart 4y ago

We can’t release the essay writing language model. Lazy children will use it to write their essays for them! We can’t release the ai-generated text detection model. Lazy teachers will use it to falsely accuse children of cheating! The problem here appears to be lazy people. Can we train an AI to detect lazy people? I promise not to lazily rely on it without thinking.

ren_engineer 4y ago

>false positive rate of 9% bringing the Roman decimation to the classroom based on AI, this is the future

la64710 4y ago

Exactly IMHO it is irresponsible to release such classifier with a title that touts the desired feature and totally do not spell its limitations. At least precede such title with experimental or something.

tshaddox 4y ago

It's almost as if you need to give exams in person and watch the students if you don't want them to cheat. This is fundamentally no different than cheating by writing notes on your hand in an exam or paying someone to write a take-home essay for you. It's cheaper than the latter, but that just means the lazy curriculum finally needs to be updated.

kilgnad 4y ago

This isn't that dystopian. The dystopian outcome is when there's a classifier that rates the quality of the text and that this classifier becomes indistinguishable from the AI-generated classifier because AI generated text is beginning to be superior to human generated text.

flandish 4y ago

In the same way deepfake video should not be allowed as evidence, thereby ensuring no video is allowed… we can apply that to text as well. We’re entering an uncanny valley before a period of “reset” with self taught (to stay on subject here) people re-learning for the sake of learning. In 30 years we will be in an educational renaissance of people learning “like the old masters did in the 1900’s.”

EGreg 4y ago
Nah. In 30 years it will be as useless to learn most subjects as it is right now to learn crocheing and knitting, or learning times tables or using an abacus. People are wayyyy too optimistic, just like in the 1900s they thought people would have flying cars but not the Internet, or how Star Trek’s android Data is so limited and lame. Bots will be doing most of the work AND have the best lines to say, AND make the best arguments in court etc. You don’t even need to look to AI for that. The best algorithms are simply uploaded to all the bots and they are able to do 800 things, in superhuman ways, and have access to the internet for whatever extra info they need. When they swarm, they’ll easily outcompete any group of humans. For example they can enter this HN thread and overwhelm it with arguments. No, the old masters were needed. Studying will not be. The Eloi and Morlocks is closer to what we can expect.
- flandish 4y ago
  As someone who’s known how to crochet and knit since he as 6… I disagree.
- tokai 4y ago
  Apparently knitwear is forecasted to have a CAGR of 12% the rest of the decade. With hand knitted garments commanding the high prices. It's definitely not the worst cottage industry one can chose.
  - EGreg 4y ago
    https://xkcd.com/1102/ https://xkcd.com/1102/

jupp0r 4y ago

This is worse than useless, if taking base rate fallacy into account.

saltysnowball 4y ago

This is already an issue, I'm a student in college right now and even technical professors are operating with full confidence in systems like turnitin which try their hand at plagiarism detection (with often much higher false negative/false positive rates). The problem was even more prevalent in high school where teachers would treat it as a 100% certainty. Thus, I think that OpenAI making atleast a slightly better classification algorithm won't make the state of affairs any worse.

screye 4y ago

Hilariously, this has already happened with music composition. Especially drumming. Since the advent of drum machines, a lot of younger players have started playing with the sort of precision that drum machines enable. eg: The complete absence of swing, and clean high-tempo blasts/rides. So you'd get accusations of drummers not being able to play their own songs, because traditional drummers think such technically complex and 'soulless' performances couldn't possibly be human. Only to then be proven wrong, when it turns out that younger players can in fact do it. The machine conditions man.

mr_toad 4y ago

Maybe they should start asking questions that AI can’t answer, instead of having students regurgitate what they’ve memorised.

headsoup 4y ago

Ah but as with AI generally before now, 'you can't stop progress.' It'll end up being used and falling into an arms race of better AI vs better detection, all the while losing the point of why it is there at all in the first place.

blueblimp 4y ago

> true positive rate of only 26% and a false positive rate of 9% That's uninformative enough that I'm surprised they launched this publicly at all.

deltree7 4y ago

Also, there will exist Prompt => AIGen (White Hat) => Obfuscate(Black Hat) => Final Text

Fomite 4y ago

I think the much more proximate threat is that fear of ChatGPT kills a lot of progress that's been made in making exam material more accessible (take home tests, etc.) to a broader audience of students.

deleted 4y ago

[deleted]

rvba 4y ago

I guess students will get recorded writing their homework, say on a tablet. Then of course the AI can whisper the student what to write to your ear. So perhaps homework will have to be done at school? School that checks its students with a metal detector when they enter. (Some schools use them already to check for guns?) On a side note Im very shocked how lax is everything in those proffessional chess tournaments. It feels there are many ways to cheat and they dont try to do anything against cheating. They should use metal detectctors (to detect computers inside stomach or tooth), they should host everything inside a bunker (so no radio), without audience (who can do various tricks) and in a secured environment (all cameras chcecked to be sure they are legit). Those chess tours look like cheating galore for me, although I dont play chess.

p-e-w 4y ago

Let me soothe your fear: This isn't a novel cheating technology, it's a technology that will make humans obsolete. Neither teachers nor students are going to matter in the future. Most or all of the population is going to be enslaved for all practical purposes, either to an all-powerful super-elite, or to AI itself. Any worries about how mundane things like education are going to be impacted by petty cat-and-mouse games are going to become irrelevant, because education itself is going to be irrelevant, along with everything else that once defined our world.

bilater 4y ago

yup - you can beat em using simple things like mixing up words to throw off the word distribution. GPT-Minus1 is an exmaple. https://gptminus1.vercel.app/ https://gptminus1.vercel.app/

dirtyid 4y ago

I imagining future that involves programs monitors students writing in proctored setting to establish some sort of individual finger print and use that to match against future writing assignments, again persistently monitored for authenticity. Clippy going to pop up in the corner to warn you when you've been behaving too artificial. Whatever that means.