Naked Science Forum
General Discussion & Feedback => Just Chat! => Topic started by: Eternal Student on 14/04/2024 02:45:59
-
Hi.
Did you know that a lot of Universities run quite a lot of submitted material, like essays, through a computer to check for plagiarism? You probably did know that.
Did you know they now also include a quick assessment of the probability that this content was substantially created by some LLM (Large language Model - like Gemini, ChatGPT etc.)? You may have known that but LLM's are quite new so they've only recently started it.
If you submit an essay with a high chance of having been generated by a LLM then it can be rejected or the person could be called in for a viva examination.
However, it seems to be bit one sided. There's one student I know who felt strongly that the feedback and comments on their essays were very generic and not especially useful. They ran the feedback through a LLM checker and - well, you guessed it. Perhaps they should reject the feedback and insist the marker comes in for a face-to-face meeting where the essay is properly discussed for 10 minutes?
On the other hand, perhaps it's not all bad. Eduaction may end up becoming a game or a test to see who can game the system better - but we will have grauates who can play the game very, very well. Students will be working not just to prevent triggering an LLM detection but to also get future iterations of their LLM assisted essays adjusted in line with output generated by the markers LLM system. This will help the markers AI system pick up key elements and hopefully grade it more highly. We'll have future graduates who are highly skilled with this new LLM technology.
That will be important because LLM and AI will only continue to develop, so our new graduates will need to be good with this technology. There's got to be a bright side to everything.
Best Wishes.
-
a LLM checker
What is a LLM checker? What are the markers that trigger a positive?
How much better are future LLMs going to get such that such tests need to be ever more complex? After all, what a student does is essentially what an LLM does: Research some source(s) and regurgitate the material in altered words. Short of an actual 'new reasearch' project, that's pretty much it.
The checker seems to be a form of the Turning test. Was this text generated by a human or no? All the harder since there is only one-way text and no interaction.
-
Literary criticism, and to some extent the whole subject of literary studies, has been neatly summed up as "hundreds of pages of bad English written about a few pages of good English", and I guess the same could be said in any other language, so the value of essays in that area must be close to zero and their purpose is difficult to imagine.
Undergraduate essays in factual (e.g. geography) or presumed factual (history) subjects really boil down to a single question: "have you read the book?" So AI has revealed that a great deal of student and teacher time is, frankly, wasted by regurgitation of secondhand facts and opinions.
On the presumption that studying the humanities has some purpose, it is therefore incumbent on those who profit from teaching them to introduce some form of laboratory practical. The pub quiz format is ideal since it tends to the requisite gaussian curve of marks with a large number of questions and candidates, and does test retention of facts (including the written opinions of third parties).
Having established the candidate's diligence in reading or attending lectures, we now need to determine whether he can form and express an opinion clearly, so a very brief viva will waste less of everyone's time than writing and marking a thousand word essay. Though written, Churchill's memo to Montgomery: "Pray let me have, on half a sheet of paper, your plan for invading Italy" sets the tone for a good viva question. From the examiner's point of view it would be a lot more fun than adjudicating a brass band contest, where the scope for interpretation of the test piece is rather limited.
-
Postscript:
The pub quiz idea is even better than I thought. Traditionally, you choose a team name or personal nom de plume, and everyone marks someone else's papers. Now if the identifiers are properly anonymised and coded, you could do this for a hundred students and save thousands of hours of examiners' time.
Reshuffle four times, ignore the highest and lowest, and award the sum of the middle three marks, and you have a completely fair marking system. It's used in ice skating and boxing, where it really matters because money and medals are at stake, unlike Eng Lit where your future career as a politician, bus driver or playgroup knicker lady doesn't depend on the class or even the subject of your degree.
-
Computers assessing the output of other computers. No need for students.
-
There is already an absurd "public outcry" (journalise for a couple of crank letters to the Daily Mail) because some UK schools will have to close for lack of pupils. Without students, what would Eng Lit, Media Studies and Philosophy lecturers do all day? And how would teenagers fill their time between school and adult unemployment?
-
Media Studies
Media studies is all the rage now, the internet has made it a mandatory feature for every business, internet site etc. Who needs maths when you have social media?
-
Very timely. Last night I bought a drink for ?4.50 and offered the bartender a ?20 note as I had left my credit card at home. He opened the cash register and stood completely baffled for at least a minute, confessing that he couldn't work out what to do next.
The world has changed hugely in the last four years.