- cross-posted to:
- fuck_ai@lemmy.world
- cross-posted to:
- fuck_ai@lemmy.world
The research from Purdue University, first spotted by news outlet Futurism, was presented earlier this month at the Computer-Human Interaction Conference in Hawaii and looked at 517 programming questions on Stack Overflow that were then fed to ChatGPT.
“Our analysis shows that 52% of ChatGPT answers contain incorrect information and 77% are verbose,” the new study explained. “Nonetheless, our user study participants still preferred ChatGPT answers 35% of the time due to their comprehensiveness and well-articulated language style.”
Disturbingly, programmers in the study didn’t always catch the mistakes being produced by the AI chatbot.
“However, they also overlooked the misinformation in the ChatGPT answers 39% of the time,” according to the study. “This implies the need to counter misinformation in ChatGPT answers to programming questions and raise awareness of the risks associated with seemingly correct answers.”
You must log in or register to comment.
ChatGPT and github copilot are great tools, but they’re like a chainsaw: if you apply them incorrectly or become too casual and careless with them, they will kickback at you and fuck your day up.
I guess it depends on the programming language… With python, I got very fast great results. But python is all about quick and dirty 😂
In Rust, it’s not great. It can’t do proper memory management in the language, which is pretty essential.
Not a programmer by any means (haven’t done any since college) but I’ve asked it for help in writing Jira queries or Excel mess and it’s been pretty solid with that stuff.
I just use it to get ideas about how to do something or ask it to write short functions for stuff i wouldnt know that well. I tried using it to create graphical ui for script but that was constant struggle to keep it on track. It managed to create something that kind of worked but it was like trying to hold 2 magnets of opposing polarity together and I had to constantly reset the conversation after it got “corrupted”.
Its useful tool if you dont rely on it, use it correctly and dont trust it too much.
This has been true for code you pull from posts on stackoverflow since forever. There are some good ideas, but they a. Aren’t exactly what you are trying to solve and b. Some of the ideas are incomplete or just bad and it is up to you to sort the wheat from the chaff.
I couldn’t have said it better
Yes there are mistakes, but if you direct it to the right direction, it can give you correct answers
It can, it also sometimes can’t unless you ask it “could it be x answer”
In my experience, if you have the necessary skills to point it at the right direction, you don’t need to use it at the first place
AI is a tool, not a solution.
Yesterday, I wrote all of this, working javascript code https://github.com/igorlogius/gather-from-tabs/discussions/8 And I don’t know a lick of javascript I know other languages but that barely was needed. I just gave it plain language instructions and reported the errors until it worked.
it’s just a convenience, not a magic wand. Sure relying on AI blindly and exclusively is a horrible idea (that lots of people peddle and a quite a few suckers buy), but there’s room for a supervised and careful use of AI, same as we started using google instead of manpages and (grudgingly, for the older of us) started tolerating the addition of syntax highlighting and even some code completion to even the most basic editors.
Removed by mod
Bruh, where in my comment did I tell people not to use it?
Removed by mod
Bad reading comprehension is bad.
Removed by mod
Actually the 4o version feels worse than the 4. Im getting tons of wrong answers now…
Yeah, it’s not supposed to be better than 4 for logic/reason/coding, etc… its strong points are it’s natural voice interaction, ability to react to streaming video, and its fast and efficient inference. The good voice and video are not available to many people yet. It is so efficient that it is going to be available to free users. If you want good reasoning, then you need to stick with 4 for now, or better yet, switch to something like Claude Opus. If you really want strong reasoning abilities, then at this point, you need a setup using agents, but that requires some research and understanding.
They’ve done studies: 48% of the time, it works every time.
This is part of the AI will replace jobs , and AI will get conscious , AI can program and automate everything. It’s bullshit. It’s a tool to help is not replacing anything . If companies start with the slogan they had with the cloud , we will be in a trouble. Because is fake
Still the same shit study that does not even name the version they used…? The one posted here 1 or 2 days ago?
I’m the footnotes they mention GPT-3.5. Their argument for not testing 4 was because it was paid, and so most users would be using 3.5 - which is already factually incorrect now because the new GPT-4o (which they don’t even mention) is now free. Finally, they didn’t mention GPT-4 Turbo either, which is even better at coding compared to 4.
Still the same shit study that does not even name the version they used…?
The answer to your question would be evident to you if you had taken the time to read what you are deeming “the same shit study.” The study mentions the version used on multiple occasions:
For each of the 517 SO questions, the first two authors manually used the SO question’s title, body, and tags to form one question prompt and fed that to the free version of ChatGPT, which is based on GPT-3.5.
Additionally, this work has used the free version of ChatGPT (GPT-3.5) for acquiring the ChatGPT responses for the manual analysis.
Hence, for this study, we used the free version (GPT-3.5) so that the results benefit the majority of our target populations.
Please ensure you have read the study before making uninformed remarks.
The one posted here 1 or 2 days ago?
I have already checked for duplicates within this community before posting, and the post you are talking about is not present.
Once again, please ensure your facts are accurate before making incorrect statements.
Just like answers on the Internet, you have to read the output and not just paste it blindly. I find the answers are usually useful, even if they aren’t completely accurate. Figuring out the last bit is why we are paid as programmers.
Not to mention that if the code fails you can often tell ChatGPT “here’s what happened” and it can debug its own code correctly much of the time.
So it is incorrect and verbose, but also comprehensive and using a well-articulated language style at the same time?
Also “study participants still preferred ChatGPT answers 35% of the time”, meaning that the overwhelming majority (two-thirds) did not prefer the bot answers over the human(e), correct ones, that maybe were not phrased as confidently as they could have been.
Just say it out loud: ChatGPT is style over substance, aka Fox News. 🦊
ChatGPT is not good enough to use as a substitute for (whatever), but can be a useful tool for someone who can quantify its output.
Me too \s
48% is still better than the Punxsutawney Phil.
If you ask the wrong questions you get the wrong results. If you don’t check the response for accuracy, you get invalid answers.
It’s just a tool. Don’t use it wrong because you’re lazy.
Lemmy is trying really, really hard to convince you that coding is going to be a viable career in 5 years.
Lemmy is trying real hard to convince you that AI is going to do everyone’s job in 5 years—including yours
Well, I do it 99% of the times
