Small rant : Basically, the title. Instead of answering every question, if it instead said it doesn’t know the answer, it would have been trustworthy.
Have it cite it’s source.
Isn’t this just a restatement of the halting problem?
I’d love to agree with you - but when people say that LLMs are stochastic parrots, this is what they mean…
LLMs don’t actually know what the words they’re saying mean, they just know what words are most likely to be next to each other based on training data.
Because they don’t know the meaning of what they’re saying, they also don’t know the factuality of what they’re saying - as such they simply can’t self-fact check.
Sure but that would mean it would have to know anything.
It would have to know that it doesn’t know, and it doesn’t.
LLMs don’t know if what they’re telling you is true or not, out of the box. It’s gonna give you an answer that statistically looks like a sequence of words that should come in response to the sequence of words (the prompt) you gave it. It doesn’t know what the words you said mean, and it doesn’t know what it answered means either. One of my favorite interactions I’ve had with Claude shows this limitation quite well…
How many r’s are there in the word strawberry?
There are 2 r’s in the word “strawberry”.
Are you sure?
You’re right to ask for confirmation. I apologize for my mistake in the previous response. Let me correct that:
There is actually only 1 ‘r’ in the word “strawberry”.
Thank you for prompting me to double-check. It’s important to provide accurate information.
Wow, GPT4o gave me this after the same questions:
“Yes, I am sure. The word “strawberry” has two “r”s: one after the “t” and another near the end of the word.”
But GPT4 got it right from the start.
What’s absolutely crazy about that is:
Prompt: write code that checks how many r’s are in the word strawberry
Response:
using System; class Program { static void Main() { string word = "strawberry"; char targetChar = 'r'; int count = 0; foreach (char c in word) { if (c == targetChar) count++; } Console.WriteLine($"The letter '{targetChar}' appears {count} times in the word \"{word}\"."); } }My first thought is that you could write a program that does something like this:
- Receive prompt “how many times does R appear in the word ‘strawberry’”
- Run the prompt through LLM saying “reword the following into a command to generate code that will answer the question” or something like that
- Run the results through LLM
- Compile and run the results
- Provide the output to the user
Of course, the biggest problem with this system is that a person could fool it into generating malicious code.
if it is able to accept that it doesn’t know an answer.
It will never be able to do this because it is not capable of knowledge
That would require ChatGPT to know that it’s talking bullshit. It’s not a knowledge database, it’s a digital parrot.
I work with plenty of people who don’t even do that. They just keep making stuff up like they do… But they’re confident in their incorrect answers, so people listen to them.
Doesnt the bot already imply that it could be wrong?
This wasn’t an intentional feature; they’re actually trying to train it with fine-tuning to add this as an ability. It’s one area that highlights the difference between it imitating the text it’s been seeing, instead of actually understanding what it’s saying – since most of its training data is of the form “(ask a question) (response to question)” overwhelmingly more often than “(ask a question) (say you don’t know, the end)”, it is trying to be a good imitator and do the same, and come up with some plausible nonsense even if it doesn’t know the answer.
And sometimes that’s exactly what I want, too. I use LLMs like ChatGPT when brainstorming and fleshing out fictional scenarios for tabletop roleplaying games, for example, and in those situations coming up with plausible nonsense is specifically the job at hand. I wouldn’t want to go “ChatGPT, I need a description of the interior of a wizard’s tower is like” and get the response “I don’t know what the interior of a wizard’s tower is like.”
At one point I messed around with a lore generator that would chop up sections of “The Dungeon Alphabet” and “Fire on the Velvet Horizon” along with some other stuff, and feed random sections of them into the LLM for inspiration and then ask it to lay out a little map, and it pretty reliably came up with all kind of badass stuff.
