Small rant : Basically, the title. Instead of answering every question, if it instead said it doesn’t know the answer, it would have been trustworthy.
Small rant : Basically, the title. Instead of answering every question, if it instead said it doesn’t know the answer, it would have been trustworthy.
This wasn’t an intentional feature; they’re actually trying to train it with fine-tuning to add this as an ability. It’s one area that highlights the difference between it imitating the text it’s been seeing, instead of actually understanding what it’s saying – since most of its training data is of the form “(ask a question) (response to question)” overwhelmingly more often than “(ask a question) (say you don’t know, the end)”, it is trying to be a good imitator and do the same, and come up with some plausible nonsense even if it doesn’t know the answer.
And sometimes that’s exactly what I want, too. I use LLMs like ChatGPT when brainstorming and fleshing out fictional scenarios for tabletop roleplaying games, for example, and in those situations coming up with plausible nonsense is specifically the job at hand. I wouldn’t want to go “ChatGPT, I need a description of the interior of a wizard’s tower is like” and get the response “I don’t know what the interior of a wizard’s tower is like.”
At one point I messed around with a lore generator that would chop up sections of “The Dungeon Alphabet” and “Fire on the Velvet Horizon” along with some other stuff, and feed random sections of them into the LLM for inspiration and then ask it to lay out a little map, and it pretty reliably came up with all kind of badass stuff.