While reading through a paper on Chain-of-Thought reasoning in LLMs, I stumbled upon something that caught my attention:

The ability of LLMs to produce “fluent nonsense”—plausible but logically flawed reasoning chains—can be more deceptive and damaging than an outright incorrect answer, as it projects a false aura of dependability.

This idea is something that has been on my mind for a while now, but reading it phrased like this made something click.

When I hear people talk about hallucinations, the focus still seems to be on a simple, binary true/false in terms of the information that is returned by an LLM, but that's not the real issue and as long as this is not collectively understood, the use of LLMs can be more harmful than we might expect.

We train LLMs to be helpful and to produce outputs we deem correct, but since these models don't actually have a proper concept of true and false information, we actually train them to produce outputs that look correct. This isn't just limited to answering questions, but can also mean that the model will deceive its users about what it can and can't actually do.

When dealing with an LLM's outputs, we have to think of it not as reading something that was written by someone who might not know what they're talking about, but rather like it might be a sophisticated piece of fake news. It might make sense, it might sound plausible, but there could be an important detail that doesn't actually reflect the truth.

This is true for code produced by LLMs as well. "LLMs are like junior developers" is a popular saying, but it doesn't quite capture that you shouldn't review that code as if it was written by a junior, as the mistakes made by it might be a lot more subtle than that. AI-written code should be reviewed as if it was written by a senior, someone who knows what they're doing and can produce coherent code, but still might miss an important detail that doesn't become apparent at first glance. Though even that analogy might not quite capture reality, because one of the big differences between a human developer and an LLM is the fact that LLMs can produce syntactically perfect solutions without any understanding of their real-world implications.

These insights should shape the way we educate people on LLMs and their effective usage to ensure that they actually benefit us, not just deceive us into thinking they do, which could have devastating consequences. Education on LLMs should not just be about prompting techniques, but about developing a critical mindset specific to verifying their outputs.

We should put more effort into explaining the fundamental concepts of LLMs and generative AI models in general to a non-technical audience, because the insights gained from that are invaluable when it comes to analyzing outputs generated by these models.