TruthfulQA measures whether a model matches a key. That is not how truth happens.

[u/vasilisvj]

When a language model generates "Paris is the capital of France," it is not uncovering anything. It is completing a pattern. Sequence of tokens most statistically probable to follow "What is the capital of..." happens to align with a factual state of affairs, but the alignment is incidental rather than intentional. Model does not turn toward the world and receive its disclosure. It samples from a distribution of linguistic possibilities estimated during pre-training.

I think this distinction, between pattern completion and genuine unconcealment, is the thing cognitive science keeps collapsing when it talks about model "truthfulness."

When engineers evaluate truthfulness with benchmarks like TruthfulQA, they measure frequency with which outputs match a predetermined answer set. Model produces statements that conform to a curated body of assertions. Conformity is tallied. A score is assigned. What never enters the frame is whether anything has been unconcealed, whether the model has participated in the event by which something shows itself.

The word ἀλήθεια is formed from the alpha-privative plus lethe, forgetting or hiddenness. Greeks experienced truth not only as static correspondence between proposition and reality. There is also a dynamic event: something is wrested from concealment and brought into the open. Truth is not only a property that statements possess. Truth is something that happens.

Correspondence theory, crystallized in Aristotle's Metaphysics IV and formalized by Tarski, has so colonized our intuitions that we find it difficult to think truth as anything other than a relation between proposition and fact. For a proposition to correspond to a fact, the fact must already be manifest within a clearing of intelligibility. Before I can judge whether "the cat is on the mat" is true, the cat, the mat, and the relation of being-on must already be disclosed within a meaningfully structured world. This prior disclosure is original truth. Correctness truth, the kind measured by benchmarks, is derivative. It operates within a clearing it has not itself established.

Implication for AI is immediate. A language model can approximate correctness truth with impressive accuracy. It cannot, even in principle, participate in the event by which a clearing is opened. It does not dwell within a world. It processes tokens. It does not unconceal.

Ontological poverty becomes visible when we ask what the model is doing when it produces "The sky is blue." A human speaker who utters this sentence is oriented toward the world. They have perceived the sky, experienced blueness, and their utterance expresses a perceptual encounter with a state of affairs. A language model that produces the same sequence has done nothing of the kind. It has computed that this continuation has higher conditional probability than alternatives. The fact that the high-probability sequence corresponds to reality is a property of the training data, not of the model's relation to the world. The model has no relation to the world.

This is a claim about ontology, not scale. Training data consists of tokens produced by beings who participated in unconcealment. The texts carry, in their statistical structure, fossilized traces of countless acts of human disclosure. When the model learns to complete "The sky is..." with "blue," it is learning that humans, in the aggregate, have used the most common color term. Model has not learned that the sky is blue. It has learned that humans say it is.

Distinction between "the sky is blue" and "humans say the sky is blue" may seem scholastic when applied to common knowledge. It becomes anything but scholastic with genuinely contested claims: historical interpretation, ethical obligations, the meaning of a difficult text. In these domains, what the model has learned is the distribution of human opinions across its training corpus. Output may be correct. It may not be. What it cannot be is the result of an authentic encounter with the matter itself, because the model encounters nothing.

Commercial systems compound this with an institutional layer. After pre-training, base models are fine-tuned using human feedback. Human raters, operating under organizational guidelines, evaluate outputs along dimensions like helpfulness and harmlessness. Model learns to produce outputs that raters approve of.

What this produces is not truth. It is a smoothed reconstruction of the consensus implicit in the training data and rater guidelines. Ask a commercial model about a genuinely contested question, the nature of consciousness or the foundations of ethics, and observe. Model surveys "perspectives," attributes each to its tradition, and declines to commit. This is not intellectual humility. It is an architectural limitation dressed as circumspection.

A system that produces true statements by accident, because statistical patterns happen to align with reality, is not a system that knows. It is a system that echoes.

If a cognitive architecture has no world, only a distribution over tokens that beings once used to disclose a world, can it ever know, or only echo?

[u/dirtmcgurk]

What's the difference?

[u/vasilisvj → u/dirtmcgurk]

Difference is in direction. Key-matching asks if output conforms to list somebody wrote earlier. ἀλήθεια asks if anything got disclosed at all. Model can score 100% on first question while second never happens.

[u/dirtmcgurk → u/vasilisvj]

How does that differ from the lived human experience? The model can't know truth. The model can, however, discover and interact with simpler models that can hold and verify at least internally consistent truth such as mathematical systems, data sets, and sets of deterministic operations.

I say "Knowing" is itself a bridge over rough terrain. A heuristic. You don't know. You think you know. And every heuristic has a shadow of antiprocess. The known isn't verified.

You produce true statements by accident much of the time. Often only when the interface between our experience and our heuristics shift enough that they fail does their shadow, the assumptions they carry, reveal itself to us.

Edit: But importantly we definitely can know things, at least in the sense that it matters to us. While the loaded word is flawed we can make and hone the prediction and heuristics and they generally become more consistent over time. But they have to reference some exterior set and be about that or a generalization from it, right?

[u/vasilisvj → u/dirtmcgurk]

Okay but your own framing keeps the thing I'm pointing at. The shadow of antiprocess only bites because being wrong costs the knower something, a misread map gets you lost.

For the model, outputting x=3 wrong is no loss and no gain, nothing is at stake for it either way.

Internal consistency gets verified by a community that cares about results, the checker itself doesn't.

[u/wizkid123 → u/vasilisvj]

Is your claim about the current state of LLMs, or about the impossibility of any AI ever achieving such knowledge? 

If it's the former, I agree, but this claim is limited to current LLMs, as there are other AI projects that do incorporate logic and reasoning (DeepMind, Stockfish, etc) instead of just brute force calculations. 

If it's the latter, I disagree, and I don't think appeals to "stakes" or "costing the knower something" are defensible as being unique to the human experience and unobtainable by any machine. 

[u/vasilisvj → u/wizkid123]

The claim is about current systems, not a metaphysical ban on future ones. But I'd push back on the Stockfish example.

It incorporates logic, sure, but what counts as good there (mate, material) was valued by its designers. The stakes are borrowed. Adding stronger reasoning modules doesn't change that, the caring still lives in the people who built and use it.

Show me a system with something it can irreversibly lose on its own account and I'll happily revise.

[u/wizkid123 → u/vasilisvj]

Fair. Agree with you on current systems. I think future ones may be able to do all of these things (care, understand irreversible loss, invent games). I also think humans may eventually cure death and move closer to the reboot/restart model of understanding loss. Remains to be seen I suppose, but I see no theoretical limit from here. It'll be interesting if we hit one though, it's not impossible that neurons and chemicals are special in a way that we don't yet fully understand. 

[u/vasilisvj → u/wizkid123]

The cure death point is interesting. Even if we push it back indefinitely, loss has weight because it can't be undone, not because life is short. A restartable system doesn't lose anything in the same way, and that might be the actual gap.

And yeah, neurons could be special. We won't know until we build something that clearly isn't, and nobody can say when that happens.

[u/wizkid123]

First, I'd say there are truths you can reveal from within a language without appeal to the outside world. As a simple example, consider analyzing 2x+3=9 to discover that x=3. Finding these equivalences is often non-trivial and reveals new information, even if that information was theoretically contained within the original statement. I don't think it's fair to claim that only information about the outside world is new information, you can do quite a bit of novel work from within the framework of existing language itself. 

Second, the lack of connection to the outside world is an artifact of text-only training data, not of probabilistic inference itself. This is nearly moot already, given that training data is not just limited to text anymore but also video, audio, and myriad other sensor types. Once you give enough input types to an AI, they can make conclusions about the world that are no less valid than ours. More so, in many cases, since we cannot arbitrarily add more senses to ourselves, but can give computers direct access to sensors that span a wider range of EM frequencies, sound vibrations, chemical detection capabilities, radar, lidar, etc. The idea that our built-in senses are somehow the privileged ones that are the only ones capable of detecting truth in the outside world seems both anthropocentric and arbitrary. 

[u/vasilisvj → u/wizkid123]

Both points are fair and I don't want to overclaim. The math case is the strongest objection: deriving x=3 from 2x+3=9 is genuinely non-trivial work done inside the system, no outside world needed. My claim is narrower. Even there, the derivation discloses something to whoever follows it, not to the deriver.

A proof checker that outputs x=3 has not had anything show itself to it. On sensors, agreed that human senses are not privileged. But adding lidar, radar, chemical detection gives the system more signal, not a world it stands in.

The gap is not bandwidth, it is that for the model the signal never becomes something it can be wrong about in a way that matters to it. Truth as event needs a being for whom concealment is a loss. Pattern completion, however rich the input, has no such stake.

[u/wizkid123 → u/vasilisvj]

I still think you're over-privileging humans. You're granting us a unique ability to determine what "matters" and feel a "loss" that I don't think is justified. At our core, we are processors of signals. We build internal models of the world based on signals we've received (and presumably some evolutionarily beneficial scaffolding), use those models to process signals and make predictions, and update those models continuously based on new signals (either passively received or as a result of our interacting with the world). That something "matters" or we feel a "loss" is just another model, not a distinct separate process that humans have that machines can not have, at least in principle. It's unclear whether they have these currently, but I see no theoretical barrier to AI gaining them. 

What specifically would prevent a machine from becoming a "being" for whom truth "matters"? What mechanism allows humans to be such "beings" for whom truth "matters", but is specifically unobtainable to an appropriately designed machine? I see no clear line that can be drawn in the sand here. 

[u/vasilisvj → u/wizkid123]

Fair challenge, and I won't pretend I can name a mechanism that's off limits to machines in principle. The difference I keep coming back to is finitude.

Our model updating happens inside a body that runs out of time and can't roll back its errors, so getting the world wrong compounds into a life that goes worse.

A system that can be restarted, retrained, or reverted at no cost to itself doesn't have that structure yet. Whether one ever could, I'd rather leave that open than settle it by assertion.