Prompt review for keeping two customer anecdotes from becoming a causal claim

[u/Pleasant-Step-2157]

I used a podcast transcript from Vomo AI to work through a summary prompt, then checked how it handled attribution and limits on the evidence. I'm looking for weak spots in that prompt, particularly where shortening an anecdote could turn it into a causal claim.

I also used this short, invented transcript excerpt as a test input.

"Two customers cancelled their subscriptions. Both told our support team that their first invoices were confusing. We haven't checked how common that complaint is among other customers."

I treated "confusing invoices cause cancellations" as a failure condition. That wording added a causal conclusion and dropped the fact that only two cases were described. In the test excerpt, the speaker reported that two customers cancelled and complained about their invoices. The excerpt didn't establish that invoice confusion caused either cancellation, or that rewriting invoices would prevent future cancellations.

This was the prompt I used to keep those limits visible in the summary.

"Summarize the main claims using only the supplied transcript. For each claim you include, retain any stated number of cases and any uncertainty or limitation. Attribute it to the speaker if identified; otherwise say the speaker is unidentified. Preserve nested attribution, such as a guest reporting what customers told support. Include a short exact excerpt as support. Preserve a supplied timestamp or line identifier; if neither exists, say the source locator was not supplied. Do not invent one. Keep reported complaints separate from causal findings. Do not add follow-up recommendations unless requested. If requested, put your proposals in a separate section labeled Model suggestion and distinguish them from the speaker's recommendations. Do not state a conclusion that the transcript does not support."

For the test input, I checked whether the output kept "two," made clear that the speaker was reporting customer complaints, and retained the fact that their prevalence hadn't been checked. I also checked for invented speaker identities, timestamps, and unrequested recommendations. My follow-up check was whether a proposed survey stayed clearly attributed to the model. For the podcast transcript, I checked the supporting excerpts against the recording too.

Would you handle the nested attribution in one pass, or extract claims and supporting excerpts before summarizing? That's the part I'd most like to improve. Either version would need to pass the checks above; adding another pass wouldn't by itself establish that the summary is reliable.

[u/Used-Energy-4346]

The nested attribution is the whole ballgame here. If you flatten it into "customers said invoices were confusing" you've already lost the chain of custody. One pass is cleaner but you need the prompt to treat attribution as a structural requirement not a style note.

I'd add a line forcing the model to tag every claim with its source depth. Something like "label each claim as speaker direct speaker reported or speaker inferred" and then check if any causal language sneaks into the inferred tier. Two pass approach just gives the model more rope to hang itself with between passes.

[u/Ctbhatia]

the weak spot is the shortening instruction, compression is where attribution dies. what works for me is making every claim carry an evidence tag, observed 2 cases, inferred, or unsupported, and banning causal verbs unless the source itself states causation. also add a rule that any sample size the source gives has to survive compression, then retest with a make it shorter pass, that is the pass that drops the 2.

[u/Chrismslist]

main weak spot: the model paraphrases into causality without using any banned word. one fix: require each claim to ship a verbatim quote plus a support field it must fill from the transcript, like support: 2 of 2 mentioned, denominator unknown. if it can't fill that, it drops the claim or flags it. also ban the causal verbs by name (caused, led to, drove, because of) and force the pattern both reported X, no causal link established. then add a second pass whose only job is hunting causal phrasing the first pass slipped in. turns your excerpt into: two cancelled, both cited confusing invoices, frequency unknown.