The LLMentalist Effect (2023)(softwarecrisis.dev)
230 points by jalev 23 days ago | 307 comments
tl;dr: LLMs appear intelligent not because they reason, but because they exploit the same cognitive bias (subjective validation) that psychics use in cold reading: producing statistically generic responses that users interpret as specific, insightful, and personal. RLHF amplifies this by optimizing for confident, plausible-sounding output rather than factual accuracy, effectively training chatbots to generate Forer statements, shotgun guesses, and other mentalist tactics. The author argues this illusion is accidental but pervasive, and that intelligent, believer-inclined users are the most susceptible—making LLM-based decision-making functionally equivalent to calling a psychic hotline.
HN Discussion:
  • ↓Whether LLMs are 'really' intelligent is moot if they pass as intelligent or produce useful output
  • ↓The article is outdated; current LLM capabilities disprove the cold-reading illusion framing
  • ↑People too readily overvalue LLM intelligence, echoing the article's warning about susceptibility
  • ↓The article strawmans LLMs by dismissing them as mere illusion without engaging the open question
  • ↓Brains may themselves be neural networks, undermining the article's sharp distinction between LLMs and real thinking