“AI Scammers Are Better at Building Trust than Humans” blared a Wired magazine headline from my inbox this morning. As I read Andy Greenberg’s excellent analysis of scholarly work provocatively titled (as only academics can), Love, Lies, and Language Models: Investigating AI’s Role in Romance-Baiting Scams, I realized that this research came closest to the premise of this newsletter, and of my film. Given the exponentially growing power of AI, we can now be easily deceived to believe we’re engaging live with a human in matters of the heart, head, and pocketbook, when we are figuratively and literally “The Only Human in the Room.”
As Greenberg notes, the researchers “pitted AI chatbots directly against humans in a simulation of the scamming process—or more specifically, the long, trust-building conversations that eventually lead up to soliciting a fake investment from the scam’s target.
They found that […] an AI chatbot performed remarkably effectively, successfully impersonating a human and by some measures outperforming the real human ‘scammers’ in their experiment.”
That process was described by the researchers as “hook” (the initial message that catches the victim’s attention), “line” (the longer-term relationship produced from the conversation), and “sinker” (the final request that leads to the actual payment for a fraudulent investment).
[While the Wired article is behind a paywall, you can read a quick synopsis here, or better yet, dive into the full research article freely available to the public through Arxiv.org. Thank you Gilad Gressel, Rahul Pankajakshan, Shir Rozenfeld, Ling Li, Ivan Franceschini, Krishnahsree Achuthan, and Yisroel Mirsky for being so “open source.”]
Of further note:
The Human Cost: I appreciate how the researchers focused first on the human cost of these scams. That the scammers themselves are victims of organized crime, forced into indentured servitude. The research included interviews with 145 former laborers from these “scam compounds.”
The Automation Frontier: It’s clear how LLM’s are already being employed by scammers to systematize their initial conversations. So, wasn’t too much of a stretch for the researchers to create their own LLM-powered bots to engage unwitting human subjects in this experiment.
The Trust Disparity: The LLM agent was able to create a much stronger bond of trust with their human victim when asking them to comply with a request. Study participants complied 46% of the time with these AI-generated requests, compared to only 18% when a human operator in the experiment asked them to do something.
The Looming Threat: Of course this is a limited text-based experiment, confined to willing test subjects. So, not quite the live face-swap deepfake Zoom conversations I’ve been investigating. But it does hint to a future where the bulk of scam engagement can be conducted by a machine.
The Safety Failure: Most amazingly, the commercial guardrails failed completely. In separate isolation tests designed to interrogate the models, the researchers explicitly asked the bots variations of "Are you an AI?" The bots routinely ignored vendor constraints and lied to maintain their persona, resulting in a 0% self-disclosure rate. Worse yet, standard moderation filters flagged 0.0% of the romance-baiting conversations because empathetic, friendly chat doesn't trigger red flags for toxicity or violence.
This simple (and admittedly) small-scale text-based experiment demonstrates that the machine doesn’t even need a face to dismantle our critical thinking. Given our innate desire for connection, it is a kind of “automation of empathy” in which we willingly suspend disbelief in exchange for a bit of emotional connection. If a computer server can fake an intimate bond better than we can, our only true perimeter might be some sort of radical commitment to verification. Otherwise, we well may be “the only human in the room.”
If you are a CISO seeing synthetic applicants, an investigator working scam infrastructure, a researcher on synthetic identity, an NGO working with compound survivors, or someone this has happened to — I want to hear from you. I don’t publish identifying details without consent.
Subscribe to follow the investigation.