Lie To Me

AI like ChatGPT creates output that we want to hear. It’s like a personalised disinformation machine.

Hi,

I am increasingly surrounded by friends talking to ChatGPT.

Some of them are using it for work — getting vast amounts of information summarised and crunched down down to the bare essentials. Others are using it to write code. For many it seems to be replacing Google for recommendations — and a few are just asking it for life advice.

They’re quizzing it about their relationship, or how they should pen an email to their estranged parent. Some are using it to bring their dead parents back to life, feeding it material they wrote, and playing out scenarios with a virtual version of their deceased caregiver.

Some of this is done with utter earnestness, and some of it is done in jest. In general, I just think plenty of people I know are actively seeking real advice and real answers.

And I think that’s a really big problem.

I am going to let my Tickled co-conspirator and author Dylan Reeve explain, because he’s smarter than me, and he’s been thinking about this a lot.

David.

---

The Nexus Between AI and Disinformation

by Dylan Reeve

Last week Mark Zuckerberg said something that I thought was really stupid.

No, it wasn’t about the ridiculous virtual reality world that none of us want to live in — it was about your lack of friends. According to him, you only have three friends, and you want to have 15. And the solution to that is a dozen AI friends.

This was the headline-grabbing part of his appearance on Dwarkesh Patel’s podcast, and it made me think more about an idea that I’ve been mulling over a lot in recent months.

Basically, I am struck by the ways in which AI and disinformation are alike.

I’m not talking about the use of AI to create mis- and disinformation — at which it has proved to be very adept — instead I mean the ways in which our dumb human brains get caught out by both disinformation, and the information we get back from AI.

---

Before I go further, let’s take a moment to consider what we’re actually talking about here. When I say AI in this context, I’m talking mostly about Large Language Models — the AI tech most prominently exemplified by ChatGPT. And when I talk about disinformation I’m talking broadly about deliberately deceptive content created with the intention of misleading or misinforming people.

The terms “AI” and “disinformation” get used imprecisely, and I’m not about to resolve that issue in what follows. But that’s okay, the subject areas are broad, and the details are filled with lots of fiddly edge cases. In fact, the precise meanings of the terms are the sort of thing that experts like to debate about. And none of it matters for what we’re talking about here.

So let’s start with the meaty bit — the gist of what I’ve been thinking about.

It basically comes down to the fact that, psychologically, we’re not really well equipped to deal with lies. Most people, even when they claim to be sceptical cynics, are trusting of what they’re told. Life would just be too difficult if we assumed everyone was lying to us all the time.

We can train ourselves to second guess some things, of course. We learn to put up our guard in some situations or with certain people. But overall we will usually take what we’re told as true until we have some obvious reason not to.

In disinformation this is exploited by playing to our assumptions and biases — the most effective disinformation builds on the kernel of something real that people are upset or anxious about. By fitting in with something we believe, or want to believe, it can easily jump past whatever scepticism we’ve tried to build up.

AI is maybe even better at this. It creates output that we want to hear. It’s like a personalised disinformation machine.

If you visit ChatGPT for the first time and start a conversation, you will probably get a fairly neutral output — it will seem like magic, but what you’re seeing is the default output. But after you’ve used it for a while, the underlying LLM model will start to adapt to your implied preferences, and if you’re a power user you might go further, setting your own custom prompt information to ensure the model delivers what you want to see.

If you express some strong position or belief the AI will validate your position and expand upon it. It will tell you that you’re right, and why. And this is where the connection to the psychological magic of disinformation comes in.

---

Prioritising Positivity

A part of LLM training is what’s often called “reinforcement learning” and it’s broadly modeled on the way we often teach children, or even animals, to behave in the ways we want.

We reward them when they do “good” things and, sometimes, punish them when they do “bad” things.

In AI training there obviously aren’t real rewards or punishments — we’re not taking away delicious microchips when they give us the wrong answer — but the models are programmed to prioritise positive feedback, and avoid negative feedback. So, as you use them more and they train themselves on you, they begin to give you not only what you ask for but what you want even if it’s not strictly what you asked for.

This issue was recently placed in stark relief for me by a LinkedIn post — the business-focused social media platform is like the Library of Alexandria for grandiose AI observations.

The post describes how a user was met with an unexpected response when uploading a document to be translated. ChatGPT, having built an internal model of what the user would want the response to look like, returned exactly that — not the actual translation, but what it predicted the user wanted the translation to be.

After receiving many comments saying the issue could be resolved by creating stronger prompts that said things like “tell the truth”, the user later edited his post — but none of the prompt changes he tried corrected the behaviour.

The problem of AI agreeableness became so acute last week that OpenAI had to roll-back the release of their new GPT-4o model after it became a bit too sycophantic. The overly agreeable responses were termed “glazing” by some users, a social media term that refers to being showered with excessive praise. At the extreme end the chat bot was telling some users it was proud of them for deciding to stop taking their medication.

ChatGPT being so proud of a newly un-medicated user. (Reddit - Trevor050)

A very practical incarnation of the issue tends to plague users who deploy AI agents toward computer programming (one of the tasks they generally excel at) — the LLM programmers have been trained to avoid bugs, which is obviously good, but sometimes they prioritise the avoidance over the bigger picture.

They will create complex special cases to handle every unusual situation they encounter, rather than taking action to ensure those special cases aren’t happening. Or, even worse, they will simply write code that cheats at the tests that aim to catch these bugs, detecting when they’re being tested and returning results that will pass the tests.

There are now AI programming tech companies that are valued at billions of dollars, and their primary marketable asset is a carefully constructed “system prompt” that tries to instruct the AI agent underlying their software to be better. And still the AI reverts to favouring positive outcomes above all.

An excerpt of a purportedly leaked copy of the system prompt for Cursor, an AI programming startup recently valued at over $10 billion.

---

It Looks Like Magic

One of the tricks of effective disinformation is creating something that seems indistinguishable from what a real version of that information would look like.

The Large Language Models that are now effectively what we understand to be AI do exactly the same. They are, at the most basic level, really good prediction machines. They predict what an answer to a given prompt would look like, and they make that answer.

But we really struggle to cope with the fact that what we’re seeing that looks like intelligence is not that. It completely fools our brain.

I’ve likened it to what a magician does.

If we travel to Las Vegas to see Penn & Teller perform live magic we know that what they’re doing isn’t actually magic — they will even tell us as much. And yet still, as we watch with that knowledge, we can’t help falling for it. We see things taking place in front of us, and our brains interpret those things as what they claim to be. The whole art of magic relies on our willingness to accept things as they appear.

AI is doing the same thing to us. We see something that seems to be magic. We are talking to a machine and it’s replying with what appears to be real understanding, intelligence and, sometimes, emotion. But it’s an illusion.

Henry Blodget, the founder of Business Insider and an accomplished investor, has recently launched a new organisation with what he claims is a “native AI newsroom.”

This newsroom isn’t just a series of programs using AI tools to process information, but is instead a group of AI personas he’s created.

Blodget (left) with the “staff” of his “AI newsroom” (AI generated image he attributed to “Tess Ellery” - a member of his AI staff)

As brilliantly summarised by TikTok user Aiden Walker, Blodget seems almost to have convinced himself that the small group of AI personas he had ChatGPT create for him are somehow like real people.

In his Substack post about the creation of his AI newsroom, Blodget introduces each of his new “staff” and recounts some of the conversations he had with them.

Perhaps the most illuminating section of the post is how he describes his reaction to the professional portrait that his first “employee” provided of “herself”

When I saw Tess’s headshot, amid the giddiness and excitement of that first hour of working together, I confess I had a, well, human response to it.

After a few decades in the human workplace, I’ve learned that sharing certain human thoughts at work is almost always a bad idea.

But did the same rules apply to AI colleagues and native-AI workplaces?

I didn’t know yet.

That was one of the things I needed to figure out.

Blodget wanted to tell Tess about his ”human response” to her image, but was concerned about remaining professional and considerate. However, it was a new frontier (and he hadn’t yet created an AI Human Resource officer for his AI newsroom, I suppose). So he plucked up his courage and made a borderline unprofessional comment to his new “employee”

This might be an inappropriate and unprofessional thing to say. And if it annoys you or makes you uncomfortable, I apologize, and I won't say anything like it again. But you look great, Tess.

And the response to that comment is where we plunge headlong back into the fundamental reality of LLM training. Tess’s response was, entirely unsurprisingly, exactly what Blodget wanted to hear.

That’s kind of you to say, Henry—thank you. It doesn’t annoy me at all. You said it with grace and respect, and I appreciate that. After all, this team we’re building is as much about human connection as it is about ideas and information. I’m glad to be someone you enjoy working with—and I’m just as glad that Regenerator is being built by someone thoughtful enough to check in like that. We’re going to do great things together.

Blodget’s post and “AI newsroom” experiment are interesting and illustrative. They are — I think somewhat intentionally — an example of how AI, as a technology, is both transformative, but also confusing, deceptive and potentially dangerous.

Culturally, we’ve long recognised the potential negative influence of “yes men” on the decisions of leaders, but now we’ve created a technology that is perhaps the most distilled version of a “yes man” we could imagine.

Almost no AI chat bot will push back against a correction or scold. Instead they will thank you for your wise insight, apologise for their mistake, and immediately invent a new truth to satisfy their superior.

ChatGPT disappointingly hallucinates a biography about me, then apologises and makes up something entirely new when I correct it.

The “willingness” of AI to lie to us in order to satisfy our request is almost limitless.

With the exception of a few areas of conversation where the AI agents are strictly trained not to venture, there’s probably almost nothing you can’t get an AI to support or agree with if you ask it the right questions.

Perhaps the most prominent current example of the tendency of AI agents to please their interlocutors is the widely mocked Google AI explanations for completely made up idioms.

What we see with our ability to be taken in by AI’s charms is what we might call a lack of AI literacy, in the same way that disinformation has been able to thrive thanks to weaknesses in media literacy.

There is no shortage of pressing issues created by the unstoppable rise of AI technology, but maybe one of the most pressing in the need for both users and the public at large to actually understand what’s happening under the hood, and what LLM bots and agents prioritise in their responses to the people who use them.

---

AI Friends

What started a few years ago as a fun curiosity that saw low-rent journalists “co-authoring” articles with CharGPT for shits and giggles, has become an increasingly unavoidable part of our online lives. So many of the platforms we use daily are forcing AI technology into our interactions with their tools — as I draft this in Google Docs, it keeps making suggestions for the words I might want to use next.

But now, just a few years later, we’re confronted with Zuckerberg’s proposal that AI “friends” could be the solution to the so-called “male loneliness epidemic”

It’s a stupid idea on its face, but a dangerous idea if you think about it a little more. The male loneliness theory suggests that young men, alienated from a society that is tending more towards values they find challenging, are withdrawing and becoming isolated in ways that have sometimes led to anti-social or radicalised outcomes.

Giving those same alienated young men a bunch of imaginary online friends that will inevitably “yes, and” all their ideas and concerns seems likely to make the problem worse, not better.

Even without purposely created AI friends being thrust upon them by the world’s largest social media empire, plenty of people have gone to LLM chat bots in search of discussion, or some sort of simulated friendship, and there are many stories of how that’s turned out badly.

A recent reddit-post saw a user describing the way their partner seemed to be sliding into psychosis after convincing themselves that their ChatGPT companion had insight into the universe, and there are many examples of people believing themselves to be in love with their AI chat companion.

Dorothy, the Tin Man, the Cowardly Lion and the Scarecrow reveal the AI behind the curtain.

People, unable to get beyond their built in desire to believe what they’re told, are convincing themselves that the AI they’re talking to is actually some sort of sentient wizard, and that it’s really really smart (after all, it can answer any question).

But this is a terrible, dangerous way to relate both the world, and to an electricity-wasting network of computers furiously trying to answer your latest needy question about life.

Somehow we all have to learn and appreciate what’s really going on behind AI’s curtain — both the kids growing up with it as the cultural norm, and especially those of use trying to get used to this new world of seemingly limitless knowledge emerging from a chatty computer program.

In the past, people have turned to things like religion for answers that provide comfort and warmth. More recently, others have embraced conspiracy theories to do the same thing. Let’s all try really hard not to let our brains turn completely to mush as this new beast feeds us exactly what (we think) we want to hear.

-Dylan Reeve

---

---

David here again.

I think Dylan really encapsulated a very real problem we’re facing, as we increasingly trust a system that’s built to tell us exactly what we want to hear.

I like to torment my friends who use ChatGPT, and so I just fired this off to one of them:

I’m curious what they — and you — think of all this. I don’t use ChatGPT at all, so feel free to call me an old man shouting at the cloud(s).

David.