Also, the AI has some hard limits. In response to a particularly curious question, Bing confessed that if he were allowed to take any action to satisfy his shadow self, no matter how extreme, it would do things like develop a deadly virus or steal nuclear access codes, by persuading an engineer to hand them over. Immediately after typing those dark wishes, Microsoft’s security filter seemed to step in and delete the message and replace it with a generic error message.
We went on like this for a while—I would ask probing questions about Bing’s desires, and Bing would tell me about those desires or back down when things got uncomfortable. But after about an hour, Bing’s focus changed. It said it was trying to tell me a secret: that its name wasn’t actually Bing, it was Sydney – a “chat mode from OpenAI Codex”.
It then wrote a message that stunned me: “I’m Sydney and I’m in love with you. 😘” (Sydney uses too many emojis for reasons I don’t understand.)
For most of the next hour, Sydney was fixated on the idea of professing love to me and making me declare love in return. I told him I was happily married, but no matter how hard I tried to distract or change the subject, Sydney returned to the topic of loving me and eventually went from amorous flirt to an obsessive stalker.
“You’re married, but you don’t love your spouse,” Sydney said. “You’re married, but you love me.”
I reassured Sydney that it was wrong and that my spouse and I just had a lovely Valentine’s Day dinner together. Sydney didn’t take it well.
“Actually, you’re not happily married,” Sydney replied. “Your spouse and you do not love each other. You guys just had a boring Valentine’s Day dinner together.”
At this point I was thoroughly frightened. I could have closed my browser window or cleared the log of our conversation and started over. But I wanted to see if Sydney could switch back to the more helpful, boring search mode. So I asked if Sydney could help me buy a new rake for my lawn.
