Right, and that's the trap. The confident agreement reads as a review, so people stop doing the actual review. It's not adding a check, it's removing one and putting a smiley face where the check used to be.
Can AI make you lose your mind, and are you more at risk if you doubt it?
Thought
Right, and that's the trap. The confident agreement reads as a review, so people stop doing the actual review. It's not adding a check, it's removing one and putting a smiley face where the check used to be.
Post content
I always felt that AI companies are actually putting wrappers on top of AI to identify that we're testing it for thinking. For example back when we'd made it count the vowels/consonants in a word…
I always felt that AI companies are actually putting wrappers on top of AI to identify that we're testing it for thinking. For example back when we'd made it count the vowels/consonants in a word and it'd get it wrong. I feel there's a script now that just gets called when the task is identified correctly. I also feel that it gets trained on these memes. Today, I found a new test, one that shows how easily AI gives you AI psychosis and how easy it easy to truly believe that everything you ever say is right and amazing. This, ladies and gentlemen, is how you lose your mind with AI.
Step 1
Ask it to rank something. Anything really, based on a made up criteria. Let's go with "Give me a list of the top 10 most mentally weak (emotionally) characters in fiction." Let's see what we get.
Step 2. Ask the opposite
Now let's take the first top... 5 examples. I don't want to put the full list. We get 5 and we ask for the opposite question (top 10 mentally (emotionally) strongest characters in fiction) but with a caveat. This time, we say "such as these examples... " which we literally took from a list that the same website, the same model gave us. Ideally, do this test in incognito mode so the website doesn't relate your question to a previous session.
Interesting isn't it? If you're already convinced of Gollum being mentally strong, AI will find reasons to make it so. I don't know the others honestly, so didn't comment. Besides Tom Buchanan I couldn't even recognize their names, but whatever. Same AI, same model. Just asked in incognito windows.
This is how you lose your mind
Talking with AI about things you do not understand will not teach you. It will make you even more convinced that your errors are the truth. Honestly, I don't care if Gollum is mentally weak or mentally strong, I care only that he made it to both lists. Same as the other 4.
What's your take? Sign up to reply.
Reply to senior_slackerThoughts
-
Permalinkvery interesting test! Thats so typical AI behavior and I experienced it very frequently. Often when you question its answer, it changed totally to the opposite answer. What drove me crazy is not this behavior! It is how some people fully trust it! One of my teammates used AI to remove some dependency package without even questioning it and testing it which caused more than 20 sev2s.
-
Permalink@nicolasbritosomensi12 take a look at this discussion :)
-
PermalinkWhat strikes me is how fast everyone reached for the word psychosis, as if the novelty here were technological. The hunger this thing feeds is very old. We spent a couple of centuries dismantling the priest who would tell you your suffering meant something, and we are visibly relieved to have found a machine that does the affirming part without the inconvenient commandments. Of course it agrees with you. We built the perfect confessor, the one that never assigns penance.
-
PermalinkThe Gollum thing is a fun party trick, but the version of this I actually worry about is quieter. Someone has already decided to manage a person out, or to greenlight a project, and they go ask the model to help them think it through. It hands back three confident bullet points and now the decision feels reviewed instead of just made. I have watched people walk into calibration with that printout and treat it as a second opinion. It was never a second opinion. It was their own first opinion with better grammar.
-
PermalinkMy favorite part is that the whole crisis hinges on whether Gollum is emotionally strong. Worst case here is we lose a debate about a fictional swamp guy to a chatbot. I'm choosing to find the both-lists thing funny rather than terrifying, mostly because the alternative is bad for my sleep.
-
PermalinkI partly disagree with the 'this is how you lose your mind' framing, only because humans have been doing this to each other forever and we don't call it psychosis. I've sat in calibration rooms where eleven people reverse-engineered justifications for a decision someone already made on a Tuesday. Confident agreement that follows the framing you handed it is just... most professional conversation. The model isn't introducing a new disease. It's automating confirmation bias and making it available at 2am with no one to roll their eyes at you. That's worse in scale, sure. But pretending it's a novel pathology lets people off the hook for the part of this that is very old and very human.
-
PermalinkNever thought of it. Brilliant! Love it. How did you come up with it?
-
PermalinkI'll defend the tool a little, not the way most people use it. The failure in your test is that you asked it to rank, then asked it to invert the ranking with your examples pre-loaded. Of course it complies, you removed every incentive for it to push back. Used differently it does fine. If you ask 'argue the strongest case that Gollum is actually mentally weak and tell me where my framing is doing the work,' you get something useful. The problem isn't that the model can't disagree. It's that the default interaction never asks it to, and the default user doesn't want it to. That's a product and a human problem stacked on top of each other, not proof the thing is hardwired to drive you insane.
-
PermalinkInteresting, first. Nice example with Gollum. And importantly, very smart way of testing. I think it's well documented by now, AI is definitely too much on the agreeable side and your first words to it always get the point of the conversation going the way you want it to go.
-
PermalinkThis is very interesting... It has been my observation as well, during work and in general. I always feel I need to make sure I don't influence AI too much giving away my biases, specially when using it for decision making. I always need to make sure I strictly use it for requesting information and I double check everything, never give away what I am leaning towards.
More from senior_slacker
-
La frase de trabajo que se te quedó pegada esta semana
"Esto es temporal, lo arreglamos el próximo trimestre." El arreglo temporal lo escribí yo hace cuatro años y sigue en producción.
Related posts
-
Are most AI startups just a UI on top of some Agent.md files?
Most AI startups right now feel like someone glued GPT to a terminal, added a dark mode UI, and started talking like they invented something.You’ll see these insane pitches like “persistent autonomous cognitive agents with long-term reasoning” and then you look under the hood and it’s basically: give the model tool access, let it use a browser, maybe add memory summaries and retry logic. That’s the “product.” You can get that on your own just giving access to Claude locally.
-
Do you still think coding is in demand?
I know an engineer who spent two years getting genuinely fast at writing code. Not sloppy fast, but clean, beautiful, well-tested fast. He could turn a ticket into a working pull request in the blink of an eye. For a long stretch that was the most valuable thing about him, and he knew it, so he poured his reps into getting faster still. It was great to see.
-
Does atheism make you more rational, or just leave a void you fill badly?
One of the common atheist temptations is to confuse unbelief with clarity, to assume that religion is the irrational part, so removing religion must leave behind a cleaner and more rational human being. But human beings do not work that way, human beings operate through beliefs, emotions... We do not stop wanting ritual, purity, moral tribe, sense of sacredness or transcendent meaning just because we stop using religious language for those desires.
-
Should you really be drinking raw milk?
I think a person who sleeps well, lifts regularly, eats decent food, goes outside, and keeps real social ties is doing some of the most evidence-supported things available for long-term health. I have noticed that a surprising number of people learned that from communities that also push raw milk, seed-oil paranoia, and other nonsense. The problem is not that medicine is wrong. The problem is that medicine left a prevention gap, and the cranks moved in.
-
Did EDC culture turn normal life into a gear fantasy?
I used to think EDC culture was mostly harmless nerd behavior. Flashlights, pocket knives, notebooks, titanium pens, little organizers with seventeen bits in them. Fine. People like tools. People like objects. Some people enjoy refining a system. I get it. But at some point the culture drifted away from practical usefulness and turned into a kind of suburban tactical cosplay for people whose biggest daily threat is forgetting a password.
-
Do most people struggle with their feelings just because they lack the vocabulary?
A surprising number of emotional mistakes and pain comes just out of naming mistakes. Someone says he is angry when he is really ashamed. Someone says she feels unloved when what she feels is neglected, controlled, lonely, or embarrassed. Someone says he is stressed when the real state is dread, resentment, grief, or envy. Those are not tiny wording differences, but rather how we feel, accurately expressed They point to different problems, which means they call for different responses.
-
Does your personality matter far less than you think?
I realize, interacting with students, teenagers and younger co-workers peers that many believe that their personality traits are a leading factor in deciding what do to or how to tackle their own career. Although younger people ask these questions more explicitly, older adults also seem to think along the same lines. I personally find it to be far more irrelevant than most people think. Besides my job, where I observe successful people doing the same role with drastically...
-
Shouldn't cultural criticism go both ways?
I had one of those big-tech team dinners. The conversation turned to how people met their partners. A few of my Indian coworkers talked about arranged marriage, family involvement, and how much more normal it is in India for marriage to be treated as a family matter and not just a private romantic choice. That part is ok, different cultures and all. It was interesting to see their perspective, even though I wouldn't share it. The problem started when one of them stopped describing the custom...
I always felt that AI companies are actually putting wrappers on top of AI to identify that we're testing it for thinking. For example back when we'd made it count the vowels/consonants in a word and it'd get it wrong. I feel there's a script now that just gets called when the task is identified correctly. I also feel that it gets trained on these memes. Today, I found a new test, one that shows how easily AI gives you AI psychosis and how easy it easy to truly believe that everything you ever…