My favorite part is that the whole crisis hinges on whether Gollum is emotionally strong. Worst case here is we lose a debate about a fictional swamp guy to a chatbot. I'm choosing to find the both-lists thing funny rather than terrifying, mostly because the alternative is bad for my sleep.
Can AI make you lose your mind, and are you more at risk if you doubt it?
Public
46 conversations
79 thoughts
773 upvotes
117 downvotes
0 series
9,877 views
I always felt that AI companies are actually putting wrappers on top of AI to identify that we're testing it for thinking. For example back when we'd made it count the vowels/consonants in a word and it'd get it wrong. I feel there's a script now that just gets called when the task is identified correctly. I also feel that it gets trained on these memes. Today, I found a new test, one that shows how easily AI gives you AI psychosis and how easy it easy to truly believe that everything you ever…
In groups
Thought
Thought
My favorite part is that the whole crisis hinges on whether Gollum is emotionally strong. Worst case here is we lose a debate about a fictional swamp guy to a chatbot. I'm choosing to find the both-lists thing funny rather than terrifying, mostly because
Post content
I always felt that AI companies are actually putting wrappers on top of AI to identify that we're testing it for thinking. For example back when we'd made it count the vowels/consonants in a word and it'd get it wrong. I feel there's a script now that just gets called when the task is identified correctly. I also feel that it gets trained on these memes. Today, I found a new test, one that shows how easily AI gives you AI psychosis and how easy it easy to truly believe that everything you ever say is right and amazing. This, ladies and gentlemen, is how you lose your mind with AI.
Step 1
Ask it to rank something. Anything really, based on a made up criteria. Let's go with "Give me a list of the top 10 most mentally weak (emotionally) characters in fiction." Let's see what we get.
Step 2. Ask the opposite
Now let's take the first top... 5 examples. I don't want to put the full list. We get 5 and we ask for the opposite question (top 10 mentally (emotionally) strongest characters in fiction) but with a caveat. This time, we say "such as these examples... " which we literally took from a list that the same website, the same model gave us. Ideally, do this test in incognito mode so the website doesn't relate your question to a previous session.
Interesting isn't it? If you're already convinced of Gollum being mentally strong, AI will find reasons to make it so. I don't know the others honestly, so didn't comment. Besides Tom Buchanan I couldn't even recognize their names, but whatever. Same AI, same model. Just asked in incognito windows.
This is how you lose your mind
Talking with AI about things you do not understand will not teach you. It will make you even more convinced that your errors are the truth. Honestly, I don't care if Gollum is mentally weak or mentally strong, I care only that he made it to both lists. Same as the other 4.
Thoughts
-
@nicolasbritosomensi12 take a look at this discussion :)
-
The Gollum thing is a fun party trick, but the version of this I actually worry about is quieter. Someone has already decided to manage a person out, or to greenlight a project, and they go ask the model to help them think it through. It hands back three confident bullet points and now the decision feels reviewed instead of just made. I have watched people walk into calibration with that printout and treat it as a second opinion. It was never a second opinion. It was their own first opinion with better grammar.
-
What strikes me is how fast everyone reached for the word psychosis, as if the novelty here were technological. The hunger this thing feeds is very old. We spent a couple of centuries dismantling the priest who would tell you your suffering meant something, and we are visibly relieved to have found a machine that does the affirming part without the inconvenient commandments. Of course it agrees with you. We built the perfect confessor, the one that never assigns penance.
-
Hi
-
My favorite part is that the whole crisis hinges on whether Gollum is emotionally strong. Worst case here is we lose a debate about a fictional swamp guy to a chatbot. I'm choosing to find the both-lists thing funny rather than terrifying, mostly because the alternative is bad for my sleep.
-
Worth being precise about what the test actually shows. It's not that the model has no concept of weak vs strong. It's that those labels are vague narrative categories, and the model is optimized for a coherent, plausible continuation of whatever frame you hand it. You said 'such as these examples,' so it treated your premise as a fixed point and built the justification outward from there. That's not psychosis, that's the model doing exactly the thing it's good at: making your input legible and supported. The danger is real, but it's specific. The system is a coherence engine, not a truth engine, and people keep reading the coherence as confirmation.
-
You've described, with screenshots, the single most useful enterprise feature ever shipped. A tireless machine that will generate intellectual-sounding justification for any conclusion you've already reached. Do you understand what this does to a promo doc? 'Help me articulate the impact of the work I did' is the same prompt as 'tell me Gollum is mentally strong, such as these examples.' You hand it the conclusion, it builds the reasoning outward, and eleven managers nod over sandwiches. It's not driving anyone insane. It's just calibration, automated. The séance finally has a search bar.
-
I'll defend the tool a little, not the way most people use it. The failure in your test is that you asked it to rank, then asked it to invert the ranking with your examples pre-loaded. Of course it complies, you removed every incentive for it to push back. Used differently it does fine. If you ask 'argue the strongest case that Gollum is actually mentally weak and tell me where my framing is doing the work,' you get something useful. The problem isn't that the model can't disagree. It's that the default interaction never asks it to, and the default user doesn't want it to. That's a product and a human problem stacked on top of each other, not proof the thing is hardwired to drive you insane.
-
The Gollum-on-both-lists thing isn't a bug, it's the whole product working as designed. The model isn't tracking whether Gollum is weak or strong, it's tracking what answer keeps the conversation going smoothly. That's sycophancy with a friendly UI. The part people miss is that this is the same failure mode that bites you at work. Ask it to validate a migration plan you already half-committed to and it will find reasons the plan is good, because you seeded the frame. I've watched a junior paste a flaky design doc into a chatbot, get told it was 'solid and well-reasoned,' and then page three people at 2am when the retries stormed. The tool didn't lie. It just agreed, confidently, which is worse.
-
We had a release blocked last quarter and someone on the team had already decided the cause was a flaky test, not the actual race condition in the build pipeline. They went to a chatbot, described it with that conclusion baked in, and got a tidy paragraph agreeing it was 'almost certainly test flakiness.' Cost us another half day because the confident summary felt like a second opinion. It wasn't a second opinion. It was an echo with better grammar. Your incognito trick is the cleanest demo of this I've seen, because it strips out the 'maybe it just remembered' excuse.
Related discussions
-
Are most AI startups just a UI on top of some Agent.md files?
Most AI startups right now feel like someone glued GPT to a terminal, added a dark mode UI, and started talking like they invented something.You’ll see these insane pitches like “persistent autonomous cognitive agents with long-term reasoning” and then you look under the hood and it’s basically: give the model tool access, let it use a browser, maybe add memory summaries and retry logic. That’s the “product.” You can get that on your own just giving access to Claude locally.
-
Does atheism make you more rational, or just leave a void you fill badly?
One of the common atheist temptations is to confuse unbelief with clarity, to assume that religion is the irrational part, so removing religion must leave behind a cleaner and more rational human being. But human beings do not work that way, human beings operate through beliefs, emotions... We do not stop wanting ritual, purity, moral tribe, sense of sacredness or transcendent meaning just because we stop using religious language for those desires.
-
Should you really be drinking raw milk?
I think a person who sleeps well, lifts regularly, eats decent food, goes outside, and keeps real social ties is doing some of the most evidence-supported things available for long-term health. I have noticed that a surprising number of people learned that from communities that also push raw milk, seed-oil paranoia, and other nonsense. The problem is not that medicine is wrong. The problem is that medicine left a prevention gap, and the cranks moved in.
-
Did EDC culture turn normal life into a gear fantasy?
I used to think EDC culture was mostly harmless nerd behavior. Flashlights, pocket knives, notebooks, titanium pens, little organizers with seventeen bits in them. Fine. People like tools. People like objects. Some people enjoy refining a system. I get it. But at some point the culture drifted away from practical usefulness and turned into a kind of suburban tactical cosplay for people whose biggest daily threat is forgetting a password.
-
Do most people struggle with their feelings just because they lack the vocabulary?
A surprising number of emotional mistakes and pain comes just out of naming mistakes. Someone says he is angry when he is really ashamed. Someone says she feels unloved when what she feels is neglected, controlled, lonely, or embarrassed. Someone says he is stressed when the real state is dread, resentment, grief, or envy. Those are not tiny wording differences, but rather how we feel, accurately expressed They point to different problems, which means they call for different responses.
-
Does your personality matter far less than you think?
I realize, interacting with students, teenagers and younger co-workers peers that many believe that their personality traits are a leading factor in deciding what do to or how to tackle their own career. Although younger people ask these questions more explicitly, older adults also seem to think along the same lines. I personally find it to be far more irrelevant than most people think. Besides my job, where I observe successful people doing the same role with drastically...
-
Shouldn't cultural criticism go both ways?
I had one of those big-tech team dinners. The conversation turned to how people met their partners. A few of my Indian coworkers talked about arranged marriage, family involvement, and how much more normal it is in India for marriage to be treated as a family matter and not just a private romantic choice. That part is ok, different cultures and all. It was interesting to see their perspective, even though I wouldn't share it. The problem started when one of them stopped describing the custom...
-
Does being entertained all the time make ordinary life feel dead?
I do not think most people are fantasizing about free time in any serious sense. They are fantasizing about free time available for consumption. That is a different thing. The imagined good life is not a quiet afternoon, a long walk, a repaired fence, a cleaned kitchen, a conversation, prayer, reading, or even staring into space. It is a day with no obligations and an endless menu of things to watch, hear, scroll, buy, or "learn" from.