October 10, 2025
AI’s Uncanny “Seahorse Problem”
And the incredible insight we learn from it?

By Ignacio de Gregorio
14 min read
Have you ever found yourself knowing what to say but struggling to find the words to express it? Like trying to speak a language you've just begun to learn. You know what you want to say, but you can't say it.
Sounds familiar? Well, what if I told you AIs can "feel" the same way?
And what's more, this has a very particular connection with a fascinating fact about AI models and, among all possible things, seahorses.
I'm sure nothing of this makes sense to you right now. But if you stick till the end, not only will it, but I guarantee you won't look at models like ChatGPT the same way ever again, revealing in the process AI's own version of the "Mandela effect" and an important lesson to take away from all this:
AI models know way more than they show.
AI explained in first principles and simple words, for those allergic to hype but hungry for knowledge. Join today.
TheWhiteBox by Nacho de Gregorio The newsletter to stay ahead of the curve in AI
Here You Go! Wait… What?
I recently read a fascinating blog that covers one of the weirdest things you can see in AI models these days: ask them to give you a seahorse emoji.
The blog is a bit too technical for most, so I've decided to give you the intuitions behind this great piece, although I do recommend you head to the original blog once you finish my article; it's really great.
As for the seahorse issue, I suggest you try it yourself, but I'll simply paste my conversation with ChatGPT going bananas over this seemingly simple question.
What is actually going on?
You're probably as surprised as I was, for several reasons:
- The model consistently fails to give the seahorse emoji back
- It appears to be surprised by its own mistakes.
This looks very confusing until you realize that, well, the emoji doesn't even exist, so they obviously cannot generate it.
But ChatGPT's reaction above isn't the ordinary one; it's not the "I'm sorry I can't do this." Instead, it appears that it's willing to comply, failing again, again, and again, each time even more confused by its own responses.
But why is this? And why isn't the model immediately realizing the seahorse emoji does not exist? This may seem like ChatGPT is just being dumb for not acknowledging this, but the real reason goes much, much deeper. In fact, I'll go and say it: I'm about to prove to you it's not the model's fault.
Before we explain the fascinating reasons why ChatGPT behaves so erratically for this straightforward question, let's answer first why models don't know the emoji exists, as this reveals one of the common pitfalls in modern AI use.
AI's "Mandela Effect"
The reason all models seem to believe a seahorse emoji exists is, ironically, because most people do, in a very illustrative example of the Mandela effect.
The Mandela Effect is a psychological phenomenon in which a large group of people collectively misremember a fact or event in the same way.
It's named after Nelson Mandela, because many people falsely remember him having died in prison during the 1980s, when in fact he was released in 1990 and lived until 2013.
Other famous examples I would definitely fall for include:
- In Star Wars: The Empire Strikes Back, many fans swear that Darth Vader says, "Luke, I am your father," when the actual line is, "No, I am your father."
- People often recall the Monopoly Man wearing a monocle, but he never has one.
- In Snow White, the Evil Queen's line is often quoted as "Mirror, mirror on the wall…" but in fact she says, "Magic mirror on the wall…"
- The children's TV show is remembered as "Looney Toons" by many, but it's actually "Looney Tunes."
- In The Silence of the Lambs, people often recall Hannibal Lecter greeting Clarice with "Hello, Clarice," but the actual line is "Good morning."
- The Pikachu tail from Pokémon is often imagined with a black tip, though it's always been entirely yellow — as someone who grew up with Pokémon, shocked was an understatement as to how I felt.
- "Houston, we have a problem" is quoted in movies, used in headlines, and repeated so often that it's become cultural shorthand for "something's gone wrong." However, the actual line spoken by astronaut Jack Swigert was: "Okay, Houston, we've had a problem here."
And the list goes on.
So, the answer to the first question, why do models falsely believe a seahorse emoji exists, is mainly because their training data includes many such claims. There are literally entire articles about people swearing that the emoji existed.
This also perfectly illustrates the delicate balance between truth and lies in AI models. Their respect for the truth only goes as far as what their training data holds as true, and you must be very aware of this when engaging with them.
But this doesn't come close to explaining why the models exhibit erratic behavior, going down a rabbit hole of emoji generations they believe are seahorses, only to realize they aren't after they generated them.
Some models like Claude or Gemini immediately try too (they also initially believe it exists) but can acknowledge its non-existence using web search tools. But this doesn't solve the problem I'm about to explain to you, it just hides it.
Why does this happen? And the answer lies in the very nature of these products; in a way, ChatGPT is letting its underlying models down. I know that made zero sense right now, but bear with me.
The Tokenizer, the Great Unknown
A largely misunderstood component of modern AI models is the tokenizer. It's an external piece (relative to the model) that is present in all models (but hidden from you under the ChatGPT app) and plays a crucial yet unknown role to most.
The reason the tokenizer exists is that AI models run on computers, so they can't see text characters in the same way we do. Computers only work with numbers. For instance, when you send ChatGPT the question, 'What is the capital of Romania?', the model doesn't see those characters, but a list of fixed tokens.
But what is a token? Let's answer that in first principles, not with your average "AI influencer" superficial explanation.
Since models can't see text characters, only numbers, we need something to transform them (or images, or videos) into numbers. That something is the tokenizer. The process looks a little bit complex, but stay with me.
The tokenizer first takes the user's request and transforms it into a set of bytes. Using our example above, 'What's the capital of Romania?' becomes a list of bytes: 57 68 61 74 27 73 20 74 68 65 20 63 61 70 69 74 61 6c 20 6f 66 20 52 6f 6d 61 6e 69 61 3f.
These bytes are then 'merged' into tokens, grouping them by how frequently those bytes appear together in the English language. Therefore, tokens are just groups of bytes that usually represent syllables, subwords, or even entire words. According to OpenAI's tokenizer, that sequence has six tokens, shown in different colors below:
Fun fact: The fact tokenizers are based on how common bytes are grouped together in Western languages (particularly English) is a heavy western bias that causes Western models to be particularly bad in non-western languages.
This builds what becomes the model's token vocabulary. Just like humans have a set of words/syllables they can use to express themselves in a given language, AI models are also limited in such a way. But the important thing here is that this vocabulary is predefined before we actually train the model; it's imposed on it. Hold this thought for later because it's crucial for all this to make sense.
And how does ChatGPT know which token each word in the user's request belongs to? This is exactly what the tokenizer does.
Those integers you see at the bottom right in the image above are the indices of a token table known as the 'embeddings matrix', which are nothing but vectors (a list of numbers), each representing a single token. Here's where the 'tokens' become numbers.
In other words, the token 'What's' is assigned a numerical ID (45350), which is the row in the embedding table number 45350, transforming it into a sequence of numbers such as [2, -1.2, 4, 7.8…], which is what the actual model, like GPT-5 in the case of ChatGPT, uses to work with your request.
Technical insight (not required to follow along): As you may guess, once the model is going to output a response, that response is also 'unembedded' using a table that performs the opposite operation of the one you're seeing above, literally called the 'unembedding table'. This table, or matrix, is the transpose (i.e., instead of going from text to numbers, it transforms numbers to text) of the above matrix.
To summarise, your input is chunked into tokens, which are groups of bytes organized by a tokenizer based on their frequency in the most common languages in the model's training data, and these tokens are then converted into lists of numbers that the model can understand.
Luckily for you, here is where complex topics about numbers and transformations end, as we don't need to go into why words are expressed as vectors today.
The takeaway here is that before ChatGPT starts to process your request, the tokenizer preprocesses the sequence to make it understandable to the model. Hence, my goal here was for you to understand the role of the tokenizer. Succinctly, the two main takeaways here are:
- Models have a fixed and limited vocabulary of these tokens that they can process and/or generate
- This vocabulary is imposed on the model (in the same way you don't invent your own language but learn a pre-existing one) and controlled by a component called a tokenizer that is external to the model (the model isn't necessarily aware of its existence).
And these two insights have quite the consequences.
Models Know More Than They Show
Building on the lessons of the previous section, let's use a human example to start to get to the crux of the issue. Here's when things are going to begin to fall in place.
Picture this: You're walking down Plaza Mayor in Madrid, Spain, desperate to try this amazing thing called "Iberic ham", cured pork that your friends back in the States described as "to die for" (can't blame them).
You sit down at a table, and amidst the rush hour, a stressed and considerably underpaid waiter comes to you and asks what you want. You're capable of saying "quiero cerveza," which means "I want beer," but you also want that glorified cured pork.
However, you completely forgot how it's said in Spanish; while you have a vivid representation in your mind of what you want, you can't express it because you don't know the Spanish term.
You panic, and instead opt for the closest thing you can imagine to describe it, like "quiero ham," your best Spanglish effort (and surprisingly effective, actually). Luckily for you, the waiter nods and smiles. She understood you, that 'good enough' try was, well, "enough."
Now, try to picture yourself in that uncomfortable scenario, especially moments before the waiter gives you the sign that she understood you. That tension, you feel it, right?
Hold that sensation, because what you're feeling is eerily similar to what's happening with ChatGPT in its conversation with me a few screenshots earlier.
It does know, but it can't express it.
The fact is, when you see ChatGPT incapable of generating the seahorse emoji, it's not because it doesn't "know" what it is; it does, but the tokenizer curbs the capacity of AI models to express themselves, only appreciated in edge cases like emojis.
Let me explain.
To figure out what word to predict next, models like ChatGPT do a series of 'meaning gatherings'. That is, to predict what to respond to your input, they gather information at the sequence level to capture what you told them, while also looking into their own knowledge to add additional relevant context.
For instance, to the user question, "In what bank does Jamie Dimon work?" the model:
- uses the context of the phrase to identify that we're talking about financial institutions, not a riverbank,
- While also using its own knowledge about the concept 'Jamie Dimon', realizing you're referring to Jamie Dimon from JPMorgan.
And this "progressive adding of meaning", where the model adds information from both dimensions (sequence and internal knowledge), is continuous, meaning it's not performing this search over a discrete set of possible answers, but exploring a "world of possibilities," allowing it to combine different concepts to form new ones.
In simple words, the things a model can "think" internally are not constrained by the things it has seen during training (or, importantly to today's topic, that it can say). It can combine two totally separate things, like taking Shakespeare poetry and iPhones and write a poem about the latter using the style of the former.
This is why the models are so creative at times: they are not forced to think of just a set of possibilities we predefine for them. Instead, they get to roam free and create new concepts on the fly.
Which is to say, models can generate internal concepts (think of this as "thoughts") that are not necessarily represented in this limited, externally-imposed vocabulary. This, as we'll see in a minute, is a "blessing" models have that sadly remains hidden from us.
As they are still forced to use the prefixed tokenizers, their "wild thoughts" always end up being expressed using a predefined set of possible words and characters that are imposed on the model.
Put another way, just as you might have the idea but lack the words to express how much you like Paris in Icelandic, AI models also have this expressiveness limitation.
But wait, we humans know how to shush when we don't have the words; we can acknowledge that absence of words in our vocabulary. On the other hand, my earlier conversation with ChatGPT showed it not only predicting the wrong emoji, but panicking over it, as if it couldn't hold itself from answering.
So, why can't ChatGPT contain itself? And the answer is not because models "can't help it"; the problem is, again, the tokenizer.
Preparing to say something you can't
The excellent blog that incentivized me to write this one includes a very well-done analysis of how models are shaping meaning internally.
As I was saying before, models progressively build meaning across their internal layers. Thus, if we ask Llama 3 70B the same question, the model's understanding of what's coming next (fifth column below, right in the middle) clearly becomes 'seahorse' in the last layers.
This means the model is clearly thinking about seahorses. However, the final output is a fish emoji, which is precisely what was happening to ChatGPT and why it went nuts.
Put another way, the model is thinking it's predicting a seahorse emoji, but it's predicting a fish emoji!
But.. why?
And the answer is, you guessed it: the external tokenizer again. And here's where we're going to understand why the tokenizer being external to the model carries so many negative connotations.
The shortest answer is that the model outputs something it didn't intend because of how the sampling mechanism works (the last part of ChatGPT's process by which a token is sent to your screen).
Specifically, in the model's last layer, we take ChatGPT's "hunch" of what might come next and collapse it into one of the possible options its vocabulary allows (the closest one, actually).
That is, we're telling the model, "Okay, cute thought, but this is the set of possible options you can generate; choose the one that is closest to what you're thinking." Put another way, what ChatGPT gives back to you isn't the exact thought the model generated, but the closest token in the vocabulary.
For instance, the fish emoji, 🐠, which is what the model gave back when being asked for a seahorse, is actually the character set 'ĠðŁIJł', which is most likely the closest thing the AI model can generate to a seahorse, as the latter does not exist in the model's vocabulary.
That is why the model output something it did not expect; it believed it could express its thoughts, but the tokenizer could not convey them.
Technical explanation (not required to continue reading): The last layer in the model is a linear projection using the unembedding table and the output of the last layer (last activation, ChatGPT's hunch as I described it earlier).
Mathematically, the unembedding matrix has size {vocabulary, model's hidden dimension}, where each row represents a token in the vocabulary, and the activation is a vector of size {model's hidden dimension, 1}.
As the last activation is multiplied by each row (each token), this results in a scalar value for each one (this means the outcome of the multiplication has a size of {vocabulary, 1}).
Therefore, you can think of each scalar in the resulting vector as a similarity score between the last activation and each token. In other words, the last activation vector serves as a look-up of the closest token in the entire vocabulary.
We are asking: "Which token is most similar to what I'm thinking right now?" The token with the highest similarity is chosen as the model's prediction for the next word.
Fascinatingly, when resorting to "quiero ham" in our Spanish example, you prove humans are also doing a close-enough vocabulary lookup between what they want to say and the closest thing (semantically speaking) in their vocabulary to it.
And the only difference between you and the AI is that your tokenizer, the mechanism that takes your semantic thought and expresses it in the desired language, is internal.
Hence, you are aware of your own 'tokenizer limitations', so you can choose not to express your thoughts if you believe they will be adequately represented. Conversely, ChatGPT can't do that because it doesn't actually know it can't generate a seahorse emoji, even if it can describe it internally, which is why you see ChatGPT consistently predicting new emojis and panicking about them; it's not "aware" of its own vocabulary limitations.
Intuitively, ChatGPT's case is very similar to using your smartphone's real-life translation system to say something in Lithuanian that doesn't quite exist (but you don't know it doesn't exist), so the smartphone goes for the closest existing option.
To fully drive this home, I can show you proof that the model actually understands your request and what you asked for, using ChatGPT's multimodal capabilities. If instead of asking for the emoji, you ask it to draw it, it simply does:
And this also explains why the model immediately realizes it has generated the wrong emoji. Once it generates the emoji, it is sent back to the model, instantly realizing it gave you a dragon emoji.
And the reason it tries again and again is that the model is convinced what you're about to see is a seahorse emoji, only to get betrayed by the tokenizer's expressive limitations.
Tokenizer-Free LLMs, for when?
I hope you're as fascinated as I am about this very particular "issue" with modern generative AI models.
All this gives for an intriguing read, but we must acknowledge the issue at hand: tokenizers severely limit the expressiveness of AI models, raising the question: what more do they know that is hidden from us due to tokenizer limitations?
This is problematic because the models might appear less intelligent than they actually are; they might be generating interesting ideas internally, that they can't express due to a vocabulary limitation.
Unsurprisingly, tokenizer-free models have been discussed for some time, and this is further proof that AI models should not be constrained by external components that curb their expressiveness.
But if there's something you can take from this piece, I hope, is that from now on you'll appreciate how interesting these things are, not just mere 'next-word predictors', but as complex programs we still know basically nothing about beyond the fact that they work.
If you enjoyed the article, I share similar thoughts in a more comprehensive and simplified manner on my LinkedIn (don't worry, no hyperbole there either). As a reminder, you can also subscribe to my newsletter.