Language models already know that kiki is spiky

Ask people which of two invented words, kiki or bouba, names a jagged shape and which names a blobby one, and they mostly agree: kiki is the spiky one. The agreement holds across languages, and it has been reported in children too young to read. It is a well-replicated finding in the psychology of sound and meaning, and it is odd, because nothing about the letters k, i, b, o or u has a shape.

Large language models never hear anything. They read. So it is worth asking whether a model trained only on text has the same lean, and if it does, where inside the model it lives. That is what five experiments went after, across five independently pretrained model families, and the answer turned out to be less about the network and more about its dictionary.

A direction built out of nonsense

The method is deliberately blunt. Start with 194 invented nonsense words, half of them written to look spiky, with clusters like k and t, and half to look round, with letters like m, l, b and doubled o and u. Feed each one to the model, take the activation vector it produces at some layer, and average within each group. Subtract one average from the other. What is left is a single direction in the model's internal space: a line whose positive end is "spiky-looking word" and whose negative end is "round-looking word". Any word can be scored by projecting it onto that line.

The interesting part is the test. The direction is built only from words that do not exist, then used to sort 80 ordinary English adjectives the model never saw during the fit, half of which describe spiky things and half round things. In Llama 3.1 8B that transfer separates the real adjectives with an AUROC of 0.8775 for the base model and 0.8869 for the instruction-tuned one. (AUROC is the chance that a randomly chosen spiky word scores above a randomly chosen round word; 0.5 is a coin flip and 1.0 is perfect.) Two null tests say this is not luck: 2,000 random directions in the same space do this well with probability 0.0005, and refitting the axis on shuffled labels gives p = 0.0050. Run the whole thing backwards, fitting on real adjectives and testing on the invented words, and the separation is 0.9924.

So the model has a spiky-round axis, and it generalizes from fake words to real ones. The question this report is about is where that axis comes from.

It is already in the embedding table

Before a transformer does anything, it looks each token up in an embedding matrix, a fixed table of vectors learned during pretraining. That table encodes nothing about context or reasoning. It is the closest thing a language model has to a dictionary of raw word pieces.

Build the same axis using only those lookups, with every transformer block skipped, and it still sorts real words: between 0.7350 and 0.7888 AUROC across all ten model variants tested. Whatever else the network does with word shape, this much of it is present in the dictionary.

You can see why by scoring the model's entire vocabulary against the axis and reading off the extremes. They are spelling fragments, not concepts.

This raises an obvious worry: maybe the invented words and the real words simply share spelling, and the axis is a lookup table for shared chunks. That is measurable. The overlap between the subword pieces of the invented words and those of the real test words is between 0.0132 and 0.0379 by Jaccard index, which is to say almost nothing. Two crude shortcuts confirm the axis is doing more than counting letters: guessing from the word's first letter alone gets 0.5750, and guessing from its length gets between 0.5006 and 0.5703, against 0.5 for a coin flip.

Four families out of five

The same recipe was run on five independently pretrained families. Four of them show the pattern clearly: Gemma 4 12B base reaches 0.8981, Mistral NeMo 12B base 0.8900, Llama 3.1 8B base 0.8775, and OLMo 2 13B base 0.8613, the last one much deeper in the stack (layer 33 of 40, where Llama peaks at layer 10 of 32).

Instruction tuning is not uniformly kind to the axis. Gemma 4 12B Instruct reads 0.6531 mid-stack where its base model reads 0.8981, while its embedding table still reads 0.7650. The information is there in that model; one mid-stack direction just picks up less of it.

Qwen2.5-7B does not. Under the identical recipe its mid-stack axis reads 0.5588, and a shuffled-label null says that is nothing at all: p = 0.98. Rescaling each activation dimension before fitting, which helps when a few dimensions carry enormous values and drown out the rest, lifts it only to 0.6544. The informative part is that Qwen's embedding table is perfectly ordinary: 0.7488 at layer 0, with the same sensible poles you can see in the figure above. So this is not a model that lacks the representation. It is a model whose mid-stack geometry hides it from this particular method.

How big is it on words people rated

Everything so far is scored against word lists written for this experiment, which is a soft target: whoever picks the words picks the answer. The fifth experiment replaced them with published human data, 1,756 English nouns from a norming study by Sidhu and colleagues in 2021, each rated by people for whether the thing it names is spiky or round.

The axis still predicts, and it is clearly smaller than the number from the adjective list written for these experiments. On a subset matched for word length and frequency, separation runs from 0.5728 to 0.7200 depending on the variant; the rank correlation with the human ratings runs from 0.1278 to 0.2649 in absolute value. Nine of the ten variants clear a shuffled-label null; Llama 3.1 8B base is the exception, at p = 0.0520, just outside. Word length and frequency do not explain the effect: controlling for both barely moves the correlations, for instance 0.2649 to 0.2496 for Mistral NeMo 12B base.

Below, every one of the 1,756 nouns is placed by what people said and by what the model computed. Search for a word, click a suggestion, or take a random one.

What the axis is reading

The vocabulary poles already gave this away: the axis responds to letters. Watching it work on the human-rated nouns makes it concrete. People place fork at the very spiky end of the ratings (0.1508, on a scale where lower is spikier), because forks are spiky; Llama's axis puts it on the round side at -0.8149, because the letters are not. cheesecake, which is not spiky, projects to +1.0687 on the strength of its k and its double e. In Mistral NeMo the most spiky-projecting noun of all 1,756 is tee, at +1.8736, with kite close behind at +1.1876. At the other end, wool and moon are the roundest-projecting words Llama has (-2.6267 and -1.9895) on the strength of their double o.

Read together with the embedding result, this is a coherent account: what the models have is a mapping from spelling to shape, largely fixed in the token table and partly refined by the network, which lines up with human judgments because human judgments about spiky and round words also track letters and sounds. It is not a representation of what the object looks like.

What this does not show

• The transformer's contribution is bounded, not measured. The mid-stack number is the best layer chosen by looking at the evaluation; layer 0 got no such search. Treat the difference as an upper bound.
• Spelling statistics are not fully controlled. The subword-overlap check rules out the invented words and the real words literally sharing tokens. It does not rule out subtler letter-frequency regularities that both sets inherit from English. The first experiment flagged this and it was never closed.
• The p-values are not corrected for the layer search. They are computed at the layer that was selected on the evaluation.
• Nothing here is causal. Whether pushing a model along this direction changes the words it writes was not tested, in either direction.
• The effect against human ratings is moderate. A rank correlation around 0.2649 in absolute value is a real but modest relationship, and the 0.8869 figure from the thread's own adjective list is not the effect size to quote for it.
• Five families, one method, English. One way of building the axis, one language, ten model variants. Qwen is a reminder that the method fails on a normal model for reasons that have nothing to do with whether the model knows the fact.

The lean is real, it survives the move to published human ratings at a smaller size, and it starts in the dictionary. If you want to know why a language model thinks kiki is spiky, most of the answer is already sitting in the table it uses to read the letters.