An explanation posited by Danko Nikolić, a neuroscientist at the Frankfurt Institute for Advanced Studies, that the experience of synaesthesia (the mixing of sensory inputs, e.g. seeing a colour when hearing a sound) may be mediated semantically. In order words, it is the idea of the input, not the direct experience of the input itself that determines which sense gets activated. For instance, S in the contexts of letters may produce red, while S next to numbers looks like a 5 and so might elicit yellow instead.
A 2012 study suggested that this model of synaesthesia might be not too far from the truth. When participants were asked to learn a few letters of Glagolitic script, of which they had no prior conceptual knowledge, the rapid of wiring between brain areas relating to grapheme regions and colour regions implied the use of higher-level conceptual connections.

It’s interesting to think about this model and its implications in how conceptual thought develops. Nikolic proposes the idea that children need to form abstract ideas of things that don’t exist in the physical world and that this may come about through first associating a concept with a sensory device like colour or shape. There might be something to this. I definitely remember using physical ‘anchors’ like clock hands to help me think about numbers when I was learning math in school.