Prof. Janet Hui-wen Hsiao’s new study shows that, across 18 languages, shared subword tokens predict human semantic priming beyond distributional and morphological similarity—though larger vocabularies obscure these cues.
#LLMs #SubwordTokens #SemanticPriming #DistributionalSemantics #WordMeaning
#LLMs #SubwordTokens #SemanticPriming #DistributionalSemantics #WordMeaning
Byte-pair Encoding Captures Aspects of Word Meaning that Distributional Semantics Overlooks
Abstract. Large language models learn word meanings through distributional semantics (i.e., patterns in text), and distribution does not always capture information about taxonomy and function, which a...
direct.mit.edu
November 9, 2026 at 2:05 AM
Prof. Janet Hui-wen Hsiao’s new study shows that, across 18 languages, shared subword tokens predict human semantic priming beyond distributional and morphological similarity—though larger vocabularies obscure these cues.
#LLMs #SubwordTokens #SemanticPriming #DistributionalSemantics #WordMeaning
#LLMs #SubwordTokens #SemanticPriming #DistributionalSemantics #WordMeaning