Terminology is essential for scientific research. Depending on its purpose, it may be diagnostic, expressing judgments made by researchers, or descriptive, refering to empirical observations. Recent d...

The term colexification has become an important concept in lexical typology in particular and linguistic typology in general. Surprisingly, it is not consistenly used by all scholars in the field. Instead, two main interpretations have emerged, a broad version that uses colexification as a cover term for polysemy, vagueness, and homophony, and a narrow version that restricts the term to polysemy and vagueness. We discuss the ambiguity and reflect about the usefulness of introducing alternative terms instead. ## 1 Introduction The introduction of the term _colexification_ to the field of lexical typology by François (2008), can be seen as marking a turning point in the field of lexical typology. Before, lexical typology was a niche discipline pursued by a small number of scholars, concentrating on very specific domains of the lexicon of human languages, such as color, kinship, and body. After the introduction of the term in a widely recognized volume on lexical typology by Vanhove (2008), devoted to the role that polysemy plays in investigating semantic shift, both the available data and the methodology by which the data could be analyzed increased drastically. In 2010, Cysouw (2010) showed, how recurrent polysemies can be visualized and analyzed as a network. In 2011, Urban (2011) presented the idea that certain directional tendencies of semantic change can be inferred from the existence of examples in which semantic similarities are overtly marked in individual language varieties. In 2014, the _Database of Cross-Linguistic Colexifications_ (CLICS, https://clics.clld.org) was published (List et al., 2014), building on Cysouws’ idea of polysemy networks which were derived from a collection of 300 comparative wordlists that were automatically searched for _colexifications_ , that is, cases where different concepts in the same language were expressed by identical word forms. With these new resources and ideas to visualize and explore semantic relations in large-scale accounts of cross-linguistic data, lexical typology grew into a popular subfield of linguistic typology (Koptjevskaja-Tamm et al. 2007, Koptjevskaja-Tamm et al. 2015, Sjöberg et al. forthcoming). While previous studies had compared how the lexicons of human languages organize meaning in different ways, had relied on case studies and small-scale examples involving hand-selected languages, it was now possible to study meaning organization through the lens of a large part of the world’s languages with the help of technically and theoretically considerably simple means. ## **2 From Polysemy to Colexification** François originally defined _colexification_ as a relation that holds for the senses that are expressed by identical word forms in one and the same language. > A given language is said to colexify two functionally distinct senses if, and only if, it can associate them with the same lexical form. (François, 2008, p. 170) From this definition, it is clear that colexification is intended as a _cover term_ _for the previously established terminology around_ __polysemy__ _,___vagueness__ _, and_ __homophony__. In linguistic terminology, there are several terms that can be used to _specific_ cases where two senses are associated with the same lexical form, such as _polysemy_ and _vagueness_ , referring to senses associated with the _same_ form, and _homophony_ , referring to senses associated with different forms that came to sound alike at some point in the history of a language. From the perspective of linguistic terminology, the term was useful in two regards. Since, on the one hand, it replaces largely _explanative terminology_ with _descriptive terminology_ _(see Jacques & List, 2019, p. 141 on this distinction)_, it facilitates, on the other hand, the investigation of lexical semantics across languages, since it shifts the burden of proof from the stage of data collection to the stage of data analysis. Explanative terminology (one could also say “diagnostic terminology”) does not only describe a phenomenon, but also tries to explain it at the same time. Polysemy is usually _explained_ as the result of semantic change when individual forms acquire more senses, while homophony is _explained_ as the result of sound change leading to phonological merger in individual forms of a language. Determining that two senses are colexified in a given language does not explain _why_ this is the case, but merely notes this as a fact inferred from empirical observations. This has drastic consequences for the investigation of lexical semantics across languages. While any cross-linguistic study on polysemy, vagueness, or homophony, would have had to distinguish these phenomena beforehand, reducing the study to the overall phenomenon of distinct senses being expressed by the same form enabled large-scale investigations based on standardized comparative wordlist collections. In such a facilitated workflow, one would no longer have to search individual languages for instances of polysemy in a fixed set of meanings, but could rather search for _colexifications_ , that is, cases where two or more concepts in a comparative wordlist are expressed by identical translational equivalents. How these cases could be _classified_ would be a second step, where early research immediately showed that cases of polysemy and vagueness tend to recur across different language families, while cases of homophony tend to be restricted to individual languages and individual language families (List et al., 2013). As a result, the distinction between polysemy and vagueness on the one hand and homophony on the other hand, can be readily established _after_ colexification patterns have been assembled for a sufficiently large number of languages and language families. This procedure was the basic idea behind the CLICS database, which provides an automated procedure to harvest colexification patterns from cross-linguistic wordlist collections that was modified and improved across the database’s four different major versions (List et al., 2014, 2018; Rzymski et al., 2020; Tjuka et al., 2025). When using colexification as a cover term for polysemy, vagueness, and homophony, CLICS assembles individual examples from different languages of all these phenomena and later allows to single out cases of homophony by focusing on the most frequently observed _patterns_. ## **3 Intended Definition of Colexification** Recent discussions at the workshop on _Semantic Shifts and the Dynamics of Lexification Patterns_ organized by Alexandre François, Anna Zalizniak, and Anna Smirnitskaya as part of the _16th International Conference of the Association for Linguistic Typology in Lyon_ (July 1-3, 2026) made it clear that the term _colexification_ is used in two different ways by linguists from different backgrounds (or “schools”). On the one hand, there are linguists who work either in the tradition of François’ original work or in the context of the _Database of Semantic Shifts_ (https://datsemshift.ru, Zalizniak et al., 2023), a large-scale catalogue of various attested kinds of semantic shifts that have been collected for more than 20 years now directly from the linguistic literature (Zalizniak, 2018). On the other hand, there are linguists working in the context of the CLICS database. The former tradition restricts the term _colexification_ to cases of polysemy and vagueness, excluding cases of homophony from the definition. In the context of the work of the latter tradition, _colexification_ was always a cover term for _all_ cases where one form has multiple senses in a given language. Given that the tradition restricting colexification to polysemy and vagueness is represented by the scholar who introduced the term colexification, there are good reasons to follow the narrower definition of colexification, even if the original text introducing the term does not explicitly exclude homophony as one of the aspects covered by the term. Even in the case of the CLICS database, which was profiting so much from the broader version of colexification in order to justify its workflow, the narrow version of colexification might be useful and even help to clarify past studies. From these studies on emotion concepts (Jackson et al., 2019), body part terms (Tjuka, 2024; Tjuka et al., 2024), perception and cognition (Georgakopoulos et al. 2022), and lexical creativity (Brochhagen et al., 2023), it is quite clear that the main intention of the database is to _infer_ colexifications in the narrow sense, filtering out cases of homophony by rather simple quantitative means. However, treating colexification as a mere cover term for vagueness and polysemy means at the same time, that the two major advantages of the term, the shift from explanative to descriptive terminology and the facilitation that this shift provides for quantitative approaches, would vanish, at least in parts. It would also mean that terminological frameworks that attempt to view colexification in the larger context of _coexpression_ would have to be revised, especially since cases of _cogrammification_ as defined by Haspelmath (2023) also do not ask _how_ a pattern emerged, but rather conclude that a pattern can be _determined_ from the data. Additionally, it may be questionable whether it is actually possible to strictly distinguish between homophony on the one hand and polysemy and vagueness on the other hand in all cases (consider the ambiguity of English _ear_ , as referring to a body part or the part of a plant, which is often seen as some kind of polysemy, although it is the result of phonological merger). Thus, despite François’ original intention, one may argue that the broad version of colexification is actually in line with the established practice in linguistic typology. As often, when it comes to terminology, it seems that the field will have to live with some compromise solutions that emerge from the research practice. Abandoning the broad definition of colexification would require us to employ a new term, which may be seen as a minor problem, but given the conflict with the practice in general discussions on coexpression and semantic maps, we would be forced to revise the terminology in this area more generally and broadly. Retaining the broad definition for colexification as the norm has the disadvantage that in most scholars’ _practice_ , homophony is essentially ignored, no matter whether they see colexification in its broad or narrow version. It seems for the time being, that it is best to leave everything as is, while being prepared to mention the ongoing conflict with the terminology explicitly, where needed. ## **4 Beyond Colexification** While François agreed in the discussion about the term _colexification_ at the workshop at the ALT conference, that the original paper (François, 2008) does _not_ explicitly exclude homophony from the definition, reading the original paper with the revised narrow definition in mind clarifies those parts that go beyond _strict colexification_ proper. > In particular, “strict colexification” (same lexeme in synchrony) should be carefully distinguished from “loose colexification” (covering all other cases mentioned here). (François, 2008, p. 171) _Strict colexification_ is contrasted with _loose colexification_. Under the latter term François (2008) subsumes a variety of other, more or less similar, relations between concepts mediated through their word forms. Among these are _diachronic semantic change_ , in which a word form _w_ expresses a concept A at a point in time _t1_ but refers to concept B at a later point in time _t2_ (at which stage concept A may either still be attested or have been lost). Cases involving diachronic semantic change are comparable to strict colexification in that the concepts relate through an identical word form; unlike strict colexification, however, these word forms belong to different historical stages. Owing to this similarity Georgakopoulos & Polis (2021) do not treat the two cases as entirely distinct, but instead introduced the term _diachronic strict colexification_ , which captures both the diachronic dimension of the phenomenon and the fact that the same form is involved. While the broad definition of colexification may be seen as problematic here, since cases in which word forms overlap in parts, are the norm rather than the exceptions in natural languages, even if the word forms involved do not display any etymological connections, it is clear from the examples given by François that colexification is used in the _narrow_ sense. This means that François does not consider cases where two word forms show some coincidental overlap in some parts, but rather presupposes that loose colexifications can only be determined if one can prove the historical unity of the shared material in question. While it seems basically convincing to restrict colexification along these lines, recent research devoted to so-called _partial colexifications_ , that is, colexification patterns involving _parts_ of the linguistic form (List et al., 2022), seem to show again, that the notion of colexification as a descriptive term has concrete advantages for computational studies. Thus, List (2023) showed how the idea of _synchronic asymmetries_ in _overt marking_ proposed by Urban (2011) can be operationalized in a quantitative framework and applied to larger collections of comparative wordlists by searching automatically for those cases in which one word occurs at the beginning or the end of another word in a given language. Since these substring relations are called _affixes_ in computer science, List proposes to term these relations _affix colexifications_ , as a specific case of _partial colexification_ , emphasizing that the inference of these relations across different forms in a given language does not necessarily reflect valid semantic relations, but showing that potentially valid forms can again be inferred by filtering _patterns_ across multiple languages and language families. Partial colexification can be seen as more restrictive than loose colexification, given that the latter refers to etymological relations, while the former explicitly points to the synchronic identity of _parts_ of the linguistic form in a given language. That cross-linguistically recurring affix colexifications can give hints to valid and potentially interesting semantic relations is further illustrated by a study of Bocklage et al. (2025). In an empirical evaluation, the authors show that filtering affix colexifications by frequency of occurrence across language families can indeed help minimizing the number of false positives. Abandoning the term _partial colexification_ due to its conflict in coverage with the term _loose colexification_ may again seem to be the easiest way to deal with the terminological problems here. But again, we must observe that first, the definition of loose colexifications does not explicitly exclude cases of coincidence in form overlap. Second, abandoning the broad colexification definition would yield conflicts with the general coexpression terminology in linguistic typology, given that coexpression cannot rule out coincidence as easily as lexical typology can rule out homophony. As a result, it may be useful to emphasize that the terms _loose colexification_ and _partial colexification_ are not compatible, given that they reflect different scholarly traditions, the former being at least in part explanative (definitely diagnostic), and the latter being deliberately descriptive. ## **5 Conclusion** From what has been discussed so far, it will probably be clear that the terminology with respect to and around the term _colexification_ has made some infelicitous turns so far. As a result, we are – yet another time – faced with terminological traditions in our field that display certain conflicts among scholars, differences in opinions, and also sheer misunderstandings. On the one hand, the young history that the term _colexification_ experienced so far, can be seen as representative for typical terminology struggles in science. On the other hand, one should not ignore the momentum of serendipity reflected in the fate of the term _colexification_ , given that the reinterpretation of the partially explanative term as a purely descriptive term has given scholars the chance to develop simple but very effective quantitative approaches to lexical typology. Future research will show if additional solutions can be found. For now, all we can recommend is that scholars who work with colexifications make clear in which tradition they allocate their terminology. **References** Bocklage, K., Georgakopoulos, T., Dam, K. P. van, Ciucci, L., Blum, F., Kučerová, A., Rubehn, A., Stephen, A., Snee, D., & List, J.-M. (2025). Testing the Potential of Automatically Inferred Affix Colexifications for Linguistic Typology. in _Humanities Commons_ (pp. 1–35). https://doi.org/10.17613/a06m1-c9939 Brochhagen, T., Boleda, G., Gualdoni, E., & Xu, Y. (2023). From language development to language evolution: A unified view of human lexical creativity. _Science_ , _381_(6656), 431–436. https://doi.org/10.1126/science.ade7981 Cysouw, M. (2010). Drawing Networks from Recurrent Polysemies. Comment on “polysemous qualities and universal networks” by loïc-michel perrin (2010). _Linguistic Discovery_ , _8_(1), 281–285. François, A. (2008). Semantic maps and the typology of colexification: intertwining polysemous networks across languages. in M. Vanhove (Ed.), _From polysemy to semantic change_ (pp. 163–215). Benjamins. Georgakopoulos, T. and Polis, S. (2021): Lexical diachronic semantic maps. Mapping the evolution of time-related lexemes. _Journal of Historical Linguistics_ 11(3). 367-420. https://doi.org/10.1075/jhl.19018.geo Georgakopoulos, T., Grossman, E., Nikolaev, D., and S Polis (2022): Universal and macro-areal patterns in the lexicon: A case-study in the perception-cognition domain. _Linguistic Typology_ 26(2): 439–487. https://doi.org/10.1515/lingty-2021-2088 Haspelmath, M. (2023). Coexpression and synexpression patterns across languages: comparative concepts and possible explanations. _Frontiers in Psychology_ , _14_ , 1–12. https://doi.org/10.3389/fpsyg.2023.1236853 Jackson, J. C., Watts, J., Henry, T. R., List, J.-M., Mucha, P. J., Forkel, R., Greenhill, S. J., Gray, R. D., & Lindquist, K. (2019). Emotion semantics show both cultural variation and universal structure. _Science_ , _366_(6472), 1517–1522. https://doi.org/10.1126/science.aaw8160 __ Jacques, G., & List, J.-M. (2019). Save the trees: Why we need tree models in linguistic reconstruction (and when we should apply them). _Journal of Historical Linguistics_ , _9_(1), 128–166. https://doi.org/10.1075/jhl.17008.mat Koptjevskaja-Tamm, M., Vanhove, M., and Koch, P. (2007). Typological approaches to lexical semantics. Linguistic Typology, 11(1), 159–185. https://doi.org/10.1515/LINGTY.2007.013 Koptjevskaja-Tamm, M., Rakhilina, E., and M. Vanhove (2015): The Semantics of Lexical Typology. In Nick Riemer (ed.), _The Routledge Handbook of Semantics_ (pp. 434–454). London: Routledge. List, J.-M. (2023). Inference of partial colexifications from multilingual wordlists. _Frontiers in Psychology_ , _14_(1156540), 1–10. https://doi.org/10.3389/fpsyg.2023.1156540 List, J.-M., Forkel, R., Greenhill, S. J., Rzymski, C., Englisch, J., & Gray, R. D. (2022). Lexibank, A public repository of standardized wordlists with computed phonological and lexical features. _Scientific Data_ , _9_(316), 1–31. https://doi.org/10.1038/s41597-022-01432-0 __ List, J.-M., Greenhill, S. J., Anderson, C., Mayer, T., Tresoldi, T., & Forkel, R. (2018). CLICS². An improved database of cross-linguistic colexifications assembling lexical data with help of cross-linguistic data formats. _Linguistic Typology_ , _22_(2), 277–306. https://doi.org/10.1515/lingty-2018-0010 List, J.-M., Mayer, T., Terhalle, A., & Urban, M. (2014). _CLICS: Database of Cross-Linguistic Colexifications. Version 1.0_ (version 1.0.0). Forschungszentrum Deutscher Sprachatlas. http://clics.lingpy.org List, J.-M., Terhalle, A., & Urban, M. (2013). Using network approaches to enhance the analysis of cross-linguistic polysemies. _Proceedings of the 10th International Conference on Computational Semantics – Short Papers_ , 347–353. Rzymski, C., Tresoldi, T., Greenhill, S., Wu, M.-S., Schweikhard, N. E., Koptjevskaja-Tamm, M., Gast, V., Bodt, T. A., Hantgan, A., Kaiping, G. A., Chang, S., Lai, Y., Morozova, N., Arjava, H., Hübler, N., Koile, E., Pepper, S., Proos, M., Epps, B. V., … List, J.-M. (2020). The Database of Cross-Linguistic Colexifications, reproducible analysis of cross- linguistic polysemies. _Scientific Data_ , _7_(13), 1–12. https://doi.org/10.1038/s41597-019-0341-x Sjöberg, A., Georgakopoulos, T., and Koptjevskaja-Tamm, M. (forthcoming): Typology and universals of word meaning. In D. Geeraerts & D. Glynn (Eds.), _The Cambridge handbook of lexical semantics_. Cambridge University Press. Tjuka, A. (2024). Objects as human bodies: cross-linguistic colexifications between words for body parts and objects. _Linguistic Typology_ , _0_(0). https://doi.org/10.1515/lingty-2023-0032 Tjuka, A., Forkel, R., & List, J.-M. (2024). Universal and cultural factors shape body part vocabularies. _Scientific Reports_ , _14_(10486), 1–12. https://doi.org/10.1038/s41598-024-61140-0 Tjuka, A., Forkel, R., Rzymski, C., & List, J.-M. (2025). Advancing the Database of Cross-Linguistic Colexifications with New Workflows and Data. _Proceedings of the 16th International Conference on Computational Semantics_ , 1–15. https://aclanthology.org/2025.iwcs-main.1 Urban, M. (2011). Asymmetries in overt marking and directionality in semantic change. _Journal of Historical Linguistics_ , _1_(1), 3–47. Vanhove, M. (Ed.). (2008). _From polysemy to semantic change_. Benjamins. Zalizniak, A. A. (2018). The Catalogue of Semantic Shifts: 20 years later. _Russian Journal of Linguistics_ , _22_(4), 770–787. https://doi.org/10.22363/2312-9182-2018-22-4-770-787 Zalizniak, A. A., Smirnitskaya, A., Russo, M., Mikhailova, T., Bobrik, M., Gruntov, I., Orlova, M., Bibaeva, M., & Voronov, M. (2023). _Database of Semantic Shifts [Version from 20/12/2023)_. Institute of Linguistics at the Russian Academy of Sciences. https://datsemshift.ru/ **Cite this article as:** Johann-Mattis List, Katja Bocklage, and Thanasis Georgakopoulos (2026): “The ambiguity of the colexification term” in _Computer-Assisted Language Comparison in Practice_ , 9.2: 85-92 [first published on 19/08/2026], URL: https://calc.hypotheses.org/9547, DOI: 10.15475/calcip.2026.2.2. **Download the article as PDF:** calcip-09-2-2.pdf **Copyright information** : This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. **Funding Information** : This project has received funding from the European Research Council (ERC) under the European Union’s Horizon Europe research and innovation programme (Grant agreement No. 101044282). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Johann-Mattis List Seit Anfang 2023 leite ich den Lehrstuhl für Multilinguale Computerlinguistik in Passau. In meiner Forschung nehme ich generell einen datenbasierten,… Read more View all posts Katja Bocklage View all posts Thanasis Georgakopoulos View all posts * * * The text only may be used under licence Creative Commons Attribution 4.0 International. All other elements (illustrations, imported files) are “All rights reserved”, unless otherwise stated. * * * OpenEdition suggests that you cite this post as follows: Johann-Mattis List, Katja Bocklage, Thanasis Georgakopoulos (August 19, 2026). The Ambiguity of the Colexification Term. _Computer-Assisted Language Comparison in Practice_. Retrieved August 19, 2026 from https://calc.hypotheses.org/9547 * * * * * * * *

The term colexification has become an important concept in lexical typology in particular and linguistic typology in general. Surprisingly, it is not consistenly used by all scholars in the field. Ins...

Formal notations for the handling of sound laws in historical linguistics have been employed by linguists for a long time. So far, however, linguists have barely tried to describe their notation practices formally. As a result, a lot of variation can be found in the practice of sound law notation. This specification can be seen as a first attempt to provide an exhaustive documentation for a particular sound law notation scheme. This notation scheme accompanies an online-tool in which sound laws can be applied to actual words in order to test the consequences of sound change in practice. ## **1 The MISOL Sound Law Notation Scheme** MISoL (_Modeling and Inducing Sound Laws_ , List 2026, https://misol.edictor.org) is an interactive online tool in which user can define sound laws and apply them to concrete words in order to test the consequences of sound change in practice. MISoL’s scheme for sound law notation is based on the common practice of sound law notation, as it can be found in the historical linguistics literature. It differs from some notation variants, however, in so far as the scheme does not employ feature systems. What MISoL offers instead is a sound class system that allows users to define their own sound classes that can be later used to denote groups of sounds when it comes to defining individual sound laws. The goal of this description is to provide a clear specification of the notation system underlying MISoL. By providing this specification along with concrete examples of how the notation is supposed to be implemented, I hope to provide the basis to test the tool consistently and develop it further in the future. The MISoL tool consists of four major components accessible in four different tabs of the web interface. The first component defines sound classes and sound laws. The former allow to group sounds into arbitrary units, and the latter allow to define how sounds in an ancestral language change into sounds in a descendant language in a certain context. The second component allows to convert words in the ancestral language into words in the descendant language (also known as “forward reconstruction”, Sims-Williams 2018), and the third component allows to guess from which words in the ancestral language a given word in the target language has evolved. The fourth component allows to import and export data in text form, enabling users to store their analyses, parse the data with additional software tools, or to compare different approaches to solve the same problem in phonological reconstruction. The components are summarized in Figure 1. Figure 1: Major components of the MISoL workflow. In the following, all four components, essential for the tool, will be described in detail, along with concrete examples that show how the tool is expected to react. **2 Defining Sound Classes** The table of sound classes defines sounds in a very simple way, by assigning one or more sounds or sound classes to a certain sound class label on each line. Sound classes are read in from the first to the last line and definitions are stored at the time a line is parsed. As a result, sound class assignments can be combined, and an existing sound class can be assigned to another class. The assignment of one or more sounds or sound classes to a given sound class is represented in the form: name = sound1 sound2 sound3 The name of a sound class must be alphanumeric, similar to typical Python variables, and should not start with a number. The =-sign must have a space to the left and to the right. All sounds (or sound groups referenced by invoking an existing sound class name) must be separated by a space. Thus, the following examples for sound class definitions are all _wrong_ : name=sound1 sound2 sound3 name =sound1 sound2 sound3 name= sound1 sound2 sound3 1name = sound1 sound2 sound3 Internally, a sound class is an ordered list of sounds. By assigning sounds to a sound class, the sounds are made available to the MISOL system to act as source and target of sound change processes and to be referenced in sound laws. In addition to referencing groups of sounds with the help of sound class variables, all individual (“terminal”) sounds are also represented in as sound classes. The difference is that these classes have the same label as the sound iself and that they contain only one element (the sound they refer to). Furthermore, each final symbol that is identified as a sound (and not a sound class name) by the MISOL system is also assigned to its own class with one single manner. Thus, the following line will define as many as 5 internal sound classes, of which four have one single member, and the first targets the four only sounds in the system. my_class = a b c d Thus, internally, this will result in the following key-value representation: { "my_class": ["a", "b", "c", "d"], "a": ["a"], "b": ["b"], "c": ["c"], "d": ["d"] } When parsing sound laws, MISOL automatically checks for sounds that have not been referenced in the sound class table and adds them as individual sounds to the table of sound classes. As a result, you do not need to define sound classes in order to define sound laws. In order to check which sound classes have been defined in MISOL, click on the SHOW CLASSES AND LAWS button, after having inserted your sound class definitions and your sound laws in the _Classes and Laws_ tab. A table will open and present you all sound classes that have been defined, including both those classes that you defined actively, as well as those classes that were inferred automatically from the sound laws you defined. When you check the sound classes in MISOL, you will see that the list of classes shows three sound classes in the beginning, which are provided independently of what you have defined or not. These reserved classes, are the symbols ^, $, and –. ^ refers to the beginning of a sequence and can be used in the context string of a sound law. The same holds for $ referring to the end of a sequence. – refers to a specific sound law in which an element is lost (rather than being replaced by something). It can also be used as a source sound (in the case of epenthesis, which must be actively modeled) or as a target sound in a sound law. Other than for this specific purpose, the symbols should not be used. Sound classes are a way to model distinctive features that define individual sounds. The difference between feature-bundle representations for sounds in sound change models is that features are flexibly defined on the fly, and modeled rather as “tags” of individual sounds, or a shortcut to reference the sounds that are tagged with a certain sound class name in an ordered manner. In our opinion, this comes quite close to the way feature bundles are used intuitively by linguists so far, since one can define one’s sound system in a convenient manner, and provide major distinctions that may play a role in sound laws, such as voicing distinctions of consonants: voiced = b d g voiceless = p t k Another important aspect of sound classes is that they can be used as a shortcut for a group of sounds in sound laws, which often consist of an abstract set of independent sound changes, rather than an individual sound change that occurs in one context alone. As a result, one can refer to both individual sounds and to sound classes in the sound law descriptions of MISOL. As a further note on the way in which sound classes are handled in MISOL, consider the following assignments: voiced = b d g voiceless = p t k consonant = voiced voiceless m n ŋ This translates internally to the following major sound class representations: { "voiced": ["b", "d", "g"], "voiceless": ["p", "t", "k"], "consonant": ["b", "d", "g", "p", "t", "k", "m", "n", "ŋ"] } Thus, if a sound class like voiced has been assigned to a list of sounds, the label can be reused in order to assign the same group of sounds to another sound class. Internally, all sound classes are only represented as a group of terminal sounds, and only sound laws can be reused in assignments if they have already be defined. As a result, the following order of assignments would be problematic: consonant = voiced voiceless m n ŋ voiced = b d g voiceless = p t k Since voiced and voiceless have not been introduced yet with their target group of sounds, the interpreting code of MISOL would treat them as individual sounds (which can be represented by any string combination, provided it does not contain a space). Invididual sounds, however cannot be assigned to another group of sounds, since they are internally assigned to a group of one sound only, so the program throws a warning here and ignores the corresponding line. Since MISOL does not care how you define your sounds, it offers the possibility to work with groups of sounds as well as with individual sounds when dealing with sound change. In order to make sure that we distinguish groups from individual sounds, the recommendation is to use a dot . between sounds in a sound sequence in order to indicate that one is not dealing with individual phonemes. Thus, the final or rhyme of a Chinese word like [kwaŋ] could then be conveniently written as a.ŋ. MISOL, however, will treat this sequence of grouped sounds as an individual sound class (potentially a terminal one) and not assign it a specific semantics. Since sound classes are ordered lists of sounds, nothing speaks against it if you assign the same sound multiple times to one and the same sound class. This may be important in cases where _mergers_ are described in complex sound laws that deal with more than one input sound. There are two symbols which are automatically defined as sounds, which cannot be assigned to sound class groups: ^ represents the beginning of each word and $ the end. # is reserved as a comment marker. ## **3 Sound Laws in MISoL Notation** A sound law is an abstract formula that shows how one or more sounds in an ancestral language are converted to one or more sounds in a descendant language. It has the general formula: source > target / context The number of source sounds and target sounds must be identical and the context is optional and can be omitted: source > target The change marker > must be preceded and followed by a space. So the following lines would be erroneous and lead to errors. source> target source >target source>target The same strict rules apply for the marker separating the context, the slash, which must be preceded and followed by at least one space. So again, the following lines will all yield errors and as a result, the line will be ignored. source > target/ context _ source > target /context _ source > target/context ### 3.1 Source and Target in Sound Laws Source and target can be either a single sound, sound class, or list of sounds (indicated by square brackets), or a sequence of sounds. If the source and the target are a list of sounds or a sound class, they must be of the same length. Otherwise, MISOL will throw an error. Thus, the following sound law is fine. [a b] > [c d] Internally, it will be represented as _two_ sound laws: a > c b > d However, the following is wrong. [a b] > c This means, if you want to define a _merger_ , you must repeat the sound in the list of sounds as follows: [a b] > [c c] If a sequence of sounds is provided (a sequence is defined by passing more than one sound, list of sounds, or sound class, without enclosing them in brackets), this will also be interpreted internally by invoking two or more separate sound laws, where preceding or following context is resolved automatically by MISOL. This, means, the following code a b > c d / x _ y is equivalent to writing a > c / x _ b y b > d / x a _ y A sequence of sounds can consist of individual sounds, sound classes, or lists of sounds. Thus, you could write a sound law as the following one: [p t k] [a i u] > [p p p] [a a a] This will turned all plosives to the sound [p] and all vowels to [a] if the former are followed by the latter. If you want to combine sound classes to form a list of sounds, you can put them into square brackets. Thus, you could define the following sound classes: unvoiced = p t k voiced = b d g and then combine them to match unvoiced and voiced plosives in a single list in a sound law. [unvoiced voiced] > [p p p p p p] In this sound law, all plosives are turned to a [p]. However, if you write unvoiced voiced > [p p p p p p] the sound classes are interpreted as consecutive sounds and the parsing won’t work. The following law, however, would work, since you define two lists that follow each other. unvoiced voiced > [p p p] [p p p] Allowing to define consecutive sounds is thus a mere shortcut but it can come in handy in those cases where it seems difficult to define complex sound laws. If you pass a sound class, a single sound, or a list of sounds does not make a difference. Thus, if you have defined a sound class ptk as shortcut for the sounds p, t, and k, the following two statements are equivalent: ptk > ptk [p t k] > [p t k] The same holds for sequences of sounds: ptk ptk > ptk ptk [p t k] [p t k] > [p t k] [p t k] Mixing is also possible. ptk [p t k] > [p t k] ptk Note, however, that it is essential that the source and the target always contain the _same amount of sounds_ and the _same amount of positions_. If you want to indicate the loss of a sound, use the – as gap marker: ptk ptk > ptk [- - -] Note in this example, that you cannot write a single gap symbol, but must assemble a group (or define a sound class with the group before), since we require to have one target sound for each source sound and vice versa. This means also, that you must repeat a sound when using sound class notations to formulate sound laws, where a merger happens. [p t k] > [p p p] If you want to indicate that one sound turns into two sounds, which could happen in the case of epenthesis, you must provide the two sounds that replace the one sound in the original separated by a dot, as follows: n > n.d / _ r [a e i o u] MISOL will internally replace the sound by the sequence n.d in the respective context, but the final output will provide the sound in merged form. MISOL will in all cases represent sound laws individually, on the basis of one source sound corresponding to one target sound in one individual context. ### 3.2 Context in Sound Laws The context typically has the form: left_context _ right_context Here, _ represents the source sound. Both left and right context can be omitted. left_context _ _ right_context Context in left and right context is identically defined by a segmental representation of the sound sequence proceeds to the left in the left context and to the right in the right context. In this way, theoretically even very long ranging contexts can be modeled. If one wants to change an [s] followed by [p, t, k] and a vowel to [ʃ], one can write: s > ʃ / _ [p t k] vowel Here, the square brackets [ and ] are used to indicate that the three sounds p, t, and k represent a group that can alternatively occur in the second position following the source sound. A full-fledged toy example that would model that an s becomes voiced when followed by a vowel and turns into a ʃ when followed by a consonant and a vowel, one could define the following sound classes: consonant = p t k b d g s z ʃ ptk = p t k b d g vowel = a e i o u These could then be used in four sound laws: s > ʃ / _ [p t k] vowel s > z / _ vowel p t k b d g > p t k b d g vowel > vowel These would turn a word like s t a b into ʃ t a b but would turn s a b into z a b. Defining groups of sounds with square brackets can be done in a very flexible manner, and even sound classes can be placed inside brackets in order to form new groups of sounds. One can, for example, define two groups of sound classes for voiced and voiceless sounds as follows: ptk = p t k bdg = b d g These can then be used in combination in a sound law. a > e / _ [ptk bdg] 1. 1. 1. 3. ### 3.3 Using Tiers in Sound Laws MISOL is based on the idea that a sound sequence is often best represented as a sequence consisting of multiple _tiers_ (similar to multi-tiered annotation of texts in linguistic examples, such as interlinear-glossed text), that is, a matrix in which different aspects of the sequence are treated in segmental form. Tone, for example, can often be thought of as representing the whole syllable of a word in some tone languages, rather than only one of the sounds in the syllable, or the vocalic nucleus. MISOL supports using multi-tiered sequences in two ways. First, one can define multi-tiered sequences in a very flexible fashion by just providing a matrix of symbols with as many tiers as one wants to use instead of using only one tier alown. A word in a tonal language, consisting of two syllables with two distinct tones, could thus be represented in the following form: p a ŋ t a n ¹ ¹ ¹ ² ² ² In a similar way, one can represent stress in a word, e.g., by using the symbol 1 for stressed syllables and the symbol 0 for unstressed syllables. f a t ə r 1 1 0 0 0 Different tiers apart from the first segmental tier (called segments in MISOL) can be addressed in the context definitions in all positions by using the symbol @, followed by the name of the tier, in front of the group of sounds (marked by square brackets), preceding or following the sound in question (or the sound itself). Thus, to indicate that an unstressed [t] turns into a [d], one can write: t > d / @stress[0]_ When applying this sound law in forward reconstruction, one must provide both tiers (the segments tier and the stress tier) and pass the names of these tiers in the text field to the right of the field where one inserts the sound sequences to be modified, as shown in Figure 2. Figure 2: Using multi-tiered sequence representations in sound laws. The most common use-cases for sound laws with tiers is to define the specific tier value that holds for the segment that one intends to change (such as we have seen in the example). Other use cases, however, are also possible, when thinking of cases where a certain tier value holds for preceding or following sounds. Instead of actively _defining_ and _passing_ new tiers for individual segmental representations of sound sequences, one can also make use of inbuilt functions in MISOL that _compute_ tiers automatically. An example is again the use of a specific tonal tier (called tone), which is computed from the segmental representation of tones using superscript letters at the end of each syllable. Thus, passing a sound sequence such p a ŋ ⁵ d a n ¹ will automatically yield a virtual representation such as the following one internally in the MISOL program: p a ŋ ⁵ d a n ¹ ⁵ ⁵ ⁵ ⁵ ¹ ¹ ¹ ¹ As a result, tonal tiers can be used, as long as the tone values are indicated by superscript numbers at the end of the syllable in each sequence. In order to invoke these tonal tiers, one must indicate this actively when using the forward reconstruction method (or the backward reconstruction), by adding @tone as an additional tier, as shown in Figure 3, assuming the following sound law: d > t / @tone[¹]_ Figure 3: Using multi-tiered sequence representations in sound laws with pre-computed tiers. Apart from tonal tiers (called tone in MISOL), MISOL currently offers three more tiers that can be automated, one tier that checks for the nasality of whole words (returning 1 if a words contains a nasal sound or a nasal vowel, and 0 otherwise), one tier for the initial sound in a word (returning the initial sound for each letter in the word, called initial), and one tier that handles the stress patterns of the word (requiring a specific annotation that uses stress markers to mark syllable boundaries in an explicit manner). As an example, consider the following three sound laws that all use one of the three automated tiers. d > t / @tone[¹]_ p > b / @initial[k]_ b > m / @nasal[1]_ Figure 4 shows, how these can be applied to individual sequences, and where in the tool one needs to provide the information on the tiers that one intents to use. In order to handle stress, stress must be marked in a specific fashion that differs from the current handling in the International Phonetic Alphabet. First, stress markers must be placed in front of every syllable in a word, not only in front of stressed syllabes. Second, stress markers receive their own slot, they should be separated by a space from the rest of the sequence. Third, stress markers must start with either the IPA stress marker ˈ or the quotation mark ‘ (for convenience), or the secondary stress marker ˌ, but stress markers can be expanded by adding arbitrary symbols, allowing to mark different kinds of stress, such as “unstressed by following a stressed syllable” (which would be needed for Verner’s law. Thus, when defining the following sound law, one can handle Verner’s law as well, by passing the sequence ˈ f a ˌ t e r, in which we assume that the secondary stress marker refers to unstressed syllables following a stressed syllable. [p t k] > [f θ x] / @stress[ˌ]_ Figure 4: Comparing different pre-computed tiers in MISoL. ### 3.4 Consecutive Sound Laws The basic idea of MISOL is that sound laws often happen at the same time. For this reason, MISOL does not chain individual sound laws in order in order to let them derive a new sequence to which the next sound law is applied, but rather applies them all at once to the original context. In most cases, these synchronous sound laws are sufficient and much easier to handle than consecutive sound laws. In some cases, however, it is clear that sound laws were active at different stages, called _layer_ in MISOL. Thus, MISOl allows you to “layer” your sound laws by assigning them to different layers which are then executed in the order in which you provide them. As a simple example, consider how Latin _generu-_ became _gendre_ in French. While one could model this change in synchronous sound laws in many ways (for example also by just replacing _e_ by a _d_ when occurring between _n_ and _r_), it is much easier and also closer to actual sound change to think of two different major changes that took place here. First, _generu_ becomes _genru_ , and then the epenthetic _n_ emerges. In order to model these sound laws in MISOL, you must assign individual sets of sound laws to a layer, by adding the layer name, placed in an equal sign, separated with a space, to the left and the right of the layer label. = Layer 1 = e > - / n _ r = Layer 2 = n > n.d / _ r [a e i o u] When applying these sound laws, MISOL will display all internal results, allowing you to track all intermediate forms that lead to the final form proposed by the tool, according to your sound laws. ### 3.5 Limitations in MISOL’s Formal Representations An obvious limitation for which there exists no current solution in MISOL is that context in MISOL cannot be selectively applied to the source values. This means essentially that a process like assimilation of [n] before voiced plosives [b d g] must be represented in three distinct lines. n > m / _ b n > n / _ d n > ŋ / _ ŋ The reason why this is needed is that we cannot assign three different target values to three different contexts, as one might, however, expect (and I am sure there are enough examples in the literature, where people formulate sound laws in this way). [n n n] > [m n ŋ] / _ [b d g] As of now, there is no other way but to spell all sound laws out, even if one feels that they belong together. It may, however, be possible that we find a solution for this problem in the future. **4 Forward Reconstruction (Tab “Forward Reconstruction”)** Forward reconstruction in MISOL is available in different flavors. What all approaches have in common is that MISOL first uses all available information on sound classes and sound laws in order to construct a virtual context window in which sound laws are supposed to take place. This window can be thought of as a multi-tiered sequence representation in which context is not handled on the horizontal axis, but precomputed for each segment in a sequence and represented in individual tiers, each corresponding to one specific context. For a sound law by which voiceless initials are voiced in intervocalic positions, for example, we would need two tiers apart from the base tier in order to represent context to the left of each segment and context to the right of each segment. The sound law could be represented as follows in MISOL (without using any sound classes), we add additional sound laws for completeness. [p t k] > [b d g] / [a i u] _ [a i u] [b d g] > [b d g] [a i u] > [a i u] When dealing with a new sequence b a p a now, MISoL has already inferred from the sound law, that we need two additional tiers, and will now represent the new sequence accordingly, by providing for each segment its right context and its left context, as shown in Table 1 below. **Table 1:** Internal segmental representation in MISoL. Tier | 1 | 2 | 3 | 4 ---|---|---|---|--- Segments | b | a | p | a Segments_Left | ^ | b | a | p Segments_Right | a | p | a | $ From the sound laws shown above, MISOL will construct vectors that represent individual contexts. These sound laws are based on the Cartesian product of the different sounds that can appear in the left and the right context and thus capture all eventualities, as shown in Table 2 below, that shows the individual vectors for the three sound laws (in abbreviated form). **Table 2:** Individual sound law vectors. ID | Law | Segments | Segments_Left | Segments_Right | Target ---|---|---|---|---|--- 1 | 1 | p | a i u | a i u | b 2 | 1 | t | a i u | a i u | d 3 | 1 | k | a i u | a i u | g 4 | 2 | b | Ø | Ø | b 5 | 2 | d | Ø | Ø | d 6 | 2 | g | Ø | Ø | g 7 | 3 | a | Ø | Ø | a 8 | 3 | i | Ø | Ø | i 9 | 3 | u | Ø | Ø | u When iterating over each position in the multi-tiered sequence, MISOL will try to find which of the laws (as shown in the table) provides a vector that matches the current vector in the sequence. The symbol Ø is a wildcard marker and matches with every sound. For our example, we can contrast the actual tiers with the pre-computed individual sound laws as shown below in Table 3. **Table 3:** Contrasting actual tiers with pre-computed individual sound laws. Tier | 1 | 2 | 3 | 4 ---|---|---|---|--- Segments | b | a | p | a Segments_Left (Source / Context) | ^ / Ø | b / Ø | a / **a** i u | p / Ø Segments_Right (Source / context) | a / Ø | p / Ø | a / **a** i u | $ / Ø Sound Law / ID | 2 / 4 | 3 / 7 | 1 / 1 | 3 / 7 Target | b | a | b | a This is the major procedure used by MISOL in order to turn one source sequence into one target sequence. The method (1) precomputes the context for the sequence, which allows it to (2) iterate over each position regardless of the order, searching for matching patterns and their corresponding output. ### 4.1 Strict and Ordered Reconstruction Mode The procedure as outlined here is what is called the “strict” mode in MISOL when using forward reconstruction. It is called “strict” with respect to the mode of reconstruction, since it does not tolerate that different sound laws match the same context and yield different output (MISOL will explicitly mark these cases). Users can choose between the strict mode and the “ordered” mode, in which the matching process is modified in such a way that in the case of competing sound laws, the law that was defined first, wins. This makes coding sound laws much more convenient, since one can first define a very strict law and later define a general law that would hold for all other cases not touched by this first law. s > ʃ / _ [p t k] s > s When passing a sequence s p a s a to this sound law, it would yield the output s|ʃ p a s a in strict mode, and ʃ p a s a in ordered mode, using the pipe to indicate competing sounds (see List et al. 2023 for this notation). ### 4.2 Treatment of Missing Sound Laws Scholars typically omit the “boring” sound laws from their descriptions, specifically those cases of sound change, where no sound change happens. Thus, we rarely find a sound law as the following in the literature. t > t MISOL tolerates the omission of not providing sound laws in those cases where sounds don’t change. The _Forward Reconstruction_ tab offers users to select whether missing sound laws should be _marked_ or _ignored_. If they are ignored, the original sound is used unchanged. If they are marked, the sound will be preceded by an exclamation mark and marked in red. ### 4.3 Tiers in Consecutive Sound Change Processes At the moment, you can only pass explicit tiers (such as accent) once. Since their main purpose is to be able to explain the initial change, they cannot be used in consecutive sound law processes, since this would require us to add a routine by which tiers from a source word turn into tiers from a target word. I have no concrete idea of how I would handle the evolution of multi-tiered sequences in which all tiers defined at an initial stage are propagated down to later stages. As a result, the tiers can be seen as a first idea that I consider worth to be tested, while they are not yet in a stage to allow for a realistic linguistic reconstruction that takes the complexity of sound sequences into account. ## **5 Backward Reconstruction** Backward reconstruction in MISOL is currently restricted to sound laws defined for one layer alone. That means, it is not possible to carry out a backwards reconstruction when multiple layers of sound laws have been defined. In order to test backwards reconstruction, you can load SAMPLE DATA that MISOL provides in the CLASSES AND LAWS tab. If you press on that tab, MISOL will load some sample data into all relevant fields. The data mimics a variety similar to Middle Chinese, with reflexes similar to the ones found in some Chinese dialects. In the tab on Backword Reconstruction, you will see that one form has been listed that is used to illustrate backward reconstruction. The form p a ⁵⁵. If you press on FIND PROTO FORMS, the program will suggest the following distinct proto-forms as shown in Figure 5. The algorithm is fairly simple. It will assemble all potential predecessor segments for all target segments and then construct all possible proto-forms. In a second step, it will apply the sound laws in a forward fashion in order to see if the same form p a ⁵⁵ will turn out as the result. Only those forms that match this output will be retained as possible proto-forms. Figure 5: Possible proto-forms for the target form pa⁵⁵ in the sample data. You can modify the method in order to see which forms were tested, by toggling the SHOW IMPERFECT button. Here, the results will be the same for the form p a ⁵⁵, since we work with a tonal tier in the data (marked by inserting tone into the second field). If you modify the form, replacing the tone ⁵⁵ by the tone ²¹⁴, the SHOW IMPERFECT option will show that there are more possible proto-forms that have been inspected using the greedy procedure, but which were discarded afterwards, by marking those segments that contradict the result. This is shown in Figure 6. Figure 6: Possible proto-forms for the target form pa⁵⁵ in the sample data. ## **6 Import and Export** Import and export of sound laws is based on a very simple text-based format that uses sections to distinguish the different parts of the data. If you have been working with the tool and want to save your sound laws, just go to the Import and Export tab and provide a name for your dataset. If you then click on EXPORT DATA, a button appears, with the name of the data and the ending .txt. Clicking that button allows you to download the data. Figure 7: Import and export tab of the online tool. In order to import a dataset, you just have to click on the Browse button below in the Import and Export tab and then go to Classes and Laws and click on LOAD DATA. This will load the data and you can start with forward reconstruction, and potentially also backward reconstruction. You can find sample files as part of the Codeberg repository (https://codeberg.org/digling/misol) of the tool in the folder examples. You can also download files for nasals.txt and verner.txt directly. ## **7 Outlook and Conclusion** With this attempt of providing a clear specification of the notation system underlying the MISoL tool, I hope to facilitate not only the application of the tool by those interested in modeling and testing sound laws, but also the further development of the tool. The current specifications has clear limitations that may have to be addressed in future work. Obvious lacks are the integration of multi-tiered sequence representations across multiple layers of sound change and the integration of multiple layers in backwards reconstruction. Whether these problems can be solved, however, remains unclear, since some both seem rather hard to be addressed by now, and already the introduction of tiers into the sound law notation scheme may introduce complications that one may want to avoid when trying to model sound laws consistently. Nevertheless, with this specification, a first step has been made towards writing down in a hopefully transparent manner the major ideas that underlie the MISoL tool. In the future, I do not only hope to solve the remaining problems, but also to use this specification as the basis for an additional implementation in a scripting language, that would allow to use the notation system in computer-assisted workflows independently of the online tool. ## Supplementary Data The MISoL software package can be found online at https://misol.edictor.org. The corresponding code is curated on Codeberg (https://codeberg.org/digling/misol, Version 1.0). ## **References** List, Johann-Mattis (2026): _MISoL: Modeling and Inducing Sound Laws with an Interactive Web-Application [Software Tool: Version 0.3]._ Passau: Chair of Multilingual Computational Linguistics at the University of Passau. https://misol.edictor.org List, J.-M., N. Hill, R. Forkel, and F. Blum (2023): Representing and computing uncertainty in phonological reconstruction. In: Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change. Singapur. 22-32. https://aclanthology.org/2023.lchange-1.3 Sims-Williams, Patrick (2018): Mechanising historical phonology. Transactions of the Philological Society. 116.3. 555-573. https://doi.org/10.1111/1467-968X.12138 **Cite this article as:** Johann-Mattis List (2026): “Towards a Specification for the Formal Notation of Sound Laws” in _Computer-Assisted Language Comparison in Practice_ , 9.2: 65-83 [first published on 15/07/2026], URL: https://calc.hypotheses.org/9435, DOI: 10.15475/calcip.2026.2.6. **Download the article as PDF:** calcip-09-2-1.pdf **Copyright information** : This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. **Funding Information** : This project has received funding from the European Research Council (ERC) under the European Union’s Horizon Europe research and innovation programme (Grant agreement No. 101044282). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Johann-Mattis List Seit Anfang 2023 leite ich den Lehrstuhl für Multilinguale Computerlinguistik in Passau. In meiner Forschung nehme ich generell einen datenbasierten,… Read more View all posts * * * The text only may be used under licence Creative Commons Attribution 4.0 International. All other elements (illustrations, imported files) are “All rights reserved”, unless otherwise stated. * * * OpenEdition suggests that you cite this post as follows: Johann-Mattis List (July 15, 2026). Towards a Specification for the Formal Notation of Sound Laws. _Computer-Assisted Language Comparison in Practice_. Retrieved July 15, 2026 from https://calc.hypotheses.org/9435 * * * * * * * *

This study presents a Croatian frequency-derived wordlist mapped to Concepticon concept sets, based on the most frequent nouns, verbs, and adjectives extracted from the hrWaC web corpus. The resulting dataset connects corpus-based Croatian vocabulary to Concepticon’s cross-linguistic framework and includes lexicalizations from nine additional languages for each mapped item. ## **1 Introduction** Standardized lexical datasets have become essential for systematic comparison across languages. Lexibank and Lexibank 2 (List et al. 2022; Blum et al. 2025) are just one of the resources that have made it possible to combine wordlists from diverse sources and connect them through shared identifiers. Concepticon (List et al. 2016, List et al. 2025) is another example that provides stable, language-independent concept set identifiers which allow diverse concept lists to be linked across languages and research traditions. Both Concepticon and the related NoRaRe database (Tjuka et al. 2023) have grown steadily, but language representation remains uneven. Many languages, including Croatian, are currently absent as mapped languages. Croatian is a South Slavic language with a strong linguistic tradition and several high-quality digital resources covering usage, frequency, and semantic structure within the language. For example, HR-CLARIN provides language resources and technologies intended to support research in the humanities and social sciences. On the other hand, Croatian Psycholinguistic Database (Peti-Stantić et al. 2021), provides detailed information about word properties including subjective frequency, age of acquisition, and concreteness for a large set of Croatian lemmas. These resources are useful for research within Croatian, but they are not linked to cross-linguistic infrastructures and do not use shared identifiers. This means that Croatian lexical data cannot easily be compared with data from other languages, or integrated into Concepticon or similar comparative frameworks. This contribution maps a corpus-derived Croatian frequency list to existing Concepticon concept sets. Instead of starting from a curated concept list, as is typical for Concepticon contributions, the starting point here is the most frequent Croatian word forms, asking which of them correspond to existing concept sets. The result is a dataset based on actual language use that is at the same time linked to Concepticon’s cross-linguistic framework. ## **2 Background** ### **2.1 Concepticon and NoRaRe** Concepticon is particularly well suited for this kind of work because it already provides a large, structured inventory of cross-linguistic concept sets that Croatian vocabulary can be mapped onto directly. Rather than building a new comparison framework, mapping Croatian to Concepticon connects it to an infrastructure that already links over 400 concept lists from 56 languages. Its limitations are also worth noting here. Concepticon is not a full formal ontology, as its semantic relations are designed for navigation and exploration rather than to encode strict semantic truths (Tjuka et al. 2023). For the present work, this means that mapping decisions do not need to resolve every ambiguity. The goal is a plausible, defensible link between a Croatian form and an existing concept set, not to make universal claims about meaning. Coverage is a more significant limitation: because Concepticon was built primarily from typological and historical lists, contemporary and domain-specific vocabulary is often absent from its concept inventory. This structural gap defines what the mapping can and cannot show, and is why the unmapped portion of the dataset is treated as a finding rather than a failure. NoRaRe (Tjuka et al. 2023) extends Concepticon with psycholinguistic and cognitive data, in the form of norms, ratings, and semantic relations drawn from published experimental studies. The two resources are linked through the Concepticon ID, meaning that any concept present in Concepticon can be associated with additional information from NoRaRe. Adding Croatian to Concepticon therefore also opens a pathway toward future integration with NoRaRe. ### **2.2 Frequency-Based Approaches and Learner-Oriented Motivation** Frequency lists derived from large corpora are widely used in vocabulary research and language teaching because they provide empirical evidence of which words speakers most commonly encounter in real texts (Kilgarriff 2010). A small number of high-frequency items typically covers a large proportion of running words in texts (Nation 2001), making frequency a reasonable starting point for vocabulary selection. For learners, though, a frequency list alone is not enough. It shows which words are common, but not how they relate to each other in meaning. Working with frequency-based approaches also reveals inherent limitations: while frequency lists record how often a form appears, they do not specify what it means in a given context. A single frequent form may have several distinct meanings, and without context there is no way to know which one is intended (Fulgosi and Tuđman Vuković 2001). Linking high-frequency items to concept sets helps with this by grouping synonyms and near-synonyms under a shared concept, making polysemy visible, and allowing multilingual comparison that goes beyond just translation pairs (Krišto 2026). Recent research has also argued that wordlist development has been dominated by English and a small set of well-resourced languages, and has called for broader, language-diverse approaches (Dang and Webb 2025). The Croatian frequency list presented here is a step in this direction. Within Croatian linguistics specifically, frequency-based and corpus-based resources exist but remain language-internal in their design. The Croatian web dictionary Mrežnik (Hudeček and Mihaljević 2019), for example, shows how corpus evidence can support curated lexicographic resources, but it is not designed for cross-linguistic concept alignment or interoperability with infrastructures such as Concepticon. ## **3 Materials and Methods** ### **3.1 Materials** The dataset was compiled from hrWaC (Croatian Web Corpus), a large collection of texts from Croatian web domains reflecting contemporary written usage on the internet (Ljubešić and Erjavec 2011), chosen because it is large, freely accessible, and covers a broad range of contemporary language. The 1,000 most frequent nouns, verbs, and adjectives were then extracted using Sketch Engine, giving an initial list of 3,000 items. These three word classes were chosen because Concepticon contains concept sets across all three categories, and working with only one would have left a large part of the available concept inventory unexplored. ### **3.2****Data Cleaning** After extraction, the raw wordlists were manually reviewed and cleaned. Several types of items were removed: proper names (such as people, cities, and nationalities), acronyms and administrative terms with no stable Concepticon equivalent, highly specific religious terms (such as names of holidays and religious figures), and items misclassified by the corpus tagger, such as verbs appearing on the adjective list. General terms such as _crkva_ “church” or _bog_ “god” were kept, while strongly culture- or institution-specific items were excluded. Native speaker intuition played an important role in this step, as it helped determine whether items were valid lexical entries suitable for later mapping. The cleaning reduced the dataset from 3,000 to 2,668 items: 981 verbs, 923 nouns, and 764 adjectives. ### **3.3 Translation into English** Translating the Croatian wordlists into English was a necessary step because Concepticon uses English as its meta-language for concept glosses, labels, and associated metadata. The translation combined online tools (Google Translate) with physical and digital dictionaries for cases where automated tools gave unclear results. All translations were manually checked by the author, a native Croatian speaker with advanced English proficiency, after which a second reviewer, a linguist with a degree in English, verified the translations for consistency and accuracy. Translation and mapping turned out not to be fully separable stages. In several cases, consulting Concepticon’s concept definitions led to changes in earlier translation choices, because the definitions made clear that a particular English gloss had a narrower or broader meaning than initially assumed. This kind of back-and-forth is difficult to avoid when working with a polysemous source language and a concept inventory built around specific meanings. ### **3.4 Automated Mapping with PyConcepticon** The mapping process combined automated methods with manual verification. As a first step, Python and the PyConcepticon library (Forkel et al. 2024) were used to generate initial candidate mappings. The tool takes English glosses as input and returns a single best-match Concepticon concept set per gloss. Because Concepticon concept sets are indexed via English labels, the quality of automated mapping depends directly on the quality of the translations produced in the previous step. The automated suggestions provided a useful starting point but required systematic correction. The tool does not perform sense disambiguation and therefore cannot resolve context-dependent meanings. Synonymy and polysemy also required attention: several Croatian words sometimes pointed to the same concept, and a single Croatian word could have more than one English translation, not all of which had a match in Concepticon. For these reasons, all automated mappings were manually reviewed. ### **3.5****Manual Mapping and Mapping Categories** Words that were not automatically matched, or whose automated suggestions seemed uncertain, were mapped manually. This relied on a combination of native speaker intuition, bilingual dictionaries, and, where possible, comparisons with Concepticon mappings in related languages. During the annotation process, items were organized in four working categories: automatically mapped and verified (PyConcepticon), manually mapped, uncertain (many of which were later resolved), and unmapped. These categories are not retained in the final dataset but were useful for managing the workflow, especially for difficult and borderline cases. All mappings were carried out using Concepticon Version 3.4.0. ### **3.6****Adding Frequency Information and Multilingual Equivalents** After the mapping stage, corpus frequency information was added to each item in the dataset using Sketch Engine’s built-in frequency tool. Each entry includes its absolute corpus frequency, its rank in the cleaned list, and the rank of its associated Concepticon concept set where applicable. This means that each mapping can be read alongside how frequently the word actually appears in contemporary Croatian. As a final step, lexicalizations from nine additional languages were added for each mapped item by consulting the corresponding Concepticon entries: Serbian, Hungarian, Czech, German, Italian, French, Spanish, Russian, and Ukrainian. These languages were selected because they are regionally relevant, because some are spoken as minority languages, and because they are the languages most commonly taught in Croatian schools (Kapović 2022). This gives the dataset a multilingual dimension that goes beyond Croatian-English glossing and makes it more useful for cross-linguistic comparison. ## **4 Analysis** ### **4.1 Mapping Coverage** Of the 2,668 items in the cleaned dataset, 1,262 (47.3%) were mapped to at least one Concepticon concept set, covering 856 unique concept set identifiers (Figure 1). In addition, 12 items were assigned two Concepticon IDs, so the dataset contains 1,274 total Concepticon assignments (mapping links). The remaining 1,406 items (52.7%) remained unmapped. Figure 1: Mapped and unmapped items by word class. Verbs show the highest mapping rate, adjectives the lowest. ### **4.2 Unmapped Items: Derivational Mismatches and Modern Vocabulary** The majority of unmapped items fall into two broad patterns. The first one involves derivational and part-of-speech mismatches. Many unmapped Croatian items are semantically close to an existing concept set but appear in a derived form (as relational adjectives, agent nouns, or nominalized verbs) that does not match the lexical category Concepticon most commonly represents. Relational adjectives are particularly frequent in this group: forms such as _ljubavni_ “love-related”, _školski_ “school-related”, and _vjers_ _ki_ “religious” are close in meaning to concept sets such as LOVE, SCHOOL, and RELIGION, but mapping them would mean collapsing the “related to X” meaning into the base noun concept, which was avoided in order to keep the mappings consistent and transparent. The second pattern concerns modern institutional vocabulary. A large number of unmapped items belongs to domains that are prominent in web-based language use but largely absent from Concepticon’s concept inventory: technology (e.g., web, online, portal), sports (e.g., football, fan, champion), politics and governance (e.g., politician, parliament, coalition), finance and money (e.g., paycheck, investment, budget), religion (e.g., believer, bishop, parish), and media and culture (e.g., art, history, cinematic). These clusters were identified through manual coding, grouping items by shared topic. This reflects a structural difference between the two resources: Concepticon was built primarily from lists motivated by typological and historical goals, where the focus is on stable, cross-culturally recurrent meanings, while a web corpus naturally surfaces vocabulary from contemporary public and institutional life. A smaller group of unmapped items points to a different kind of problem: not a mismatch in word class or domain, but a mismatch between a universal concept and a culturally specific meaning. For example, _dvojica_ „two male persons“ could not be mapped to TWO because Concepticon’s concept set does not encode the human and gender-specific restriction that is central to the Croatian form. These cases are small in number but illustrative: they show that some Croatian words carry cultural or grammatical specificity that sits outside what a language-independent concept inventory can currently represent. ### **4.3****Mapped Items: Many-to-One Clusters and Polysemy** Within the mapped subset, the most notable pattern is many-to-one mapping, where multiple Croatian forms linked to a single Concepticon concept set. This happens most often with near-synonyms, aspectual verb pairs, and prefix-derived verb variants. The concept set UNDERSTAND (ID 1536) attracted the largest cluster, with six Croatian forms: _kužiti, razumjeti, shvatiti, shvaćati, skužiti,__and_ _uvidjeti_. These forms share a conceptual core but differ in register, aspect, and usage context, which remain visible at the form level even though Concepticon groups them under one concept. Similar patterns appear with descriptive adjectives such as _velik, ogroman, golem,_ and _krupan_ all mapping to BIG (ID 1202), and with verb families such as _završiti, završavati,_ and _dovrš_ _iti_ all mapping to FINISH (ID 1766). Table 1 gives an overview of where this clustering is most pronounced. **Table 1.** Top 10 Concepticon concept sets by number of mapped Croatian forms, with Croatian variants listed. **Rank** | **Concepticon ID** | **Concept name** | **# Croatian forms** | **Croatian forms** ---|---|---|---|--- 1 | 1536 | UNDERSTAND | 6 | kužiti, razumjeti, shvatiti, shvaćati, skužiti, uvidjeti 2 | 3004 | FORMER | 6 | bivši, dosadašnji, nekadašnji, prethodan, prijašnji, tadašnji 3 | 7 | GATHER | 5 | okupiti, okupljati, prikupiti, prikupljati, skupiti 4 | 680 | BREAK (DESTROY OR GET DESTROYED) | 5 | probiti, puknuti, pući, razbiti, slomiti 5 | 692 | BRING | 5 | donijeti, donositi, dovesti, dovoditi, ponijeti 6 | 707 | REMEMBER | 5 | pamtiti, prisjetiti, sjetiti, sjećati, zapamtiti 7 | 1309 | ASK (INQUIRE) | 5 | pitati, upitati, zamoliti, zapitati, zatražiti 8 | 1529 | LAST (FINAL) | 5 | finalan, konačan, krajnji, posljednji, zadnji 9 | 1752 | LEAVE | 5 | odlaziti, ostaviti, ostavljati, otići, prepustiti 10 | 1778 | APPROACH | 5 | bližiti, približavati, pristupati, pristupiti, prići A small number of items, 12 in total, were assigned two Concepticon IDs rather than one, for different reasons. The clearest group consists of genuinely polysemous Croatian forms, where one word covers two conceptually distinct meanings. Several of these reflect well-known cross-linguistic patterns: _mjesec_ maps to both MONTH (ID 1370) and MOON (ID 1313), _jezik_ to both LANGUAGE (ID 1307) and TONGUE (ID 1205), and _zemlja_ to both COUNTRY (ID 1300) and EARTH/SOIL (ID 1228). A second group reflects Concepticon’s finer-grained sense inventory rather than polysemy in Croatian itself. _Dan_ , for example, was mapped to both DAY (NOT NIGHT) (ID 1225) and DAY (24 HOURS) (ID 1260) because Concepticon distinguishes two senses that a context-free lemma cannot. Similarly,_kraj_ and _završetak_ were linked to both END (OF SPACE) (ID 742) and END (OF TIME) (ID 743). In all these cases, allowing two mappings makes the ambiguity explicit rather than forcing a single interpretation that the data cannot support. ### **4.4 Multilingual Coverage** Of the 856 unique concept sets covered by the mapped Croatian items, 831 (97.1%) contain at least one lexical entry in the nine additional language columns. Coverage is however uneven: German is the best represented with forms for 748 concept sets, followed by Russian (722), French (705), and Spanish (647). Hungarian (515) and Italian (396) occupy a middle position, while Czech (82), Serbian (80), and Ukrainian (8) are sparsely represented. These differences reflect the number and scope of Concepticon lists available for each language rather than any absence of the concepts in those languages. For example, Ukrainian forms appear almost exclusively for colour terms, as the Ukrainian data in Concepticon derive primarily from a colour term study (Jonauskaite et al. 2020). ## **5 Conclusion and Outlook** This contribution mapped the 1,000 most frequent nouns, verbs, and adjectives from the hrWaC web corpus to existing Concepticon concept sets. After cleaning and translation, the final dataset contains 2,668 items, of which 1,262 (47.3%) were successfully mapped to 856 unique Concepticon concept set identifiers, with lexicalizations from nine additional languages. The mapping results show that Concepticon covers stable, cross-linguistically recurrent meanings well, but provides weaker coverage for morphologically derived forms and for vocabulary typical of contemporary web-based language use. The unmapped items are not a failure of the mapping process but a finding in their own right, as they identify specific conceptual areas where Concepticon’s current inventory diverges from everyday modern usage. Within the mapped subset, many-to-one mappings are common, and most mapped concepts are also represented in at least one additional language in Concepticon, which confirms that the Croatian items were primarily linked to well-established conceptual cores. Several limitations of the study are worth acknowledging. The dataset is based on a web corpus, so the frequency distribution reflects genres typical of online discourse, including news, public commentary, and informational texts. Also, spoken or informal vocabulary is likely underrepresented. The mapping relied on English as an intermediate language, which means that language-specific nuances may not always be fully captured by a single English gloss. The manual verification was also carried out by a single annotator, and the dataset would benefit from inter-annotator agreement checks on borderline cases in future work. As for future directions, the frequency-first workflow used here could be applied to other languages currently absent from Concepticon, enabling broader cross-linguistic comparison of frequency-concept alignment. Methodologically, distributional semantic methods such as word embeddings could support mapping decisions for polysemous and borderline cases where context is unavailable. The full methodology and extended analysis are documented in the underlying Master’s thesis (Krišto 2026), to which the reader is referred for further detail. ## **References** Blum, Frederic and Barrientos, Carlos and Englisch, Janina and Forkel, Robert and Greenhill, Simon and Rzymski, Christoph and List, Johann-Mattis (2025): Lexibank 2: pre-computed features for large-scale lexical data. Open Research Europe 5. 126. _https://do_ _i.org/10.12688/openreseurope.20216.2_ Dang, Thi Ngoc Yen and Webb, Stuart (2025): Applications of word lists in second language learning and teaching. Language Teaching 58. 1–21. Forkel, Robert and Rzymski, Christoph and List, Johann-Mattis (2024): PyConcepticon [Python library, Version 3.1.0]. Max Planck Institute for Evolutionary Anthropology: Leipzig. _https://pypi.org/project/pyconcepticon_ Fulgosi, Sanda and Tuđman Vuković, Nina (2001): Relevantnost frekvencije jezične uporabe pri opisu strukture leksema. Suvremena lingvistika 27. 73–85. Hudeček, Lana and Mihaljević, Milica (2019): Croatian web dictionary – Mrežnik vs. Croatian linguistic terminology – Jena. In: Proceedings of the 7th International Conference The Future of Information Sciences INFuture2019: Knowledge in the Digital Age. 22–31. _https://doi.org/10.17234/INFUTURE.2019.4_ Jonauskaite, Domicele and Parraga, C. Alejandro and Quiblier, Michael and Mohr, Christine (2020): Feeling blue or seeing red? Similar patterns of emotion associations with colour patches and colour terms. Perception 11. _https://doi.org/10.1177/2041669520902_ _484_ Kapović, Mate (2022): Strani jezici u formalnom obrazovanju u Hrvatskoj. Strani jezici 51. 283–309. Kilgarriff, Adam (2010): Comparable corpora within and across languages, word frequency lists and the KELLY project. Lexical Computing Ltd, Brighton. Krišto, Anja and Mambrini, Francesco (2026): Integrating Croatian into Concepticon: a corpus-based frequency mapping of Croatian vocabulary. Master’s thesis, Università Cattolica del Sacro Cuore, Milan. _https://doi.org/10.17613/p4fqh-03m92_ List, Johann-Mattis and Cysouw, Michael and Forkel, Robert (2016): Concepticon: A resource for the linking of concept lists. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016). 2393–2400. _https://aclanthol_ _ogy.org/L16-1379_ List, J.-M., A. Tjuka, F. Blum, A. Kučerová, C. Barrientos Ugarte, C. Rzymski, S. Greenhill, and R. Forkel (2025): CLLD Concepticon [Dataset, Version 3.4.0]. Max Planck Institute for Evolutionary Anthropology: Leipzig. _https://concepticon.clld.org_ List, Johann-Mattis and Forkel, Robert and Greenhill, Simon J. and Rzymski, Christoph and Englisch, Johannes and Gray, Russell D. (2022): Lexibank: A public repository of standardized wordlists with computed phonological and lexical features. Scientific Data 9. 44. _https://doi.org/10.1038/s41597-022-01432-0_ Ljubešić, Nikola and Erjavec, Tomaž (2011): hrWaC and slWaC: Compiling web corpora for Croatian and Slovene. In: Proceedings of the 14th International Conference on Text, Speech and Dialogue. 395–402. _https://doi.org/10.1007/978-3-642-23538-2_50_ Nation, I.S.P. (2001): Learning Vocabulary in Another Language. Cambridge University Press, Cambridge. Peti-Stantić, Anita and Anđel, Maja and Gnjidić, Vedrana and Keresteš, Gordana and Ljubešić, Nikola and Masnikosa, Irina and Tonković, Mia and Tušek, Jana and Willer-Gold, Jana and Stanojević, Mateusz-Milan (2021): The Croatian psycholinguistic database: Estimates for 6000 nouns, verbs, adjectives and adverbs. Behavior Research Methods 53. 1799–1816. _https://doi.org/10.3758/s13428-020-01533-x_ Tjuka, Annika and Forkel, Robert and List, Johann-Mattis (2023): Curating and extending data for language comparison in Concepticon and NoRaRe. Open Research Europe 2. 1–11. _https://doi.org/10.12688/openreseurope.15380.3_ **Cite this article as:** Krišto, Anja (2026): “Integrating Croatian into Concepticon: a Corpus-Based Frequency Mapping of Croatian Vocabulary” in _Computer-Assisted Language Comparison in Practice_ , 9.1: 55-64 [first published on 08/06/2026], URL: https://calc.hypotheses.org/9392, DOI: 10.15475/calcip.2026.1.6. **Download the article as PDF:** calcip-09-1-6.pdf **Copyright information** : This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Anja Krišto View all posts * * * The text only may be used under licence Creative Commons Attribution 4.0 International. All other elements (illustrations, imported files) are “All rights reserved”, unless otherwise stated. * * * OpenEdition suggests that you cite this post as follows: Anja Krišto (June 8, 2026). Integrating Croatian into Concepticon: a Corpus-Based Frequency Mapping of Croatian Vocabulary. _Computer-Assisted Language Comparison in Practice_. Retrieved June 8, 2026 from https://calc.hypotheses.org/9392 * * * * * * * *

This tutorial shows how to build beautiful maps using polygon data of language distributions and the geospatial software QGIS. For this purpose, I show in the first part how to extract a list of Glottocodes from CLDF datasets and how to use command-line tools to extract the polygon data from Glottography datasets. The second part focuses on the visualization of the extracted data with QGIS. In this part, I also show to include non-linguistic data like archaeological sites and how to prepare the map for printing. The tutorial is fully implemented with free and open software and intends to make map-making accessible for linguists and other researchers. ## 1. Introduction A common representation of languages in quantitative studies is through the use of point coordinates in maps and statistical models. Since languages are distributed across areas, this critique certainly has a point. If some random mean of location is chosen for languages which are distributed across vast areas, two languages appear far further away from each other than they actually are. On the other hand, it is unclear how to model such distances, since only speakers at borders or overlapping areas are actually in contact, and the speakers in the mean locations or the far extremes are not. An example of such a discussion can be observed in the reviews and answers to the reviewers in (Guzmán Naranjo & Jäger 2024). Regarding the presentation of languages in maps, the answer to the critique is often “but we don’t have any other data”, especially in large quantitative studies with a worldwide sample. Alas, after the publications of multiple polygon datasets of the Glottography project (Ranacher et al. 2025, 2026), this is no longer true (if it ever was). The project published multiple datasets of existing publications like the “World Atlas of Languages” (Asher & Moseley 2018). In contrast to other collections of language distributions like Ethnologue, the Glottography data comes as open-access repositories and is not behind a paywall. In its current form, the project already covers more than 5,300 languages from 29 different publications and is likely to increase in coverage in the future. The presentation of the data using the Cross-Linguistic Data Formats (CLDF, Forkel et al. 2018) supports good integration with other existing datasets in CLDF like Lexibank (Blum et al. 2025) or Grambank (Skirgård et al. 2023). It is part of this tutorial to show how this integration can be leveraged through the command-line interface and SQLite queries. The first question I asked myself as someone who has a map in almost all his articles was: “How can I make nice maps with such polygons”? While there are some online-tools which provide quick maps, I was not satisfied by this solution because of the lack of control, difficulties in reproduction, and unclear long-term reliability that these tools provide. For comparative purposes, I will briefly present one of those tools later on. While those online tools offer quick solutions and are easily accessible, QGIS offers a broad variety of customization for maps that provide complex representations of multiple data types. Especially when it comes to important publications, such maps can make the difference in how (re)viewers perceive the quality of the work and provide a more realistic representation of the distribution of languages across the world. This contribution thus seeks to provide a short tutorial on (a) how to extract the data for a set of languages, (b) to import the data to different visualization tools, and (c) how to make a beautiful map of language distributions using QGIS. ## 2. Software Requirements We will be using QGIS (v4.0.1, Dawson et al. 2026), which is an open-source software for spatial visualization (https://qgis.org/download/). It is a very powerful tool, of which we will only make use of a small amount of its features. You can install the application from the link above on all common operating systems. The tutorial requires a basic setup that involves a functional installation of Python 3 (>3.10) and GIT (https://git-scm.com/). With Linux and MacOS systems, this is straightforward and will not be covered here. For the usage with Windows, I recommend to have a look at the corresponding tutorial (Snee 2024). We also recommend the installation of SQLite (https://sqlite.org/). For the initial data processing, we will be running Python from the command line. Here, we recommend using a virtual environment. On MacOS, you can set up one such environment with the following commands: # Create an environment within the folder venv/ python3 -m venv venv/your-virtual-environment-1 # Activate the environment source venv/your-virtual-environment-1/bin/activate Now, we will be installing a couple of packages that we need to run the code using the Python package managing tool `pip`: pip install pycldf pip install csvkit pip install cldfgeojson We will also use multiple datasets that are curated on GitHub, which you can clone directly via `git` on the command-line: git clone https://github.com/Glottography/asher2007world git clone https://github.com/Glottography/queixalos2000linguas git clone https://github.com/pano-takanan-history/blumpanotacana All the code code and data that are used for this tutorial are curated on Codeberg, which you can also clone in the same way: git clone https://codeberg.org/calc/tutorial-visualizing-polygons ## 3. Extracting the Polygon Data In this part, I will briefly explain how to extract the polygons for (1) languages from existing CLDF datasets and (2) how to extract a set of glottocodes directly from Glottolog, either through an explicit list or based on language-families. Both tasks are fundamentally the same: We extract a list of Glottocodes from a dataset and then extract the corresponding polygons from Glottography repositories. ### 3.1 Extracting Polygons from Wordlist Collections For retrieving the polygon data from Glottography datasets such as the “Atlas of the World’s Languages” (https://github.com/Glottography/asher2007world, Asher & Moseley 2018), we need a CSV-file with a column for Glottocodes. Such a file representing the LanguageTable of CLDF is standard in all datasets of Lexibank and I will exemplify this with the `blumpanotacana` dataset (Blum et al. 2024a) which we have cloned above. The command to extract the polygon data proceeds in four steps: (1) Identify the Glottocodes in the input file, (2) Remove the header, (3) Extract the polygons of a Glottography dataset based on the provided set of Glottocodes, and (4) Save the file in `.geojson` format. csvcut -c Glottocode blumpanotacana/cldf/languages.csv | \ csvformat -E | \ cldfbench geojson.geojson --no-glottolog queixalos2000linguas/cldf - \ > data/bpt_polygons.geojson One quick way of visualizing the output is to upload the content of this file to https://geojson.io/next/, which provides a basic map using an OpenStreetMap background and some basic customization options. This is great for a quick view on the extracted data, but not sufficient for most publications. I show an example of this in Figure~1. Figure 1: Map of languages in the blumpanotacana dataset uploaded to the website for easy visualization. The visualization was automatically created by drag&drop of a .geojson file. ### 3.2 Querying Glottolog for a List of Glottocodes Another option is to extract Glottocodes for specific languages or language families directly from Glottolog using an SQLite query. For example, I was approached by a researcher who approached me about a map of Arawak, Carib, and Tupi languages across the Amazon. Since the point coordinates hide much of the true extension of the families, I recommended going for polygons instead, making use of the newly published datasets. To extract data from Glottolog (v5.3, Hammarström et al. 2026), we clone the data and create an SQLite database. This step presupposes you have `sqlite3` installed to be used on the command line. The usage of SQLite queries with large datasets like Glottolog or Lexibank has proven worthwhile because it is a lot quicker than iterating through a CSV or TSV file. It also allows performing data preprocessing tasks during the data extraction (Blum et al. 2024b). Through `pycldf`, we first create an SQLite database of Glottolog and then open it via `sqlite3` to run the query. # Clone dataset git clone https://github.com/glottolog/glottolog-cldf # Create SQLite database cldf createdb glottolog-cldf/cldf/cldf-metadata.json glottolog.sqlite3 # Open database sqlite3 glottolog.sqlite3 You should be prompted within a sqlite3 console now: INFO <cldf:v1.0:StructureDataset at glottolog-cldf/cldf> loaded in glottolog.sqlite3 SQLite version 3.51.0 2025-06-12 13:14:41 Enter ".help" for usage hints. sqlite> In the next step, we build a short SQLite query to retrieve the languages of our choice from Glottolog. In this case, we want to include language families from the Arawak, Carib, and Tupi language families. For the purpose of illustration, I have also added two language isolates to show how we can add individual glottocodes from languages to the query. Within the sqlite3 console, we first choose settings so that the results of the query are stored as CSV with headers. .mode csv .headers on .output data/glottocodes.csv Then, we are ready to run a query to extract a CSV-file from three different languages families by providing the family-level glottocode. This extracts all Glottocodes of languages that belong to the respective family. Alternatively, we can also select language-level Glottocodes. You can also extract other columns of tables that are part of Glottolog, like language family, point coordinates, or others, that are then also stored to the output CSV-file. This saving of the list is done in the last line of the console prompt. SELECT l.cldf_id AS Glottocode, -- Example for additional columns l.Family_ID FROM LanguageTable AS l WHERE -- List of language-family glottocodes l.Family_ID IN ('araw1281', 'cari1283', 'tupi1275') OR -- List of language-level glottocodes l.cldf_glottocode IN ('taus1253', 'muni1258') ; .output stdout After extracting our list of desired Glottocodes, we can use this CSV (more than 400 entries!) to query the Glottography repository to extract all available polygon data for the languages in the list. This time, we will be using the “As línguas amazônicas hoje” dataset (Queixalos & Renault-Lescure 2000), which is focused on languages in the Amazon. The command is similar to the one from above, only using different input, output, and geojson file names. csvcut -c Glottocode polygons.csv | \ csvformat -E | \ cldfbench geojson.geojson --no-glottolog asher2007world/cldf/traditional - \ > data/amazon_polygons.geojson If we only want data from a specific language family, we can also use a prompt that is based on `csvkit`, which we have installed previously. This is a bit slower than the SQLite query, but if we are only interested in a single family, this can be a reasonable alternative. csvgrep -c Parameter_ID -r "^classification$" \ glottolog-cldf/cldf/values.csv | \ csvgrep -c Value -m cari1283 | \ csvcut -c Language_ID | \ csvformat -E | \ cldfbench geojson.geojson --no-glottolog asher2007world/cldf/traditional - \ > data/carib_polygons.geojson Until now, I have shown different ways of extracting geojson data from Glottography projects, using either data from a single CLDF dataset, or extracting data for language families directly from Glottolog. Now we are ready to turn to the visualization of the extracted data. ## 4. Using GIS for Visualizing Language Polygons ### 4.1 Setting up QGIS We will be using GIS for the map making, which stands for “geographic information system”. This acronym refers to systems that allow the editing of geospatial data. For this tutorial, I have used the OpenSource application QGIS (v4.0.1). Some configurations might look slightly different in older versions, but the basic functionality is the same. When opening the application, you can choose between a blank template or an OpenSteetMap Basemap. We choose the latter. You can also add map data from other resources (e.g. https://www.tracestrack.com), but this will not be covered as part of the tutorial. ### 4.2 Loading the Data Within a QGIS project, you have several toolbars. We will only use a small amount of functions for the moment. I will visualize the basic tools for our purpose using the `amazon_polygons.geojson` file with the data from Arawak, Carib, and Tupi languages. In the very first step, you can simply drag&drop the geojson-file of your choice to QGIS. You should now already see the distributions of those languages on the map after the drag&drop operation. ### 4.3 Basic Visualization Tools The first customization is to colour the distributions according to language family. To do this, we double-click the layer-name in the bottom left pane to open the layer properties. Here, we select “Symbology” to change the representation of each polygon. At the top, currently “Single Symbol” is chosen by default. We change this to “Categorized” and select “family” as “Value”. A click on “Classify” at the bottom created a random colour ramp based on the different values on the “family” variable on the input data. This setting changes the fill-colour of all polygons. The three steps are shown in Figure 2. Figure 2: Distribution of Arawak, Carib, and Tupi languages across the Amazon as visualized in QGIS. The application interface shows how the fill-colour for each language family can be customized. ### 4.4 Bonus: Adding Archaeological Sites in QGIS QGIS begins to excel once we combine different data types. For example, we can add data for archaeological sites that come in TSV or CSV-files. For this purpose, I have created a CSV-file from the archaeological sites published in the article “Archaeological expansions in tropical South America during the late Holocene: Assessing the role of demic diffusion” (Gregorio de Souza, Alcaina Mateos & Marco 2020) which originally came in ODS-Format (https://pmc.ncbi.nlm.nih.gov/articles/instance/7185720/bin/pone.0232367.s001.ods). The CSV-file could be made part of the tutorial repository, since the data is thankfully published under a CC-license. To include this file, add a new layer in QGIS and import this file by selecting “Layer > Add Layer > Add Delimited Text Layer”. Here, you browse through your file-system to choose the input file and select the correct encoding. In the next step, you have to select the Longitude (X) and Latitude (Y) columns and choose a Geometry system (CRS). Now, you have a second layer with the archaeological sites and can again choose a colouring system of your choice, as well as other customizations like opacity, size, and stroke for a symbol of your choice. To give just one example, you could use different SVG markers for sites related to different archaeological traditions. It is also possible to import custom SVG’s. The import of the CSV file is illustrated in Figure~3. Figure 3: Importing CSV data within QGIS. It is important to set the Geometry Definition for coordinates and the CRS (coordinate reference system). ### 4.5 Print Layout The final task is a print layout. Doing this for a first time, it seemed a bit unintuitive and unnecessarily complex to me. But soon I realized that QGIS is worth it, since it actually offers you a lot of layout options that make it easy to include a map cutout and a legend to the final version of your map. Also, the richness of customization options opens up many possibilities of building high-quality maps. To begin with this process, either select “Cmd + P”, or select “Project > New Print Layout” from the main toolbar. This opens a second window, which will be your print layout. First, we select “Add Map” from the symbols at the left and drag across the whole layout page to include our main map. The map gets added exactly as you see it in your map interface. Second, we select “Add Legend”, and again drag the area in the layout where we want to put the legend. The legend automatically accesses our selected symbology (shape, colour) for the respective layers of language families and archaeological sites. Now, we already have map and legend ready for print. Great! Towards the right, you find a lot of customization options for all added items. For the legend, we want to change the names of the layers to something more insightful. For this, we first change to “Manual” under “Legend Items” and select new names for each legend to “Archaeological Traditions” and “Language Families”. You can also customize background, frames, and all other kinds of data that is shown. This allows you to change the text and icons of all the symbology visualized in the legend. In a final step, we want to add a bigger map as a cutout in a top corner. For this, we first need to select the Map 1 from the items panel and lock the layers, so that they don’t get changed when we change the original map. Then, we switch back to said main map, deactivate the language and archaeology layers by unticking the box before their name and zoom out to the desired view of the world. We can now return to the Print Layout and add a second map. Again we can select the area where we want to include the map, for which I chose the top right corner. Through selecting “Add Shape” in the left item panel we can add a rectangle that indicates the area of our map we have visualized. Simply drag the rectangle above your little world map and move it to the corresponding area. There is probably also some automated way of referencing the extent of the cut-out map of which I am not currently aware. But considering the offered complexity of QGIS, I’d be very surprised if this was not the case. The final map now has the original language distributions, the archaeological sites, a legend, and a world-map in the top corner. Through “Cmd + P” or by clicking the “Export PDF” symbol in the toolbar, you can now export a PDF file of your print layout for dissemination and publishing. The options for the print layout are shown in Figure 4. Figure 4: Final print layout including the main map, a legend, and a world-map indicating the extent of the selected map. Such visualizations can help us to build hypotheses about historical expansions of language and culture, putting linguistic data into spatial context. In the case of the three Amazonian language families, we see interesting overlaps of the distribution of language families and different archaeological traditions. These links have been proposed in the archaeological literature and often rely on such spatial overlays of linguistic and archaeological distributions (Iriarte 2024). Other possibilities involve topographic maps, which put more emphasis into geological information such as mountains and rivers and could be used to reference possible migration or contact routes. To understand the spatial element behind such distributions is the necessary first step for a more sophisticated modelling of space. ## 5. Conclusion This tutorial provided an introduction to annotating language distributions with QGIS and showed how to combine different kinds of input data. The toolset offered by QGIS seems complex at first, but offers a wide variety of customization options that make it possible to create high-quality maps. The visualization of languages as polygons provides a more realistic account of the distribution of language families and can lead to a better understanding of contact, expansion, and diversification. The polygon data is also the necessary first step for incorporating language distributions in spatial models for computational analysis, potentially linking up the analysis with other kinds of geospatial data like rivers or mountains. Again, QGIS can be a valuable tool for the visualization of such information. ## Acknowledgements Special thanks to Robert Forkel, who provided much of the code of the command-line applications and to Antonia Lieschke, who answered all my QGIS questions. ## References Asher, R. E. & Christopher Moseley. 2018. _Atlas of the world’s languages_. 2nd edn. London: Routledge. https://doi.org/10.4324/9781315829845. Blum, Frederic, Carlos Barrientos, Johannes Englisch, Robert Forkel, Simon J. Greenhill, Christoph Rzymski & Johann-Mattis List. 2025. Lexibank 2: Pre-computed features for large-scale lexical data. _Open Re_ _search Europe_ 5(126). 1–27. https://doi.org/10.12688/openreseurope.20216.2. Blum, Frederic, Carlos Barrientos, Roberto Zariquiey & Johann-Mattis List. 2024a. A comparative wordlist for investigating distant relations among languages in Lowland South America. _Scientific Data_ 11(1). https://doi.org/10.1038/s41597-024-02928-7. Blum, Frederic, Ludger Paschen, Robert Forkel, Susanne Fuchs & Frank Seifart. 2024b. Consonant lengthening marks the beginning of words across a diverse sample of languages. _Nature Human Behaviour_ 8. 2127–2138. https://doi.org/10.1038/s41562-024-01988-4. Nyall Dawson, Jürgen Fischer, Matthias Kuhn, Alessandro Pasotti, Denis Rouzaud, mhugent, Alexander Bruy, et al. 2026. qgis/QGIS: 4.0.1. Zenodo. https://doi.org/10.5281/zenodo.19401447. Forkel, Robert, Johann-Mattis List, Simon J. Greenhill, Christoph Rzymski, Sebastian Bank, Michael Cysouw, Harald Hammarström, Martin Haspelmath, Gereon A. Kaiping & Russell D. Gray. 2018. Cross-Linguistic Data Formats, advancing data sharing and re-use in comparative linguistics. _Scientific Data_ 5(1). 1–10. https://doi.org/10.1038/sdata.2018.205. Gregorio de Souza, Jonas, Jonas Alcaina Mateos & Madella Marco. 2020. Archaeological expansions in tropical South America during the late Holocene: Assessing the role of demic diffusion. _PLOS ONE_ 15(4). 1–32. https://doi.org/10.1371/journal.pone.0232367. Guzmán Naranjo, Matías & Gerhard Jäger. 2024. Euclide, the crow, the wolf and the pedestrian: Distance metrics for linguistic typology. _Open Research Europe_ 3(104). 1–32. https://doi.org/10.12688/openreseurope.16141.2. Hammarström, Harald, Robert Forkel, Martin Haspelmath & Sebastian Bank. 2026. _Glottolog database (v5.3)_. Leipzig: Max Planck Institute for Evolutionary Anthropology. https://doi.org/10.5281/zenodo.18840935. Iriarte, José. 2024. _The archaeology of Amazonia: A human history_. London: Bloomsbury Academic. https://doi.org/10.5040/9781350270770. Queixalos, Francisco & Odile Renault-Lescure. 2000. _As línguas amazônicas hoje_. São Paulo: IRD/Instituto Socioambiental/MPEG. http://www.cartographie.ird.fr/linguas2.html. Ranacher, Peter, Robert Forkel, Nour Efrat-Kowalsky, Matthias Urban, Antonia Hehli, Micha Franz, Gregory Biland, et al. 2025. A global and interoperable dataset of linguistic distributions derived from the Atlas of the World’s Languages. _Scientific Data_ 12(1466). 1–10. https://doi.org/10.1038/s41597-025-05828-6. Ranacher, Peter, Robert Forkel, Nour Efrat-Kowalsky, Matthias Urban, Antonia Hehli, Micha Franz, Gregory Biland, et al. 2026. Glottography: An Open-Source Geolinguistic Data Platform for Mapping the World’s Languages. Journal of Open Humanities Data 12(47). 1–16. https://doi.org/10.5334/johd.459. Skirgård, Hedvig, Hannah J. Haynie, Damián E. Blasi, Harald Hammarström, Jeremy Collins, Jay J. Latarche, Jakob Lesage, et al. 2023. Grambank reveals the importance of genealogical constraints on linguistic diversity and highlights the impact of language loss. _Science Advances_ 9(16). 1–15. https://doi.org/10.1126/sciadv.adg6175. Snee, David. 2024. Using CLDFBench and PyLexibank on Windows. University of Passau 7(2). 103–109. https://doi.org/10.15475/CALCIP.2024.2.6. **Cite this article as:** Blum Frederic (2026): “Visualizing Language Polygons with (Q)GIS” in _Computer-Assisted Language Comparison in Practice_ , 9.1: 41-53 [first published on 04/05/2026], URL: https://calc.hypotheses.org/9287, DOI: 10.15475/calcip.2026.1.5. **Download the article as PDF:** calcip-09-1-5.pdf **Copyright information** : This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. **Funding Information** : This project has received funding from the Max Planck Society. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. * * * The text only may be used under licence Creative Commons Attribution 4.0 International. All other elements (illustrations, imported files) are “All rights reserved”, unless otherwise stated. * * * OpenEdition suggests that you cite this post as follows: Frederic Blum (May 4, 2026). How To Visualize Language Polygons with QGIS (How to Do X in Linguistics 15). _Computer-Assisted Language Comparison in Practice_. Retrieved May 4, 2026 from https://calc.hypotheses.org/9287 * * * * * * * *

This study gives a brief overview on formal aspects of etymological analysis, by providing a modified workflow for the classical comparative method in historical language comparison. This workflow is contrasted with the current state-of-the-art in computational historical linguistics, pointing out where computational methods and interactive tools for annotation are lacking, and where they are available already. ## 1 Introduction It has been more than 10 years now that my dissertation was published, in which I presented a computational method that was supposed to detect cognates in comparative wordlists, based on an automated analysis of potential sound correspondences (List 2014). Now, ten years later, I know much more about the problem of cognate detection in specific and etymological analysis in general, but I still do not feel that the problem can be considered as solved. However, despite the slow progress in my personal and my colleagues’ attempts to automatize the comparative method, I see the general _workflow_ that one should follow in order to carry out a _formal_ etymological analysis, much clearer now. For this reason, I thought it is time to share my views on this workflow, in order to also provide a detailed account on those parts, where we still lack full-fledged automated approaches, while at the same time pointing to those aspects that might become relevant in the future. Some of these problems have already been mentioned as part of an essay published two years ago (List 2024a), in which I discussed several _open problems_ in the field of comparative linguistics and historical language comparison. The current attempt to summarize the major workflows involved in _formal etymological analysis_ can be seen as an attempt to follow up on this work, by contrasting the solutions we have with the solutions we need. Since the workflow for etymological analysis involves several steps that consist themselves of at times complex sub-tasks, I will restrict myself here to presenting the general workflow for etymological analysis. In later studies, I may zoom in to address particular problems in more detail. ## 2 Background The central workflow of the comparative method has been discussed in many studies. A very influential workflow can be found in the study of Ross (1996) (pages 6f). Inspired by this workflow, I have proposed a five-step approach (see Figure 1), starting from the (1) proof of relationship, followed by the (2) identification of cognates, and the (3) identification of sound correspondences, and cumulating in the (4) reconstruction of proto-forms and (5) an internal classification of the language family in question (List 2014, 58). Figure 1: Workflow of the traditional comparative method based on Ross and Durie (1996) taken from List (2014). It was based on this workflow that I later tried to automatize central parts of the comparative methods. A first step in this direction was my work on phonetic alignment and cognate detection (List 2014), which can be assigned to step (2). My work on correspondence patterns (List 2019) can be seen as central for step (3). We have used this work to build initial methods for the supervised reconstruction of proto-forms and for the prediction of reflex forms (Bodt and List 2022). While these methods are all fully automated, I have also tried to establish formal tools that allow to conduct these stages of the workflow manually, but in a way that would account for a formal – computer-readable – treatment of the individual steps when applying them to a standardized dataset. Here, most ideas went into the EDICTOR tool (https://edictor.org, List 2017), that has since then been constantly enhanced (List and Dam 2024, List et al. 2025). ## 3 Workflow When working on fully automated workflows for different aspects of the comparative method, I soon realized that none of the methods would actually yield the result that I had originally hoped for. Instead of solving a problem once and for all time, each solution would point to additional problems. This has not changed even today, even if I think that we have made great progress in the retrospective. Right now, however, I would revise the workflow that I had proposed back in 2014 and expect much smaller steps to be carried out – often also in an iterative fashion. 1. initial data preparation -> convert data to standard phonetic transcriptions 2. internal cognate analysis -> segment words into morphemes, carry out internal reconstruction and allomorphic and allophonic analysis, identify language-internal cognates, gloss individual morphemes for their particular meaning 3. external cognate analysis -> identify cognate morphemes, align them, identify potential exceptions 4. external correspondence analysis -> identify correspondence patterns, refine them, mark exceptions 5. linguistic reconstruction -> assign proto-forms to correspondence patterns 6. sound law induction -> determine the sound laws that lead from a proto-form to the individual language varieties 7. phylogenetic reconstruction -> determine major subgroupings of the languages Most of these steps have been discussed already by me in past studies, but I figured I have so far never found the time to clarify in a much more explicit manner, how I think that these different approaches could or should be best combined to form a full-fledged workflow for historical language comparison. We can arrange the seven steps in the manner shown in Figure 2. Figure 2: Revised workflow for historical language comparison. This workflow keeps the major division into two phases of the workflow that I proposed earlier, with one phase being devoted to the identification of cognates and correspondences and one being devoted to the identification of proto-forms and subgroupings. However, both steps are now extended by another step. The identification of cognate sets and correspondence patterns is now extended by what one could best call _internal reconstruction_. The phonological and phylogenetic reconstruction stage is no accompanied by a detailed search for sound laws. All three subtasks must be solved – as before – in an iterative fashion during which researchers or algorithms jump back and forth through different representations of the original data. ## 4 Workflow Examples In the following, I will try to provide more detailed examples on this revised workflow, pointing both to software solutions and to interface and modeling solutions that can be used to address the individual tasks constituting the two major phases of historical language comparison. Before we discuss the phases of cognate and correspondence detection and phonological and phylogenetic reconstruction, however, I will briefly try point to the first stage of data preparation, which is crucial for any kind of linguistic analysis, but all to often, unfortunately, treated with much less care, both by researchers and by computational solutions. ### 4.1 Data Preparation Data preparation constitutes a crucial but also frequently overlooked aspect of formal etymological analysis. Thus, in order to allow for an automated handling of word forms as _sound sequences_ we must insist that sounds are represented in standardized phonetic transcriptions in which each sound is clearly defined. Transcription systems, however, are rarely clearly defined and rely on ad-hoc extensions by scholars who feel that the transcription lacks the sounds they want to transcribe or who simply ignore the rules. Since rules are never explicitly imposed upon transcriptions and also rarely checked, this has led to a considerable diversity of transcription practices and also implementations, as can be seen from the variation underlying cross-linguistic databases and data collections (Anderson et al. 2018, 2023). With the _Cross-Linguistic Data Formats_ initiative (CLDF, https://cldf.clld.org), we have tried to lay the foundation of standardized datasets in cross-linguistic applications (Forkel et al. 2018). When developing the recommendations that were later adapted in individual CLDF versions, we relied on practical experience resulting from concrete attempts to handle cross-linguistic data both manually – trying to compile new datasets – and automatically – trying to analyze existing datasets. These recommendations, however, were not necessarily developed with CLDF as endpoint in mind, but rather reflect practical requirements for cross-linguistic data that we observed in practice. Based on these observations, we would now require that a common cross-linguistic dataset, compiled for etymological analysis, has the form of a comparative _wordlist_ provided in _long table_ format. This means, that we work with at least one table that stores the data, with each row reflecting an individual _word form_ (Form) in a individual _language variety_ (Language) that represents the translation of an individual _concept_ (Concept). The word form itself can be represented in three different ways. In the form of the original _value_ (Value) that one may observe in a dictionary (containing additional non-standard information, like brackets, but also mixed entries in which several forms are provided, separated by slashes or decorated with additional ad-hoc annotations), in the form of the actual _form_ (Form) that one wants to use as the basis of an analysis, and in _segmented form_ (Segments), fully transcribed, with individual sounds being segmented by spaces and morpheme boundaries (including word boundaries) inside the form being marked by a plus symbol (+). While the form may contain idiosyncratic transcriptions, the segmented form in CLDF _must_ account for standard transcription systems, including the B(road) IPA defined by the catalog of Cross-Linguistic Transcription Systems (https://clts.clld.org, Anderson et al. 2018). This final step of analysis, the actual _segmentation_ thus not only segments a given form of idiosyncratic transcriptions into individual speech sounds, it also standardizes speech sounds. This step of converting forms as they are usually provided in the literature into standardized segmented representations is usually done in a semi-automated fashion with the help of _Orthography Profiles_ (Moran and Cysouw 2018, see also Forkel and List 2024), but could in theory also be done manually. Depending on the language family in question, the segmentation of word forms into speech sounds is not enough, and additional segmentations of sound sequences into sequences representing individual meaning-bearing units – morphemes – are required. While this step might fall inside the stage of data preparation, practical experience shows that it is often much easier to combine this step with the _internal cognate analysis_ that will be discussed in more detail in the following section. ### 4.2 Cognate and Correspondence Detection Starting with Swadesh’s work on lexicostatistics (Swadesh 1952, 1955), the compilation of cognate-annotated comparative wordlists has become a common practice in historical linguistics. In a comparative wordlist, words from different languages are arranged according to the concepts they express and later compared for cognacy. An alternative to this annotation practice consists in the collection of etymologically related words regardless of the meanings they express. The most common format for such collections is the _etymological dictionary_ , which adds hypothetical proto-forms to all previously identified cognate sets. Both representations have advantages and disadvantages. Etymological dictionaries suffer from the fact that they only pick _positive evidence_ for genetic language relations. Meanings of individual word forms – _reflexes_ of proto-forms in the classical terminology – are usually insufficiently reported and barely standardized, often, scholars even strip off suffixes and prefixes from the reflex forms, making it very difficult to verify the origins of the entries that made it into the etymological dictionary. Comparative wordlists have the disadvantage of missing out on details. Semantic shift is a common phenomenon, but rarely reported in comparative wordlists. The annotation of cognate sets that goes beyond the concept slots that constitute their organisational basis can be tedious in practice. Furthermore, due to the original selection of a limited _number_ of basic concepts, serving as the main comparanda, many potentially interesting cognate sets are deliberately ignored. While I understand the arguments of both sides, I see much better chances in enhancing the annotation of comparative wordlists in order to make up for their shortcomings than in trying to address the shortcomings of etymological dictionaries. As a result, the basic format that I consider as relevant for etymological analysis is always the comparative wordlist that consists of a list of concepts that are in turn translated into a list of languages. In order to identify cognate sets and sound correspondences in such wordlists, three distinct analysis steps are important, which are deeply intertwined with each other. The first step consists in the detailed annotation of individual word forms in individual language varieties, with the goal of segmenting the word forms into individual morphemes, thereby identifying allophones and allomorphs and carrying out what is known as an _internal reconstruction_ of the individual language varieties. The second step consists in the identification of cognate morphemes, which have to be aligned with each other after initial identification. This step is pretty well explored, but quite a few problems remain, especially when carrying out the second step without paying attention to the first step of language-internal analysis. The third step consists in the identification of correspondence patterns. Here, one must cluster all identical _alignment sites_ that have been identified in step 1 and 2 of the cognate and correspondence detection workflow into partitions that show no conflicts with each other. Conflicts are those cases where one language shows two reflexes for the same _correspondence pattern_. Step 1 is typically ignored, but it is of crucial importance, since internal reconstruction decreases language-internal variation (Fox 1995, pp. 211-213), and can thus be seen as an important pre-requisite for any _external_ comparison of word forms. Thus, when comparing word forms in Germanic languages, it would not make much sense to use two sounds `[`ç`]` and `[`x`]` to represent the grapheme `<`ch`>` in German, since both pronunciations can be seen als allophones of the same phoneme. Retaining their phonetic pronunciations without prior language-internal analysis would only increase the variation when trying to identify sound correspondences across Germanic languages. The same holds for allomorphic variation that can be traced to language-internal processes. Language-internal variation must be reduced before comparing one language with another, in order to reduce the complexity of the external comparison. Another important aspect of language-internal comparison, that is often disregarded, consists in the identification of language-internal cognates or _word families_. Since our basic workflow does not start from previously identified cognate sets, but rather from a comparative wordlist, in which word forms are selected due to the meanings they express, we may encounter many cases in which a morpheme _recurs_ across many word forms. For this reason, a proper internal analysis should not only identify allophones, allomorphs, and morpheme boundaries, but also tag those cases where morphemes recur across word forms, making sure that morphemes are only compared once _across_ languages. The second step of the workflow, the actual identification of cognate morphemes through external language comparison is pretty well explored, with many attempts of automation and formalization. For me, the essential part of this stage consists in three individual tasks, the actual _partitioning_ of morphemes into a first set of _partial cognate sets_ (see List et al. 2016 for details on this notion), the identification of cognates recurring _across meaning slots_ in the wordlist (in Wu et al. 2020, we have called this _cross-semantic cognate detection_ at times, but I am hesitant of using the term nowadays), and the consecutive _multiple phonetic alignment_ of these morphemes (see List 2014 on multiple phonetic alignment). In order to avoid stretching the size of this study too much, I won’t comment too much on both steps, but emphasize that I consider them as actually solved for those cases where high-quality comparative wordlists with morpheme-segmented word forms resulting from a thorough internal comparison are provided. Tools like EDICTOR (List et al. 2025; List and van Dam 2024) facilitate this step greatly. Once morphemes have been assigned to individual cognate sets and consecutively aligned morphologically, we arrive at the third step of the workflow, which requires us to partition the _alignment sites_ , that is, the _columns_ of all _alignments_ reflecting all cognate sets, into _correspondence patterns_. While individual alignments reflect true correspondences, provided one trusts that the alignments have been carried out properly, thee alignments do not reflect _patterns_ , given that they are but _one_ instance of a correspondence. In order to count as a _pattern_ , a larger number of alignment sites is needed, that must confirm the recurrence of the correspondences observed. Although they play a crucial role in the traditional literature, correspondence patterns have been ignored for a long time in computational approaches. In 2019, I presented a first workflow that helps to derive correspondence patterns from aligned cognates automatically (List 2019). While this workflow was an eye-opener for myself, helping us to address several additional tasks, such as _supervised phonological reconstruction_ (List et al. 2022) and _reflex prediction_ (Bodt and List 2022), it has not received much attention beyond the work of my own group. The reason may have been that the workflow involves larger amounts of data that are difficult to be handled manually, thus making the actual verification of findings in the interaction with classical linguistics rather difficult. With EDICTOR 3, new functionalities have been made available that help scholars to carry out the annotation of correspondence patterns in a computer-readable but manual manner (List and van Dam 2024). Unfortunately, we lack clear-cut examples in which actual data have been analyzed from scratch. In practice, it has also turned out to very difficult to account for the different layers of annotation that a full-fledged etymological analysis requires. While for me it is pretty clear that a very detailed annotation is needed that accounts for the three major aspects and multiple smaller problems involved in all individual steps, we currently lack the full-fledged examples where this annotation has been applied to a larger dataset. I hope to be able to provide such examples in the future. Without such examples, I fear, it will be very difficult to enhance both the interactive annotation tools and the algorithms that could help to automatize particular tasks. ### 4.3 Phonological and Phylogenetic Reconstruction The third and in the present workflow final phase consists again of three steps that are deeply intertwined with each other. While it may be clear that the first step of linguistic (or phonological) reconstruction and the second step of sound law induction should go hand in hand, I see a similar need to integrate the final step of phylogenetic reconstruction with the previous steps, given that phylogenies play a crucial role in informing the reconstruction of proto-sounds and the induction of sound laws. The first step, linguistic (or more precisely phonological) reconstruction can be seen as an additional _clustering_ or _partitioning_ operation applied this time to the _correspondence patterns_. Correspondence patterns show – in the definition of the term that I have used in my own work – for each language only _one_ reflex sound. This means that conflicts in inferred correspondence patterns must be flagged as _exceptions_. Exceptions however, could in theory be resolved as a later stage, especially when they show a high degree of systematicity, as we know from Grimm’s studies (Grimm 1822) that were later resolved – among others – by Verner (1877). While exceptions in correspondence patterns can be considered a specific case, we can also easily think of correspondence patterns that differ strong enough so that we would not merge them (and accepting multiple reflex sounds for the same language variety). This does not, however, mean, that these patterns must reflect _different proto-sounds_. We know very well that conditioning context can easily trigger different reflex sounds in sound change. As a result, two or more distinct correspondence patterns can often be expected to correspond to the same sound in the proto-language. Carrying out a phonological reconstruction analysis thus has two consequences. First, we assign each correspondence pattern in our data to a certain proto-sound of which we assume that it was present in the proto-language. Second, by assigning identical proto-sounds to different correspondence patterns, we set the stage for the identification of conditioning context that explains the differences in the reflexes of individual language varieties. Formally, we can thus say that we cluster or partition correspondence patterns into proto-sounds. Once this step has been carried out, we are left with distinct proto-forms for each aligned cognate sets in our data. We must now identify the conditioning contexts that explain the split of a proto-sound into two or more reflex sounds in the same language variety. I call this step _sound law induction_ (see also List 2024b), since sound laws provide a concrete explanation for the change of a sound based on conditioning context. The basic task of the second step in phonological and phylogenetic reconstruction thus consists in the identifification of those contexts (or more broadly speaking those _sound laws_) that explain why the same proto-form splits into different reflexes. Since splits and mergers (mergers are less relevant for us in this context, since they have no consequences on the reconstruction practice) can occur at any stage during language evolution, it would be wrong to assign their occurrence only to the individual development of a language variaty after having split off its subgroup. Instead, we must assume that splits and mergers can happen throughout the whole evolution of a language family, on each internal node that a phylogenetic reconstruction might propose. As a result, it is crucial to substantiate any findings regarding the proto-phonology and the sound laws that constitute a language family by providing a detailed phylogenetic reconstruction that explains how the proto-language split into branches and individual language varieties over time. While I have initial ideas of how phonological reconstruction can be carried out formally, based on previously identified corresponcence patterns (see List et al. 2022), as well as an initial idea of modeling sound laws computationally (List 2024b), I have neither concrete ideas for the _induction_ of sound laws from comparative data, nor for the consistent integration of phylogenetic reconstruction with the other two steps of the workflow. I am, however, convinced, that a successfully implemented model of etymological analysis needs to account for all three steps in this stage of the workflows. Future research will show how well these ideas aspects can be integrated with each other. ## 5 Outlook This little study has not provided any answers to currently open questions in historical language comparison. What I wanted to provide instead was an overview on the detailed workflow of the comparative method that I consider as fruitful to pursue in the future. I am sure that this overview is by no means the last word on the matter. On the contrary, I expect that the workflow will be enriched by more details with time. It is also quite possible that different language families will require different treatments of certain aspects. In any case, I have the hope that future studies will not only help to improve the current state of the art with respect to the conceptualization of the workflow, but also with respect to the implementation of interactive annotation tools and automated workflows that help to make human annotation more efficient. ## References Anderson, Cormac, Tiago Tresoldi, Thiago Costa Chacon, Anne-Maria Fehn, Mary Walworth, Robert Forkel, and Johann-Mattis List. 2018. “A Cross-Linguistic Database of Phonetic Transcription Systems.” _Yearbook of the Poznań Linguistic Meeting_ 4 (1): 21–53. https://doi.org/10.2478/yplm-2018-0002. Anderson, Cormac, Tiago Tresoldi, Simon J. Greenhill, Robert Forkel, Russell D. Gray, and Johann-Mattis List. 2023. “Variation in Phoneme Inventories: Quantifying the Problem and Improving Comparability.” _Journal of Language Evolution_ 8 (2): 149–68. https://doi.org/10.1093/jole/lzad011. Bodt, Timotheus Adrianus, and Johann-Mattis List. 2022. “Reflex Prediction. A Case Study of Western Kho-Bwa.” _Diachronica_ 39 (1): 1–38. https://doi.org/10.1075/dia.20009.bod. Forkel, Robert, and Johann-Mattis List. 2024. “A New Python Library for the Manipulation and Annotation of Linguistic Sequences.” _Computer-Assisted Language Comparison in Practice_ 7 (1): 17–23. https://doi.org/10.15475/calcip.2024.1.3. Forkel, Robert, Johann-Mattis List, Simon J. Greenhill, Christoph Rzymski, Sebastian Bank, Michael Cysouw, Harald Hammarström, Martin Haspelmath, Gereon A. Kaiping, and Russell D. Gray. 2018. “Cross-Linguistic Data Formats, Advancing Data Sharing and Re-Use in Comparative Linguistics.” _Scientific Data_ 5 (180205): 1–10. https://doi.org/10.1038/sdata.2018.205. Grimm, Jacob. 1822. _Deutsche Grammatik_. 2nd ed. Vol. 1. Göttingen: Dieterichsche Buchhandlung. List, Johann-Mattis. 2014. _Sequence Comparison in Historical Linguistics_. Düsseldorf: Düsseldorf University Press. https://doi.org/10.1515/9783110720082. ———. 2017. “A Web-Based Interactive Tool for Creating, Inspecting, Editing, and Publishing Etymological Datasets.” In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics. System Demonstrations_ , 9–12. Valencia: Association for Computational Linguistics. https://edictor.org. ———. 2019. “Automatic Inference of Sound Correspondence Patterns Across Multiple Languages.” _Computational Linguistics_ 45 (1): 137–61. https://doi.org/10.1162/coli_a_00344. ———. 2024a. “Modeling Sound Change with Ordered Layers of Simultaneous Sound Laws.” [Preprint, not peer-reviewed]._Humanities Commons_. https://doi.org/10.17613/4n5z-9y52. ———. 2024b. “Open Problems in Computational Historical Linguistics [Version 2; Peer Review: 5 Approved].” _Open Research Europe_ 3 (201): 1–27. https://doi.org/10.12688/openreseurope.16804.2. List, Johann-Mattis, and Kellen Parker van Dam. 2024. “Computer-Assisted Language Comparison with EDICTOR 3 [Invited Paper].” In _Proceedings of the 5th Workshop on Computational Approaches to Historical Language Change_ , edited by Nina Tahmasebi, Syrielle Montariol, Andrey Kutuzov, David Alfter, Francesco Periti, Pierluigi Cassotti, and Netta Huebscher, 1–11. Bangkok, Thailand: Association for Computational Linguistics. https://aclanthology.org/2024.lchange-1.1. List, Johann-Mattis, Nathan W. Hill, and Robert Forkel. 2022. “A New Framework for Fast Automated Phonological Reconstruction Using Trimmed Alignments and Sound Correspondence Patterns.” In _Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change_ , 89–96. Dublin: Association for Computational Linguistics. https://aclanthology.org/2022.lchange-1.9. List, Johann-Mattis, Philippe Lopez, and Eric Bapteste. 2016. “Using Sequence Similarity Networks to Identify Partial Cognates in Multilingual Wordlists.” In _Proceedings of the Association of Computational Linguistics 2016 (Volume 2: Short Papers)_ , 599–605. Berlin: Association of Computational Linguistics. https://anthology.aclweb.org/P16-2097. List, Johann-Mattis, Kellen Parker van Dam, and Frederic Blum. 2025. _EDICTOR 3. An Interactive Tool for Computer-Assisted Language Comparison [Software Tool, Version 3.1]_. Passau: MCL Chair at the University of Passau. https://edictor.org. Moran, Steven, and Michael Cysouw. 2018. _The Unicode Cookbook for Linguists: Managing Writing Systems Using Orthography Profiles_. Berlin: Language Science Press. https://langsci-press.org/catalog/book/176. Ross, Malcolm D. and Durie, Mark. 1996. “Introduction.” In _The Comparative Method Reviewed: Regularity and Irregularity in Language Change_ , edited by Mark Durie, 3-38. New York: Oxford University Press. Swadesh, Morris. 1952. “Lexico-Statistic Dating of Prehistoric Ethnic Contacts: With Special Reference to North American Indians and Eskimos.” _Proceedings of the American Philosophical Society_ 96 (4): 452–63. ———. 1955. “Towards Greater Accuracy in Lexicostatistic Dating.” _International Journal of American Linguistics_ 21 (2): 121–37. https://www.jstor.org/stable/1263939. Verner, Karl A. 1877. “Eine Ausnahme Der Ersten Lautverschiebung.” _Zeitschrift Für Vergleichende Sprachforschung Auf Dem Gebiete Der Indogermanischen Sprachen_ 23 (2): 97–130. Wu, Mei-Shin, Nathanael E. Schweikhard, Timotheus A. Bodt, Nathan W. Hill, and Johann-Mattis List. 2020. “Computer-Assisted Language Comparison. State of the Art.” _Journal of Open Humanities Data_ 6 (2): 1–14. https://doi.org/10.5334/johd.12. **Cite this article as:** List, Johann-Mattis (2026): “Foundations of Formal Etymological Analysis” in _Computer-Assisted Language Comparison in Practice_ , 9.1: 19-30 [first published on 25/03/2026], URL: https://calc.hypotheses.org/9208, DOI: 10.15475/calcip.2026.1.3. **Download the article as PDF:** calcip-09-1-3.pdf **Copyright information** : This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. **Funding Information** : This project has received funding from the European Research Council (ERC) under the European Union’s Horizon Europe research and innovation programme (Grant agreement No. 101044282). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. * * * The text only may be used under licence Creative Commons Attribution 4.0 International. All other elements (illustrations, imported files) are “All rights reserved”, unless otherwise stated. * * * OpenEdition suggests that you cite this post as follows: Johann-Mattis List (March 25, 2026). Foundations of Formal Etymological Analysis. _Computer-Assisted Language Comparison in Practice_. Retrieved March 26, 2026 from https://doi.org/10.58079/15xvf * * * * * * * *

Multilingual lexical data is typically stored in a wide variety of forms, based on many idiosyncratic decisions that vary from dataset to dataset. Here, a simple but efficient solution for the manipulation of lexical data in multilingual wordlists will be introduced. This solution, the PyLexibank FormSpec, was originall developed for the conversion of various kinds of lexical data to Cross-Linguistic Data Formats, but it can also be used as a standalone. This study offers a basic tutorial that illustrates how the FormSpec can be put to concrete use. ## 1 Introduction When working with lexical data in multilingual settings, one encounters a huge variety of ways in which similar kinds of information about lexical forms are encoded. While handling such cases may often require targeted solutions that may in the worst case consist in manual corrections of individual data points, our work with the Lexibank repository (List et al. 2022) has also allowed us to detect certain recurring idiosyncracies in lexical data that can be handled with unified approaches. These approaches have been integrated into the PyLexibank software package (Forkel et al. 2021) and provide important help in converting various kinds of cross-linguistic data with lexical forms to Cross-Linguistic Data Formats (Forkel et al. 2018). In this small study, I will introduce one particular solution that deals with lexical forms before their conversion to phonetic transcriptions. This solutions, as simple as it may seem, has helped us a lot in constructing the Lexibank repository that by now aggregates standardized data from more than 120 different datasets (Blum et al. 2025). ## 2 Background While the basic information that scholars want to provide in a dictionary or a wordlist can be described in pretty simple and straightforward terms, the techniques that scholars use in order to _mark_ this kind of information in concrete datasets vary greatly. While the major information that we need to provide when listing word forms in a given language consists in a triple of _language_ , _form_ , and _meaning_ (List 2014; Gévaudan 2007), linguistic practice adds various forms of complexity and inconsistency to this triple structure. Language names are rarely used in a standardized form, making it at times difficult to identify the varieties in question. Short glosses used to represent meanings are often highly idiosyncratic and can at times only be understood from the larger context of the concept list in which they are assembled (List, Cysouw, and Forkel 2016). Forms are given in a mix of transcriptions, orthographic information, often expanded by additional information that can often only be understood when taking detailed contextual information into account. As an example, consider cases where data are given in tabular form, where columns represent languages and rows represent meanings, and word forms are placed into the respective cells. This format can be found in numerous publications and is considered some kind of a standard among many linguists. The problem of the format is, that it invites inconsistencies regarding the representation of the lexical forms. These inconsistencies surface in numerous occasions. Thus, if more than one word is found to express a given meaning in a given language, scholars use various ways to code for this, using characters like comman, semicolon, or slash as a delimiter when listing multipel word forms, with many datasets using different delimiters without any clear semantics attached to them. Another problem consists in the use of brackets, which are also used in multiple variants, ranging from square brackets over normal brackets to curly braces. Here again, semantics of bracket use are rarely consistent, ranging from reading variants over pronunciation differences to metainformation that relates to the language or the concept in question rather than to the word form itself. An additional problem consists in the explicit marking of missing data, which varies also greatly, ranging from empty cells over dashes invarious forms to explicit entries, such as “no data” or “missing entry”. While inconsistencies may seem to be unproblematic when inspecting data directly by eyeballing them, they may cause huge problems when trying to digest data with the computer. If missing data is marked by an entry “no entry”, for example, it can easily occur that this entry will make its way into the final database, leading to the false impression that the word for “apple” in some language variety is “no entry” instead of being simply missing from the record. The problems arising from variation in lexical entries in cross-linguistic datasets may not only sound funny but also evitable. One would expect computationally versed people to be able to spot or predict such problems when trying to convert a dataset to some standardized format. When dealing with idiosyncrasies of individual data, however, it is helpful to make use of some standardized routines that help to solve problems that often recur across different datasets in a unified way. ## 3 Getting Started with the PyLexibank FormSpec When developing the framework that would later be used to feed the Lexibank repository with data (Blum et al. 2025; List et al. 2022), we started out with individual solutions to deal with inconsistencies in lexical entries. Lexical entries were thus dealt with on a case-to-case basis, using standard routines for text manipulation offered by Python. When adding more data, however, we began to realize that certain problems with lexical forms would recur with a certain regularity. Entries for missing data would be marked idiosyncratically, multiple forms within the same cell would be separated with different separation symbols, and brackets would force us to apply at times quite complex regular expressions. In order to address these problems, a new functionality to handle lexical forms flexibly in a unified way was added to PyLexibank (Forkel et al. 2021), the library that we used to convert data that we would obtain in raw form from published resources into Cross-Linguistic Data Formats (Forkel et al. 2018). This _FormSpec_ , as it is called in PyLexibank, addresses the three major problems summarized above. It deals with brackets (preferably removing everything that is inside a bracket, given that both additional morphemes and metainformation can both not be reliably interpreted when standardizing a form entry). It deals with separators used to describe several variants within the same cell of a data entry. Finally, it also deals with missing data, allowing users to provide a list of the symbol combinations used to indicate that a cell contains _no_ data. Additionally, the FormSpec provides some basic _cleaning operations_ of lexical forms, stripping certain characters from the form and applying standard Unicode normalization (Moran and Cysouw 2018: 17). While the _FormSpec_ is automatically applied whenever you use CLDFBench (Forkel and List 2020) and PyLexibank to create a CLDF dataset, you can also test its functionality directly in an interactive Python session. In order to get started, all you need is a fresh installation of the PyLexibank package, which you can easily obtain with the help of the Python package index `pip`. $ pip install pylexibank Equipped in this form, all you need to load the _FormSpec_ is to import it from your interactive Python session or from within a Python script. from pylexibank import FormSpec In order to _use_ the _FormSpec_ , you must _initialize_ it first. This means, you predefine its behavior in cleaning a given lexical form. The call signature of the class is as shown below. class FormSpec(builtins.object) | FormSpec( | brackets={'(': ')'}, | separators=(';', '/', ','), | missing_data=('?', '-'), | strip_inside_brackets=True, | replacements=NOTHING, | first_form_only=False, | normalize_whitespace=True, | normalize_unicode=None | ) -> None We define pairs of brackets by means of a dictionary in which the key is the opening bracket and the value is the closing bracket. This would not work with cases where a bracket is defined by the same start and end symbol, but our experience shows that most datasets would use traditional brackets for which start and end symbols are defined. The separators handle multiple forms for the same concept. Missing data are passed as a list (or more strictly speaking, a tuple, according to the call, but a list will also be accepted). If the option `strip_inside_brackets` is set to `True`, this means that the algorithm deletes content inside brackets. With respect to the order of execution, note that in cases where a separator, used as a separator of multiple word forms, is also passed inside a bracket, the algorithm would not split the text at this point, but first identify the brackets in the text and then apply the segmentation operation. The option `first_form_only` will yield only the first form of multiple potential forms, when set to `True`. Normalization can be done with respect to whitespace (deleting and unifying whitespace) and Unicode (where one would have to choose between `NFD` and `NFC`). The option `replacements` allows to define a list consisting of tuples of source-target strings, where the source string is what will be replaced and the target string is the replacement. Having initialized the _FormSpec_ by calling the class with particular parameters, one can use it by calling its `split`-method with two arguments, the first argument being always `None` when using it outside the context of CLDFBench, while the second argument is the string one wants to manipulate. This is illustrated in the following example. >>> fs = FormSpec() >>> for form in fs.split(None, "this, is; a (form)"): ... print(form) this is a ## 4 Usage Examples When dealing with the _FormSpec_ , it is important to be aware about the order by which actions are carried out when using the functionality. In the following, we will go through some examples that illustrate basic use-cases. We start with the handling of brackets, which are – as I mentioned before – defined as a dictionary (opening bracket as a key, closing bracket as the value). This allows us to define all kinds of potentially strange brackets that could occur in one’s data. >>> fs = FormSpec(brackets={"<": ">", "{": "}"}) >>> fs.split(None, "this <really?>, is, an {example}") ['this', 'is', 'an'] The `missing_data` argument allows you to specify any string that could occur as missing data. The _FormSpec_ generally assumes that whitespace to the left or the right of the string will be stripped. >>> fs = FormSpec(missing_data=("???", "?")) >>> fs.split(None, "???, really,?, ") ['really'] For separators, there is a particular restriction that only single-character strings can be used as a separator. Thus, passing a string of more than one character will throw an error. Depending on the data, however, one can find workarounds that would nevertheless allow us to separate even strings where multiple characters have been used as a separator. As an example, consider the following output, where three slashes have been used as a separator. >>> fs = FormSpec(separators=(',', ';', '/')) >>> fs.split(None, "hallo /// welt / hier / bin //// ich") ['hallo', 'welt', 'hier', 'bin', 'ich'] According to the way in which _FormSpec_ works, the internal splitting process will only return those forms that consist of at least one character that is not a whitespace character. The _FormSpec_ splits the string in the example into 10 different forms, but only five are returned, since they are note empty. If you want to use the option to replace strings by other strings during the form conversion with the _FormSpec_ , it is important to keep in mind that the replacement is carried out _after_ all splitting operations have been carried out. This limits the possibilities of application, on the one hand, but it also reduces complexity, since the replacements are quite restricted and they do not interfere with the process of splitting a string into several forms. As an example, consider the following lines, where the replacement of the string `/x/` to the string `/` is not carried out, given that `/` is also defined as a character that triggers the string to be split into parts. >>> fs = FormSpec(separators=(',', ';', '/'), replacements=[('/x/', '/')]) >>> fs.split(None, "hallo /// welt / hier / bin /x/ ich") ['hallo', 'welt', 'hier', 'bin', 'x', 'ich'] ## 5 Outlook Although the _FormSpec_ is based on a limited number of options, the functionality has proven very useful in practice, especially when populating the Lexibank repository (Blum et al. 2025). It seems that the decision to limit the scope of the method to a very dedicated range of options, deciding, among others, against the possibility to apply regular expressions, was helpful, given that the results triggered by the current _FormSpec_ can still be easily understood when considering input and output strings. With more complex operations, we would quickly loose the possibility to trace individual decisions made in the code we used to convert raw data into standardized CLDF data points. ## References Blum, Frederic, Carlos Barrientos, Johannes Englisch, Robert Forkel, Simon J. Greenhill, Christoph Rzymski, and Johann-Mattis List. 2025. “Lexibank 2: Pre-Computed Features for Large-Scale Lexical Data [version 2; peer review: 3 approved].” _Open Research Europe_ 5 (126): 1–24. https://doi.org/https://doi.org/10.12688/openreseurope.20216.2. Forkel, Robert, Simon J Greenhill, Hans-Jörg Bibiko, Christoph Rzymski, Tiago Tresoldi, and Johann-Mattis List. 2021. _PyLexibank. The Python Curation Library for Lexibank [Software Library, Version 2.8.2]_. Geneva: Zenodo. https://doi.org/10.5281/zenodo.2630582. Forkel, Robert, and Johann-Mattis List. 2020. “CLDFBench. Give Your Cross-Linguistic Data a Lift.” In _Proceedings of the Twelfth International Conference on Language Resources and Evaluation_ , 6997–7004. Luxembourg: European Language Resources Association (ELRA). http://www.lrec-conf.org/proceedings/lrec2020/pdf/2020.lrec-1.864.pdf. Forkel, Robert, Johann-Mattis List, Simon J. Greenhill, Christoph Rzymski, Sebastian Bank, Michael Cysouw, Harald Hammarström, Martin Haspelmath, Gereon A. Kaiping, and Russell D. Gray. 2018. “Cross-Linguistic Data Formats, Advancing Data Sharing and Re-Use in Comparative Linguistics.” _Scientific Data_ 5 (180205): 1–10. https://doi.org/10.1038/sdata.2018.205. Gévaudan, Paul. 2007. _Typologie Des Lexikalischen Wandels: Bedeutungswandel, Wortbildung Und Entlehnung Am Beispiel Der Romanischen Sprachen_. Tübingen: Stauffenburg. List, Johann-Mattis. 2014. _Sequence Comparison in Historical Linguistics_. Düsseldorf: Düsseldorf University Press. https://doi.org/10.1515/9783110720082. List, Johann-Mattis, Michael Cysouw, and Robert Forkel. 2016. “Concepticon. A Resource for the Linking of Concept Lists.” In _Proceedings of the Tenth International Conference on Language Resources and Evaluation_ , edited by Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, 2393–2400. Luxembourg: European Language Resources Association (ELRA). https://aclanthology.org/L16-1379/. List, Johann-Mattis, Robert Forkel, Simon J. Greenhill, Christoph Rzymski, Johannes Englisch, and Russell D. Gray. 2022. “Lexibank, a Public Repository of Standardized Wordlists with Computed Phonological and Lexical Features.” _Scientific Data_ 9 (316): 1–31. https://doi.org/10.1038/s41597-022-01432-0. Moran, Steven, and Michael Cysouw. 2018. _The Unicode Cookbook for Linguists: Managing Writing Systems Using Orthography Profiles_. Berlin: Language Science Press. https://langsci-press.org/catalog/book/176. **Cite this article as:** List, Johann-Mattis (2025): “Manipulating Lexical Forms with the PyLexibank FormSpec” in _Computer-Assisted Language Comparison in Practice_ , 8.2: 87-93 [first published on 28/10/2025], URL: https://calc.hypotheses.org/8877, DOI: 10.15475/calcip.2025.2.4. **Download the article as PDF:** calcip-08-2-4.pdf **Copyright information** : This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. **Funding Information** : This project has received funding from the European Research Council (ERC) under the European Union’s Horizon Europe research and innovation programme (Grant agreement No. 101044282). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. * * * OpenEdition suggests that you cite this post as follows: Johann-Mattis List (October 27, 2025). Manipulating Lexical Forms with the PyLexibank FormSpec. _Computer-Assisted Language Comparison in Practice_. Retrieved October 27, 2025 from https://calc.hypotheses.org/8877 * * * * * * * *

This study illustrates how semantic embeddings can be added to and retrieved from NoRaRe. By that, it provides a template for handling vector data and makes popular methodology in semantic modeling available for cross-linguistic comparison. # 1 Introduction The success of word embedding techniques has lead to a complete shift to a distributional framework in computational semantics, which is underlined by the fact that word embeddings build the backbone of modern Large Language Models. Following the distributional hypothesis, vector representations for words (i.e., embeddings) are learned from surrounding words. Words with similar meanings and/or functions will therefore appear in similar contexts, leading to similiar representations, while words that are found in different contexts are considered dissimilar and thus appear distant from each other in the embedded vector space. Since they are demonstrably powerful, yet readily interpretable, static word embeddings (meaning that one word maps to one vector representation, in contrast to contextual word embeddings) trained with models like Word2Vec (Mikolov et al. 2013), GloVe (Pennington et al. 2014), and FastText (Bojanowski et al. 2017) remain a popular choice for the computational investigation of semantic similarity and relations between word forms. While certainly powerful and highly expressive, word embeddings cannot be directly plugged into models for computer-assisted language comparison. Research in comparative linguistics relies on the presence of comparative concepts — items under the same gloss in different languages should refer to the same sense or concept (even if the gloss in the metalanguage is potentially ambiguous). For example, the English word _bark_ can refer to the exterior part of a tree or the vocalization of a dog. The word embedding for _bark_ captures both meanings — simply using that embedding for a wordlist where _bark_ only refers to the part of the tree would thus lead to an imprecise representation of the expressed concept due to the homonymy in the English gloss. To address this problem, we recently presented a new technique for training language-agnostic _concept embeddings_ from cross-lingual colexification networks (Rubehn and List 2025). In this post, I will describe how those concept embeddings were integrated into the Database of Norms, Ratings, and Relations (NoRaRe, Tjuka et al. 2022, https://norare.clld.org/) alongside multilingual FastText word embeddings (Grave et al. 2018). By that, I will provide a general tutorial on how to integrate and retrieve vector data with NoRaRe. # 2 Integrating Concept Embeddings into NoRaRe Following last month’s tutorial by List (2025), we start by expanding the `norare.tsv` and `datasets.tsv` files that contain basic information about all datasets collected in NoRaRe, as well as adding new references to `references/references.bib`. In `datasets.tsv`, I simply append one new line where I provide meta-information concerning the new dataset under the name `Rubehn-2025-ConceptEmbeddings`. This name is used as unique identifier for the dataset throughout the entire integration process. Similarly, I extend `norare.tsv` by describing the type of data I am contributing. Since for each concept I am contributing several vector representations (3 types of concept embeddings based on different types of colexification + FastText embeddings in 9 different languages), each of them has to be described separately in a new line. Finally, I add BibTeX entries for the sources of the data I am contributing (in this case Grave et al. 2018; Rubehn and List 2025) to `references/references.bib`. Having dealt with the basic files, it is time to contribute the actual dataset. For this, the first step is to create a new directory `datasets/Rubehn-2025-ConceptEmbeddings` where all relevant information is stored. This directory itself will contain three files: `Rubehn-2025-ConceptEmbeddings.tsv`, `Rubehn-2025-ConceptEmbeddings.tsv-metadata.json`, and `norare.py`. Note how the dataset identifier is recurring in the namespace. `Rubehn-2025-ConceptEmbeddings.tsv-metadata.json` is a metadata file conforming to CSVW standards (Gower 2021). Simply speaking, it allows (and requires) me to define which columns the corresponding TSV file (where all data is written to) contain, and which data type is expected in each column. This ensures data consistency and allows for smooth data retrieval later on. Concretely, I use this file to define that each entry consists of a Concepticon ID, the corresponding Concepticon gloss (the concept inventory that NoRaRe operates on is defined by Concepticon; List et al. 2025, https://concepticon.clld.org/), and then all different types of embedding vectors described above. For all columns containing embeddings, I define `"json"` as data type: embeddings are therefore directly represented as lists of floats. Finally, we get to the core of the dataset: the `norare.py` script that produces the file `Rubehn-2025-ConceptEmbeddings.tsv`, which contains all the actual data in the end. In `norare.py`, I have to define two functions that correspond to shell commands defined by `pynorare` (List and Forkel 2024): `download` and `map`. As the names suggest, these two functions define the behavior for 1. downloading the raw data and 2. mapping them to NoRaRe. For the concept embeddings, it is straightforward to define the downloading and mapping behavior, since the embeddings already represent Concepticon concepts directly. The only trick I had to employ here was to round all numbers to 4 decimals to prevent the file from becoming too large. This is a simple quantization technique that is commonly done to compress data (Gray and Neuhoff 1998) without losing relevant information. # 3 Integrating Multilingual Word Embeddings into NoRaRe Mapping multilingual FastText embeddings (Grave et al. 2018) requires some more handling of data, which is essentially due to the fact that embedded words have to be mapped to the concepts defined by Concepticon, and this mapping is usually not a 1-to-1 relation. As a first step, again, we download the data from https://fasttext.cc/docs/en/crawl-vectors.html. Now that we have obtained the embeddings for _words_ in different languages, we have to map them to the relevant _concepts._ This can be done via the MultiSimLex database (Vulić et al. 2020) that provides parallel translations for 1,888 cues in various different languages. Since the elicited cues have already been linked to Concepticon (List 2021) and the translations are parallel, MultiSimLex offers a good resource for systematically linking words from different languages to Concepticon concepts. Now that we have a principal method for mapping words to concepts, we encounter another problem that we need to deal with: sometimes, there are multiple translations for the same concept. For example, the concept CAR can be expressed by the Russian words _avtomobil’_ and _mashina,_ and both translations are actually found in the MultiSimLex data. In those cases, we define the vector representing a concept in a language as the weighted average of the corresponding word vectors. As a simplified example, assume that we find 1 occurrence of _avtomobil’_ with the vector `[2, 3]` and two occurrences of _mashina_ with the vector `[1, 6]`: the resulting vector would be `[(1*2 + 2*1) / 3, (1*3 + 2*6) / 3] = [1.3333, 5]`. With solutions to these practical issues, we now have a robust and consistent way of handling and mapping word embedding data to Concepticon; so we can simply implement the described behavior in the `map` function. Having implemented both the downloading and mapping routine for both, the concept embeddings and the word embeddings, pynorare offers convenient shell commands to create the desired TSV file, in which all information is finally represented. $ norare download Rubehn-2025-ConceptEmbeddings # download data $ norare map Rubehn-2025-ConceptEmbeddings # map data # 4 Retrieving Embeddings from NoRaRe Thanks to the pynorare API and the CSVW specifications, it is now straightforward to retrieve the different embeddings. Start by creating a fresh virtual environment, install pynorare via pip and clone the latest version of the Concepticon and NoRaRe datasets: $ python -m venv venv $ source venv/bin/activate # for *NIX-based systems; if you use Windows, you need to run the 'Activate.ps1' script instead $ git clone --depth=1 https://github.com/concepticon/concepticon-data $ git clone --depth=1 https://github.com/concepticon/norare-data Now that you’re set up, you can easily retrieve the described embeddings, as illustrated in the code snippet below. Thanks to the CSVW specifications, all data are already represented as objects of the correct type — the embedding vectors are already lists of floats, and the Concepticon ID’s are already integers. from pynorare import NoRaRe from pyconcepticon import Concepticon # set up Concepticon and NoRaRe API c = Concepticon("concepticon-data") norare = NoRaRe("norare-data", concepticon=c) # retrieve data embedding_data = norare.datasets["Rubehn-2025-ConceptEmbeddings"] concept_embeddings = {concepticon_id: entry["embeddings_full_affix"] for concepticon_id, entry in embedding_data.items()} ft_embeddings_es = {concepticon_id: entry["fasttext_es"] for concepticon_id, entry in embedding_data.items()} # 5 Conclusion and Outlook In this study, I have briefly illustrated how vector data can be added to and retrieved from NoRaRe by the concrete example of concept embeddings (Rubehn and List 2025) and multilingual word embeddings (Grave et al. 2018). I am optimistic that this addition will be useful for future research representing semantics on a conceptual level, since distributional semantic representations have proven themselves as powerful tools already. Furthermore, the inclusion of embeddings serves as yet another template for how complex data types can be handled by NoRaRe (Ahmedović 2025). # References Ahmedović, Mira (2025): Handling Non-Standard Datasets in NoRaRe: A Practical Guide. _Computer-Assisted Language Comparison in Practice_ 8.1. 17–23. https://doi.org/10.15475/calcip.2025.1.3 Bojanowski, Piotr and Edouard Grave and Armand Joulin and Tomas Mikolov (2017). Enriching word vectors with subword information. _Transactions of the Association for Computational Linguistics_ 5. 135-146. https://doi.org/10.1162/tacl_a_00051 Forkel, Robert and Johann-Mattis List (2024). _PyNoRaRe [Python package, version 1.1.0]._https://pypi.org/project/pynorare/ Gower, Robin (2021): CSV on the Web. Stirling: Swirrl. https://csvw.org Grave, Edouard and Piotr Bojanowski and Prakhar Gupta and Armand Joulin and Tomas Mikolov (2018). Learning word vectors for 157 languages. In _Proceedings of the Eleventh International Conference on Language Resources and Evolution (LREC 2018)._https://aclanthology.org/L18-1550/ Gray, Robert M. and David L. Neuhoff (1998). Quantization. _IEEE Transactions on Information Theory_ , 44.6. 2325-2383. https://doi.org/10.1109/18.720541 List, Johann-Mattis (2021). Mapping Multi-SimLex to Concepticon. _Computer-Assisted Language Comparison in Practice_ 4.3. 1-8. List, Johann-Mattis (2025). Illustrating Data Curation in NoRaRe with the Help of Templates. _Computer-Assisted Language Comparison in Practice_ 8.2. List, Johann Mattis and Annika Tjuka and Frederic Blum and Alžběta Kučerová and Carlos Barrientos Ugarte and Christoph Rzymski and Simon Greenhill and Robert Forkel (eds.) (2025). _CLLD Concepticon 3.4.0 [Data set]_. Zenodo. https://doi.org/10.5281/zenodo.14923561 Mikolov, Tomas and Kai Chen and Greg Corrado and Jeffrey Dean (2013). Efficient estimation of word representations in vector space. _arXiv preprint arXiv:1301.3781._https://doi.org/10.48550/arXiv.1301.3781 Pennington, Jeffrey and Richard Socher and Christopher D. Manning (2014). GloVe: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)._ 1532-1543. https://doi.org/10.3115/v1/D14-1162 Rubehn, Arne and Johann-Mattis List (2025). Partial colexifications improve concept embeddings. In _Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)._ 20571-20586. https://aclanthology.org/2025.acl-long.1004 Tjuka, Annika and Robert Forkel and Johann-Mattis List (2022). Linking Norms, Ratings, and Relations of Words and Concepts Across Multiple Language Varieties. _Behavior Research Methods_ 54. 864–884. https://doi.org/10.3758/s13428-021-01650-1 Vulić, Ivan and Simon Baker and Edoardo Maria Ponti and Ulla Petti and Ira Leviant and Kelly Wing and Olga Majewska and Eden Bar and Matt Malone and Thierry Poibeau and Roi Reichart and Anna Korhonen (2020): Multi-SimLex: A large-scale evaluation of multilingual and cross-lingual lexical semantic similarity. _Computational Linguistics_ 46.4. 847-897. https://doi.org/10.1162/coli_a_00391 **Cite this article as:** Rubehn, Arne (2025): “Integrating Semantic Embeddings with NoRaRe” in _Computer-Assisted Language Comparison in Practice_ , 8.2: 81-86 [first published on 17/09/2025], URL: https://calc.hypotheses.org/8783, DOI: 10.15475/calcip.2025.2.3. **Download the article as PDF:** calcip-08-2-3.pdf **Copyright information** : This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. **Supplementary Materials** : The data described here is available as part of the NoRaRe database, which is currently curated on GitHub (https://github.com/concepticon/norare-data) and regularly archived with Zenodo (all versions available at https://doi.org/10.5281/zenodo.3957680). The data created here will be part of the next release Version 1.2. For details, the contribution can be inspected via the pull-request on GitHub at https://github.com/concepticon/norare-data/pull/282. **Funding Information** : This project has received funding from the European Research Council (ERC) under the European Union’s Horizon Europe research and innovation programme (Grant agreement No. 101044282). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. * * * OpenEdition suggests that you cite this post as follows: Arne Rubehn (September 17, 2025). Integrating Semantic Embeddings into NoRaRe. _Computer-Assisted Language Comparison in Practice_. Retrieved September 17, 2025 from https://calc.hypotheses.org/8783 * * * * * * * *

This study describes the process of digitizing legacy materials into a computer-readable format for the purposes of computational typology and computer-assisted historical reconstruction. It presents a comparative wordlist that is made available in the formats recommended by the Cross-Linguistic Data Formats initiative. # 1 Needham’s “ _A collection of a few Môshâng Nâgâ words”_ The original publication, _A collection of a few Môshâng Nâgâ words_ (Needham 1897) provides the first written account of Muishaung (Glottolog: mosa1240), a Tibeto-Burman language spoken Arunachal Pradesh, India as well as neighbouring regions of Sagaing Region, Myanmar. Data were collected during a trip made by Needham to the Muishaung area in 1881. Today Muishaung is estimated to have around 2000 speakers in India, with an uncertain population size in Myanmar. This is one of a handful of texts written by Needham at the time. The others include similar descriptions of Tai Khamti and Singpho. Needham’s text includes 264 concepts — although with some minor misunderstandings around which concepts were actually being provided by speakers — along with basic grammatical notes on topics such as gender, verbal morphology and the pronominal system. Aside from being the earliest known account of any Tangsa-Nocte variety, Needham’s text provides valuable insights into historical sound change in the region; Muishaung is one of over two-dozen closely related language varieties within Tangsa-Nocte, but significantly it is the most phonologically divergent of the group. For example, it has innovative dental stops, including a split between /n/ and /n̪/ not seen elsewhere in the group, analogous to the /t/ and /t̪/ pair as reflexes of _*t_ and _*ð_ respectively. It also shows a split between /g/ and /ɣ/ as reflexes of *ɣ, and finally, Muishaung has been undergoing a process of vowel fracture not seen elsewhere among the closely related varieties. Access to historical texts such as Needham’s allow us to better understand the timing of such changes and the possible mechanisms behind them. For this reason, proper analysis of such descriptions is important. By developing computer-readable versions of such data sets, we are better able to include them in comparative work. With somewhat inconsistent orthographic conventions (many of which are never described by the author) along with some clear mistakes in the data which were elicited, the full value of Needham’s text is not apparent without a more in-depth investigation. This was done in van Dam & Mossang (2025), which analyzed the entirety of the text. This study undertook careful investigation into the terms given through comparison to modern-day forms of the concepts and reconstructed proto-forms. The full study is available as an open-access publication in the _Journal of Asian and African Studies_ published by the Tokyo University of Foreign Studies. # 2 Data Availability as a CLDF Data Set In the process of analyzing the text, a digital version of the lexical data along with terms found in the grammatical notes was created in a flat tabular format. These were transcribed exactly as they occurred in the original text, including the use of circumflexes for marking distinctions in vowel quality and an underlined ⟨n⟩ for vowel nasalization. Needham’s transcription was then converted to IPA and paired with modern-day pronunciations for cases in which cognates are attested today. In those instances where no modern-day cognate was found, the term which replaced the form in Needham’s time was given. In an additional step, the data from the flat tabular format were converted to the formats recommended by the Cross-Linguistic Data Formats (CLDF) initiative (Forkel et al. 2018, https://cldf.clld.org), using the workflow for the handling of comparative wordlists developed for the Lexibank repository (List et al. 2022, Blum et al. 2025, https://lexibank.clld.org). The CLDF dataset also includes all comments given in the original text, of which where were a few, along with notes by the authors of the 2025 study indicated cases where Needham may have elicited a term other than what was intended. For example in eliciting ‘flea’, the term given was actually one for ‘cat’, perhaps the result of gesturing toward a flea-infested feline at the time of elicitation. An example of the Forms table, with some columns removed here for the sake of saving space, is seen here: Local_ID | Form | Segments | Comment | Source ---|---|---|---|--- MuishaungNeedham-1_above-1 | rʌŋ | r ʌ ŋ | Shâng´gê is distant from Môshâng about 20 miles, much less as the crow flies. J.N. | Needham1897 MuishaungModern-1_above-1 | rɐuŋ₂ | r ɐu ŋ ₂/²³¹ | | VanDam2025 MuishaungNeedham-2_acid-1 | ɑ.hiˀ | ɑ + h i ˀ/ʔ | | Needham1897 MuishaungModern-2_acid-1 | ə₀hi₂ | ə ₀/⁰ h i ₂/²³¹ | | VanDam2025 MuishaungNeedham-3_all-1 | wʌ.tɒŋ | w ʌ + t ɒ ŋ | | Needham1897 MuishaungModern-3_all-1 | βə₀tɐuŋ₂ | β ə ₀/⁰ t ɐu ŋ ₂/²³¹ | | VanDam2025 An orthography profile was also created, and all concepts were mapped to the corresponding CONCEPTICON IDs (List et al 2025) where applicable. Language varieties are also linked to their corresponding Glottocodes (Hammarström et al 2025) to facilitate language identification. # 3 Next Steps In addition to forms given for Muishaung, the original text also includes a number of words from the Shecyü variety under the name Shâng´gê, Needham’s representation of the common exonym Shangke (Glottolog: sank1250). Occasionally forms are also given for Singpho, a distantly related Tibeto-Burman language, as well as Tai Hkamti, a Kra-Dai variety. In addition to Needham’s Muishaung description, he also published texts on Singpho and Tai Hkamti, and was thus knowledgable about both. He regularly included reference to these languages in cases where he felt the term in Muishaung was borrowed from one or the other, although in some cases with Singpho he was simply identifying cognate terms. In a future version of the data set, these forms will be fully encoded with both Needham’s orthographic representation and its corresponding IPA form, as well as the modern-day equivalents. This is intended to be published as an update to the current dataset in order to further support computer-assisted cross-linguistic comparative work. # References Blum, Frederic, Carlos Barrientos, Johannes Englisch, Robert Forkel, Simon Greenhill, Christoph Rzymski, and Johann-Mattis List (2025): Lexibank 2: pre-computed features for large-scale lexical data [version 2; peer review: 3 approved]. _Open Research Europe_ 5.126. 1-19. https://doi.org/10.12688/openreseurope.20216.2 van Dam, Kellen Parker and Kelim Mossang, Wanglung. (2025a). A Classified Account of J. F. Needham’s A Collection of A Few Môshâng Naga Words. In: Journal of Asian and African Studies 2025 (109), 111-145. https://doi.org/10.57275/ilcaajaas.2025.109_111 van Dam, Kellen Parker and Kelim Mossang, Wanglung (2025b). Supplementary materials for van Dam & Kelim Mossang 2025 [Data set, Version 1.0.0]. In Journal of Asian and African Studies (Vol. 109, pp. 111–145). Zenodo. https://doi.org/10.5281/zenodo.14053893 Forkel, Robert, Johann-Mattis List, Simon Greenhill, Christoph Rzymski, Sebastian Bank, Michael Cysouw, Harald Hammarström, Martin Haspelmath, Gereon Kaiping, and Russell D. Gray (2018): Cross-Linguistic Data Formats, advancing data sharing and re-use in comparative linguistics. _Scientific Data_ 5.180205. 1-10. https://doi.org/10.1038/sdata.2018.205 Hammarström, Harald, Robert Forkel, Martin Haspelmath, and Sebastian Bank (2025): Glottolog [Dataset, Version 5.2], Leipzig: Max Planck Institute for Evolutionary Anthropology. https://glottolog.org List, Johann Mattis, Annika Tjuka, Frederic Blum, Alžběta Kučerová, Carlos Barrientos, Christoph Rzymski, Simon Greenhill, and Robert Forkel (2025): _CLLD Concepticon_ [Data set, Version 3.4.0]. Leipzig: Max Planck Institute for Evolutionary Anthropology. https://concepticon.clld.org List, Johann-Mattis, Robert Forkel, Simon Greenhill, Christoph Rzymski, Johannes Englisch, and Russell D. Gray (2022): Lexibank, A public repository of standardized wordlists with computed phonological and lexical features. _Scientific Data_ 9.316. 1-31. https://doi.org/10.1038/s41597-022-01432-0 Needham, J. F. (1897). _A collection of a few Moshang Naga words_. Shillong: Assam Secretariat Printing Office. https://archive.org/details/collectionoffewm00needrich **Cite this article as:****** van Dam, Kellen Parker (2025): “Digitizing Legacy Lexical Data of Muishaung for Computer-Assisted Language Comparison” in _Computer-Assisted Language Comparison in Practice_ , 8.2: +++ [first published on 23/07/2025], URL: https://calc.hypotheses.org/8624, DOI: 10.15475/calcip.2025.2.1. **Download the article as PDF:** calcip-08-2-1.pdf **Copyright information** : This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. **Supplementary Materials** : Code and data are curated on GitHub (https://github.com/phonemica/needhammuishaung, Version 1.0.0) and archived with Zenodo (https://doi.org/10.5281/zenodo.14053893). * * * OpenEdition suggests that you cite this post as follows: Kellen Parker van Dam (July 23, 2025). Digitizing Legacy Lexical Data of Muishaung for Computer-Assisted Language Comparison. _Computer-Assisted Language Comparison in Practice_. Retrieved July 23, 2025 from https://calc.hypotheses.org/8624 * * * * * * * *

12 Beiträge, 243 Folge ich, 99 Follower ·

NoRaRe, the Database of Cross-Linguistic Norms, Ratings, and Relations, is a resource that curates multiple datasets containing information on various properties of words and concepts. When researchers contribute their data, the format and structure can vary widely, presenting challenges for seamless integration. Here, I offer practical guidance for addressing common issues such as data being placed in different sheets, headers in unexpected rows, or datasets contained within `zip-`files. The strategies shared here offer a foundational approach to understanding and adapting NoRaRe’s flexibility to accommodate the idiosyncrasy of each dataset. # **1 Introduction** NoRaRe – the Database of Cross-Linguistic Norms, Ratings, and Relations – is a compilation of datasets that offer information on various properties of words and concepts, such as emotional ratings, concreteness, frequency, and more (Tjuka et al. 2022a, Tjuka et al. 2023, NoRaRe). The current version, Version 1.1, can be found here. The datasets are gathered from studies in linguistics and psychology. By linking datasets to the standardized concept sets of the Concepticon (List et al. 2016; Tjuka et al. 2023), NoRaRe allows researchers to explore cross-linguistic similarities and differences while adding context-specific data from published studies. The database is open-access and all researchers are welcome and encouraged to add data that are relevant. A previous contribution contains a general guide to adding datasets to NoRaRe (Tjuka 2021) and the data are publicly available on Github and _Zenodo_. When adding data, it quickly becomes evident that no two datasets are exactly alike in terms of their format and structure. Some researchers put their data on the second sheet of a spread sheet, others have headers on the second or third line, while yet others upload their electronic supplementary material in `zip-`files for ease of accessibility of many files at once. Van Zantwijk (2023) provides guidance for preparing better datasets. The following offers a resource for handling special cases – and combinations of them – effectively. # **2****Accessing Data from Different Sheets** When a dataset’s main data is not on the first sheet of an Excel file, running the command `norare map name-of-dataset` will fail. By default, NoRaRe assumes the primary data resides in the first sheet. This issue can be solved as follows. In the `norare.py` file: * Adjust the file name in the `sheet_list` object. * Adjust the script to specify the correct sheet index that contains the dataset. * Adjust the `dataset.extract_data` function and add `sheet` as the first argument. The following example is a chunk of code from the Coso-2023-Emotions dataset (Ćoso et al. 2023). def map(dataset, concepticon, mappings): sheet_list = dataset.get_excel('CROWD-5e.xlsx', 1, dicts=False) sheet = [dict(zip(sheet_list[0], row)) for row in sheet_list[1:]] dataset.extract_data(sheet, concepticon, mappings, gloss='ENGLISH', language='en') Here, the `sheet_list` object has an argument after the name of the file, namely`1`, which tells the program where to start extracting the data from – recall that Python uses zero-based indexing so 1 specifies the second sheet. The `sheet` argument tells the program where in the second sheet to look for the header, with `sheet_list[0], row` indicating the first row of the sheet. The second part of the `sheet` argument, namely `for row in sheet_list[1:]` tells the program to extract the actual data from every row after the header. # **3 Unusual Headers and Values** Another common occurrence is that some researchers place column headers in lines other than the first because they either split the data into sub data types or add metadata in the first rows of the sheet. This issue will also become apparent once running `norare map name-of-dataset`. NoRaRe assumes that headers are placed in the first line. Furthermore, unexpected null values in the original dataset make correct treatment of the data impossible for the program. A workaround, which mainly consists of adjusting the `norare.py` file, is shown below. In the `norare.py` file: * Add a `def replace_null` function, which tells the program how to handle null values. Adjust it for the specific unexpected string in the original data, `#NULL!` in the example below. * Adjust the file name in the `sheet_list` object. * Adjust the script and define the `valid_fields` list object. List all columns that are in the dataset in the order that they appear in. The names do not have to be identical to the original file but their order is important to ensure correct extraction. Avoid duplicates. * Add the first number of the first row of data to the loop that iterates over all rows (`for row in sheet_list`). * Adjust the `sheet.append` function to replace null values correctly and to iterate over all columns specified in `valid_fields` appropriately. * Add the `sheet` argument in the `dataset.extract_data` function. In the `metadata.json` file: * Use the names in the `valid_fields` list object from the `norare.py` file as the `titles` fields when setting up the metadata. The following example is a chunk of code from the Bonin-2018-Concreteness dataset (Bonin et al. 2018). def replace_null(v): if v == '#NULL!': return None else: return v def map(dataset, concepticon, mappings): sheet_list = dataset.get_excel('13428_2018_1014_MOESM3_ESM.xlsx', 0, dicts=False) sheet = [] valid_fields = ['items', 'English translation', 'mean', 'sd', 'Mean context availability', 'SD context availability', 'Mean valence', 'SD valence', 'Mean arousal', 'SD arousal'] for row in sheet_list[2:]: sheet.append({k: replace_null(v) for k, v in zip(valid_fields, row[:10])}) dataset.extract_data(sheet, concepticon, mappings, gloss='FRENCH', language='fr') # **4****Handling a Mix of Issues a****t Once** Cases that combine the previous two issues require the same approach previously discussed. An example of such a case is the Repetto-2023-Sensorimotor dataset (Repetto et al. 2023), which additionally also contains Part of Speech (POS) data. This information is added to the `metadata.json` file as a separate column. In the `norare.py` file: * Adjust the file name in the `sheet_list` object. * Define the column names in the `valid_fields` list object, avoiding duplicates. * Adjust the `for row in sheet_list` loop to iterate over the data after the header. * Adjust the `sheet +=` statement to iterate over all `valid_fields`. * Add the `sheet` argument to the `dataset.extract_data` function. * Add the `pos` tag to the `dataset.extract_data` function. The `pos_mapper` dictionary contains the shorthand used in the original data on the left and the corresponding NoRaRe category of the POS on the right hand-side. Specify the column name of the POS as it appears in the `metadata.json` file. In the `metadata.json` file: * Use the names in the `valid_fields` list object from the `norare.py` file as the `titles` fields when setting up the metadata. * Define the POS column. The following chunk of code is an example from the Repetto-2023-Sensorimotor dataset (Repetto et al. 2023). def map(dataset, concepticon, mappings): sheet_list = dataset.get_excel('sensorimotor dataset.xlsx', 1, dicts=False) valid_fields = [ 'Ita_Word', 'WordClass', 'Let_ITA', 'FreqColfis', 'Ln_Colfis', 'FreqRepub', 'Ln_FreqRep', 'N_OrtNeig', 'MeanFreq_Neig', 'N_Part_p', 'M_Fam', 'SD_Fam', 'M_Ima', 'SD_Ima', 'M_Con', 'SD_Con', 'N_Part_a', 'M_Val', 'SD_Val', 'M_Aro', 'SD_Aro', 'M_Dom', 'SD_Dom', 'N_Part1', 'M_head', 'SD_head', 'M_foot/leg', 'SD_foot/leg', 'M_hand/arm', 'SD_hand/arm', 'M_mouth/throat', 'SD_mouth/throat', 'M_torso', 'SD_torso', 'N_Part2', 'M_int', 'SD_int', 'M_taste', 'SD_taste', 'M_smell', 'SD_smell', 'M_touch', 'SD_touch', 'M_audition', 'SD_audition', 'M_vision', 'SD_vision', 'Max_strength.action', 'Exclusivity.action', 'Dominant.action', 'Max_strength.perceptual', 'Exclusivity.perceptual', 'Dominant.perceptual', 'Max_strength.sensorimotor', 'Exclusivity.sensorimotor', 'Dominant.sensorimotor' ] sheet = [] for row in sheet_list[2:]: sheet += [dict(zip(valid_fields, row[:len(valid_fields)]))] dataset.extract_data( sheet, concepticon, mappings, gloss='ITALIAN', language='it', pos=True, pos_mapper={'n': 'Person/Thing', 'v': 'Action/Process'}, pos_name="ITALIAN_POS" ) # **5 Data in a Zip-File** Some datasets are contained in a `.zip` file with additional information. By adjusting the `norare.py` file, we work around this issue. In the `norare.py` file: * Adjust the `def download(dataset)` function and change `dataset.download_file` to` dataset.download_zip`. * Adjust the download link. * Adjust the `.zip` file name. * Adjust the name of the file that is supposed to be extracted from the `.zip` file. Use this file name throughout the rest if the `norare.py` file. The following chunk of code is an example from the Syssau-2009-Valence dataset (Syssau and Monnier 2009). def download(dataset): dataset.download_zip( 'https://staticcontent.springer.com/esm/' 'art%3A10.3758%2FBRM.41.1.213/MediaObjects/' '13428_2010_410100213_MOESM1_ESM.zip', 'syssau_monnier_norms.zip', 'syssau_monnier_norms.xls' ) # **6 Conclusion** Some datasets come with their idiosyncrasies and not all of them fit perfectly into NoRaRe’s default workflows. Whether it is dealing with data placed in different sheets, unconventional headers, `.zip` files, or even combinations of these deviations from the norm, there is always a work-around. By applying the solutions outlined here, accurate integration of datasets can be managed. The examples presented above aim to illustrate a general approach: understanding the dataset structure, making targeted adjustments in the `norare.py` and `metadata.json` files and utilizing NoRaRe’s flexibility to suit your individual dataset. With this in mind, tackling new and unexpected formats becomes a learning process, building upon the tools and techniques shared here. # **References** Bonin, Patrick, Alain Méot & Aurélia Bugaiska. 2018. Concreteness norms for 1,659 French words: Relationships with other psycholinguistic variables and word recognition times. _Behavior Research Methods_ 50(6). 2366–2387. https://doi.org/10.3758/s13428-018-1014-y Ćoso, Bojana, Marc Guasch, Irena Bogunović, Pilar Ferré & José A. Hinojosa. 2023. CROWD-5e: A Croatian psycholinguistic database of affective norms for five discrete emotions. _Behavior Research Methods_ 55(8). 4018–4034. https://doi.org/10.3758/s13428-022-02003-2 List, Johann-Mattis, Michael Cysouw & Robert Forkel. 2016. Concepticon: A resource for the linking of concept lists. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk & Stelios Piperidis (eds.), Proceedings of the Tenth International Conference on Language Resources and Evaluation, 2393–2400. Portorož, Slovenia: European Language Resources Association. https://aclanthology.org/L16-1379/ Repetto, Claudia, Claudia Rodella, Francesca Conca, Gaia C. Santi & Eleonora Catricalà. 2023. The Italian Sensorimotor Norms: Perception and action strength measures for 959 words. _Behavior Research Methods_ 55(8). 4035–4047. https://doi.org/10.3758/s13428-022-02004-1 Syssau, Arielle & Catherine Monnier. 2009. Children’s emotional norms for 600 French words. _Behavior Research Methods_ 41(1). 213–219. https://doi.org/10.3758/BRM.41.1.213 Tjuka, Annika. Adding data sets to NoRaRe: A guide for beginners. Computer-Assisted Language Comparison in Practice 4(8). https://calc.hypotheses.org/2890 Tjuka, Annika, Robert Forkel & Johann-Mattis List. 2022a. Linking norms, ratings, and relations of words and concepts across multiple language varieties. _Behavior Research Methods_ 54(2). 864–884. https://doi.org/10.3758/s13428-021-01650-1 Tjuka, Annika, Robert Forkel & Johann-Mattis List. 2022b. NoRaRe. A database of Cross-Linguistic Norms, Ratings, and Relations for Words and Concepts (Version 1.1). Leipzig: Max Planck Institute for Evolutionary Anthropology. https://doi.org/10.5281/zenodo.14924483 Tjuka, Annika, Robert Forkel & Johann-Mattis List. 2023. Curating and extending data for language comparison in Concepticon and NoRaRe. _Open r_ _esearch Europe_ 2(141): 1-13. https://doi.org/10.21956/openreseurope.17368.r32031 van Zantwijk, Mathilda. 2023. Five Recommendations for Creating Spreadsheets. Computer-Assisted Language Comparison in Practice 6(2): 93–96,****https://doi.org/10.58079/m6m3 **Cite this article as:** Ahmedović, Mira, “Handling Non-Standard Datasets in NoRaRe: A Practical Guide,” in _Computer-Assisted Language Comparison in Practice_ , 8.1: 17-23 [first published on 12/03/2025], URL: https://calc.hypotheses.org/8331, DOI: 10.15475/calcip.2025.1.3. **Download the article as PDF:** calcip-08-1-3.pdf **Copyright information** : This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. **Supplementary Materials** : Data and code are curated on GitHub (https://github.com/concepticon/norare-data/tree/v1.1) and archived with Zenodo (https://doi.org/10.5281/zenodo.14924483). **Acknowledgements:** I would like to thank Dr. Annika Tjuka for supervising the creation of this practical guide. * * * OpenEdition suggests that you cite this post as follows: Mira Ahmedovic (March 12, 2025). Handling Non-Standard Datasets in NoRaRe: A Practical Guide. _Computer-Assisted Language Comparison in Practice_. Retrieved March 12, 2025 from https://calc.hypotheses.org/8331 * * * * * * *

<p>EDICTOR3 offers many ways of comparing language data with computer-assisted methods. This study offers a short overview of how to run EDICTOR3 locally, without the need for uploading the data to a server or being connected to the internet, while maintaining all the functionalities. In a first step, we will show how one can download a Lexibank dataset and create different types of files that one can use with EDICTOR. We will then proceed to present the possibility of running an EDICTOR server locally and to edit the dataset that one has downloaded.</p> <div> <p><span id="more-8143"></span></p> </div> <h2><strong>1 Introduction</strong></h2> <p>Computer-assisted methods for language comparison have come a long way in recent years (Wu et al. 2020). Methods for automatizing steps of the comparative method have been developed, and many tools and workflows have already been presented in this series. Here, we will focus on the EDICTOR3 application (List et al. 2025, <a href="https://edictor.org">https://edictor.org</a>). In contrast to previous versions which were run online, the accompanying Python 3 package (<a href="https://pypi.org/project/edictor">https://pypi.org/project/edictor</a>) offers the possibility to run your own EDICTOR server locally. The article will lead you through the necessary steps to work with this application, and showcase different ways of preparing your data.</p> <p>We will use the <code>blumpanotacana</code> dataset (Blum et al. 2024) for this article. The dataset features data from 21 language varieties of several language families, the largest being the Pano-Takana family. All languages of the dataset are spoken in the Amazon region in South America. The repository follows the Cross-Linguistic Data Formats (CLDF, <a href="https://cldf.clld.org">https://cldf.clld.org</a>, Forkel et al. 2018), specifically the Lexibank structure (<a href="https://lexibank.clld.org">https://lexibank.clld.org</a>, List et al. 2022). It has a couple of files for the maintenance of the repository, as well as folders for metadata (<code>etc/</code>), the raw data (<code>raw/</code>), and the CLDF data (<code>cldf/</code>). All data is stored in tabular formats and linked through certain identifiers. Using CLDF ensures that the data is easily accessible and re-usable with different tools, exemplified through its use in this article. A detailed description how to create such a dataset for lexical comparisons and how to provide the necessary standardization is given in Blum et al. (2024).</p> <h2><strong>2 First steps</strong></h2> <h3><strong>2.1 Preliminary Information<br/></strong></h3> <p>Windows users might have to run some additional steps. Please follow the instructions provided by Snee (2024), which should be sufficient to follow all steps described in the following. Note also that — as an application written for Python 3 — depending on your configuration, you may have to use the command <code>python3</code> instead of the command <code>python</code> in order to follow all steps discussed in the following. Since Python 3 has been the new standard Python version for a very long time now, with Python 2 applications still in use being the exception, we use the command <code>python</code> throughout this tutorial, assuming that it should be no problem to adjust the command in your individual settings.</p> <h3><strong>2.2 Installing EDICTOR3</strong></h3> <p>Before installing EDICTOR3, we need to make sure to have a clean Python environment. We achieve this by setting up a virtual environment exclusively to be used with EDICTOR3. Mac and Linux users, you can prepare your virtual environment with the help of the following commands. Windows users find additional information on how to set up a virtual environment online (<a href="https://pypi.org/project/edictor/">https://pypi.org/project/edictor/</a>).</p> <pre class="wp-block-code">$ python -m pip install virtualenv $ python -m venv venv/edictor $ source venv/edictor/bin/activate</pre> <p>We can now install the EDICTOR3 application. We can again simply use thy Python package manager <code>pip</code> to install the <code>edictor</code> package.</p> <pre class="wp-block-code">$ pip install edictor</pre> <p>It is recommended to run the installation with enabling support for some other Python packages. This makes sure to have the full functionality of EDICTOR3 available, to which we will turn later. You can run the following command to install the extended packages LingPy (<a href="https://pypi.org/project/lingpy/">https://pypi.org/project/lingpy/</a>, List and Forkel 2023), LingRex (<a href="https://pypi.org/project/lingrex">https://pypi.org/project/lingrex/</a>, List and Forkel, 2024), and LexiBase (<a href="https://pypi.org/project/lexibase">https://pypi.org/project/lexibase</a>) along with EDICTOR.</p> <pre class="wp-block-code">$ pip install "edictor[lingpy]"</pre> <p>Note that you can also install the three packages directly using pip.</p> <pre>$ pip install lingpy<br/>$ pip install lingrex<br/>$ pip install lexibase</pre> <p>If you now run the command <code>edictor server</code>, a browser window should open where you can access some base files and examples. Windows users will have to open the browser window directly, by passing the default URL <a href="http://localhost:9999">http://localhost:9999</a>.  However, we do not have our own data ready yet, so we can close the window. This step is only for verifying that the installation has worked.</p> <h2><strong>2.3 P</strong><strong>reparing the Data</strong></h2> <p>We can now turn to the preprocessing of the data we want to analyze. The first step is to download the data. As mentioned before, we will examplify this with the <code>blumpanotacana</code> dataset. However, you may chose any other Lexibank dataset and replace the name accordingly (check the GitHub organization at <a href="https://github.com/lexibank/">https://github.com/lexibank/</a> for individual Lexibank datasets). The datasets are all curated on GitHub, and the released versions are stored on Zenodo. If you have <code>git</code> installed on your computer, you can simply clone the repository from GitHub:</p> <pre class="wp-block-code">$ git clone https://github.com/pano-takanan-history/blumpanotacana $ cd blumpanotacana</pre> <p>Otherwise, you can also download the dataset directly from Zenodo (<a href="https://zenodo.org">https://zenodo.org</a>), via its DOI (<a href="https://doi.org/10.5281/zenodo.10214016">https://doi.org/10.5281/zenodo.10214016</a>). The exact command might differ between operating systems. The following example is from MacOS 15.2 for v1.2 of <code>blumpanotacana</code>.</p> <pre class="wp-block-code">$ curl https://zenodo.org/records/13149588/files/pano-tacanan-history/blumpanotacana-v1.2.zip --output blumpanotacana.zip $ unzip blumpanotacana.zip &amp;&amp; mv pano-tacanan-history-blumpanotacana-5532c2a/ blumpanotacana/ $ cd blumpanotacana</pre> <h2><strong>3 Accessing the Data</strong></h2> <p>We have two options to access the data with EDICTOR: (a) using a TSV-file, or (b) using an SQLite database. For the first option, you only need to create a TSV-file using the same <code>edictor</code> package with the following command:</p> <pre class="wp-block-code">$ edictor wordlist --dataset=cldf/cldf-metadata.json --name=blumpanotacana --addon="cognacy:cogid,alignment:alignment"</pre> <p>The command uses the <code>wordlist</code> sub-command, which is installed when selecting the <code>[lingpy]</code> option during installation of the <code>edictor</code> package, or when directly installing LingPy along with EDICTOR3. The first argument is the metadata file of the dataset, and <code>--name</code> provides the name of the file you create. The list in <code>--addon</code> provides additional columns, based on data from the CLDF dataset. In this case, we want the existing cognate judgements and manual alignments to be included in the EDICTOR dataset. You now receive an output file <code>blumpanotacana.tsv</code>. If you now run <code>edictor server</code>command again, select the tab <code>Files</code> in the web interface and then <code>Open File «blumpanotacana.tsv»</code>. You can now edit and analyze the data.</p> <p>The second option is to load the dataset as an SQLite database. For this purpose, we need to create such a database (again with the help of the <code>edictor</code> package). This merely requires you to add the option <code>--sqlite</code> to the command shown above. This command will create an SQLite database called <code>blumpanotacana.sqlite3</code>. </p> <pre><span style="font-family: andale mono, monospace">$ edictor wordlist --dataset=cldf/cldf-metadata.json --name=blumpanotacana --sqlite --addon="cognacy:cogid,alignment:alignment"</span></pre> <p>The advantage of using SQLite databases is that any modifications that you make to your database when editing it can be stored immediately, without requiring you to saving the data actively. When working with SQLite datasets, EDICTOR will also automatically create a version history that tracks each modification done manually to the data.</p> <p>However, the additional features and complexity that come with the SQLite database make it necessary to provide some information to EDICTOR how to use it. This is provided in the form of a configuration file. We recommend to store this kind of file in a new folder, which one can call <code>edictor/</code>. Within that directory, we can create a new folder where we will store all the necessary configuration files.</p> <pre class="wp-block-code">$ mkdir edictor</pre> <p>The configuration is stored in JSON format. for our example, you can create a file <code>config.json</code> that looks like the following:</p> <pre class="wp-block-code">{ "sqlite": "./", "links": [ { "url": "edictor.html", "data": { "file": "blumpanotacana", "remote_dbase": "blumpanotacana", "doculects": "Matses|Kakataibo|Amahuaca|Moseten|Isconahua|Tacana", "columns": "DOCULECT|FAMILY|CONCEPT|FORM|TOKENS|COGID|COGIDS|ALIGNMENT|PATTERNS|NOTE" }, "name": "Panoan (local)"} ] }</pre> <p>The <code>sqlite</code> argument specifies the location where the database is stored. The value <code>./</code> means that it is in the current folder. Make sure that this is the case, and move the <code>blumpanotacana.sqlite3</code> to the <code>edictor/</code> folder.</p> <pre>$ mv blumpanotacana.sqlite3 edictor/<br/>$ cd edictor</pre> <p>The information in <code>file</code> and <code>remote_dbase</code> needs to be adjusted according to your database file. The <code>doculects</code> attribute specifies which language varieties you want to edit, and the attribute in <code>columns</code> points to the columns you want to be able to work with when editing your data in EDICTOR. Lastly, the <code>name</code> specifies the name under which you can open the database in EDICTOR. Note that you can use this to create different subsets of the same SQLite database, e.g. by specifying a second access within the <code>links</code> list in the <code>config.json</code> file. This subset accesses the same local database file (<code>blumpanotacana.sqlite3</code>), but a different list of languages, creating a virtual subset of the data.</p> <pre class="wp-block-code">{ "sqlite": "./", "links": [ { "url": "edictor.html", "data": { "file": "blumpanotacana", "remote_dbase": "blumpanotacana", "doculects": "Matses|Kakataibo|Amahuaca|Moseten|Isconahua|Tacana", "columns": "DOCULECT|FAMILY|CONCEPT|FORM|TOKENS|COGID|COGIDS|ALIGNMENT|PATTERNS|NOTE" }, "name": "Panoan (local)"}, "url": "edictor.html", "data": { "file": "blumpanotacana", "remote_dbase": "blumpanotacana", "doculects": "Araona|Tacana|Cavinena", "columns": "DOCULECT|FAMILY|CONCEPT|FORM|TOKENS|COGID|COGIDS|ALIGNMENT|PATTERNS|NOTE" }, "name": "Tacanan (local)"} ] }</pre> <h2><strong>4 Saving Data</strong></h2> <p>Most importantly, if you decided to go for the TSV-based approach, you need to explicitly save and download your data. You can do so in the top right, where small icons indicate the respective functions. Alternatively, press CTRL + S (save) and then CTRL + E (download) on your keyboard. EDICTOR will always create a backup file with a time stamp that contains the data as it was before saving it the last time, and replace the content of the original file with the new content. This means, even if you decide that you work without SQLite, you can conveniently edit your data and go back in history to older versions. Since EDICTOR always stores a full version of your TSV file, however, the data may grow with time, so that you may want to actively curate your TSV file using some management system.</p> <p>If you are running EDICTOR3 with an SQLite file, this becomes unnecessary. All changes are stored directly in the database, and you do not need to save or download the changes you made. You can even check the history of your annotations, but currently, you can only do so by querying the BACKUP table of the SQLite database. This table will always store an old version of a given cell content as a triple of the identifier for the word, the name of the column, and the previous value, along with information on the time of modification and the user. Querying it is thus possible to check what happened, if problems emerge, but it requires some experience with SQL and SQLite databases.</p> <h2><strong>5 Outlook</strong></h2> <p>This tutorial has provided a basic introduction into how to run EDICTOR locally on your computer, even without a working internet connection. The full functionality of EDICTOR3 itself is described in List and van Dam (2024). A similar workflow as the one presented in this study can be used to to analyze one’s own data, or one can extract different combinations of data from the Lexibank repository to import them in EDICTOR.</p> <h2><strong>References</strong></h2> <p><span style="font-size: 10pt">Blum, F. and Barrientos, C. (2024). “A New Dataset with Phonological Reconstructions in CLDF” in Computer-Assisted Language Comparison in Practice, 6.1: 43–51 [first published on 21/06/2023]. <a href="https://doi.org/10.15475/calcip.2023.1.6">https://doi.org/10.15475/calcip.2023.1.6</a></span></p> <p><span style="font-size: 10pt"> </span></p> <p><span style="font-size: 10pt">Blum, F., Barrientos, C., Zariquiey, R., &amp; List, J.-M. (2024). A comparative wordlist for investigating distant relations among languages in Lowland South America. Scientific Data, 11(1). <a href="https://doi.org/10.1038/s41597-024-02928-7">https://doi.org/10.1038/s41597-024-02928-7</a></span></p> <p><span style="font-size: 10pt"> </span></p> <p><span style="font-size: 10pt">Forkel, R., List, J.-M., Greenhill, S. J., Rzymski, C., Bank, S., Cysouw, M., Hammarström, H., Haspelmath, M., Kaiping, G. A., &amp; Gray, R. D. (2018). Cross-Linguistic Data Formats, advancing data sharing and re-use in comparative linguistics. Scientific Data, 5(1), 1–10. <a href="https://doi.org/10.1038/sdata.2018.205">https://doi.org/10.1038/sdata.2018.205</a></span></p> <p><span style="font-size: 10pt"> </span></p> <p><span style="font-size: 10pt">List, J.-M., Forkel, R., Greenhill, S. J., Rzymski, C., Englisch, J., &amp; Gray, R. D. (2022). Lexibank, a public repository of standardized wordlists with computed phonological and lexical features. Scientific Data, 9(1), 1–16. <a href="https://doi.org/10.1038/s41597-022-01432-0">https://doi.org/10.1038/s41597-022-01432-0</a></span></p> <p><span style="font-size: 10pt"> </span></p> <p><span style="font-size: 10pt">List, J.-M, van Dam, K. P., &amp; Blum, F. (2024). EDICTOR 3. An Interactive Tool for Computer-Assisted Language Comparison [Software Tool, Version 3.1]. MCL Chair at the University of Passau. <a href="https://edictor.org">https://edictor.org</a></span></p> <p>List, J.-M. and K. van Dam (2024): Computer-Assisted Language Comparison with EDICTOR 3 [Invited Paper]. In: Proceedings of the 5th Workshop on Computational Approaches to Historical Language Change. 1-11. <a href="https://aclanthology.org/2024.lchange-1.1">https://aclanthology.org/2024.lchange-1.1 </a></p> <p>List, J.-M. and R. Forkel (2024): LingRex: Linguistic reconstruction with LingPy [Software, Version 1.4.2]. Max Planck Institute for Evolutionary Anthropology: Leipzig. <a href="https://pypi.org/project/lingrex">https://pypi.org/project/lingrex </a></p> <p>List, J.-M. and R. Forkel (2023): LingPy. A Python library for quantitative tasks in historical linguistics [Software Library, Version 2.6.13]. Version 2.6.13. MCL Chair at the University of Passau: Passau. <a href="https://pypi.org/project/lingpy">https://pypi.org/project/lingpy </a></p> <p><span style="font-size: 10pt"> </span></p> <p><span style="font-size: 10pt">Snee, D. (2024). “Using CLDFBench and PyLexibank on Windows” in Computer-Assisted Language Comparison in Practice, 7.2: 103-109 [first published on 18/12/2024], <a href="https://doi.org/10.15475/calcip.2024.2.6">https://doi.org/10.15475/calcip.2024.2.6</a></span></p> <p><span style="font-size: 10pt"> </span></p> <p><span style="font-size: 10pt">Wu, M.-S., Schweikhard, N. E., Bodt, T. A., Hill, N. W. &amp; List, J.-M. (2020). Computer-Assisted Language Comparison: State of the Art. Journal of Open Humanities Data, 6(1), 2. <a href="https://doi.org/10.5334/johd.12">https://doi.org/10.5334/johd.12</a></span></p> <p> </p> <div class="wpcp" style="background-color: aliceblue;border: 1px solid black;border-radius: 5px;padding: 5px"> <p><b>Cite this article as:</b> Blum, Frederic, “How to Run EDICTOR 3 Locally,” in <em>Computer-Assisted Language Comparison in Practice</em>, 8.1: 1-8 [first published on 27/01/2025], URL: <a href="https://calc.hypotheses.org/8143">https://calc.hypotheses.org/8143</a>, DOI: <a href="https://doi.org/10.15475/calcip.2025.1.1">10.15475/calcip.2025.1.1</a>.</p> <p><b>Download the article as PDF:</b> <a href="https://ojs3.uni-passau.de/index.php/calcip/article/view/355/327">calcip-08-1-1.pdf</a><br/><br/><b>Grant Information:</b> This project has received funding from the European Research Council (ERC) under the European Union’s Horizon Europe research and innovation programme (Grant agreement No. <a href="https://doi.org/10.3030/101044282">101044282</a>). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.</p> <p><b>Copyright information</b>: This is an open access article distributed under the terms of the <a href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution License</a>, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.</p> </div> <hr/><div class="citation"><p><span class="citation-title">OpenEdition suggests that you cite this post as follows:</span><br/>Frederic Blum (January 27, 2025). How to Run EDICTOR 3 Locally. <em>Computer-Assisted Language Comparison in Practice</em>. Retrieved January 28, 2025 from https://doi.org/10.58079/1363c</p></div> <hr/> <ul class="soc"> <li><a class="x share-hypotheses" href="https://twitter.com/share?text=How+to+Run+EDICTOR+3+Locally&amp;url=https%3A%2F%2Fcalc.hypotheses.org%2F8143&amp;via=hypothesesorg"><svg aria-hidden="true" class="svg-inline--fa fa-x-twitter" data-fa-i2svg="" data-icon="x-twitter" data-prefix="fab" focusable="false" role="img" viewbox="0 0 512 512" xmlns="http://www.w3.org/2000/svg"><path d="M389.2 48h70.6L305.6 224.2 487 464H345L233.7 318.6 106.5 464H35.8L200.7 275.5 26.8 48H172.4L272.9 180.9 389.2 48zM364.4 421.8h39.1L151.1 88h-42L364.4 421.8z" fill="currentColor"></path></svg></a></li> <li><a class="facebook share-hypotheses" href="https://www.facebook.com/sharer.php?title=How+to+Run+EDICTOR+3+Locally&amp;u=https%3A%2F%2Fcalc.hypotheses.org%2F8143"><svg aria-hidden="true" class="svg-inline--fa fa-facebook-f" data-fa-i2svg="" data-icon="facebook-f" data-prefix="fab" focusable="false" role="img" viewbox="0 0 320 512" xmlns="http://www.w3.org/2000/svg"><path d="M279.14 288l14.22-92.66h-88.91v-60.13c0-25.35 12.42-50.06 52.24-50.06h40.42V6.26S260.43 0 225.36 0c-73.22 0-121.08 44.38-121.08 124.72v70.62H22.89V288h81.39v224h100.17V288z" fill="currentColor"></path></svg></a></li> <li><a class="linkedin share-hypotheses" href="https://www.linkedin.com/shareArticle?title=How+to+Run+EDICTOR+3+Locally&amp;url=https%3A%2F%2Fcalc.hypotheses.org%2F8143&amp;mini=true"><svg aria-hidden="true" class="svg-inline--fa fa-linkedin" data-fa-i2svg="" data-icon="linkedin" data-prefix="fab" focusable="false" role="img" viewbox="0 0 448 512" xmlns="http://www.w3.org/2000/svg"><path d="M416 32H31.9C14.3 32 0 46.5 0 64.3v383.4C0 465.5 14.3 480 31.9 480H416c17.6 0 32-14.5 32-32.3V64.3c0-17.8-14.4-32.3-32-32.3zM135.4 416H69V202.2h66.5V416zm-33.2-243c-21.3 0-38.5-17.3-38.5-38.5S80.9 96 102.2 96c21.2 0 38.5 17.3 38.5 38.5 0 21.3-17.2 38.5-38.5 38.5zm282.1 243h-66.4V312c0-24.8-.5-56.7-34.5-56.7-34.6 0-39.9 27-39.9 54.9V416h-66.4V202.2h63.7v29.2h.9c8.9-16.8 30.6-34.5 62.9-34.5 67.2 0 79.7 44.3 79.7 101.9V416z" fill="currentColor"></path></svg></a></li> <li><a class="mastodon share-hypotheses" href="https://mastodonshare.com?text=How+to+Run+EDICTOR+3+Locally&amp;url=https%3A%2F%2Fcalc.hypotheses.org%2F8143"><svg class="svg-inline--fa fa-mastodon" viewbox="0 0 448 512" xmlns="http://www.w3.org/2000/svg"><path d="M433 179.1c0-97.2-63.7-125.7-63.7-125.7-62.5-28.7-228.6-28.4-290.5 0 0 0-63.7 28.5-63.7 125.7 0 115.7-6.6 259.4 105.6 289.1 40.5 10.7 75.3 13 103.3 11.4 50.8-2.8 79.3-18.1 79.3-18.1l-1.7-36.9s-36.3 11.4-77.1 10.1c-40.4-1.4-83-4.4-89.6-54a102.5 102.5 0 0 1 -.9-13.9c85.6 20.9 158.7 9.1 178.8 6.7 56.1-6.7 105-41.3 111.2-72.9 9.8-49.8 9-121.5 9-121.5zm-75.1 125.2h-46.6v-114.2c0-49.7-64-51.6-64 6.9v62.5h-46.3V197c0-58.5-64-56.6-64-6.9v114.2H90.2c0-122.1-5.2-147.9 18.4-175 25.9-28.9 79.8-30.8 103.8 6.1l11.6 19.5 11.6-19.5c24.1-37.1 78.1-34.8 103.8-6.1 23.7 27.3 18.4 53 18.4 175z" fill="currentColor"></path></svg></a></li> </ul>

**Calcipotriene** is a vitamin D analog widely prescribed for the topical treatment of **plaque psoriasis**. It works by slowing down abnormal skin cell growth and restoring healthier skin turnover. Dermatologists favor calcipotriene because it’s **non-steroidal** , effective for long-term use, and suitable for sensitive areas when used correctly. Available under brand names like **Dovonex** , it comes as creams, ointments, foams, and scalp solutions. ## **What Is Calcipotriene?** **Calcipotriene** (also called calcipotriol) is a **synthetic vitamin D3 analog** used topically to treat **Psoriasis** —especially plaque-type psoriasis. It regulates skin cell production and reduces inflammation, scaling, and thickness of plaques. **Key takeaway:** Calcipotriene targets the root problem in psoriasis—**overactive skin cell growth** —without the risks associated with long-term steroid use. ## **How Calcipotriene Works** Calcipotriene binds to **vitamin D receptors (VDRs)** in skin cells (keratinocytes). This action: * **Slows keratinocyte proliferation** * **Promotes normal cell differentiation** * **Reduces plaque thickness and scaling** Unlike topical steroids that suppress inflammation broadly, calcipotriene **normalizes skin biology** , making it ideal for maintenance therapy. **Related concepts you should know:** * Vitamin D analogs * Plaque psoriasis * Topical corticosteroids * Hypercalcemia ## **Uses of Calcipotriene** ### **Approved Uses** * Mild to moderate **plaque psoriasis** (body) * Scalp psoriasis (solution or foam) ### **Off-Label / Adjunct Uses** * Maintenance therapy after steroid induction * Combination therapy with topical steroids * Chronic plaque control to reduce steroid dependence ## **Calcipotriene Dosage & How to Apply** ### **Standard Dosage** * **Adults:** Apply a thin layer **once or twice daily** * **Maximum weekly dose:** Usually **≤100 g/week** (to avoid calcium imbalance) ### **Step-by-Step Application Guide** **1. Clean & dry** the affected area 2. Apply a **thin layer only on plaques** **3. Do not rub aggressively** 4. Wash hands after application 5. Avoid face, eyes, mouth unless directed **Timeline:** * Improvement begins in **1–2 weeks** * Peak results by **6–8 weeks** ** ** ## **Available Formulations** Form| Best For ---|--- Cream| Normal skin areas Ointment| Thick plaques Foam| Large areas, easy spread Scalp solution| Hair-bearing areas ## **Calcipotriene vs Other Psoriasis Treatments** ### **Calcipotriene vs Topical Steroids** Feature| Calcipotriene| Steroids ---|---|--- Long-term safety| ✅ High| ❌ Limited Skin thinning| ❌ No| ✅ Yes Onset speed| Moderate| Fast Maintenance use| Ideal| Not ideal ### **Calcipotriene + Betamethasone (Combo Therapy)** Combination products (e.g., vitamin D analog + steroid) provide: * Faster relief * Better plaque clearance * Reduced steroid exposure ## **Side Effects of Calcipotriene** ### **Common (Mild)** * Skin irritation * Burning or itching * Redness ### **Rare but Important** **Hypercalcemia** (with excessive use) **Stop and consult a doctor if:** * Severe irritation occurs * Symptoms worsen after 2 weeks * You apply more than prescribed amounts ## **Who Should Avoid or Use With Caution** * Patients with calcium metabolism disorders * Severe renal disease * Pregnancy (use only if prescribed) * Children (dose-restricted) ## **Common Mistakes** ❌ **Using too much** → ✅ Stick to prescribed quantity ❌ **Applying on face/folds** → ✅ Use only if doctor allows ❌ **Expecting overnight results** → ✅ Give it 4–8 weeks ❌ **Stopping too early** → ✅ Complete the treatment cycle ## **Real-World Usage Example** **Scenario:** A 28-year-old with stable plaque psoriasis on elbows and knees. **Plan:** * Calcipotriene cream **once daily** * Moisturizer twice daily * Reassessment after **6 weeks** **Outcome:** Significant plaque thinning, reduced scaling, no steroid dependence. ## **FAQs** #### **Is calcipotriene a steroid?** No. It is a **vitamin D analog** , not a steroid. #### **Can calcipotriene be used long-term?** Yes, when used within recommended limits. #### **Can I use it on my face?** Generally **no** , unless specifically prescribed. #### **How long does calcipotriene take to work?** Visible improvement in **2 weeks** , full effect by **6–8 weeks**. #### **Can calcipotriene cure psoriasis?** No cure—but it **controls symptoms effectively**. #### **Is calcipotriene safe in pregnancy?** Use only if clearly needed and prescribed. #### **Can it be used with moisturizers?** Yes. Apply moisturizer **after** calcipotriene absorbs.

Formal notations for the handling of sound laws in historical linguistics have been employed by linguists for a long time. So far, however, linguists have barely tried to describe their notation pract...

This study presents a Croatian frequency-derived wordlist mapped to Concepticon concept sets, based on the most frequent nouns, verbs, and adjectives extracted from the hrWaC web corpus. The resulting...

This tutorial shows how to build beautiful maps using polygon data of language distributions and the geospatial software QGIS. For this purpose, I show in the first part how to extract a list of Glott...

The fourth version of the Database of Cross-Linguistic Colexifications (https://clics.clld.org) was published last year. Now, we launched the accompanying web application that presents the data for in...

This study gives a brief overview on formal aspects of etymological analysis, by providing a modified workflow for the classical comparative method in historical language comparison. This workflow is ...

This study illustrates how semantic embeddings can be added to and retrieved from NoRaRe. By that, it provides a template for handling vector data and makes popular methodology in semantic modeling av...

This study describes the process of digitizing legacy materials into a computer-readable format for the purposes of computational typology and computer-assisted historical reconstruction. It presents ...