The central problem in computing with Sanskrit is segmentation: because sounds change where words meet, a line of text must be split back into words before it can be searched, tagged, or translated. Rule-based tools such as Gérard Huet's Sanskrit Heritage Reader list every possible split, and a 2020 study on 21,127 test expressions raised the share of cases with the correct answer ranked first from 53.51 percent to 89.27 percent by adding a probability score. Neural models trained on a corpus of more than 560,000 hand-checked sentences now handle the same task without hand-written rules.
Key takeaways
- Sandhi, the set of sound changes at word and compound boundaries, means that one written string can often be split into words in several ways.
- The Sanskrit Heritage Reader, a finite-state tool, outputs all segmentations allowed by Pāṇini's sandhi rules, which can be a large list for long sentences.
- Re-ranking that list with word and transition probabilities put the correct split first in 89.27 percent of 21,127 test cases, up from 53.51 percent.
- A 2018 EMNLP paper released 561,596 sentences with validated splits and trained neural models that need no external linguistic resources.
- A 2022 Cambridge thesis on Pāṇini's rule conflicts was publicized with computing claims that the university's own release does not show as implemented.
Why Sanskrit text is hard to split
Sanskrit was a common language of religious, scientific, and literary writing in ancient India. Text production began in the second millennium BCE and has continued to the present, and a 19th-century catalog by Aufrecht recorded more than 40,000 Sanskrit texts known at the time, which the Hellwig and Nehrdich paper describes as only a small part of the extant literature. Tools that process this writing automatically have to deal with its sound rules first.
The obstacle is sandhi. In Sanskrit these sound changes are recorded in writing and apply both between words and between the members of a compound. Vowels may merge, as when mahā and utsava join to give mahotsava, and consonants may change to match their neighbors. A reader or program that sees only mahotsava has to work out where the boundary was and which sounds were altered.
The rule-based approach: listing every legal split
Gérard Huet at the French research institute Inria built the Sanskrit Heritage Platform, whose Reader segments Sanskrit text. In a 2020 paper, Sriram Krishnan and Amba Kulkarni describe its segmenter as a finite state transducer that returns all possible segmentations allowed by Pāṇini's sandhi rules, after checking that each word form, transition, and phase is feasible. Huet and Kulkarni also described a collection of Sanskrit web services, covering lexicons, morphological analysis, segmentation, and parsing, at the COLING 2014 conference.
Completeness has a cost. Morphological checks alone can leave very large numbers of solutions for long sentences, and the same sentence may be split several ways that differ only in grammatical labels. In the 2020 paper, the example rāmovanaṅgacchati yields four solutions that differ mainly in such details.
Adding probabilities: results on 21,127 expressions
Krishnan and Kulkarni changed the Reader in two ways. They ignored most of the grammatical phase labels, merging solutions that share the same word-level split, and they ranked the remaining candidates by the product of word and transition probabilities estimated from corpus frequencies. The table gives their reported results on 21,127 sandhied expressions.
| Measure | Original Reader | Modified Reader |
|---|---|---|
| Correct split ranked first | 53.51 percent | 89.27 percent |
| Correct split in top three | 75.12 percent | 98.3 percent |
| Total candidate solutions | 240,942 | 146,610 |
| Average solutions per input | 11.4 | 6.94 |
| Recall (correct split present at all) | 92.27 percent | 92.27 percent |
Recall did not change because the paper alters only the order of candidates, not which ones are produced. About 7.7 percent of expressions therefore still lacked the correct split in both versions.
The data-driven approach: neural models and a shared dataset
Oliver Hellwig and Sebastian Nehrdich presented neural network models at EMNLP in 2018 that split compounds and undo sandhi in one step. The models combine convolutional layers, which capture nearby sound patterns, with recurrent layers, which capture wider context, and they learn only from pairs of raw and segmented text. No hand-built features or external dictionaries are required. The authors also released a dataset of 561,596 sentences containing 4,171,682 tokens, with manually validated splits, drawn mostly from epic and scientific texts such as medicine, alchemy, and astronomy.
The data came from the Digital Corpus of Sanskrit, a lemmatized collection that Hellwig announced in February 2010 with more than 2,500,000 manually annotated entries. That corpus records the grammatical analysis of each word but not the points where strings were split or which sandhi rules applied, so an earlier project re-analyzed 560,000 of its sentences with the Heritage Reader to produce training splits. Hellwig and Nehrdich also report that a Transformer model performed almost on par with their model on one test, while needing 55 hours of training against two.
What a claim about decoding Pāṇini does and does not show
Pāṇini's Aṣṭādhyāyī is a grammar of roughly 4,000 rules, according to a University of Cambridge press release, and the rule-based tools above follow its sandhi system. Rishi Rajpopat's doctoral thesis on Pāṇini was published on December 15, 2022, and a University of Cambridge release states that it resolved how the grammar handles cases in which two rules apply at once, by reading the metarule 1.4.2 as favoring the rule that applies to the right-hand part of a word. The release says the grammar then derives words such as mantra and guru with almost no exceptions, and that it might be taught to computers.
These are the author's claims and the university's framing. The release reports no working implementation and no independent test, and the words it uses to praise the result come from unnamed experts, so the computing benefit remains a possibility, not a documented outcome.
FAQ
What is sandhi in Sanskrit?
Sandhi means sound changes at the boundaries of morphemes and words. In Sanskrit they are written down and apply inside compounds as well, so text must be segmented before its words can be looked up.
Can AI read Sanskrit manuscripts?
The sources describe tools that take typed or transliterated text. They do not cover reading images of manuscripts, which is a separate problem.
Is a neural model better than the rule-based Reader?
The sources do not support a single ranking. The Reader guarantees only splits that obey sandhi rules and misses the correct split in about 7.7 percent of cases in the 2020 test, while the neural models learn from data and inherit inconsistencies in how compounds were split in their training set.
Did the Cambridge thesis let computers learn Pāṇini's grammar?
Not according to the release. It calls this a potential application and describes no implementation.
Related reading
- AI in English language learning
- How AI affects the English language
- Optimization problems in AI
- GPT-3 and language models
More in AI Basics.
Last verified: October 2026
Sources
- Sanskrit Segmentation Revisited, Krishnan S, Kulkarni A, arXiv:2005.06383, 2020
- Sanskrit Word Segmentation Using Character-level Recurrent and Convolutional Neural Networks, Hellwig O, Nehrdich S, EMNLP 2018
- Sanskrit Linguistics Web Services, Huet G, Kulkarni A, COLING 2014 System Demonstrations
- Digital Sanskrit corpus and dictionary, Hellwig O, announcement on the INDOLOGY mailing list, February 12, 2010
- Solving grammar's greatest puzzle, University of Cambridge, December 15, 2022
- Sandhi, reference article used only for the mahotsava example