Added cardinal semiotic class for te ITN#451
Open
jaya-TN wants to merge 12 commits into
Open
Conversation
Signed-off-by: jaya-TN <seruganti@nvidia.com>
Signed-off-by: jaya-TN <seruganti@nvidia.com>
jaya-TN
marked this pull request as ready for review
July 7, 2026 10:51
jaya-TN
changed the title
folivoramanh
requested changes
Jul 21, 2026
folivoramanh
left a comment
folivoramanh
left a comment
Collaborator
There was a problem hiding this comment.
- add sparrowhawk file and run sh test (we need to make sure both pytest and sparrowhawk pass)
- add te to jenkins file
- try to optimize cardinal logic (you can look up other languages eg. Hindi, Spanish, Portuguese, etc... for both TN and ITN) to optimize graph logic, too much looping could lead to slow runtime
- do te ITN support english number (eg. 123456789?) if yes, can you add test cases for both te number and en number?
Collaborator
There was a problem hiding this comment.
why we need both hundred.tsv and hundred_words.tsv?
Collaborator
There was a problem hiding this comment.
blank files, please modify or delete it
| | (graph_ties_prefix + delete_space + graph_digit) | ||
| | self.graph_single_digit_with_zero | ||
| ) | ||
| hundred_words = [row[0] for row in load_labels(get_abs_path("data/numbers/hundred_words.tsv"))] |
| MINUS = pynini.union("ఋణాత్మక", "మైనస్").optimize() | ||
|
|
||
|
|
||
| def integer_to_telugu(n: int) -> str: |
Collaborator
There was a problem hiding this comment.
why we need a customize function for join string?
…nkins Address review feedback on Telugu ITN cardinals: - Rewrite cardinal grammar using the Hindi/Marathi fixed-width place-value chaining approach, cutting FST build from ~9.5min to <1s. - Hardcode the closed-class magnitude words (hundred/thousand/lakh/crore) inline as in the hi/mr grammars and remove the redundant *_words.tsv data files (hundred_words, thousand_words, lakh_words, crore_words); also drop unused hundred.tsv, thousands.tsv and blank labels_exception.tsv. - Add configurable numeral_system output (Telugu digits or English/Arabic digits) threaded through ClassifyFst, with test_cases_cardinal_en.txt and a dedicated pytest class. - Fix tab delimiter in digit_people.tsv; remove custom integer_to_telugu. - Add Sparrowhawk ITN test script and te stages to the Jenkinsfile, and register te in pynini_export.py.
Per review, do not mix te and te_en inside one grammar. Remove the numeral_system parameter that rendered Telugu words as Arabic digits. Telugu ITN now always outputs Telugu digits, and English/Arabic numerals already present in the input pass through unchanged (123456789 -> 123456789), mirroring the hi vs hi_en separation. The en test file now covers digit pass-through rather than word-to-Arabic conversion.
English/Arabic numbers just pass through, so they are ordinary cardinal cases rather than a separate concern. Fold the digit pass-through cases into test_cases_cardinal.txt and drop the redundant test_cases_cardinal_en.txt file and its dedicated test class.
Align Telugu ITN with the Hindi ITN grammar: convert only the number words in a phrase and leave surrounding non-number words untouched (code-switching), e.g. "ఆరు మంది ఆటగాళ్లు బయటకు" -> "౬ మంది ఆటగాళ్లు బయటకు". Remove the ASCII numeral pass-through path and the corresponding 42~42 / 123456789 test cases, since hi's cardinal has no such behaviour.
Mirror the Hindi ITN data/numbers layout by keeping the magnitude words as single-column word lists (hundred.tsv, thousands.tsv) so future Telugu semiotic classes (decimal/measure/money) can reuse them, just as hi does. The cardinal grammar continues to hardcode these closed-class words exactly like hi's cardinal.
- Support an explicit "ఒక" (one) before a bare magnitude, e.g. ఒక కోటి, ఒక లక్ష, ఒక వెయ్యి, ఒక వంద, which previously were not recognized. - Add నూట (combining hundred form) to hundred.tsv. - Extend cardinal test cases with edge cases: standalone magnitudes (నూరు/వెయ్యి/లక్ష/కోటి), counting-people forms (ఇద్దరు/ముగ్గురు/పది మంది), bare single digits, and the ఒక+magnitude forms.
Previously a case suffix (కి/కు/లో/తో/కోసం) following a number was deleted, losing meaning (ఇరవైకి -> ౨౦) and swallowing spaced words (ఇరవై లో ఐదు -> ౨౦ ౫). Now the suffix is preserved and attached to the numeral, with glued and spaced inputs normalising to the same natural written form (ఇరవైకి / ఇరవై కి -> ౨౦కి). Add test coverage for the suffix cases.
Only a glued case suffix is absorbed by the cardinal (ఇరవైకి -> ౨౦కి) so the number is still recognised; a space-separated suffix keeps its space via the word tagger (ఇరవై తో -> ౨౦ తో, ఇరవై మూడు కి -> ౨౩ కి). Update tests accordingly.
Forbid the cardinal number from consuming a trailing space so a spaced
suffix (e.g. "వంద కి") falls through to its own word token and keeps the
space ("౧౦౦ కి"), while a glued suffix ("వందకి" -> "౧౦౦కి") is still
absorbed. Fixes the residual gluing on numbers ending on a magnitude word.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
This PR adds support for the Telugu Inverse Text Normalization (ITN) Cardinal semiotic class.
Before your PR is "Ready for review"
Pre checks:
git commit -sto sign.pytestor (if your machine does not have GPU)pytest --cpufrom the root folder (given you marked your test cases accordingly@pytest.mark.run_only_on('CPU')).bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...pytestand Sparrowhawk here.__init__.pyfor every folder and subfolder, includingdatafolder which has .TSV files?Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.to all newly added Python files?Copyright 2015 and onwards Google, Inc.. See an example here.try import: ... except: ...) if not already done.PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.