Datasets
Dataset
indic-hplt-v2
by ashtok897
Indic HPLT v2 A multilingual pretraining corpus of 34,605,630 documents (~25. 5B estimated tokens, ~218 GB raw JSONL) across 13 Indic languages and English, built from HPLT Monolingual v3 high-quality web crawl data.
CC0-1.0text-generation 596 310M < n < 100M rows
published 19 May 2026
Preview
First 5 rows of default / train, from the Hugging Face dataset viewer.
| collection | prob | doc_id | lang | score | text | char_count | word_count | web-register | url |
|---|---|---|---|---|---|---|---|---|---|
| text/html | [1,0,0] | ta_2500958 | ta | 24872 | சங்கரன்கோவிலில் பைக்கை திருடி கிணற்றில் வீசிய 3 பேரை போலீஸாா் கைது செய்தனா். சங்கரன்கோவில், ஓடைத்தெருவைச் சோ்ந்தவா் ஷேக் சா்புதீன்(25). சாலையோர உணவகம் நடத்தி வருகிறாா். இவா், தனது நண்பா்களான பாரதியாா் 3 ஆம் தெரு காா்த்திகேயன் (21), சதீஸ்குமாா்(27),கோமதியாபுரம் புது 2 ஆம் தெரு ஐயப்பன் (20) உள்ளிட்ட … | 940 | 109 | {"MT":0.077,"LY":0.073,"SP":0.079,"ID":0.082,"NA":0.868,"HI":0.084,"IN":0.124,"OP":0.094,"IP":0.084,"it":0.059,"ne":0.837,"sr":0.107,"nb":0.123,"re":0.08,"en":0.061,"ra":0.064,"dtp":0.088,"fi":0.053,"lt":0.071,"rv":0.076,"ob":0.079,"rs":0.089,"av":0.057,"ds":0.068,"ed":0.089} | https://www.dinamani.com/all-editions/edition-thirunelveli/tenkasi/2021/aug/27/theft-at-sankarankoil-3-arrested-3687563.html |
| text/html | [1,0,0] | ta_1587073 | ta | 35093 | சென்னை: தமிழகம் முழுவதும் உள்ள கிறிஸ்தவ தேவாலயங்களில் இன்று (ஏப்.10) ஈஸ்டர் பண்டிகையின் முக்கிய நிகழ்வான குருத்தோலை ஞாயிறை முன்னிட்டு, குருத்தோலை பவனி நடைபெற்றது. கிறிஸ்தவ மக்களின் முக்கிய பண்டிகைகளில் ஒன்றான ஈஸ்டர் பண்டிகை கடந்த மாதம் சாம்பல் புதனுடன் தொடங்கியது. இதிலிருந்து 7 வாரங்கள் கிறிஸ்தவ மக… | 1470 | 164 | {"MT":0.052,"LY":0.073,"SP":0.081,"ID":0.065,"NA":0.796,"HI":0.063,"IN":0.161,"OP":0.133,"IP":0.131,"it":0.056,"ne":0.693,"sr":0.086,"nb":0.129,"re":0.075,"en":0.059,"ra":0.07,"dtp":0.095,"fi":0.041,"lt":0.067,"rv":0.065,"ob":0.074,"rs":0.151,"av":0.051,"ds":0.071,"ed":0.079} | https://www.hindutamil.in/news/tamilnadu/786715-palm-sunday-rally-held-all-over-tamil-nadu-christians-participated.html?utm_source=site&utm_medium=art_editor_choice&utm_campaign=art_editor_choice |
| text/html | [0.9981,0.0004,0.0001] | en_1174611 | en | 28753 | Introduction Edwin Friedman’s book, A Failure of Nerve, delves into the complexities of human relationships and the challenges that can arise within them. As I revisit this book for the second time, I’ve decided to focus on a particular paragraph that has captured my attention. In this paragraph, F… | 7442 | 1243 | {"MT":0.05,"LY":0.057,"SP":0.091,"ID":0.061,"NA":0.156,"HI":0.135,"IN":0.404,"OP":0.634,"IP":0.099,"it":0.082,"ne":0.102,"sr":0.059,"nb":0.1,"re":0.058,"en":0.062,"ra":0.158,"dtp":0.182,"fi":0.082,"lt":0.054,"rv":0.109,"ob":0.252,"rs":0.254,"av":0.209,"ds":0.077,"ed":0.08} | https://blogs.georgefox.edu/dlgp/friedman-the-second-story/ |
| text/html | [1,0,0] | pa_0272041 | pa | 9554 | ਪੰਜਾਬ ਐਗਰੀਕਲਚਰਲ ਯੂਨੀਵਰਸਿਟੀ, ਲੁਧਿਆਣਾ ਵਿੱਚ ਲੰਮਾ ਸਮਾਂ ਸੰਪਾਦਕ ਪੰਜਾਬੀ ਵਜੋਂ ਸੇਵਾ ਨਿਭਾ ਚੁੱਕੇ ਸਾਡੇ ਪੁਰਖ਼ਿਆਂ ਵਿੱਚ ਸ: ਹਰਬੰਸ ਸਿੰਘ ਦਾ ਨਾਮ ਨਵੇਂ ਪ੍ਰਸੰਗ ਵਿੱਚ ਤੁਹਾਨੂੰ ਬਹੁਤ ਓਪਰਾ ਜਾਪੇਗਾ। ਉਨ੍ਹਾਂ ਦੇ ਜੀਵਨ ਵਿਹਾਰ ਵਿੱਚ ਸ਼ਾਇਰੀ ਦਾ ਦਖ਼ਲ ਤਾਂ ਸੀ ਪਰ ਉਨ੍ਹਾਂ ਦਾ ਸਿਰਜਕ ਆਪਾ ਇਸ ਯੂਨੀਵਰਸਿਟੀ ਦੀ ਸੇਵਾ ਦੌਰਾਨ ਲਗਪਗ ਲਗਾਤਾਰ ਅਗਿਆਤਵਾਸ ਵਿੱਚ ਹੀ ਰਿ… | 5162 | 1052 | {"MT":0.143,"LY":0.611,"SP":0.162,"ID":0.118,"NA":0.416,"HI":0.076,"IN":0.25,"OP":0.159,"IP":0.097,"it":0.145,"ne":0.087,"sr":0.088,"nb":0.301,"re":0.065,"en":0.122,"ra":0.135,"dtp":0.202,"fi":0.07,"lt":0.099,"rv":0.13,"ob":0.148,"rs":0.109,"av":0.076,"ds":0.051,"ed":0.141} | http://punjabilekhak.com/details.php?id=1890 |
| text/html | [0.9999,0.0001,0] | or_1268272 | or | 25540 | ବାରିପଦା : ଆଗାମୀ ୨୬ ତାରିଖରେ ଆସିବାକୁ ଥିବା ଘୂର୍ଣ୍ଣିଝଡ \”ୟସ\”ର ମୁକାବିଲା ପାଇଁ ରାଜ୍ୟ ସରକାରଙ୍କ ପକ୍ଷରୁ ତତ୍ପରତା ପୂର୍ବକ ସମସ୍ତ ଜିଲ୍ଲାକୁ ସତର୍କ ସୂଚନା ଜାରୀ କରାଯାଇଛି । କ୍ଷତିଗ୍ରସ୍ତ ହେବାକୁ ଥିବା ଜିଲ୍ଲା ମାନଙ୍କ ମଧ୍ୟରେ ମୟୂରଭଞ୍ଜ ଜିଲ୍ଲାରେ ଜିଲ୍ଲା ପ୍ରଶାସନ ପକ୍ଷରୁ ପ୍ରସ୍ତୁତି ବୈଠକ ଅନୁଷ୍ଠିତ ହୋଇଯାଇଛି । ରାଜ୍ୟ ସରକାରଙ୍କ ସ୍ୱତନ୍ତ୍ର ର… | 1440 | 197 | {"MT":0.071,"LY":0.068,"SP":0.09,"ID":0.071,"NA":0.875,"HI":0.081,"IN":0.133,"OP":0.084,"IP":0.109,"it":0.062,"ne":0.843,"sr":0.105,"nb":0.123,"re":0.071,"en":0.052,"ra":0.071,"dtp":0.092,"fi":0.055,"lt":0.072,"rv":0.069,"ob":0.075,"rs":0.078,"av":0.065,"ds":0.075,"ed":0.084} | https://sambad.in/district/preparation-meeting-for-cyclone-667603/ |
Cite this
CC0-1.0tabulartextCite
Tags
task_categories:text-generationtask_categories:fill-masksource_datasets:hplt-project/hplt_monolingual_v3language:hilanguage:bnlanguage:talanguage:telanguage:mrlanguage:gulanguage:knlanguage:mllanguage:palanguage:urlanguage:nelanguage:orlanguage:aslanguage:enlicense:cc0-1.0size_categories:10M<n<100Mformat:parquetmodality:tabularmodality:textlibrary:datasetslibrary:dasklibrary:polarslibrary:mlcroissantarxiv:2511.01066region:usindicmultilingualpretraininghplt