ଓOpenOdia
Datasets

Dataset

hplt

by CoRover

HPLT Indic Language Corpus This repository contains selected Indic-language data from the HPLT Monolingual Dataset 3. 0, prepared and hosted by CoRover for large-scale generative language model pretraining and multilingual NLP research.

CC0-1.0text-generation 145 0

published 25 Aug 2026

Open on Hugging Face

Preview

First 5 rows of default / ben_Beng, from the Hugging Face dataset viewer.

fosrsuctsdecrawl_idlangprobtextxmlhtml_langcluster_sizeseg_langsidfilterpiidoc_scoresweb-register
./segments/1590348517506.81/warc/CC-MAIN-20200606155701-20200606185701-00588.warc.gz8787245561523977115https://www.sylhetsomachar.com/%E0%A6%85%E0%A6%A8%E0%A7%8D%E0%A6%A4%E0%A6%B0%E0%A7%8D%E0%A6%AD%E0%A7%81%E0%A6%95%E0%A7%8D%E0%A6%A4%E0%A6%BF%E0%A6%AE%E0%A7%82%E0%A6%B2%E0%A6%95-%E0%A6%89%E0%A6%A8%E0%A7%8D%E0%A6%A8%E0%A7%9F%E0%A6%A8-%E0%A6%A8%E0%A7%80%E0%A6%A4%E0%A6%BF-%E0%A6%95%E0%A7%8C%E0%A6%B6%E0%…text/html2020-06-06T18:19:48utf-8CC-MAIN-2020-24["ben_Beng","mni_Beng","npi_Deva"][1,0,0]অন্তর্ভুক্তিমূলক উন্নয়ন নীতি কৌশল ও ডিজিটাল অন্তর্ভুক্তি বাংলাদেশ বিশ্বে দ্রুত বর্ধমান অর্থনীতির একটি দেশ। অনেক আগেই ৬ শতাংশ অর্থনৈতিক প্রবৃদ্ধি অর্জনের বৃত্ত থেকে বেরিয়ে এসেছে। বর্তমানে দেশের অর্থনৈতিক প্রবৃদ্ধির হার ৮ শতাংশের উপরে। দ্রুত প্রবৃদ্ধি অর্জনের এই সাফল্যকে আন্তর্জাতিক বিভিন্ন সংস্থা …<doc fingerprint="fbb69e11a5ec810"> <main> <head rend="h1">অন্তর্ভুক্তিমূলক উন্নয়ন নীতি কৌশল ও ডিজিটাল অন্তর্ভুক্তি</head> <p>বাংলাদেশ বিশ্বে দ্রুত বর্ধমান অর্থনীতির একটি দেশ। অনেক আগেই ৬ শতাংশ অর্থনৈতিক প্রবৃদ্ধি অর্জনের বৃত্ত থেকে বেরিয়ে এসেছে। বর্তমানে দেশের অর্থনৈতিক প্রবৃদ্ধির হার …["en"]1["ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","…1c0833da58e2a531cfb4eb0ebc497f7akeep[][10,10,10,10,10,10,10,10,10,10]{"MT":0.06,"LY":0.071,"SP":0.118,"ID":0.064,"NA":0.208,"HI":0.045,"IN":0.267,"OP":0.425,"IP":0.401,"it":0.064,"ne":0.162,"sr":0.076,"nb":0.082,"re":0.043,"en":0.056,"ra":0.129,"dtp":0.118,"fi":0.06,"lt":0.086,"rv":0.084,"ob":0.457,"rs":0.096,"av":0.087,"ds":0.117,"ed":0.314}
./segments/1590348509264.96/warc/CC-MAIN-20200606000537-20200606030537-00143.warc.gz5550176434911199913http://giridarpon.com/category/politics/page/3/text/html2020-06-06T00:43:27utf-8CC-MAIN-2020-24["ben_Beng","mni_Beng","npi_Deva"][1,0,0]"জাতীয় প্রেসক্লাবে ১৯৭১ সালের ২৬ মার"<doc fingerprint=\"4ac3ea500f240cc9\">\n <main>\n <p>জাতীয় প্রেসক্["en-US"]1["ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng",ad3cebda5e1925df218fe2ec8180ed41keep[][10,10,10,10,10,10,10,10,10,10]{"MT":0.079,"LY":0.069,"SP":0.096,"ID":0.063,"NA":0.77,"HI":0.064,"IN":0.235,"OP":0.081,"IP":0.097,"
./segments/1590347458095.68/warc/CC-MAIN-20200604192256-20200604222256-00495.warc.gz67789056547557230986https://www.dailyandolonerbazar.com/2019/05/text/html2020-06-04T21:15:24utf-8CC-MAIN-2020-24["ben_Beng","mni_Beng","npi_Deva"][1,0,0]"কুমারখালী প্রতিনিধি ॥ নিরাপদ খাদ্য ন"<doc fingerprint=\"ae2bb838a75e2791\">\n <main>\n <p>কুমারখালী প্রত["bn-BD"]1["ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng",b582b0c0601d3d292412614b1b2f82f1keep[][10,10,10,10,10,10,10,10,9.6,10]{"MT":0.078,"LY":0.07,"SP":0.078,"ID":0.082,"NA":0.875,"HI":0.071,"IN":0.14,"OP":0.091,"IP":0.099,"i
./segments/1590347445880.79/warc/CC-MAIN-20200604161214-20200604191214-00177.warc.gz26382309920382100429https://bondhubrikkho.com.bd/%E0%A6%A6%E0%A7%87%E0%A6%B6%E0%A6%AA%E0%A7%8D%E0%A6%B0%E0%A7%87%E0%A6%AE-%E0%A6%B8%E0%A6%A4%E0%A6%A4%E0%A6%BE-%E0%A6%86%E0%A6%B0-%E0%A6%AC%E0%A6%99%E0%A7%8D%E0%A6%97%E0%A6%AC%E0%A6%A8%E0%A7%8D/text/html2020-06-04T17:46:06utf-8CC-MAIN-2020-24["ben_Beng","mni_Beng","npi_Deva"][1,0,0]"দুরন্ত কৈশোরে সামাজিক সাংস্কৃতিক আর "<doc fingerprint=\"7d3434e9058a7ca5\">\n <main>\n <p>দুরন্ত কৈশোরে স["en-US"]5["ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng",418f9a1ff53b55aae7c9ea509a6f50b4keep[][10,10,10,10,10,10,10,10,10,10]{"MT":0.066,"LY":0.075,"SP":0.139,"ID":0.077,"NA":0.253,"HI":0.097,"IN":0.871,"OP":0.103,"IP":0.133,
./segments/1590347436466.95/warc/CC-MAIN-20200603210112-20200604000112-00485.warc.gz2762736222004086236https://bn.wikisource.org/wiki/%E0%A6%B8%E0%A6%BE%E0%A6%B9%E0%A6%BF%E0%A6%A4%E0%A7%8D%E0%A6%AF/%E0%A6%B8%E0%A6%BE%E0%A6%B9%E0%A6%BF%E0%A6%A4%E0%A7%8D%E0%A6%AF%E0%A7%87%E0%A6%B0_%E0%A6%97%E0%A7%8C%E0%A6%B0%E0%A6%ACtext/html2020-06-03T22:07:21utf-8CC-MAIN-2020-24["ben_Beng","mni_Beng","asm_Beng"][0.9999,0.0001,0]"সাহিত্য/সাহিত্যের গৌরব\nসাহিত্যের গৌ"<doc fingerprint=\"4fa857e76d127d12\">\n <main>\n <head rend=\"h1\">সাহিত্য/স["bn"]15["ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng","ben_Beng",4f2f0a9b1772967ef25a84311ec0c14akeep[][10,10,10,10,10,10,10,10,10,10]{"MT":0.071,"LY":0.119,"SP":0.146,"ID":0.071,"NA":0.203,"HI":0.035,"IN":0.176,"OP":0.355,"IP":0.398,

Cite this

CC0-1.0Cite

Tags

task_categories:text-generationlanguage:bnlanguage:gulanguage:hilanguage:knlanguage:mllanguage:mrlanguage:mnilanguage:orlanguage:palanguage:talanguage:telanguage:urlicense:cc0-1.0region:uslanguage-modelingcausal-lmllmmultilingualindic-languageshplt