ΑiSrSSKrSSKrSSKJr /rSrSrSr Sr Sr S r S r S rS rS rSrSr\"SSSSS9S5r\"SSSSS9S5r\"SSSSS9S5r\"SSSSS9SSj5r\"SSSSS9S5rg)a WMT14 dataset. The original WMT14 dataset is too large and a small set of data for set is provided. This module will download dataset from http://paddlepaddle.bj.bcebos.com/demo/wmt_shrinked_data/wmt14.tgz and parse training set and test set into paddle reader creators. N) deprecatedzJhttp://www-lium.univ-lemans.fr/~schwenk/cslm_joint_paper/data/dev+test.tgz 7d7897317ddd8ba0ae5c5fa7248d3ff5z/http://paddlemodels.bj.bcebos.com/wmt/wmt14.tgz 0791583d57d5beb693b9414c5b36798cz1http://paddlemodels.bj.bcebos.com/wmt%2Fwmt14.tgz 0cb4a5366189b6acba876491c8724fa3zzzc*Sn[R"USS9nUVs/sH1nURRS5(dM%URPM3 nn[ U5S:XdeU"UR US5U5nUVs/sH1nURRS5(dM%URPM3 nn[ U5S:XdeU"UR US5U5nXg4sSSS5 $s snfs snf!,(df  g=f)Nc0n[U5H-up4X1:a"X2UR5R5'M, U$ U$)N) enumeratestripdecode)fdsizeout_dict line_countlines T/var/www/html/banglarbhumi/venv/lib/python3.13/site-packages/paddle/dataset/wmt14.py __to_dict!__read_to_dict..__to_dict1sD )" J 2<,,./ !. rmodezsrc.dictrztrg.dict)tarfileopennameendswithlen extractfile)tar_file dict_sizerf each_itemnamessrc_dicttrg_dicts r__read_to_dictr'0s hS )Q  ~~&&z2 INN  5zQQ]]584i@  ~~&&z2 INN  5zQQ]]584i@! * )   * )s4D$C:C:2D$C?0C?0D: D Dc^^^UUU4SjnU$)Nc 3n># [TT5up[R"TSS9nUVs/sH1nURR T5(dM%URPM3 nnUGHnUR U5GHnUR 5nUR5RS5n[U5S:waMDUSnUR5n [/U Q[PV s/sHn URU [5PM n n USn U R5n U V s/sHoRU [5PM nn [U 5S:d[U5S:aM/UQU[PnU[/UQnXU4v GM GM" SSS5 gs snfs sn fs sn f!,(df  g=f7f)Nrr rrrP)r'rrrrrr r splitrSTARTENDgetUNK_IDX)r%r&r"r#r$rr line_splitsrc_seq src_wordswsrc_idstrg_seq trg_wordstrg_ids trg_ids_nextr! file_namer s rreaderreader_creator..readerMs+Hi@ \\( -"#!"I>>**95 !"  MM$/D;;=D!%!3!3D!9J:!+ (mG ' I#(!9)!9S!9!9A! Q0!9 )mG ' IAJKA||Aw7GK7|b(CL2,= #Ls9B Mrz2.0.0zpaddle.text.datasets.WMT14rz>Please use new dataset API which supports paddle.io.DataLoader)since update_tolevelreasonc~[[RRR [ S[ 5SU5$)z WMT14 training set creator. It returns a reader creator, each sample in the reader is source language word ID sequence, target language word ID sequence and next word ID sequence. :return: Training reader creator :rtype: callable wmt14z train/trainr>paddledatasetcommondownload URL_TRAIN MD5_TRAINr!s rtrainrMqs3" &&y'9E rc~[[RRR [ S[ 5SU5$)z WMT14 test set creator. It returns a reader creator, each sample in the reader is source language word ID sequence, target language word ID sequence and next word ID sequence. :return: Test reader creator :rtype: callable rDz test/testrErLs rtestrOs3" &&y'9E rc~[[RRR [ S[ 5SU5$)NrDzgen/genrErLs rgenrQs3 &&y'9E rc>[RRR[S[ 5n[ X 5up4U(aHUR5VVs0sHupVXe_M nnnUR5VVs0sHupVXe_M nnnX44$s snnfs snnfNrD)rFrGrHrIrJrKr'items)r!reverser r%r&kvs rget_dictrXs~~$$--i)LH'<H%-^^%56%5TQAD%56%-^^%56%5TQAD%56  76s B?Bc[RRR[S[ 5 [RRR[ S[5 grS)rFrGrHrIrJrK URL_MODEL MD5_MODELr=rrfetchr\s< NN""9gyA NN""9gyAr)T)__doc__rpaddle.dataset.commonrF paddle.utilsr__all__ URL_DEV_TEST MD5_DEV_TESTrJrKrZr[r-r.UNKr0r'r>rMrOrQrXr\r=rrrds# Q 2  > . ? .     "8"J * K    $ * K    $ * K     * K     * K  B  Br