x-jPdZddlZddlZddlmZgZdZdZdZ dZ dZ d Z d Z d Zd Zd ZdZdZedddddZedddddZedddddZeddddddZedddddZdS)a WMT14 dataset. The original WMT14 dataset is too large and a small set of data for set is provided. This module will download dataset from http://paddlepaddle.bj.bcebos.com/demo/wmt_shrinked_data/wmt14.tgz and parse training set and test set into paddle reader creators. N) deprecatedzJhttp://www-lium.univ-lemans.fr/~schwenk/cslm_joint_paper/data/dev+test.tgz 7d7897317ddd8ba0ae5c5fa7248d3ff5z/http://paddlemodels.bj.bcebos.com/wmt/wmt14.tgz 0791583d57d5beb693b9414c5b36798cz1http://paddlemodels.bj.bcebos.com/wmt%2Fwmt14.tgz 0cb4a5366189b6acba876491c8724fa3zzzcd}tj|d5}d|D}t|dksJ|||d|}d|D}t|dksJ|||d|}||fcdddS#1swxYwYdS)Nci}t|D]5\}}||kr*|||<5|S)N) enumeratestripdecode)fdsizeout_dict line_countlines T/var/www/html/banglarbhumi/venv/lib/python3.11/site-packages/paddle/dataset/wmt14.py __to_dictz!__read_to_dict..__to_dict1sU )"    JD  2<,,..//rmodecPg|]#}|jd|j$S)zsrc.dictnameendswith.0 each_items r z"__read_to_dict..;@   ~&&z22 N   rrcPg|]#}|jd|j$S)ztrg.dictrrs rrz"__read_to_dict..Br r)tarfileopenlen extractfile)tar_file dict_sizerfnamessrc_dicttrg_dicts r__read_to_dictr-0s9 hS ) ) )"Q      5zzQ9Q]]5844i@@      5zzQ9Q]]5844i@@!""""""""""""""""""sBB77B;>B;cfd}|S)Nc3 Kt\ tjd5}fd|D}|D]}||D]}|}|d}t|dkrR|d}|} fdtg|tD}|d}|} fd | D} t|d kst| d krg|  t} tg| } || | fV ddddS#1swxYwYdS) NrrcRg|]#}|j|j$Sr)rr file_names rrz2reader_creator..reader..PsA>**955r rrcFg|]}|tSr1getUNK_IDX)rwr+s rrz2reader_creator..reader..]s7! Q00rr!cFg|]}|tSr1r5)rr8r,s rrz2reader_creator..reader..ds'KKKAx||Aw77KKKrP) r-r#r$r&r r splitr%STARTEND)r)r*rr line_splitsrc_seq src_wordssrc_idstrg_seq trg_wordstrg_ids trg_ids_nextr+r,r(r2r's @@rreaderzreader_creator..readerMs +Hi@@( \( - - - 9!"E  9 9MM$//99D;;==D!%!3!3D!9!9J:!++ (mG ' I"'!9)!9S!9G )mG ' IKKKKKKKG7||b((CLL2,=,= #Please use new dataset API which supports paddle.io.DataLoader)since update_tolevelreasoncttjjt dt d|S)a WMT14 training set creator. It returns a reader creator, each sample in the reader is source language word ID sequence, target language word ID sequence and next word ID sequence. :return: Training reader creator :rtype: callable wmt14z train/trainrGpaddledatasetcommondownload URL_TRAIN MD5_TRAINr(s rtrainrVqs6" &&y'9EE  rcttjjt dt d|S)z WMT14 test set creator. It returns a reader creator, each sample in the reader is source language word ID sequence, target language word ID sequence and next word ID sequence. :return: Test reader creator :rtype: callable rMz test/testrNrUs rtestrXs6" &&y'9EE  rcttjjt dt d|S)NrMzgen/genrNrUs rgenrZs6 &&y'9EE  rTc tjjtdt }t ||\}}|rzget_dict..666TQAq666rci|]\}}|| Sr1r1r]s rr`zget_dict..rar)rOrPrQrRrSrTr-items)r(reverser'r+r,s rget_dictres~$--i)LLH')<<Hh766X^^%5%566666X^^%5%5666 X rctjjtdt tjjt dtdS)NrM)rOrPrQrRrSrT URL_MODEL MD5_MODELr1rrfetchrisD N""9gyAAA N""9gyAAAAAr)T)__doc__r#paddle.dataset.commonrO paddle.utilsr__all__ URL_DEV_TEST MD5_DEV_TESTrSrTrgrhr<r=UNKr7r-rGrVrXrZrerir1rrrqs###### Q 2  > . ? .     """8"""J  * K    $  * K    $  * K      * K      * K  BB  BBBr