Αi3SrSSKrSSKrSSKJr SSKrSSKJr /rSr Sr Sr Sr S r S rS rS rSS jrSrSr\"SSSSS9SSj5r\"SSSSS9SSj5r\"SSSSS9SSj5r\"SSSSS9SSj5r\"SSSSS9S5rg)aW ACL2016 Multimodal Machine Translation. Please see this website for more details: http://www.statmt.org/wmt16/multimodal-task.html#task1 If you use the dataset created for your task, please cite the following paper: Multi30K: Multilingual English-German Image Descriptions. @article{elliott-EtAl:2016:VL16, author = {{Elliott}, D. and {Frank}, S. and {Sima"an}, K. and {Specia}, L.}, title = {Multi30K: Multilingual English-German Image Descriptions}, booktitle = {Proceedings of the 6th Workshop on Vision and Language}, year = {2016}, pages = {70--74}, year = 2016 } N) defaultdict) deprecatedz2http://paddlemodels.bj.bcebos.com/wmt/wmt16.tar.gz 0c38be43600334966403524a40dcd81ei+iKzzzc [[5n[R"USS9nUR S5HvnUR 5nUR 5RS5n[U5S:waMCUS:XaUSOUSnUR5Hn XI==S- ss'M Mx SSS5 [US 5n U R[S [S [S 3R55 [[UR!5S S S 95HCupU S-U:Xa O7U RU SR55 U RS5 ME SSS5 g!,(df  N=f!,(df  g=f)Nrmode wmt16/train enrwb c US$)Nr)xs T/var/www/html/banglarbhumi/venv/lib/python3.13/site-packages/paddle/dataset/wmt16.py__build_dict..BsAaDT)keyreverse )rinttarfileopen extractfiledecodestripsplitlenwrite START_MARKEND_MARKUNK_MARKencode enumeratesorteditems) tar_file dict_size save_pathlang word_dictfline line_splitsenwfoutidxwords r __build_dictr93s=C I hS )QMM-0D;;=D++D1J:!##'4<*Q-Z]CYY[ ! ! 1 * i $ zl"XJb "=EEGH" 9??$.$ G ICQw)# JJtAw~~' ( JJu      * )  sB E-BE>- E;> F cR[RR[RR R SUSUS35n[RRU5(a'[[US5R55U:wa [XXB5 0n[US5n[U5HNupxU(a"UR5R5XW'M.XuUR5R5'MP SSS5 U$!,(df  U$=f)Nwmt16/_.dictrb)ospathjoinpaddledatasetcommon DATA_HOMEexistsr#r readlinesr9r)r!r ) r,r-r/r dict_pathr0fdictr7r2s r __load_dictrJJs ''6$q 5)II 77>>) $ $ DD ! + + -.);X):I i %"5)IC!%!4!4!6 36$**,--/0 *     s /AD D&ct[XS:Xa[O[5n[XS:Xa[O[5nX4$)Nr )minTOTAL_EN_WORDSTOTAL_DE_WORDS src_dict_size trg_dict_sizesrc_langs r__get_dict_sizerS]s;d*:Md*:M  ''rc$^^^^^UUUUU4SjnU$)Nc 3># [TTT5n[TTTS:XaSOS5nU[nU[nU[nTS:XaSOSnSU- n[R "TSS9nUR T5HnUR5nUR5RS5n [U 5S:waMCXR5n U/U V s/sHoRX5PM sn -U/-n XR5n U V s/sHoRX5PM nn /UQUPnU/UQnXU4v M SSS5 gs sn fs sn f!,(df  g=f7f) Nr derrrrr r ) rJr%r&r'rrrr r!r"r#get)src_dicttrg_dictstart_idend_idunk_idsrc_coltrg_colr1r2r3 src_wordsr5src_ids trg_wordstrg_ids trg_ids_next file_namerPrRr,rQs rreaderreader_creator..readerhsmxA mh$.>dD J'(#(#4'!Qg+ \\( - i0{{}!ZZ\//5 z?a'&/557 J8AB 1||A. BCh '/557 Please use new dataset API which supports paddle.io.DataLoader)since update_tolevelreasoncUS;a [S5e[XU5up[[RR R [S[S5SUUUS9$)a WMT16 train set reader. This function returns the reader for train data. Each sample the reader returns is made up of three fields: the source language word index sequence, target language word index sequence and next word index sequence. NOTE: The original like for training data is: http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/training.tar.gz paddle.dataset.wmt16 provides a tokenized version of the original dataset by using moses's tokenization script: https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl Args: src_dict_size(int): Size of the source language dictionary. Three special tokens will be added into the dictionary: for start mark, for end mark, and for unknown word. trg_dict_size(int): Size of the target language dictionary. Three special tokens will be added into the dictionary: for start mark, for end mark, and for unknown word. src_lang(string): A string indicating which language is the source language. Available options are: "en" for English and "de" for Germany. Returns: callable: The train reader. r rVzIAn error language type. Only support: en (for English); de(for Germany).wmt16 wmt16.tar.gzr r,rdrPrQrR ValueErrorrSrgrBrCrDdownloadDATA_URLDATA_MD5rOs rtrainrvsqP|# 1  $3h$ M &&// gx  ## rcUS;a [S5e[XU5up[[RR R [S[S5SUUUS9$)a WMT16 test set reader. This function returns the reader for test data. Each sample the reader returns is made up of three fields: the source language word index sequence, target language word index sequence and next word index sequence. NOTE: The original like for test data is: http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/mmt16_task1_test.tar.gz paddle.dataset.wmt16 provides a tokenized version of the original dataset by using moses's tokenization script: https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl Args: src_dict_size(int): Size of the source language dictionary. Three special tokens will be added into the dictionary: for start mark, for end mark, and for unknown word. trg_dict_size(int): Size of the target language dictionary. Three special tokens will be added into the dictionary: for start mark, for end mark, and for unknown word. src_lang(string): A string indicating which language is the source language. Available options are: "en" for English and "de" for Germany. Returns: callable: The test reader. rmHAn error language type. Only support: en (for English); de(for Germany).rnroz wmt16/testrprqrOs rtestrysqN|# ?  $3h$ M &&// gx ## rcUS;a [S5e[XU5up[[RR R [S[S5SUUUS9$)a+ WMT16 validation set reader. This function returns the reader for validation data. Each sample the reader returns is made up of three fields: the source language word index sequence, target language word index sequence and next word index sequence. NOTE: The original like for validation data is: http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/validation.tar.gz paddle.dataset.wmt16 provides a tokenized version of the original dataset by using moses's tokenization script: https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl Args: src_dict_size(int): Size of the source language dictionary. Three special tokens will be added into the dictionary: for start mark, for end mark, and for unknown word. trg_dict_size(int): Size of the target language dictionary. Three special tokens will be added into the dictionary: for start mark, for end mark, and for unknown word. src_lang(string): A string indicating which language is the source language. Available options are: "en" for English and "de" for Germany. Returns: callable: The validation reader. rmrxrnroz wmt16/valrprqrOs r validationr{sqL|# ?  $3h$ M &&// gx ## rcUS:Xa[U[5nO[U[5n[RR [ RRRSUSUS35n[RRU5(dS5e[RR [ RRRS5n[XAX5$)a return the word dictionary for the specified language. Args: lang(string): A string indicating which language is the source language. Available options are: "en" for English and "de" for Germany. dict_size(int): Size of the specified language dictionary. reverse(bool): If reverse is set to False, the returned python dictionary will use word as key and use index as value. If reverse is set to True, the returned python dictionary will use index as key and word as value. Returns: dict: The word dictionary for the specific language. r r;r<r=z Word dictionary does not exist. ro) rLrMrNr?r@rArBrCrDrErFrJ)r/r-rrHr,s rget_dictr}Bs0 t| >2  >2  ''6$q 5)II 77>>) $ $H&HH $Eww||FNN11;;^LH xD ::rc[RRRR [ S[ S5 g)zdownload the entire dataset.rnroN)rBv4rCrDrsrtrurrrfetchris+ II%%'8^r)F)r )__doc__r?r collectionsrrB paddle.utilsr__all__rtrurMrNr%r&r'r9rJrSrgrvryr{r}rrrrrs" # #  ? -   .&(&R * K  3  3l * K  3  3l * K  1  1h * K  ;  ;B * K    r