x-j.FdZddlZddlZddlZddlZddlZddlmZgZ dZ dZ dZ dZ edd d d d Zedd d d dZedd d d dZedd d d dZedd d d dZdS)a IMDB dataset. This module downloads IMDB dataset from http://ai.stanford.edu/%7Eamaas/data/sentiment/. This dataset contains a set of 25,000 highly polar movie reviews for training, and 25,000 for testing. Besides, this module also provides API for building dictionary. N) deprecatedz6https://dataset.bj.bcebos.com/imdb%2FaclImdb_v1.tar.gz 7c2ac02c03563afcf9b574c7e56c153ac#Ktjtjjt dt5}|}|t| |j r| | ddt jdV|}|ddddS#1swxYwYdS)zQ Read files that match the given pattern. Tokenize and yield each file. imdbNs zlatin-1)tarfileopenpaddledatasetcommondownloadURLMD5nextboolmatchname extractfilereadrstrip translatestring punctuationencodelowersplit)patterntarftfs S/var/www/html/banglarbhumi/venv/lib/python3.11/site-packages/paddle/dataset/imdb.pytokenizer &s@ fn+44S&#FF G G4 YY[[nGMM"'**++ $$R((TVVVG__YtV%7%>%>y%I%IJJUWWUWW Bn sC%D77D;>D;c tjt}t|D]}|D]}||xxdz cc<fd|D}t |d}t t|\}}tt t|tt|}t||d<|S)z Build a word dictionary from the corpus. Keys of the dictionary are words, and values are zero-based IDs of these words. c,g|]}|dk|S)r").0xcutoffs r zbuild_dict..Js"???q1c$|d |dfS)Nr"rr$)r&s rzbuild_dict..Ls1Q4%1r))key) collections defaultdictintr itemssortedlistzipdictrangelen) rr' word_freqdocword dictionarywords_word_idxs ` r build_dictr??s ',,I  !! ! !D dOOOq OOOO !@???IOO--???I '>'>???JC$%%HE1DUE#e**$5$5667788HE HW Or)z2.0.0zpaddle.text.datasets.Imdbr"z>Please use new dataset API which supports paddle.io.DataLoader)since update_tolevelreasoncjdgfd}||d||dfd}|S)Nr-ctt|D]&}|fd|D|f'dS)Nc<g|]}|Sr$)get)r%wUNKr>s rr(z0reader_creator..load.._s';;;!a--;;;r))r append)routlabelr9rIr>s rloadzreader_creator..load]sZG$$ E EC JJ;;;;;s;;;UC D D D D E Er)rr"c3KEd{VdS)Nr$)INSsrreaderzreader_creator..readerdsr)r$) pos_pattern neg_patternr>rMrPrOrIs ` @@rreader_creatorrSSs 7 C CEEEEEE Dc1Dc1 Mr)clttjdtjd|S)a IMDB training set creator. It returns a reader creator, each sample in the reader is an zero-based ID sequence and label in [0, 1]. :param word_idx: word dictionary :type word_idx: dict :return: Training reader creator :rtype: callable zaclImdb/train/pos/.*\.txt$zaclImdb/train/neg/.*\.txt$rSrecompiler>s rtrainrYjs5$  011 011  r)clttjdtjd|S)a IMDB test set creator. It returns a reader creator, each sample in the reader is an zero-based ID sequence and label in [0, 1]. :param word_idx: word dictionary :type word_idx: dict :return: Test reader creator :rtype: callable zaclImdb/test/pos/.*\.txt$zaclImdb/test/neg/.*\.txt$rUrXs rtestr[s5$  /00 /00  r)cFttjddS)za Build a word dictionary from the corpus. :return: Word dictionary :rtype: dict z/aclImdb/((train)|(test))/((pos)|(neg))/.*\.txt$)r?rVrWr$r)r word_dictr^s&  EFF  r)cftjjtdt dS)Nr)r r r r r rr$r)rfetchr`s' N""344444r))__doc__r.rVrrpaddle.dataset.commonr paddle.utilsr__all__r rr r?rSrYr[r^r`r$r)rres ###### ?(2(  ) K    "  ) K    &  ) K    &  ) K       ) K  55  555r)