ёi S SK Jr S SKrS SKrS SKrS SKrS SKJrJr S SK r
S SK J r S SK
Jr \( a S SKJr S SKJr \S r/ rSrS r " S
S \5 rg) ) annotationsN)
TYPE_CHECKINGLiteral)_check_exists_and_download)Dataset)Patterntraintestz6https://dataset.bj.bcebos.com/imdb%2FaclImdb_v1.tar.gz 7c2ac02c03563afcf9b574c7e56c153ac \ rS rSr% SrS\S' S\S' S\S' S \S
' S \S ' S SS
jjrSS jrSS jrSS jr SS jr
SS jr Sr g )Imdb' at
Implementation of `IMDB `_ dataset.
Args:
data_file(str|None): path to data tar file, can be set None if
:attr:`download` is True. Default None.
mode(str): 'train' 'test' mode. Default 'train'.
cutoff(int): cutoff number for building word dictionary. Default 150.
download(bool): whether to download dataset automatically if
:attr:`data_file` is not set. Default True.
Returns:
Dataset: instance of IMDB dataset
Examples:
.. code-block:: pycon
>>> # doctest: +TIMEOUT(75)
>>> import paddle
>>> from paddle.text.datasets import Imdb
>>> class SimpleNet(paddle.nn.Layer):
... def __init__(self):
... super().__init__()
...
... def forward(self, doc, label):
... return paddle.sum(doc), label
>>> imdb = Imdb(mode='train')
>>> for i in range(10):
... doc, label = imdb[i]
... doc = paddle.to_tensor(doc)
... label = paddle.to_tensor(label)
...
... model = SimpleNet()
... image, label = model(doc, label)
... print(doc.shape, label.shape)
paddle.Size([121]) paddle.Size([1])
paddle.Size([115]) paddle.Size([1])
paddle.Size([386]) paddle.Size([1])
paddle.Size([471]) paddle.Size([1])
paddle.Size([585]) paddle.Size([1])
paddle.Size([206]) paddle.Size([1])
paddle.Size([221]) paddle.Size([1])
paddle.Size([324]) paddle.Size([1])
paddle.Size([166]) paddle.Size([1])
paddle.Size([598]) paddle.Size([1])
str | None data_file_ImdbDataSetModemodedict[str, int]word_idxlistdocslabelsNc 0 UR 5 S; d
SU 35 eUR 5 U l Xl U R c* U( d S5 e[ U[ [
SU5 U l U R
U5 U l U R 5 g )Nr z(mode should be 'train', 'test', but got z>data_file is not set and downloading automatically is disabledimdb) lowerr r r URLMD5_build_work_dictr
_load_anno)selfr r cutoffdownloads Y/var/www/html/banglarbhumi/venv/lib/python3.13/site-packages/paddle/text/datasets/imdb.py__init__
Imdb.__init__b s zz|
= 6dV < =
JJL " >> !
P
8 83VX DN
--f5
c [ R " [ 5 n[ R " S5 nU R U5 H nU H nX%== S-
ss' M M UR
5 Vs/ s H ofS U: d M
UPM nn[ US S9n[ [ U6 5 u p[ [ [ U[ [ U5 5 5 5 5 n
[ U5 U
S' U
$ s snf )Nz/aclImdb/((train)|(test))/((pos)|(neg))/.*\.txt$ c U S * U S 4$ )Nr( r )xs r# 'Imdb._build_work_dict.. s qteQqT]r& )key)
collections defaultdictintrecompile _tokenizeitemssortedr zipdictrangelen) r r! word_freqpatterndocwordr+
dictionarywords_r s r# r Imdb._build_work_dict~ s ++C0 ** OP>>'*C1$ +
!* 1C 11qTF]Q 1 CI+BC
Z()Sc%j(9 :;< J
Ds 0 C& C&c . / n[ R " U R 5 nUR 5 nUb [ UR UR 5 5 ( a UR UR U5 R 5 R S5 R S [ R R S5 5 R 5 R! 5 5 UR 5 nUb M S S S 5 U$ ! , ( d f U$ = f)Ns
zlatin-1)tarfileopenr nextboolmatchnameappend extractfilereadrstrip translatestring punctuationencoder split)r r= datatarftfs r# r5 Imdb._tokenize s
\\$..
)TB.
bgg.//KK ((, "4););)B)B9)MN
YY[ . * *
) s CD
Dc
[ R " SU R S35 n[ R " SU R S35 nU R S n/ U l / U l U R
U5 Hb nU R R U Vs/ s H oPR R XS5 PM sn5 U R
R S5 Md U R
U5 Hb nU R R U Vs/ s H oPR R XS5 PM sn5 U R
R S5 Md g s snf s snf )NzaclImdb/z
/pos/.*\.txt$z
/neg/.*\.txt$r/ r r( ) r3 r4 r r r r r5 rK get)r pos_pattern neg_patternUNKr> ws r# r Imdb._load_anno s jjHTYYK}!EF jjHTYYK}!EF mmG$ >>+.C II EAmm / / 7 E F KK q ! / >>+.C II EAmm / / 7 E F KK q ! / F Fs $E
$E
c [ R " U R U 5 [ R " U R U /5 4$ N)nparrayr r )r idxs r# __getitem__Imdb.__getitem__ s5 3 ("((DKK4D3E*FGGr& c , [ U R 5 $ r` )r; r )r s r# __len__ Imdb.__len__ s 499~ r& )r r r r r )Nr
T)
r r r r r! r2 r" rH returnNone)r! r2 rj r )r= z Pattern[str]rj zlist[list[str]])rj rk )rc r2 rj z1tuple[npt.NDArray[np.int_], npt.NDArray[np.int_]])rj r2 )
__name__
__module__ __qualname____firstlineno____doc____annotations__r$ r r5 r rd rg __static_attributes__r* r& r# r r ' s 2h
J L !%!(
8 &
" HH :H
r& r )
__future__r r0 r3 rP rE typingr r numpyra paddle.dataset.commonr paddle.ior r numpy.typingnptr __all__r r r r* r& r# r{ sS #
) < /
>(O 7 O r&