ϑi ^SSKJr SSKrSSKrSSKrSSKrSSKrSSKrSSKJ r J r J r J r J r JrJr SSKrSSKJrJrJr SSKJrJr SSKJrJrJr SS KJrJr SS KJ r J!r!J"r" \ (aKSSK#r#SS K$J%r%J&r& SSK'Jr( SS KJ)r) SS K*J+r+ SSK,J-r- SSK.J/r/ SSK0J1r1 \"S5r2\"S5r3"SS\ 5r4Sr5Sq6Sq7Sq8SSjr9Sr:Sr;"SS5r<"SS5r=g) ) annotationsN) TYPE_CHECKINGAnyAnyStrCallableProtocolTypeVaroverload)_current_expected_place_get_paddle_place_get_paddle_place_list)corein_dynamic_mode) BatchSamplerIterableDatasetSubset)DistributedBatchSampler_InfiniteIterableSampler)_DataLoaderIterMultiProcess_DataLoaderIterSingleProcess _DatasetKind)MappingSequence)Tensor) PlaceLike)_Place)_DataLoaderIterBase)Dataset_K_Vc\rSrSr\S Sj5r\S Sj5r\S Sj5r\S Sj5r\S Sj5rSrg) _CollateFnBcgNselfbatchs P/var/www/html/banglarbhumi/venv/lib/python3.13/site-packages/paddle/io/reader.py__call___CollateFn.__call__Cs #cgr'r(r)s r,r-r.H?Br/cgr'r(r)s r,r-r.Kr1r/cgr'r(r)s r,r-r.Ns"r/cgr'r(r)s r,r-r.SsKNr/r(N)r+z5Sequence[npt.NDArray[Any]] | Sequence[numbers.Number]returnznpt.NDArray[Any])r+zSequence[Tensor]r5r)r+zSequence[AnyStr]r5r)r+zSequence[Mapping[_K, _V]]r5zMapping[_K, _V])r+zSequence[Sequence[_V]]r5z Sequence[_V])__name__ __module__ __qualname____firstlineno__r r-__static_attributes__r(r/r,r$r$Bst  #N #  #  # B B B B  "2 "  "  " N Nr/r$<Fc UqUqgr') USE_AUTOTUNE TUNING_STEPS) use_autotune tuning_stepss r,set_autotune_configrBas LLr/c[U5S:Xa[$[U5S:Xa[US[5(deUSqg)Nrr)lenUSE_PINNED_MEMORY isinstancebool)argss r,use_pinned_memoryrIhs> 4yA~  4yA~*T!Wd";";;; Gr/c[U[[45(dU/n/nUH[n[U[R5(d([R"5nUR U5 UnUR U5 M] U$r')rFlisttuplerPlace set_placeappend)placesretptmps r,_convert_placesrTqsh ftUm , , C !TZZ((**,C MM! A 1  Jr/c>\rSrSrSrSrSrSrSrSr Sr S r g ) AuToTunecJXl[R"5S- Ulg)Nr )loadermultiprocessing cpu_countmax_num_worker)r*rYs r,__init__AuToTune.__init__s -779A=r/c[(aUR5(dURR$UR 5nUcURR$[ R "5n[ R"S5 [ R"S[URR5-5 Sn[S5n[ R"S[UR5-5 SnXPR:aXQlURU5nUS-U:aUnUnO&URUUUUR5nXs:XaOIUn[ R"S[U5-S-[U5-5 US - nXPR:aM[ R"S [U5-5 [ R"S [[ R "5U- 5-S -5 U$) Nz(========= DataLoader Auto Tune =========zUser config for DataLoader: rinfz"Tuning Range for num_workers: 0 ~ g?z num_workers: avg_cost: r z'auto_tune dataLoader best_num_workers: z AutoTuning Cost for DataLoader: z seconds)r> need_autotunerY num_workersget_autotune_loadertimeloggingdebugstrfloatr\evaluate_reader_costis_bestinfo)r*auto_tune_loaderauto_tune_startbest_num_workersmin_costrcavg_cost update_nums r,r-AuToTune.__call__s d&8&8&:&:;;** * 335  #;;** *))+ @A *S1H1H-I I < 03t7J7J3K K  ///+6 (001ABH$)##. !\\$$''  1'1$ MMk"# h-   1 K////0  5r/cv[U[-[U55n[U[ [ U55S9nU$)N)indices)minr?rDrrKrange)r*dataset batch_size num_samples sub_datasets r,get_sub_datasetAuToTune.get_sub_datasets3*|3S\B Wd53E.FG r/c  [R"UR5nURRRn[ URR[ R R5(aURRRnURX25n[ R RUUURRRURRRURRRURRRS9UlU$[ URR[ R R5(aURRRR nURX25n[ R RUUURRRS9UlU$SnU$)Nrr num_replicasrankshuffle drop_last)rrr)copyrY batch_samplerrrFpaddleiorrrnranks local_rankrrrsampler data_source)r*rYrrrs r,rdAuToTune.get_autotune_loaders4;;'[[..99  KK % %vyy'H'H  kk//77G..wCK#)99#D#D#%![[66==[[..99 1199++33== $E$F $  116993I3I J Jkk//77CCG..wCK#)99#9#9#%++33==$:$F  F r/cn/nSn[R"5n[U5HAupVUR[R"5U- 5 [R"5nMC [U5S:a[ USS5[USS5- nU$[ USS5[USS5- nU$)Nrr )re enumeraterOrDsum)r*readercostsrqstartidatas r,rjAuToTune.evaluate_reader_costs  (GA LLu, -IIKE) u:>59~E!"I6H59~E!"I6Hr/c$SnUS-nSnXd:aUS:a{X`RlURU5n[R"S[ U5-S-[ U5-5 US- nXS-U-:aU$US- nUS-nXd:aUS:aM{U$)Nrrzfor back num_workers: ragffffff?g?)rYrcrjrfrgrh) r*r best_workers best_timenum_work_boundarysteprcboundaryres r,rkAuToTune.is_bests"Q& -$(&1KK #,,V4D MM(k"# d)  AID$&11""q  H-$(r/)rYr\N) r6r7r8r9r]r-rbrrdrjrkr:r(r/r,rVrVs&>8 t  8 r/rVcL\rSrSr%SrS\S'S\S'S\S'S\S 'S\S 'S \S 'S \S'S\S'S\S'S\S'S\S'S\S'S\S'S\S'S\S'S\S'S"S#SjjrS$SjrS%SjrS%S jr S!r g)& DataLoaderi a DataLoader provides an iterator which iterates given dataset once by the batch_sampler. DataLoader supports single-process and multi-process data loading, multi-process workers will be used to load data asynchronously if :attr:`num_workers` is set as a positive number. DataLoader supports map-style dataset and iterable-style dataset. For map-style dataset(can get a sample from dataset with a given index), please see :code:`paddle.io.Dataset`. For iterable-style dataset(get samples from dataset iteratively, like a Python iterator), please see :code:`paddle.io.IterableDataset`. For :code:`batch_sampler` please see :code:`paddle.io.BatchSampler` Notes: GPU tensor operation is not supported in subprocess currently, please don't use GPU tensor operations in pipeline which will be performed in subprocess, such as dataset transforms, collate_fn, etc. Numpy array and CPU tensor operation is supported. **Disable automatic batching** In certain cases such as some NLP tasks, instead of automatic batching, handling batching manually in dataset is needed by users. For these cases, automatic batching is disabled if both :attr:`batch_size` and :attr:`batch_sampler` is set as None, each data got from :attr:`dataset` should be batched data and will be processed with function define by :attr:`collate_fn` or :attr:`default_collate_fn`. Notes: When automatic batching is disabled, :attr:`default_collate_fn` will do nothing to data from dataset. Args: dataset(Dataset): the dataset to load data from, should be an instance of subclass of :code:`paddle.io.Dataset` or :code:`paddle.io.IterableDataset`. feed_list (list(Tensor)|tuple(Tensor)|None, optional): feed Tensor list. The Tensors should be created by :code:`paddle.static.data()`. :attr:`feed_list` must be set if :attr:`return_list` is False. Default None. places(list(Place)|tuple(Place)|list(str)|None, optional): a list of Place, to put data onto, :attr:`places` can be None, if :attr:`places` is None, default place(CPUPlace or CUDAPlace(0)) will be used. Default None. If ``places`` is list of string, the string in the list can be ``cpu``, ``gpu:x`` and ``gpu_pinned``, where ``x`` is the index of the GPUs. return_list (bool, optional): whether the return value on each device is presented as a list. If :attr:`return_list=False`, the return value on each device would be a dict of str -> Tensor, where the key of the dict is the name of each fed Tensors. If :attr:`return_list=True`, the return value on each device would be a list(Tensor). :attr:`return_list` can only be True in dynamic graph mode. Default True. batch_sampler(BatchSampler|None, optional): an instance of `paddle.io.BatchSampler` to generate batch indices to draw samples from :attr:`dataset` and combine a batch. Default None. batch_size(int|None, optional): sample number in a mini-batch, a substitution parameter for :attr:`batch_sampler`, if :attr:`batch_sampler` is not set, a default `paddle.io.BatchSampler` will be used and initialize by :attr:`batch_size`, :attr:`shuffle` and :attr:`drop_last`. Default 1. shuffle(bool, optional): whether to shuffle indices order before generate batch indices, a substitution parameter for :attr:`batch_sampler` see :attr:`batch_size`. Default False. drop_last(bool, optional): whether drop the last incomplete batch dataset size is not divisible by the batch size, a substitution parameter for :attr:`batch_sampler`, see :attr:`batch_size`. Default False collate_fn(Callable|None, optional): function to generate mini-batch data by merging the sample list, None for only stack each fields of sample in axis 0(same as :attr::`np.stack(..., axis=0)`). Default None num_workers(int, optional): the number of subprocess to load data, 0 for no subprocess used and loading data in main process. Default 0 use_buffer_reader (bool, optional): whether to use buffered reader. If use_buffer_reader=True, the DataLoader would prefetch batch data asynchronously, so it would speed up data feeding and occupies a little more CPU or GPU memory, i.e., the memory of one batch input data. Default True. reader_buffer_size (int, optional): This option takes effect only when use_buffer_reader is set to True. It specifies the number of batches the buffer reader prefetches in advance. Note that Increasing this value will result in a linear increase in CPU or GPU memory usage. Default 2. prefetch_factor (int, optional): Number of batch data the DataLoader would prefetch if use_buffer_reader=True. Default 2. use_shared_memory (bool, optional): whether to use shared memory to speed up putting data into inter-process queue, set :attr:`use_shared_memory` as True only when the shared memory space on your machine(e.g. space of '/dev/shm' on Linux operating system) is large enough. Shared memory will only be enabled in multi-process mode(num_workers > 0). Default True. timeout(int, optional): the timeout value for getting data form output queue of subprocesses. Default 0. worker_init_fn(Callable|None, optional): init function which will be called with worker id on each subprocess starting if not set as None. Default None. persistent_workers(bool, optional): whether to keep the workers in the DataLoader. Default False. Returns: DataLoader: an iterable object for data iterating, each element of the generated data is a Tensor. Examples: .. code-block:: python >>> # doctest: +SOLO('can not use multiprocessing testing `paddle.io.DataLoader`') >>> import numpy as np >>> import paddle >>> import paddle.nn as nn >>> import paddle.nn.functional as F >>> from paddle.io import Dataset, BatchSampler, DataLoader >>> BATCH_NUM = 20 >>> BATCH_SIZE = 16 >>> EPOCH_NUM = 4 >>> IMAGE_SIZE = 784 >>> CLASS_NUM = 10 >>> # define a random dataset >>> class RandomDataset(Dataset): # type: ignore[type-arg] ... def __init__(self, num_samples): ... self.num_samples = num_samples ... ... def __getitem__(self, idx): ... image = np.random.random([IMAGE_SIZE]).astype('float32') ... label = np.random.randint(0, CLASS_NUM - 1, (1, )).astype('int64') ... return image, label ... ... def __len__(self): ... return self.num_samples ... >>> dataset = RandomDataset(BATCH_NUM * BATCH_SIZE) >>> class SimpleNet(nn.Layer): ... def __init__(self): ... super().__init__() ... self.fc = nn.Linear(IMAGE_SIZE, CLASS_NUM) ... ... def forward(self, image, label=None): ... return self.fc(image) ... >>> simple_net = SimpleNet() >>> opt = paddle.optimizer.SGD(learning_rate=1e-3, ... parameters=simple_net.parameters()) ... >>> loader = DataLoader(dataset, ... batch_size=BATCH_SIZE, ... shuffle=True, ... drop_last=True, ... num_workers=2) ... >>> for e in range(EPOCH_NUM): ... for i, (image, label) in enumerate(loader()): ... out = simple_net(image) ... loss = F.cross_entropy(out, label) ... avg_loss = paddle.mean(loss) ... avg_loss.backward() ... opt.minimize(avg_loss) ... simple_net.clear_gradients() ... print("Epoch {} batch {}: loss = {}".format(e, i, np.mean(loss.numpy()))) Notes: For reading iterable dataset with multiprocess Dataloader, please see :code:`paddle.io.IterableDataset` rG return_list_CollateFn | None collate_fnuse_buffer_readerintreader_buffer_sizeprefetch_factorCallable[[int], None] | Noneworker_init_fn Dataset[Any]rSequence[Tensor] | None feed_listz list[_Place]rPrcr dataset_kinduse_shared_memorytimeoutz.BatchSampler | _InfiniteIterableSampler | Nonerrauto_collate_batchNc X@lXlXlXlXlUUlXlU(d[5(d UcS5eX lUc [5n[U[[45(a [U5nO [U5n[U5UlU S:dS5eU S:a@["R$S:Xd["R$S:Xa[&R("S5 Sn XlU S:dS5eXlU(a U S:XaSUlUS:dS 5eXl[U[05(a9[2R4UlU(a[9S U35eUb [9S 5eO[2R:UlUb)US :XaU(dU(aS 5eXPlSUlO\UcSUlSUlOJUS:dS5eX`l[U[05(a[AX5UlO[CUUUUS9Ul[DRFRHRJRM5(Gas[DRFRNRPRS5nUcP[DRFRU5n[DRFRW[[YSU55S/S9nSURZ;a [9S5eUR]S[DRFR_55nURaS5n[cUR<R>U- 5Ul[UR<[@5(aSnSnO,UR<RdnUR<Rfn[iUUR>UUUUS9UlXl3UR<SLUl5SUl6[5(a[o5cSO [o5Ul6UUl8SUl9[uU5Rw5Ulg)Nz.feed_list should be set when return_list=Falserz*num_workers should be a non-negative valuerurvzDataLoader with multi-process mode is not supported on MacOs and Windows currently. Please use single-process mode with num_workers = 0 insteadz*prefetch_factor should be a positive valueFz&timeout should be a non-negative valuez5IterableDataset not support shuffle, but got shuffle=z0IterableDataset expect unspecified batch_samplerrzJbatch_size/shuffle/drop_last should not be set when batch_sampler is givenzMbatch_size should be None or a positive value when batch_sampler is not given)rrrrdp) dim_namesz;Auto-DP mode requires the mesh to include a 'dp' dimension.rT),.F ftUm , ,+F3F&v.F%f- aM!MM ? LLH $ (? MMO K&"P$PP"!2 !1%*D "!|EEE| g / / , 1 1D  KG9U( F) !- 0 0D   $?79 ) D"/ "DO  !%D "DO> - >)O'?33%=&"&2#)#' &"    + + 9 9 I I K K%%++0099;D|"..==? ))55q),-$64>>) Q99f((113G!--d3M!$"4"4"?"?-"OPDO$,,.FGG! ,,44 ..88 !8??*# "D #"&"4"4D"@   )+39J9L O$6 #D>224r/cUR[R:Xa [S5eUR(a[ UR 5$[ UR5$)Nz'length of IterableDataset not supported)rrrrrrDrrrys r,__len__DataLoader.__len__bsK    1 1 1FG G&&4--..4<<((r/cURS:Xa [U5$UR(aOURc[ U5UlUR$URR 5 UR$[ U5$)Nr)rcrrrr_resetrys r,__iter__DataLoader.__iter__ksm   q /5 5  % %~~%!> !%%'>> !.t4 4r/c"UR5$r')rrys r,r-DataLoader.__call__ws}}r/)rrrrrrrrrrrcrrPrrrrrrr)NNTNrFFNrTr r TrNF)$rrrrrPz&PlaceLike | Sequence[PlaceLike] | NonerrGrzBatchSampler | NonerrrrGrrGrrrcrrrGrrrrrrGrrrrrrGr5None)r5r)r5r) r6r7r8r9__doc____annotations__r]rrr-r:r(r/r,rr sul\!!00 &&  LAAO .29= -1(,"&"# "&7;#(%W5W5+W57 W5  W5 + W5W5W5W5&W5W5 W5 W5W5 W5 !W5"5#W5$!%W5& 'W5r) 5r/r)r<)> __future__rrrfrZrwrertypingrrrrrr r rbase.frameworkr r r frameworkrr dataloaderrrrdataloader.batch_samplerrrdataloader.dataloader_iterrrrnumberscollections.abcrr numpy.typingnptrpaddle._typingrpaddle._typing.device_liker$paddle.io.dataloader.dataloader_iterrpaddle.io.dataloader.datasetr r!r"r$QUEUE_GET_TIMEOUTrEr>r?rBrIrTrVrr(r/r,rs#   .== 1(1H4 B BOXO.   $ FFRoor/