ϑiXaSSKJr SSKrSSKrSSKrSSKJrJrJrJ r J r SSK J r J r Jr SSKrSSKJr \(aSSKJrJrJrJr SSKJr \ "S 5r\ "S 5r"S S \ \5r"S S\\5r"SS\S5rSr"SS\\\\5r"SS\\5r "SS\\5r!SSSjjr"S4S Sjjr#"SS\\5r$g)!) annotationsN) TYPE_CHECKINGAnyCallableGenericTypeVar)Never TypeVarTupleUnpack) framework) GeneratorIterableIteratorSequence)Tensor_T_TscT\rSrSrSrS SjrS SjrS Sjr\(aS Sjr Sr gSr g) Dataset+aS An abstract class to encapsulate methods and behaviors of datasets. All datasets in map-style(dataset samples can be get by a given key) should be a subclass of `paddle.io.Dataset`. All subclasses should implement following methods: :code:`__getitem__`: get sample from dataset with a given index. This method is required by reading dataset sample in :code:`paddle.io.DataLoader`. :code:`__len__`: return dataset sample number. This method is required by some implements of :code:`paddle.io.BatchSampler` see :code:`paddle.io.DataLoader`. Examples: .. code-block:: python >>> import numpy as np >>> from paddle.io import Dataset >>> # define a random dataset >>> class RandomDataset(Dataset): # type: ignore[type-arg] ... def __init__(self, num_samples): ... self.num_samples = num_samples ... ... def __getitem__(self, idx): ... image = np.random.random([784]).astype('float32') ... label = np.random.randint(0, 9, (1, )).astype('int64') ... return image, label ... ... def __len__(self): ... return self.num_samples ... >>> dataset = RandomDataset(10) >>> for i in range(len(dataset)): ... image, label = dataset[i] ... # do something cgNselfs \/var/www/html/banglarbhumi/venv/lib/python3.13/site-packages/paddle/io/dataloader/dataset.py__init__Dataset.__init__U c`[SRSURR55e)N'{}' not implement in class {} __getitem__NotImplementedErrorformat __class____name__ridxs rr$Dataset.__getitem__Xs,! , 3 3t~~66   r!c`[SRSURR55e)Nr#__len__r%rs rr.Dataset.__len___s,! , 3 34>>22   r!cgrrrs r__iter__Dataset.__iter__hsCr!rNreturnNoner+intr4rr4r7r4z Iterator[_T]) r) __module__ __qualname____firstlineno____doc__rr$r.rr1__static_attributes__rr!rrr+s$'R   /r!rc@\rSrSrSrS SjrS SjrS SjrS SjrSr g) IterableDatasetkam An abstract class to encapsulate methods and behaviors of iterable datasets. All datasets in iterable-style (can only get sample one by one sequentially, like a Python iterator) should be a subclass of :ref:`api_paddle_io_IterableDataset` . All subclasses should implement following methods: :code:`__iter__`: yield sample sequentially. This method is required by reading dataset sample in :ref:`api_paddle_io_DataLoader` . .. note:: do not implement :code:`__getitem__` and :code:`__len__` in IterableDataset, should not be called either. see :ref:`api_paddle_io_DataLoader` . Examples: .. code-block:: python :name: code-example1 >>> import numpy as np >>> from paddle.io import IterableDataset >>> # define a random dataset >>> class RandomDataset(IterableDataset): # type: ignore[type-arg] ... def __init__(self, num_samples): ... self.num_samples = num_samples ... ... def __iter__(self): ... for i in range(self.num_samples): ... image = np.random.random([784]).astype('float32') ... label = np.random.randint(0, 9, (1, )).astype('int64') ... yield image, label ... >>> dataset = RandomDataset(10) >>> for img, label in dataset: ... # do something ... ... When :attr:`num_workers > 0`, each worker has a different copy of the dataset object and will yield whole dataset samples, which means samples in dataset will be repeated in :attr:`num_workers` times. If it is required for each sample to yield only once, there are two methods to configure different copy in each worker process to avoid duplicate data among workers as follows. In both the methods, worker information that can be getted in a worker process by `paddle.io.get_worker_info` will be needed. splitting data copy in each worker in :code:`__iter__` .. code-block:: python :name: code-example2 >>> import math >>> import paddle >>> import numpy as np >>> from paddle.io import IterableDataset, DataLoader, get_worker_info >>> class SplitedIterableDataset(IterableDataset): # type: ignore[type-arg] ... def __init__(self, start, end): ... self.start = start ... self.end = end ... ... def __iter__(self): ... worker_info = get_worker_info() ... if worker_info is None: ... iter_start = self.start ... iter_end = self.end ... else: ... per_worker = int( ... math.ceil((self.end - self.start) / float( ... worker_info.num_workers))) ... worker_id = worker_info.id ... iter_start = self.start + worker_id * per_worker ... iter_end = min(iter_start + per_worker, self.end) ... ... for i in range(iter_start, iter_end): ... yield np.array([i]) ... >>> dataset = SplitedIterableDataset(start=2, end=9) >>> dataloader = DataLoader( ... dataset, ... num_workers=2, ... batch_size=1, ... drop_last=True) ... >>> for data in dataloader: ... print(data) # doctest: +SKIP("The output depends on the environment.") Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[2]]) Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[3]]) Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[4]]) Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[5]]) Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[6]]) Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[7]]) Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[8]]) splitting data copy in each worker by :code:`worker_init_fn` .. code-block:: python :name: code-example3 >>> import math >>> import paddle >>> import numpy as np >>> from paddle.io import IterableDataset, DataLoader, get_worker_info >>> class RangeIterableDataset(IterableDataset): # type: ignore[type-arg] ... def __init__(self, start, end): ... self.start = start ... self.end = end ... ... def __iter__(self): ... for i in range(self.start, self.end): ... yield np.array([i]) ... >>> dataset = RangeIterableDataset(start=2, end=9) >>> def worker_init_fn(worker_id): ... worker_info = get_worker_info() ... ... dataset: RangeIterableDataset = worker_info.dataset # type: ignore[assignment] ... start = dataset.start ... end = dataset.end ... num_per_worker = int( ... math.ceil((end - start) / float(worker_info.num_workers))) ... ... worker_id = worker_info.id ... dataset.start = start + worker_id * num_per_worker ... dataset.end = min(dataset.start + num_per_worker, end) ... >>> dataloader = DataLoader( ... dataset, ... num_workers=2, ... batch_size=1, ... drop_last=True, ... worker_init_fn=worker_init_fn) ... >>> for data in dataloader: ... print(data) # doctest: +SKIP("The output depends on the environment.") Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[2]]) Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[3]]) Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[4]]) Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[5]]) Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[6]]) Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[7]]) Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[8]]) cgrrrs rrIterableDataset.__init__ r r!c`[SRSURR55e)Nr#r1r%rs rr1IterableDataset.__iter__s,! , 3 3DNN33   r!c`[SRSURR55e)N/'{}' should not be called for IterableDataset{}r$ RuntimeErrorr'r(r)r*s rr$IterableDataset.__getitem__s, = D Dt~~66   r!c`[SRSURR55e)NrGr.rHrs rr.IterableDataset.__len__s, = D D4>>22   r!rNr3r9)r+r7r4r )r4r ) r)r:r;r<r=rr1r$r.r>rr!rr@r@ks^@    r!r@cB\rSrSr%SrS\S'S SjrS SjrS SjrSr g ) TensorDataseti%a Dataset defined by a list of tensors. Each tensor should be in shape of [N, ...], while N is the sample number, and each tensor contains a field of sample, :code:`TensorDataset` retrieve each sample by indexing tensors in the 1st dimension. Args: tensors(list|tuple): A list/tuple of tensors with same shape in the 1st dimension. Returns: Dataset: a Dataset instance wrapping tensors. Examples: .. code-block:: python >>> import numpy as np >>> import paddle >>> from paddle.io import TensorDataset >>> input_np = np.random.random([2, 3, 4]).astype('float32') >>> input = paddle.to_tensor(input_np) >>> label_np = np.random.random([2, 1]).astype('int32') >>> label = paddle.to_tensor(label_np) >>> dataset = TensorDataset([input, label]) >>> for i in range(len(dataset)): ... input, label = dataset[i] ... # do something Sequence[Tensor]tensorsc^[R"5(d [S5e[U4SjT55(dS5eTUlg)Nz1TensorDataset con only be used in imperative modec3f># UH&oRSTSRS:Hv M( g7f)rN)shape).0tensorrPs r )TensorDataset.__init__..Os. AHvLLOwqz//2 2s.1z0tensors not have same shape of the 1st dimension)r in_dynamic_moderIallrP)rrPs `rrTensorDataset.__init__JsY((**C  AH    > = >  r!cB^[U4SjUR55$)Nc3,># UH oTv M g7frr)rTrUindexs rrV,TensorDataset.__getitem__..Us>vE]s)tuplerP)rr]s `rr$TensorDataset.__getitem__Ts>>>>r!c:URSRS$Nr)rPrSrs rr.TensorDataset.__len__Ws||A$$Q''r!)rPN)rPrOr4r5)r]r7r4ztuple[Tensor, ...]r8 r)r:r;r<r=__annotations__rr$r.r>rr!rrNrN%s D?(r!rNrc^UcU$[U[[45(a [U5$U/$r) isinstancelistr_)values rto_listrj[s. } %$''E{ 7Nr!cB\rSrSr%SrS\S'S SjrS SjrS SjrSr g ) ComposeDatasetica A Dataset which composes fields of multiple datasets. This dataset is used for composing fields of multiple map-style datasets of same length. Args: datasets(list of Dataset): List of datasets to be composed. Returns: Dataset: A Dataset which composes fields of multiple datasets. Examples: .. code-block:: python >>> import numpy as np >>> import paddle >>> from paddle.io import Dataset, ComposeDataset >>> # define a random dataset >>> class RandomDataset(Dataset): # type: ignore[type-arg] ... def __init__(self, num_samples): ... self.num_samples = num_samples ... ... def __getitem__(self, idx): ... image = np.random.random([32]).astype('float32') ... label = np.random.randint(0, 9, (1, )).astype('int64') ... return image, label ... ... def __len__(self): ... return self.num_samples ... >>> dataset = ComposeDataset([RandomDataset(10), RandomDataset(10)]) # type: ignore[var-annotated] >>> for i in range(len(dataset)): ... image1, label1, image2, label2 = dataset[i] ... # do something list[Dataset[Any]]datasetsc~[U5Ul[UR5S:dS5e[UR5Htup#[ U[ 5(dS5e[ U[ 5(aS5eUS:dME[U5[URUS- 5:XaMoS5e g)Nr"input datasets should not be emptyz.each input dataset should be paddle.io.Datasetz'paddle.io.IterableDataset not supportedz"lengths of datasets should be same)rhrnlen enumeratergrr@rrnidatasets rrComposeDataset.__init__sX 4==!A%K'KK%#DMM2JAgw// @ /"'?;; 9 ;1u7|s4==Q+?'@@8@3r!c2[URS5$rb)rrrnrs rr.ComposeDataset.__len__s4==#$$r!cz/nURHnUR[X155 M! [U5$r)rnextendrjr_)rr+samplervs rr$ComposeDataset.__getitem__s2}}G MM'',/ 0%V}r!rnN)rnrmr4r5r8)r4ztuple[Unpack[_Ts]]) r)r:r;r<r=rerr.r$r>rr!rrlrlcs%N!  %r!rlc,\rSrSrSrSSjrSSjrSrg) ChainDatasetia A Dataset which chains multiple iterable-style datasets. This dataset is used for assembling multiple datasets which should be :ref:`api_paddle_io_IterableDataset`. Args: datasets(list of IterableDatasets): List of datasets to be chainned. Returns: paddle.io.IterableDataset: A Dataset which chains fields of multiple datasets. Examples: .. code-block:: python >>> import numpy as np >>> import paddle >>> from paddle.io import IterableDataset, ChainDataset >>> # define a random dataset >>> class RandomDataset(IterableDataset): # type: ignore[type-arg] ... def __init__(self, num_samples): ... self.num_samples = num_samples ... ... def __iter__(self): ... for i in range(10): ... image = np.random.random([32]).astype('float32') ... label = np.random.randint(0, 9, (1, )).astype('int64') ... yield image, label ... >>> dataset = ChainDataset([RandomDataset(10), RandomDataset(10)]) >>> for image, label in iter(dataset): ... # do something ... ... c[U5Ul[UR5S:dS5e[UR5H!up#[ U[ 5(aMS5e g)Nrrpz3ChainDataset only support paddle.io.IterableDataset)rhrnrrrsrgr@rts rrChainDataset.__init__s\X 4==!A%K'KK%#DMM2JAg77 E 73r!c#J# URH nUShvN M gN 7frr~)rrvs rr1ChainDataset.__iter__s}}G  % s #! #r~N)rnzlist[IterableDataset[Any]])r4z Iterator[Any])r)r:r;r<r=rr1r>rr!rrrs%Nr!rcL\rSrSr%SrS\S'S\S'S SjrS SjrSS jrS r g )Subsetia Subset of a dataset at specified indices. Args: dataset (Dataset): The whole Dataset. indices (sequence): Indices in the whole set selected for subset. Returns: List[Dataset]: A Dataset which is the subset of the original dataset. Examples: .. code-block:: python >>> import paddle >>> class RangeDataset(paddle.io.Dataset): # type: ignore[type-arg] ... def __init__(self, start, stop): ... self.start = start ... self.stop = stop ... ... def __getitem__(self, index): ... return index + self.start ... ... def __len__(self): ... return self.stop - self.start >>> # Example 1: >>> a = paddle.io.Subset(dataset=RangeDataset(1, 4), indices=[0, 2]) >>> print(list(a)) [1, 3] >>> # Example 2: >>> b = paddle.io.Subset(dataset=RangeDataset(1, 4), indices=[1, 1]) >>> print(list(b)) [2, 2] Dataset[_T]rv Sequence[int]indicescXlX lgrrvr)rrvrs rrSubset.__init__s   r!c:URURU$rrr*s rr$Subset.__getitem__ s||DLL-..r!c,[UR5$r)rrrrs rr.Subset.__len__ s4<<  r!rN)rvrrrr4r5r6r8rdrr!rrrs%$L /!r!rc V[R"[U5S5(a[U5S::a/n[U5H\upEUS:dUS:a[ SUS35e[ [R "[U5U-55nURU5 M^ [U5[U5- n[U5HnU[U5-nX8==S- ss'M Un[U5H'upIU S:XdM [R"SUS35 M) [U5[U5:wa [ S5e[R"[U55R5n [[!U5U5V V s/sHup[#X X- U 5PM sn n $s sn n f)a Randomly split a dataset into non-overlapping new datasets of given lengths. Optionally fix the generator for reproducible results, e.g.: Args: dataset (Dataset): Dataset to be split lengths (sequence): lengths or fractions of splits to be produced generator (Generator, optional): Generator used for the random permutation. Default is None then the DefaultGenerator is used in manual_seed(). Returns: Datasets: A list of subset Datasets, which are the non-overlapping subsets of the original Dataset. Examples: .. code-block:: python >>> import paddle >>> paddle.seed(2023) >>> a_list = paddle.io.random_split(range(10), [3, 7]) # type: ignore[arg-type, var-annotated] >>> print(len(a_list)) 2 >>> # output of the first subset >>> for idx, v in enumerate(a_list[0]): ... print(idx, v) # doctest: +SKIP("The output depends on the environment.") 0 7 1 6 2 5 >>> # output of the second subset >>> for idx, v in enumerate(a_list[1]): ... print(idx, v) # doctest: +SKIP("The output depends on the environment.") 0 1 1 9 2 4 3 2 4 0 5 3 6 8 rqrzFraction at index z is not between 0 and 1zLength of split at index z- is 0. This might result in an empty dataset.zDSum of input lengths does not equal the length of the input dataset!)mathisclosesumrs ValueErrorr7floorrrappendrangewarningswarnpaddlerandpermtolistzip _accumulater) rvlengths generatorsubset_lengthsrufracn_items_in_split remainder idx_to_add_atlengthroffsets r random_splitrs\ ||CL!$$W): )GAax4!8 (+BC #4::c'lT.A#BC   ! !"2 3 *L3~#66 y!AN 33M  )Q . )"!"7+IA{ /s3=>, 7|s7|# R  ooc'l+224G"+g"6@ @NF w&9:@  sF%c X-$rr)xys rrcsaer!c## [U5n[U5nUv UHnU"X45nUv M g![a gf=f7f)a Return running totals Args: iterable: any iterable object for example dataset. y (x): one element in the iterable object. fn (x, y): Defaults to lambdax. Yields: yields total from beginning iterator to current iterator. Example code: .. code-block:: python >>> list(_accumulate([1, 2, 3, 4, 5])) [1, 3, 6, 10, 15] >>> import operator >>> list(_accumulate([1, 2, 3, 4, 5], operator.mul)) [1, 2, 6, 24, 120] N)iternext StopIteration)iterablefnittotalelements rrrbsQ4 hBR K5"  s$ A 4A AAAAcJ\rSrSrSr\S Sj5rS SjrS SjrS Sjr Sr g) ConcatDatasetia Dataset as a concatenation of multiple datasets. This class is useful to assemble different existing datasets. Args: datasets (sequence): List of datasets to be concatenated Returns: Dataset: A Dataset which concatenated by multiple datasets. Examples: .. code-block:: python >>> import numpy as np >>> import paddle >>> from paddle.io import Dataset, ConcatDataset >>> # define a random dataset >>> class RandomDataset(Dataset): # type: ignore[type-arg] ... def __init__(self, num_samples): ... self.num_samples = num_samples ... ... def __getitem__(self, idx): ... image = np.random.random([32]).astype('float32') ... label = np.random.randint(0, 9, (1, )).astype('int64') ... return image, label ... ... def __len__(self): ... return self.num_samples ... >>> dataset = ConcatDataset([RandomDataset(10), RandomDataset(10)]) # type: ignore[var-annotated] >>> for i in range(len(dataset)): ... image, label = dataset[i] ... # do something cb/Sp!UH%n[U5nURXB-5 X$- nM' U$rb)rrr)sequencersels rcumsumConcatDataset.cumsums711AAA HHQUO FAr!c[U5Ul[UR5S:dS5eURHn[U[5(dMS5e UR UR5Ulg)Nrz(datasets should not be an empty iterablez.ConcatDataset does not support IterableDataset)rhrnrrrgr@rcumulative_sizes)rrnds rrConcatDataset.__init__srX 4==!A% 6 %A!!_55 @ 5!% DMM :r!c URS$)N)rrs rr.ConcatDataset.__len__s$$R((r!cUS:a)U*[U5:a [S5e[U5U-n[R"URU5nUS:XaUnOXRUS- - nUR UU$)Nrz8absolute value of index should not exceed dataset lengthrq)rrrbisect bisect_rightrrn)rr+ dataset_idx sample_idxs rr$ConcatDataset.__getitem__s 7tc$i Nd)c/C))$*?*?E ! J44[1_EEJ}}[)*55r!)rrnN)rz Sequence[Any]r4z list[int])rnzIterable[Dataset[Any]]r4r5r8r6) r)r:r;r<r= staticmethodrrr.r$r>rr!rrrs+$L ;) 6r!rr)rvrrrrz Any | Noner4zlist[Subset[_T]])rz Iterable[_T]rzCallable[[_T, _T], _T]r4zGenerator[_T, None, None])% __future__rrrrtypingrrrrrtyping_extensionsr r r rr collections.abcrrrrrrrrr@rNrjr_rlrrrrrrr!rrs/# :9 GG T]5=0gbk=0@w gbkw t3(GH%3(l@WU6#;/0@F2?3'2j2!WR[2!p!O O OO Of:L"" 6""JJ6GBKJ6r!