ΑiYtSSKJr SSKJrJr SSKrSSKJr SSK J r \(aSSKJ r J r "SS\ 5r g)) annotations) TYPE_CHECKINGAnyN)Layer) Placement ProcessMeshcF^\rSrSrSrSU4SjjrSSjrSrU=r$) LocalLayera" The `LocalLayer` class is a specialized `Layer` for managing distributed tensors during forward and backward passes in a parallelized training environment. It converts distributed tensors to local tensors for computation and then back to distributed tensors as output, ensuring seamless integration with distributed parallelism frameworks. Args: out_dist_attrs (list[tuple[ProcessMesh, list[Placement]]]): A list where each entry is a tuple containing the `ProcessMesh` and the list of `Placement` attributes for the corresponding output tensors. These attributes define the distribution strategy for the outputs. grad_dist_attrs (list[tuple[ProcessMesh, list[Placement]]]): Similar to `out_dist_attrs` but for gradient tensors. The tuple in the list can be None, indicating that the dist_attr of the gradient tensor is same as the corresponding input tensor. Examples: .. code-block:: python >>> from __future__ import annotations >>> import paddle >>> import paddle.distributed as dist >>> from paddle import Tensor >>> from paddle.distributed import ProcessMesh >>> class CustomLayer(dist.LocalLayer): ... def __init__(self, out_dist_attrs, grad_dist_attrs): ... super().__init__(out_dist_attrs, grad_dist_attrs) ... self.local_result = paddle.to_tensor(0.0) ... def forward(self, x): ... mask = paddle.zeros_like(x) ... if dist.get_rank() == 0: ... mask[1:3] = 1 ... else: ... mask[4:7] = 1 ... x = x * mask ... mask_sum = paddle.sum(x) ... mask_sum = mask_sum / mask.sum() ... self.local_result = mask_sum ... return mask_sum >>> # doctest: +REQUIRES(env:DISTRIBUTED) >>> dist.init_parallel_env() >>> mesh = ProcessMesh([0, 1], dim_names=["x"]) >>> dist_attrs = [ ... (mesh, [dist.Partial(dist.ReduceType.kRedSum)]), ... ] >>> local_input = paddle.arange(0, 10, dtype="float32") >>> local_input = local_input + dist.get_rank() >>> input_dist = dist.auto_parallel.api.dtensor_from_local( ... local_input, mesh, [dist.Shard(0)] ... ) >>> custom_layer = CustomLayer(dist_attrs, dist_attrs) >>> output_dist = custom_layer(input_dist) >>> local_value = custom_layer.local_result >>> gathered_values: list[Tensor] = [] >>> dist.all_gather(gathered_values, local_value) >>> print(f"[Rank 0] local_loss={gathered_values[0]}") [Rank 0] local_loss=1.5 >>> print(f"[Rank 1] local_loss={gathered_values[1]}") [Rank 1] local_loss=6.0 >>> print(f"global_loss (distributed)={output_dist}") global_loss (distributed)=7.5 >>> # This case needs to be executed in a multi-card environment >>> # export CUDA_VISIBLE_DEVICES=0,1 >>> # python -m paddle.distributed.launch {test_case}.py c:>[TU]5 XlX lg)N)super__init__out_dist_attrsgrad_dist_attrs)selfrr __class__s l/var/www/html/banglarbhumi/venv/lib/python3.13/site-packages/paddle/distributed/auto_parallel/local_layer.pyrLocalLayer.__init__ds ,.c [U5n[U5[UR5:Xd*S[U5S[UR5S35e[[U55HnXR 5(dMURUcn[ R "5(aXRXRpTO[XR5RXR5RpTO#URUSURUSpT[RRRXXE5X'M [R"U/UQ70UD6n[ R R#U5n[U5[UR$5:Xd*S[U5S[UR$5S35e/n[[U55H_nUR'[RRR)XsUR$USUR$US55 Ma [ R R+Xh5$)a Overrides the base `Layer`'s `__call__` method. Transforms distributed tensors to local tensors before computation, invokes the parent class's `__call__` method, and then transforms the outputs back to distributed tensors based on the specified distribution attributes. zThe number of inputs (z0) does not match the number of grad_dist_attrs (z).rzThe number of outputs (z8) does not match the number of distribution attributes ()listlenrrangeis_distpaddlein_dynamic_mode process_mesh placements dist_attrdist auto_parallelapidtensor_to_localr__call__utilsflattenrappenddtensor_from_localpack_sequence_as) rinputskwargsidxmesh placementoutputs list_outs dist_outss rr%LocalLayer.__call__msO f6{c$"6"677 $S[M1abefjfzfzb{a||~  7V%C{""$$'',4--//"K44"K22( #K113@@"K113>>( ,,S1!4,,S1!4$ #0044EEK '&...99&9LL((1 9~T%8%8!99 %c)n%55mnqrvsFsFoGnHHJ K 9 Y(C   ""&&99N'',Q/'',Q/ )||,,W@@r)rr)r)list[tuple[ProcessMesh, list[Placement]]]rr4returnNone)r+rr,rr5r) __name__ __module__ __qualname____firstlineno____doc__rr%__static_attributes__ __classcell__)rs@rr r s8FP/A/C/  /0A0Arr ) __future__rtypingrrrpaddle.distributed distributedr! paddle.nnrrrr rrrDs,#% !9BABAr