x-jYrddlmZddlmZmZddlZddlmZddl m Z erddlm Z m Z Gdde Z dS)) annotations) TYPE_CHECKINGAnyN)Layer) Placement ProcessMeshc,eZdZdZd fd Zd d ZxZS) LocalLayera The `LocalLayer` class is a specialized `Layer` for managing distributed tensors during forward and backward passes in a parallelized training environment. It converts distributed tensors to local tensors for computation and then back to distributed tensors as output, ensuring seamless integration with distributed parallelism frameworks. Args: out_dist_attrs (list[tuple[ProcessMesh, list[Placement]]]): A list where each entry is a tuple containing the `ProcessMesh` and the list of `Placement` attributes for the corresponding output tensors. These attributes define the distribution strategy for the outputs. grad_dist_attrs (list[tuple[ProcessMesh, list[Placement]]]): Similar to `out_dist_attrs` but for gradient tensors. The tuple in the list can be None, indicating that the dist_attr of the gradient tensor is same as the corresponding input tensor. Examples: .. code-block:: python >>> from __future__ import annotations >>> import paddle >>> import paddle.distributed as dist >>> from paddle import Tensor >>> from paddle.distributed import ProcessMesh >>> class CustomLayer(dist.LocalLayer): ... def __init__(self, out_dist_attrs, grad_dist_attrs): ... super().__init__(out_dist_attrs, grad_dist_attrs) ... self.local_result = paddle.to_tensor(0.0) ... def forward(self, x): ... mask = paddle.zeros_like(x) ... if dist.get_rank() == 0: ... mask[1:3] = 1 ... else: ... mask[4:7] = 1 ... x = x * mask ... mask_sum = paddle.sum(x) ... mask_sum = mask_sum / mask.sum() ... self.local_result = mask_sum ... return mask_sum >>> # doctest: +REQUIRES(env:DISTRIBUTED) >>> dist.init_parallel_env() >>> mesh = ProcessMesh([0, 1], dim_names=["x"]) >>> dist_attrs = [ ... (mesh, [dist.Partial(dist.ReduceType.kRedSum)]), ... ] >>> local_input = paddle.arange(0, 10, dtype="float32") >>> local_input = local_input + dist.get_rank() >>> input_dist = dist.auto_parallel.api.dtensor_from_local( ... local_input, mesh, [dist.Shard(0)] ... ) >>> custom_layer = CustomLayer(dist_attrs, dist_attrs) >>> output_dist = custom_layer(input_dist) >>> local_value = custom_layer.local_result >>> gathered_values: list[Tensor] = [] >>> dist.all_gather(gathered_values, local_value) >>> print(f"[Rank 0] local_loss={gathered_values[0]}") [Rank 0] local_loss=1.5 >>> print(f"[Rank 1] local_loss={gathered_values[1]}") [Rank 1] local_loss=6.0 >>> print(f"global_loss (distributed)={output_dist}") global_loss (distributed)=7.5 >>> # This case needs to be executed in a multi-card environment >>> # export CUDA_VISIBLE_DEVICES=0,1 >>> # python -m paddle.distributed.launch {test_case}.py out_dist_attrs)list[tuple[ProcessMesh, list[Placement]]]grad_dist_attrsreturnNonecdt||_||_dS)N)super__init__r r )selfr r __class__s l/var/www/html/banglarbhumi/venv/lib/python3.11/site-packages/paddle/distributed/auto_parallel/local_layer.pyrzLocalLayer.__init__ds1 ,.inputsrkwargsc 0t|}t|t|jks0Jdt|dt|jdtt|D]}||r|j|mt jr||j||j}}ne|| j|| j}}n&|j|d|j|d}}tj j ||||||<tj|g|Ri|}t j|}t|t|jks0Jdt|dt|jdg}tt|D]c}|tj j |||j|d|j|ddt j||S) a/ Overrides the base `Layer`'s `__call__` method. Transforms distributed tensors to local tensors before computation, invokes the parent class's `__call__` method, and then transforms the outputs back to distributed tensors based on the specified distribution attributes. zThe number of inputs (z0) does not match the number of grad_dist_attrs (z).NrzThe number of outputs (z8) does not match the number of distribution attributes ()listlenr rangeis_distpaddlein_dynamic_mode process_mesh placements dist_attrdist auto_parallelapidtensor_to_localr__call__utilsflattenr appenddtensor_from_localpack_sequence_as) rrridxmesh placementoutputs list_outs dist_outss rr(zLocalLayer.__call__ms f6{{c$"677777 S[[  befjfzb{b{   877V%%  Cc{""$$ ',4-// "3K4"3K2( #3K1133@"3K1133>( ,S1!4,S1!4$D #04EE3Kys .9999&99L((11 9~~T%8!9!9999 Kc)nn K KnqrvsFoGoG K K K:99 Y((  C   "&99cN',Q/',Q/    |,,Wi@@@r)r r r r rr)rrrrrr)__name__ __module__ __qualname____doc__rr( __classcell__)rs@rr r sfFFP//////0A0A0A0A0A0A0A0Arr ) __future__rtypingrrrpaddle.distributed distributedr$ paddle.nnrrrr rrr?s#"""""%%%%%%%% !!!!!!:99999999BABABABABABABABABABAr