x-jYddlmZddlmZmZddlZddlZddlmZddl m Z ddl m Z m Z erddlmZdd lmZdd lmZed Z dddZ d d!dZdS)") annotations) TYPE_CHECKINGTypeVarN) framework)stream)convert_object_to_tensorconvert_tensor_to_object)Tensor)task)Group_TT tensor_list list[Tensor]tensorr group Group | Nonesync_opboolreturn task | Nonec0tj||||S)a Gather tensors from all participators and all get the result. As shown below, one process is started with a GPU and the data of this process is represented by its group rank. Through the all_gather operator, each GPU will have data from all GPUs. .. image:: https://githubraw.cdn.bcebos.com/PaddlePaddle/docs/develop/docs/api/paddle/distributed/img/allgather.png :width: 800 :alt: all_gather :align: center Args: tensor_list (list): A list of output Tensors. Every element in the list must be a Tensor whose data type should be float16, float32, float64, int32, int64, int8, uint8, bool, bfloat16, complex64 or complex128. tensor (Tensor): The Tensor to send. Its data type should be float16, float32, float64, int32, int64, int8, uint8, bool, bfloat16, complex64 or complex128. group (Group|None, optional): The group instance return by new_group or None for global default group. sync_op (bool, optional): Whether this op is a sync op. The default value is True. Returns: None. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env: DISTRIBUTED) >>> import paddle >>> import paddle.distributed as dist >>> dist.init_parallel_env() >>> tensor_list = [] # type: ignore >>> if dist.get_rank() == 0: ... data = paddle.to_tensor([[4, 5, 6], [4, 5, 6]]) >>> else: ... data = paddle.to_tensor([[1, 2, 3], [1, 2, 3]]) >>> dist.all_gather(tensor_list, data) >>> print(tensor_list) >>> # [[[4, 5, 6], [4, 5, 6]], [[1, 2, 3], [1, 2, 3]]] (2 GPUs) )r all_gather)rrrrs k/var/www/html/banglarbhumi/venv/lib/python3.11/site-packages/paddle/distributed/communication/all_gather.pyrr&s\  [&% A AA object_listlist[_T]objr Nonectjs Jdt|\}}g}t|||t t |}|}tj ||g}tj |}g} t| ||t| D].\} }| t||| /dS)aH Gather picklable objects from all participators and all get the result. Similar to all_gather(), but python object can be passed in. Args: object_list (list): A list of output object. The datatype of every element in the list is same as the input obj. obj (Any): The picklable object to send. group (Group): The group instance return by new_group or None for global default group. Returns: None. Warning: This API only supports the dygraph mode. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env: DISTRIBUTED) >>> import paddle >>> import paddle.distributed as dist >>> dist.init_parallel_env() >>> object_list = [] # type: ignore >>> if dist.get_rank() == 0: ... obj = {"foo": [1, 2, 3]} >>> else: ... obj = {"bar": [4, 5, 6]} >>> dist.all_gather_object(object_list, obj) >>> print(object_list) >>> # [{'foo': [1, 2, 3]}, {'bar': [4, 5, 6]}] (2 GPUs) z4all_gather_object doesn't support static graph mode.N)rin_dynamic_moder rintmaxitemnumpynpresizepaddle to_tensor enumerateappendr ) rrrr len_of_tensorlist_len_of_tensormax_len_of_tensor numpy_data input_tensorris rall_gather_objectr2WsF  $ & &> &5S99FM!=%888C 23388::;;J:(9':;;J#J//LK{L%000{++  6 $V-?-B C C      r)NT) rrrr rrrrrr)N)rrrrrr rr) __future__rtypingrrr%r&r(r paddle.distributed.communicationrserialization_utilsr r r paddle.base.corer &paddle.distributed.communication.groupr rrr2rrr:s.#""""")))))))) 333333 %%%%%%<<<<<< B  .B.B.B.B.Bd48: : : : : : : r