BjG\ddlZddlZddlmZddlmZddlmZmZdddZ dZ ej fdZ dej fdZ dej fd Zejej fd Zejej fd Zej fd Zej fd Zej fdZddej fdZejej fdZGddeZGddeZGddeZGddeZGddeZGddeZGddeZGdd eZGd!d"eZGd#d$eZ dS)%N)Function)groupReduceOp suggestioncBd|d}|r |d|dz }t|)N torch.distributed.nn.functional.z& is not supported under torch.compile.z Use instead.) RuntimeError)namermsgs b/var/www/html/Carbon-Document/venv/lib/python3.11/site-packages/torch/distributed/nn/functional.py_not_supported_under_compilersCW4WWW- ,z,,,, s  cLtjd|d|dtddS)Nr z is deprecated, use r )category stacklevel)warningswarn FutureWarning)r rs r _deprecatedrsN M %4 % % % % % rctjrtddt ddt |||S)a Broadcasts the tensor to the whole group. ``tensor`` must have the same number of elements in all processes participating in the collective. Arguments: tensor (Tensor): Data to be sent if ``src`` is the rank of current process. src (int): Source rank. group (ProcessGroup, optional): The process group to work on. Returns: Tensor: Received tensor from the broadcast op. broadcastz3torch.distributed._functional_collectives.broadcastr)torchcompiler is_compilingrr _Broadcastapply)tensorsrcrs rrr sb" ~""$$ $ L     RSSS   C / //rctjrtdt|||S)aT Gathers a list of tensors in a single process. Arguments: tensor (Tensor): Input tensor. dst (int, optional): Destination rank (default is 0). group (ProcessGroup, optional): The process group to work on. Returns: tuple[Tensor]: List of appropriately-sized tensors with the gathered data. gather)rrrr_Gatherr)r dstrs rr#r#:s> ~""$$/$X... ==eV , ,,rctjrtdt j||g|RS)a Scatters a list of tensors to all processes in a group. Each process will receive exactly one tensor and store its data in the ``tensor`` argument. Arguments: tensors (list[Tensor]): List of tensors to scatter on the source rank. Receivers must pass ``None`. src (int, optional): Source rank (default is 0). group (ProcessGroup, optional): The process group to work on. Returns: Tensor: Output tensor from the scatter operation. scatter)rrrr_Scatterr)tensorsr!rs rr'r'KsB" ~""$$0$Y/// >#u /w / / //rctjrtdt||||S)a Reduces the tensor data across all machines. Only the process with rank ``dst`` is going to receive the final result. Arguments: tensor (Tensor): Input of the collective. dst (int): Destination rank. op (optional): One of the values from ``torch.distributed.ReduceOp`` enum. Specifies an operation used for element-wise reductions. group (ProcessGroup, optional): The process group to work on. Returns: Tensor: Output of the collective. reduce)rrrr_Reducer)r r%oprs rr+r+as@$ ~""$$/$X... ==b% 0 00rctjrtddt ddt j|||g|RS)a Reduces, then scatters a list of tensors to all processes in a group. Arguments: output (Tensor): Output tensor. input_list (list[Tensor]): List of tensors to reduce and scatter. op (optional): One of the values from ``torch.distributed.ReduceOp`` enum. Specifies an operation used for element-wise reductions. group (ProcessGroup, optional): The process group to work on. Returns: Tensor: Output of the collective. reduce_scatterz?torch.distributed._functional_collectives.reduce_scatter_tensorr)rrrrr_Reduce_Scatterr)output input_listr-rs rr/r/xsp  ~""$$ $ X    I  UF @Z @ @ @@rctjrtddt ddt ||S)a Gathers tensors from the whole group in a list. Arguments: tensor (Tensor): Tensor to be broadcast from current process. group (ProcessGroup, optional): The process group to work on. Returns: tuple([Tensor]): Output of the collective. all_gatherz;torch.distributed._functional_collectives.all_gather_tensorr)rrrrr _AllGatherr)r rs rr4r4si ~""$$ $ T    S   E6 * **rctjrtdt|||S)a Single tensor all gather. Gathers a single tensor from all ranks, and puts them in a single output tensor. Args: output_tensor (Tensor): Output tensor. It should contain correctly-sized tensors to be used for output of the collective. input_tensor (Tensor): Tensor to be broadcast from current process. group (ProcessGroup, optional): The process group to work on. If None, the default process group will be used. Examples: >>> # All tensors below are of torch.int64 dtype. >>> # We have 2 process groups, 2 ranks. >>> # xdoctest: +SKIP("incorrect want text") >>> output_tensor = torch.zeros(2, dtype=torch.int64) >>> output_tensor [tensor([0, 0])] # Rank 0 and 1 >>> tensor = torch.arange(1, dtype=torch.int64) + 1 + rank >>> tensor tensor([1]) # Rank 0 tensor([2]) # Rank 1 >>> dist.all_gather_base(output_tensor, tensor) >>> output_tensor tensor([1,2]) # Rank 0 tensor([1,2]) # Rank 1 .. warning:: `_all_gather_base` is experimental and subject to change. It is the caller's responsibility to ensure the output_tensor is correctly sized. _all_gather_base)rrrr_AllGatherBaser) output_tensor input_tensorrs rr7r7sBB ~""$$9$%7888    |U C CCrctjrtdt j||g|RS)a Each process scatters list of input tensors to all processes in a group and return gathered list of tensors in output list. Arguments: output_tensor_list (list[Tensor]): list of tensors to gather one per rank. input_tensor_list (list[Tensor]): List of tensors to scatter one per rank. group (ProcessGroup, optional): The process group to work on. Returns: tuple([Tensor]): Output of the collective. all_to_all)rrrr _AlltoAllr)output_tensor_listinput_tensor_listrs rr<r<sD ~""$$3$\222 ?5"4 I7H I I IIrctjrtddt ddt |||||S)a Each process splits input tensor and then scatters the split list to all processes in a group. Then concatenate the received tensors from all the processes in the group and return single output tensor. Arguments: output (Tensor): Gathered concatenated output tensor. input (Tensor): Input tensor to scatter. output_split_sizes: (list[Int], optional): Output split sizes for dim 0 if specified None or empty, dim 0 of ``output`` tensor must divide equally by ``world_size``. input_split_sizes: (list[Int], optional): Input split sizes for dim 0 if specified None or empty, dim 0 of ``input`` tensor must divide equally by ``world_size``. Returns: Tensor: Output of the collective. all_to_all_singlez;torch.distributed._functional_collectives.all_to_all_singler)rrrrr_AlltoAllSingler)r1inputoutput_split_sizesinput_split_sizesrs rrArAsx4 ~""$$ $ T    E   v)+z#_Gather.forward..Fs1   )*E V $ $   rrL) r%rrangerMget_world_size contiguousrNr#tuple)rQr%rr tensor_lists ` rrRz_Gather.forward=s     .3D4Ge4T4T4T.U.U   ""$$ =u % % % , , K S > > > > > Kc 7 7 7 7[!!!rcDdtj|j|jg|RfzSNNN)r(rr%r)rQ grad_outputss rrYz_Gather.backwardQs(x~cgsyP<PPPRRRrNrZr_rrr$r$<sM""\"$SS\SSSrr$c:eZdZedZedZdS)r(cJ||_||_tfdDstt jd}t j||kr&t j|t||nt j|d|||S)Nc3xK|]4}|dkV5dS)rNsize)rdtr)s r z#_Scatter.forward..\s>BBQ16688wqz000BBBBBBrrrL) r!rallAssertionErrorrrcrMrNr'list)rQr!rr)r1s ` rrRz_Scatter.forwardWs BBBB'BBBBB ! !'!*-- =u % % % , , Lg5 A A A A A Ls% 8 8 8 8 rcTdt|j|j|zSrm)r$rr!rrQrWs rrYz_Scatter.backwardes#gmmCGSY LLLLrNrZr_rrr(r(VsM  \ MM\MMMrr(c:eZdZedZedZdS)r,cz||_||_|}tj|||||S)Nr-r)r!rrPrMr+)rQr!r-rr s rrRz_Reduce.forwardls=  FCBe4444 rcVdt|j|j|fzSN)NNN)rrr!rr{s rrYz_Reduce.backwardus("Z%5%5cgsy+%V%V$XXXrNrZr_rrr,r,ksM\YY\YYYrr,c:eZdZedZedZdS)r0c||_|}td|D}tj|t ||||S)Nc3>K|]}|VdSrTrirdrus rrvz*_Reduce_Scatter.forward..s*!L!LQ!,,..!L!L!L!L!L!Lrr~)rrirjrMr/ry)rQr-rr r?s rrRz_Reduce_Scatter.forward|sb ""$$!!L!L:K!L!L!LLL FD):$;$;%PPPP rcHdt|j|zSr)r5rrr{s rrYz_Reduce_Scatter.backwards!"J$4$4SY $L$LLLrNrZr_rrr0r0{sM\MM\MMMrr0c:eZdZedZedZdS)r5c||_fdttj|D}tj||t |S)Nc8g|]}tjSr_r empty_like)rd_r s rrfz&_AllGather.forward..s1   )*E V $ $   rrL)rirrgrMrhr4rj)rQrr out_tensor_lists ` rrRz_AllGather.forwards""$$     .3D4Ge4T4T4T.U.U    u====_%%%rctj|jtjjtjjfvrXtj|j}tj||}tj tj |j|g|R}nLd|D}tj |j|g|R}tjtj|d}d|fS)NrLc6g|]}tj|Sr_r)rdr s rrfz'_AllGather.backward..s#OOO5+F33OOOrr)dim)rM get_backendrBackendNCCLXCCLrNrrr0rrrUr=sumstack)rQrorOrXrkgxss rrYz_AllGather.backwards  #) , , ,1BDLDU0V V V=sy111D!,t"455B &x|SYR\RRRBBPO,OOOK/#)[H<HHHC5;s++333BbzrNrZr_rrr5r5sH & &\ &  \   rr5c:eZdZedZedZdS)r8cf||_tj||||SrK)rrMr7ri)rQr9r:rs rrRz_AllGatherBase.forwards5  m\-D-D-F-FeTTTTrc<tj|jtjjtjjfvrtj|j}t|}|d|zdkrtd|d||dtj|jz|d<tj ||j |j }tj||tj|jntdd|dfS)NrLrzTensor with dimensions: z8 does not have first dimension divisible by world_size: devicedtypezBackend not supported!)rMrrrrrrhryrtr remptyrr_reduce_scatter_baserrU)rQrW world_sizeout_sizerXs rrYz_AllGatherBase.backwards(  #) , , ,1BDLDU0V V V,39===JK,,..//H{Z'1,,"VxVVISVV#1+)<39)M)M)MMHQK!3;;LB  %b+x|SY O O O O788 8b$rNrZr_rrr8r8sH\   \   rr8c:eZdZedZedZdS)r=c<||_fdttj|D|_tj|}t dDtj|tjj ur[ttj|D]7}d}||krt}tj |||||8n$tj |t|t |S)NcDg|]}|Sr_rs)rdrer)s rrfz%_AlltoAll.forward..s4& & & "#GAJOO  & & & rrLc3>K|]}|VdSrTrrs rrvz$_AlltoAll.forward..s*881 888888r) rrgrMrhinput_tensor_size_listrNrjrrGLOOryr'r<)rQrrr)my_rankreto_sends ` rrRz_AlltoAll.forwards< & & & & ',T-@u-M-M-M'N'N& & & "-e,,,8888888  % ( ( (DL,= = =4.U;;;<< J J<<"7mmG _Q/!5IIIII  J OW      _%%%rc`fd|jD}dtj|j|gRzS)Ncjg|]/}tj|djdj0S)rr)rrrr)rdrtros rrfz&_AlltoAll.backward..sQ    K\!_3<?;P      rrn)rr=rr)rQrorks ` rrYz_AlltoAll.backwardsS    2    iociT|TTTTTrNrZr_rrr=r=sM&&\&,UU\UUUrr=c:eZdZedZedZdS)rBc||_||_||_||_t j||||||S)N)rDrEr)rrt input_sizerDrErMrA)rQrr1rDrErCs rrRz_AlltoAllSingle.forwardsZ !2 2   1/      rc tj|j|j|j}dt |j||j|j | fzS)Nr)NNNN) rrrrrrBrrrDrEri)rQrWr s rrYz_AlltoAllSingle.backwardss N;#5[=N   (  ! ! &%&&((   +   rNrZr_rrrBrBsH  \    \    rrBc:eZdZedZedZdS)rHc||_||_|tj}t j||||S)N) memory_formatr~)rr-rPrcontiguous_formatrMrG)rQr-rr s rrRz_AllReduce.forwardsD E,CDD 2U3333 rcVdt|j|j|fzSrm)rHrr-rr{s rrYz_AllReduce.backwards(z// ;OOQQQrNrZr_rrrHrH sM\RR\RRRrrH)!rrtorch.distributed distributedrMtorch.autogradrrrrrWORLDrr#r'rUr+r/r4r7r<rArGrr$r(r,r0r5r8r=rBrHr_rrrs ###### .-------6:"'00004 ----"%+0000,$ #D#D#D#DL=BKJJJJ* + %%%%P#,ek////6        ,SSSSShSSS4MMMMMxMMM* Y Y Y Y Yh Y Y Y MMMMMhMMM"<     X   :!U!U!U!U!U!U!U!UH     h   D R R R R R R R R R Rr