Αi"SSKJr SrSrSrg))corecUS:a[US-5nUS-nO[US-5nUS-n[US-5n[US-5nUSUSU3$)Ni'id.)int)verNCCL_MAJOR_VERSIONNCCL_MINOR_VERSIONNCCL_PATCH_VERSIONs c/var/www/html/banglarbhumi/venv/lib/python3.13/site-packages/paddle/distributed/utils/nccl_utils.pyget_nccl_version_strr sq e| .Ek -DjSCZS3Y !#5"6a8J7K LLch[R"5n[U5nSnX:d SUS35eg)Ni zrThe version of NCCL is required to be at least v2.8.4 while training with pipeline/MoE parallelism, but we found vz. The previous version of NCCL has some bugs in p2p communication, and you can see more detailed description about this issue from ReleaseNotes of NCCL v2.8.4 (https://docs.nvidia.com/deeplearning/nccl/release-notes/rel_2-8-4.html#rel_2-8-4).)r nccl_versionr )rnccl_version_strnccl_version_baselines r check_nccl_version_for_p2pr!sL$$&L+L9   0 33C2DE^ ^ 0rc8[R"5nSnX:$)NiR)rr)rrs r check_nccl_version_for_bf16r.s$$&L!  00rN) paddle.baserr rrrr rs  M 1r