o ei@s^ddlZddlZddlmZddlZgdZdZddZddZ dd Z d d Z d ej eej Bd dfddZ deddfddZddeddfddd eej dej eej BdBdededeejjdBdeej dBd dfddZ d d eej ded dfddZ  d!d eej deej d dfddZeddfd eej deej ded dfddZdS)"N)Sequence) all_reducereduce broadcast all_gatherreduce_scattercCsxttjdstjddddSt}|D]%}|jrdS|s#dS|js)dS| }||vr4dS| |qdS)N_nccl_all_reducez)PyTorch is not compiled with NCCL support stacklevelFT) hasattrtorch_Cwarningswarnset is_sparse is_contiguousis_cuda get_deviceadd)tensorsdevicestensordevicerY/var/www/addictedbytheproject.nl/epg/venv/lib/python3.10/site-packages/torch/cuda/nccl.py is_availables  rcCsTtj}|d?}|d?d@}|d@}tjd}|dkr$|||fS||||fS)a Returns the version of the NCCL. This function returns a tuple containing the major, minor, and patch version numbers of the NCCL. The suffix is also included in the tuple if a version suffix exists. Returns: tuple: The version information of the NCCL. izutf-8)r r _nccl_version_nccl_version_suffixdecode)vermajorminorpatchsuffixrrrversion#s    r)cCs tjSN)r r_nccl_unique_idrrrr unique_id8s r,cCstj|||Sr*)r r_nccl_init_rank) num_ranksuidrankrrr init_rank<sr1inputsreturncCs&t|tjjr t|tjrtddS)Nz(Inputs should be a collection of tensors) isinstance collectionsabc Containerr Tensor TypeError)r2rrr_check_sequence_type@s r:cCs4t||dur |}t|tj|||||dSr*)r:r rrr2outputsopstreamscommsrrrrGs r)r<outputrootr=r>r<cCst||dur|durtdtjdtdd||}n$t|tjs7t|tj j r7tjdtdd||}n |dur?||n|}tj ||||||dS)Nz'output' and 'outputs' can not be both specified. 'outputs' is deprecated in favor of 'output', taking in a single output tensor. The signature of reduce is: reduce(inputs, output=None, root=0, op=SUM, streams=None, comms=None).z`nccl.reduce` with an output tensor list is deprecated. Please specify a single output tensor with argument 'output' instead instead.r r z\nccl.reduce with an output tensor list is deprecated. Please specify a single output tensor.) r: ValueErrorrr FutureWarningr4r r8r5r6rr _nccl_reduce)r2r@rAr=r>r?r<_outputrrrrQs.   rcCst|tj||||dSr*)r:r r_nccl_broadcast)r2rAr>r?rrrr|srcCs&t|t|tj||||dSr*)r:r r_nccl_all_gather)r2r<r>r?rrrrsrcCs(t|t|tj|||||dSr*)r:r r_nccl_reduce_scatterr;rrrrsr)rNN)NN)r5rcollections.abcr torch.cudar __all__SUMrr)r,r1r8r:rintcudaStreamrrrrrrrrs~    ,