o eih$@sddlZddlmZddlmZmZddlZddlmZddl m Z ddl m Z ddl mZm Z dZGd d d e Zd ed ed efddZGdddZe ed ddejdeejjdBd ejfddZdejd efddZdS)N)Iterable)AnyNoReturn)_State)DistributedDataParallel) _get_registrycontractc seZdZUejed<dfdd Zefdej de ej de ej d e ddf d d Z dd d Zdej de ej ddfddZdddZdddZdej deedfdee efdefddZdej deejdejdejfddZZS)_ReplicateState _ddp_weakrefreturnNcsNtt|_d|_t|_|j|_g|_d|_ d|_ i|_ g|_ dS)NF) super__init__nn ParameterListmodulehas_initialized _param_list _orig_module _param_names_no_sync _init_args _init_kwargs_comm_hook_argsself __class__q/var/www/addictedbytheproject.nl/epg/venv/lib/python3.10/site-packages/torch/distributed/_composable/replicate.pyrs    z_ReplicateState.__init__rignored_modulesignored_paramsprefixc Cst|rdS||vr dS|tkr|dnt}|jddD]\}}||vr5|j||j||q|D]\}} |j| ||||dq:dS)N.F)recurse)r#)_is_fully_sharded_ROOT_MODULE_PREFIXnamed_parametersrappendrnamed_children_collect_params) rrr!r"r#recurse_prefixnpname child_modulerrr r+"s&  z_ReplicateState._collect_paramscs"tjddfdd}|dS)NT) recursivecs:jdusJjjijd_i_dS)Nr)rinitrregister_comm_hookrrrr _lazy_initCs  z-_ReplicateState.lazy_init.._lazy_init)torch_disable_dynamo)rr4rrr lazy_initBs  z_ReplicateState.lazy_initcKs|jrdSd|_||_dd|D}|D] }t|r#||qddlm}|||d||||d|vrd|ddur[|d}t |t j rU|j dkrUd|d <n |g|d <nd|d <| dt|jfi||_t|jt|j_dS) NTcSsh|] }|D]}|qqSr) parameters).0mr.rrr Xsz'_ReplicateState.init..r)_localize_dtensor)r" device_idcpu device_ids)rrmodulesr&updater8%torch.distributed.tensor.parallel.ddpr<r+ isinstancer5devicetypepoprr_ddpweakrefref replicatestater )rrr!kwargsr" submoduler<r=rrr r2Ms,       z_ReplicateState.initcCs0|jD] \}}|jj|i|q|jdSN)rrGr3clear)r comm_args comm_kwargsrrr r3wsz"_ReplicateState.register_comm_hookcOs||_||_dSrN)rrrargsrLrrr record_init_args|s z _ReplicateState.record_init_argsrS.rLcCs:|js|jr ||j |j_|jt_|jj|i|SrN) rrr7rrGrequire_backward_grad_syncr_active_ddp_module _pre_forward)rrrSrLrrr forward_pre_hooks  z _ReplicateState.forward_pre_hookinputoutputcCsdt_|j|SrN)rrVrG _post_forward)rrrYrZrrr forward_post_hooks z!_ReplicateState.forward_post_hookr N)__name__ __module__ __qualname__rHrI__annotations__rr'rModuleset Parameterstrr+r7r2r3rTtuplerdictrXr5Tensorr\ __classcell__rrrr r sV    *     r rSrLr cOstd)NzGDDP does not support deepcopy. Please use state dict for serialization.)AssertionError)rSrLrrr unimplemented_deepcopysrkc@s0eZdZddZdeddfddZd dd ZdS) DDPcOs"|jd}|j|g|Ri|S)z Override ``__new__`` to remove the DDP class and directly construct the original class for cases like indexing into a container module. )__mro____new__)clsrSrLorig_clsrrr ros z DDP.__new__requires_gradient_syncr NcCs| t|_dS)a Sets if the module should sync gradients. This can be used to implement gradient accumulation without communication. Args: requires_gradient_sync (bool): Whether to reduce gradients for the module's parameters. N)rJrKr)rrrrrr set_requires_gradient_syncs zDDP.set_requires_gradient_synccOst|j||fdSrN)rJrKrr)rRrrr r3szDDP.register_comm_hookr])r^r_r`roboolrsr3rrrr rls  rl) state_clsrr!c Kstjdd|vrt|dttjfstdt|dt|r'td|dur.i}nt |}t |}|j |j dd|d}|durb|}||krbd d lm}m}| |||||j|j||fi||j}d ti} td |jt|f| } | |_|S) zReplicates a module Args: module (torch.nn.Module): module to replicate Example:: >>> # xdoctest: +REQUIRES(module:torch._C._distributed_c10d) >>> module = nn.Linear(3, 3) >>> replicate(module) ztorch.distributed.replicater=z6Expected device_id to be int or torch.device, but got zGCannot apply `replicate()` on a Module already managed by `fully_shard`NT) with_kwargs device_meshr)r<_reconstruct_dtensor __deepcopy__rl)r5_C_log_api_usage_oncerCintrD RuntimeErrorrEr&rcrJrKregister_forward_pre_hookrXget_get_root_meshrBr<rxregister_forward_hookr\rTrrkr^rl) rr!rLrKrw root_meshr<rxrpdctnew_clsrrr rJs>       rJcCst|}|dur dSd|vS)z+Check if module is marked with fully_shard.NF fully_shard)r)rregistryrrr r&sr&rN)rHcollections.abcrtypingrrr5torch.nnr#torch.distributed._composable_statertorch.nn.parallelrr rr'r rkrlrbrJrtr&rrrr s.     F