o ei@s4UddlZddlZddlZddlZddlmZmZmZddlm Z m Z m Z ddl m Z ddlmZmZmZmZddlZddlmZddlmZddlmZddlmZmZmZmZm Z m!Z!ddl"m#Z#dd l$m%Z%m&Z&m'Z(m)Z)m*Z*m+Z+m,Z,m-Z-dd l.m/Z/m0Z0dd l1m2Z2dd l3m4Z4dd l5m6Z7ddl8m9Z9gdZ:dZ;dZe?e@ZAee2eejBeCeDe@fZEeeEeFeEeGeEeHe@dffZIeHe@eIfZJeFeJZKeHe@eJeKBfZLe?ZMe?eeNd<ejOddZPe GdddZQe GdddeQZR   dcdejSde@d e@d!eTd"eTd#eAf d$d%ZUGd&d'd'ZVddd(d)ZWddd*dejSd+eGejXjYd,fd-eTd.e?ejSdBd/eQdBd#eRf d0d1ZZd2eHe@eIfd3eLd4eRd#dfd5d6Z[d7ejSejXjYBd8e@d#efd9d:Z\d;eHe@efd4eRd#eHe@effdd?Z_e^dejSd;eHe@eIfd4eRd#e4fd@dAZ`dBejXjYd#dfdCdDZad;eLd#eHe@eIffdEdFZbdBejXjYd;eHe@eIfd4eRd#eLfdGdHZce^dejSdIeGejXjYd,fd4eRd#eLfdJdKZddejSdBejXjYd3eLd4eRd#eLf dLdMZee^dejSdIeGejXjYd,fd;eLd4eRd#df dNdOZfddd*dejSd.e?ejSdBd/eQdBd#eHe@eIffdPdQZgddd*dejSdIejXjYeejXjYBd.e?ejSdBd/eQdBd#eLf dRdSZhddd*dejSdIejXjYeejXjYBd.e?ejSdBd/eQdBd#eGeHe@eIfeLff dTdUZidejSd;eHejSeHe@eIffeHe@eIfBd#eHe@eIffdVdWZjddXdejSd2eHe@eIfd/eQdBd#e4fdYdZZkddXdejSdIejXjYeejXjYBd3eLd/eQdBd#df d[d\ZlddXdejSdIejXjYeejXjYBd2eHe@eIfd3eLd/eQdBd#e4f d]d^ZmeddXdejSd/eQdBd#dfd_d`ZneddXdejSdIeGejXjYd,fd/eQdBd#dfdadbZodS)eN)Callable GeneratorIterable)asdict dataclassfield)chain)Anycast no_type_checkUnion) ShardedTensor)_broadcast_state_dict_distribute_state_dict_flatten_state_dict_gather_state_dict_offload_state_dict_to_cpu_unflatten_state_dict)_CHECKPOINT_PREFIX)FullOptimStateDictConfigFullStateDictConfigFullyShardedDataParallelOptimStateDictConfigShardedOptimStateDictConfigShardedStateDictConfigStateDictConfig StateDictType)._get_module_fsdp_state_if_fully_sharded_moduleFSDP_WRAPPED_MODULE)DTensor)_IncompatibleKeys)DistributedDataParallel) tree_map_only) FQNS_T PrimitiveType ValueType DictValueTypeListDictValueTypeOptimizerStateTypeStateDictOptionsget_model_state_dictget_optimizer_state_dictget_state_dictset_model_state_dictset_optimizer_state_dictset_state_dict _flat_param param_groupsparamsstater%_patched_state_dictccs@t}tzdVW|rtdSdS|rtwwN)gc isenableddisableenable) is_enabledr;q/var/www/addictedbytheproject.nl/epg/venv/lib/python3.10/site-packages/torch/distributed/checkpoint/state_dict.py _gc_contextQs  r=c@sreZdZUdZdZeed<dZeed<dZeed<dZ eed<dZ eed<dZ eed <dZ eed <d Z eed <d S)r)ap This dataclass specifies how get_state_dict/set_state_dict will work. - ``full_state_dict``: if this is set to True, all the tensors in the returned state_dict will be gathered. No ShardedTensor and DTensor will be in the returned state_dict. - ``cpu_offload``: offload all the tensors to cpu. To prevent CPU OOM, if ``full_state_dict`` is also true, then only the rank0 will get the state_dict and all other ranks will get empty state_dict. - ``ignore_frozen_params``: if the value is True, the returned state_dict won't contain any frozen parameters -- the ``requires_grad`` is False. The default value is False. - ``keep_submodule_prefixes`` (deprecated): when ``submodules`` is not None, this option indicates whether to keep the submodule prefixes from the state_dict keys. or example, if the submodule is ``module.pretrain`` and the full FQN of the parameter is ``pretrain.layer1.weight`` of the param. When this option is True, the parameter's key in the returned state_dict will be ``pretrain.layer1.weight``. If the options is False, the key will be ``layer1.weight``. Note that if ``keep_submodule_prefixes`` is False, there may be conflicted FQNs, hence there should be only one submodule in ``submodules``. - ``strict``: the ``strict`` option when ``set_state_dict`` calls model.load_state_dict(). - ``broadcast_from_rank0``: when the option is True, rank0 should receive a full state_dict and will broadcast the tensors in the state_dict/ optim_state_dict one by one to other ranks. Other ranks will receive the tensors and shard according to the local shards in the model and optimizer. ``full_state_dict`` must be set to True when using this option. This option currently only supports DTensor, not the legacy ShardedTensor. Ffull_state_dict cpu_offloadignore_frozen_paramsTkeep_submodule_prefixesstrictbroadcast_from_rank0flatten_optimizer_state_dict_fqn_modifiersdsd_fqn_modifiersN)__name__ __module__ __qualname____doc__r>bool__annotations__r?r@rArBrCrDrFstrr;r;r;r<r)\s  $      r)c@seZdZUeedZeeejBe ejBfe d<eedZ eeejBe ejBfe d<ee dZ e ee d<dZee d<dZee d<ejZee d<eedZeeje d <d S) _StateDictInfo)default_factoryfqn_param_mappingshared_params_mappingsubmodule_prefixesT handle_model handle_optim fsdp_context fsdp_modulesN)rGrHrIrdictrPrMtorchTensorr#rLrQsetrRrSrKrT contextlib nullcontextrUrlistrVnnModuler;r;r;r<rNs$   rNrETmodelnamerFskip_ddp_prefixskip_compiler_prefixreturnc s|td}d|vr |hS|d}g}|}t|D]\}} t|tr:| dkr/td| d|j}|s9|| qt|t r|t |dkrl||dt krld |t |t } r`dfdd| jDSt |t}| tkr|| t || }qt|tjjjr| d krtd | d|j}|s|| qt||rt ||| } rt|| rt || }|| | tjjjkr|t |dkrtd qt || }qd |tdhS) a This API is used to convert the name of a parameter to the FQNs. For FSDP without `use_orig_params`, the name of FlatParameter can be mapped to multiple original parameters. As a result, the return type of this function is `set[str]`. Args: module (nn.Module): the root model. name (str): the name skip_ddp_prefix (bool): whether to skip DDP's `module` prefix Returns: The canonical FQNs based on the model traversal. .modulezExpected 'module', got ''csh|]}|qSr;r;.0fqnprefixr;r< z_get_fqns.. _orig_modzExpected '_orig_mod', got 'z-Expect `_extra_state` to be the last obj name)replacersplit enumerate isinstanceDDPAssertionErrorrgappendFSDPlen _FLAT_PARAMjoingetattr_fqnsrrX_dynamo eval_frameOptimizedModulerqhasattrgetr^modules_EXTRA_STATE_KEY_SUFFIX RuntimeError) r`rarFrbrc obj_names fqn_obj_namescurr_obji curr_obj_name flat_param removed_fqnr;rmr< _get_fqnss\                   rc@s eZdZdS) _EXTRA_STATEN)rGrHrIr;r;r;r<rsrc#s<tdtjdtdtffdd |dEdHdS)Nrgcurr_fqnrdc3s||r |dnd}|D]-\}}|vrqt|r2|t|vr2|dd}n||}||EdHqt|jdd|jddD]\}}||jvrXqN||}||fVqNt|j dt j j t j j kr|t j jj}|tfVdSdS)NrfreF)recurseget_extra_state)addnamed_childrenrr}valuesr named_buffersnamed_parameters_non_persistent_buffers_set __class__r^r_rrrgrr)rgrra submodulenew_fqnobjrFrvisited_modulesr;r<rs2     z+_iterate_valid_model_state..recursere)rZr^r_rMr)r`rFr;rr<_iterate_valid_model_states "r) submodulesoptionsoptims. optim_onlyrrc Cs(|r tjdtdd|r|std|pt}i}i}t|D]?\}}t|tr)qt||} | |} | durJt t t || | ||||<n| ||<| D] } t|ts]||| <qRqt|D]\} } | D] } t tj| || <qkqet } |rt |}|D]"\}}||vrqt||} t| dkrtd| dd | Dq|jr|jstd t|}|r|jrt|j|jd }t|j|jp|jd }tj}nt |jd }t!|jd }tj"}t#j$d d}t%j&|||||d}nt#j'}t(dit)|||| |t tt*j+|| t|dkdS)zW Verify the model and options passed by the user and generates _StateDictInfo. zGetting submodules only model/optim state_dict is deprecated and will be removed in 2.5. This feature can be achieved by manually filtering out the state_dict returned from get_state_dict. stacklevelz;Optimizers are not passed in but optim_only is set to True.Nriz)Submodule FQN should only have 1 instancecss|]}|dVqdS)rfNr;rjr;r;r< Mz"_verify_options..z?full_state_dict must be True when broadcast_from_rank0 is True.)offload_to_cpu rank0_only)rc sst5tjddtdtj||||d dVWdn1s%wYWddSWddS1s=wYdS)NignorezFSDP.state_dict_type)messagecategoryrgstate_dict_typestate_dict_configoptim_state_dict_config)warningscatch_warningsfilterwarnings FutureWarningryrrr;r;r<$fsdp_state_dict_type_without_warningks  "z=_verify_options..fsdp_state_dict_type_without_warningrr)rPrQrRrUrVrSrTr;),rwarnrrr)rrurrrr rZrMupdatecopyr]itemsrXrY named_modulesrzrwrCr> ValueErrorryrVrr?rrFULL_STATE_DICTrrSHARDED_STATE_DICTr[contextmanager functoolspartialr\rNrr^r_)r`rrrrrPrQraparamfqnsrlparam_fqns_rRrgrVrrrrrUr;r;r<_verify_optionss               rmodel_state_dictoptim_state_dictinfocCs|jD]}t|}|durtdq|jr3|s3|js3|js3|jr#|js3|jr3|j s3t dt d|j rH|sH|jr>|jsH|j sHt d||D]}t|vrZt |dtdqJdS)Nz)Expected a fsdp_state with a fsdp module.z}The option indicates that model state_dict is required to save or load, but model state_dict is empty.rank = dist.get_rank()=rfzgThe option indicates that model state_dict is required to save, or load but optim state_dict is empty. z contains z6. This can happen if the model is not the root module.)rVrrwrSrRr@r?r>rBrCrdistget_rankrTr{)rrrrg fsdp_statekeyr;r;r<_verify_state_dicts^  rrapicCs,t||}|tvrtjt|j||d}|S)N)self)r}r4rrr)rrcallr;r;r<_state_dict_fns r state_dictcCs@|jr|jr tjs dnd}t||j|dS|jrt|S|S)Nr;)r)r? ranks_only)r>r?rX distributedis_initializedrr)rrrr;r;r<_maybe_full_or_cpu_state_dictsrc Csz|jsiS|t|d}Wdn1swYt|D]C}t||}t|dkrAtd|dt|d|tt |}||krhdt fdd}|||sat d |d || |||<q%|j ri}|D]&}|j D] }||s}qu|jr||||<qu|t|d} |||| <quqp|}|jr|D]\}} | jrqt||}|D]}| |qqt||S) NrriExpected 1 FQN for key '', got z: rdcSst|t|kr dS|d}|d}d}t|D]&\}}|||kr9|d7}|t|kr8|t|dkSq|dvr>qdSdS)NFrfrri)rgrqT)rzrsrt)rrl fqn_split key_splitfqn_idxkey_idxkey_namer;r;r<verifys    z%_get_model_state_dict..verifyzAn unexpected key, z, exists. FQN is )rSrUrr]keysrrzrwnextiterrKrpoprR startswithrAr@r requires_gradr) r`rrrrrlrnew_state_dictrnrrr;r;r<_get_model_state_dictsN          rc Cs|jr|s |js tiiSi}t||jD]J\}}t|||j}t|||jddd}t||D]/\}} |jrrZrrX is_tensordimrrremoverzdistributed_c10d_get_pg_default_devicerrr?rrrUr r) r`rrlocal_state_dictrvaluerfqns_with_prefixrlfqn_with_prefix load_valuerdevicesr;r;r<_load_model_state_dictst           $roptimcCs|jrdS|jD]}|tD] }|jdurdSqq|jD]}|tD] }|jr/t||_q$qg}|jD]}d|vrT||dt|dtj rPt dnd|d<q6|j dd|jD] }d|vrk| d|d<q^|j dddS)zH Initialize optim states by calling the step() with zero grads. Nlrg)closurerT) set_to_none)r3r1_PARAMSgradrrX zeros_likerxrurYtensorstepr zero_grad)r param_grouprlrsr;r;r<_init_optim_statecs:          rc sdtttfdtdtttfffdd ddi}tt|tD]\}}td|}|ttttf||q%tt |t D]&}| t }tt t|D]}|D]\}}||t d|d|<q[qUqG|S) a3 This API flattens the optimizer state_dict to support optimizer resharding for MPMD, e.g., pipeline parallelism. Without the API, the original optimizer state_dict looks like: { "state": { "layer1.weight": { "step": 10, "exp_avg": SomeTensor, "exp_avg_sq": SomeTensor }, "layer2.weight": { "step": 10, "exp_avg": SomeTensor, "exp_avg_sq": SomeTensor }, }, "param_groups": [ { "lr": 0.0, "betas": (0.9, 0.95), ..., "params": ["layer1.weight", "layer2.weight"] } ] } With this API, the optimizer state_dict looks like: { "state.layer1.weight.step": 10, "state.layer2.weight.step": 10, "state.layer1.weight.exp_avg": SomeTensor, "state.layer2.weight.exp_avg": SomeTensor, "state.layer1.weight.exp_avg_sq": SomeTensor, "state.layer2.weight.exp_avg_sq": SomeTensor, "param_groups.layer1.weight.lr": 0.1, "param_groups.layer2.weight.lr": 0.1, "param_groups.layer1.weight.betas": (0.9, 0.95), "param_groups.layer2.weight.betas": (0.9, 0.95), } The "state" section supports arbitrary levels of nesting for optimizers like Shampoo. nested_dictrnrdcsdi}|D])\}}t|}|r|d|n|}t|tr'|||q||||<q|S)a Recursively flatten a nested dictionary with dot-separated keys. Args: nested_dict: The dictionary to flatten prefix: The prefix to prepend to all keys Returns: Flattened dictionary with dot-separated keys rf)rrMrurWr)rrn flattenedrrstr_keyfull_key_flatten_state_nested_dict_raise_if_type_not_supportedr;r<r s   z=_flatten_optim_state_dict.._flatten_state_nested_dictcSs,t|tjtttfstdt|ddS)Nz[Flattening optimizer state_dict only supports tensor, int, float, dict states now. Type is rf)rurXrYintfloatrWNotImplementedErrortype)vr;r;r<rsz?_flatten_optim_state_dict.._raise_if_type_not_supportedrf)rWrMr r%r r&_STATErrr'_PGrrr]) rretrlr3 state_prefixrrkrr;r r<_flatten_optim_state_dicts,)   rc Csdtdtttfdtttffdd}i}g}t|t|i}|jD]}|tgi|tD]}|j|D]z} | |j vrXd} |D]} | tkrFq?td| d| } | |vrVd} nd} | s]q4|d t} t | t sqt d t | | | |jszq4i|| <|j|D]*}td| d|}||vr|||}|tt|| |<q||tt|| |<qq4q-tt t|d td }|D]=} | tkrq|td|d| }| |d vr||d | <q|d | |krtd |d| d |d|d | d qq |S)z This API unflattens the state_dict generated by _flatten_optim_state_dict(). Supports arbitrary levels of nesting in the state section through recursive reconstruction. See the docstring of _flatten_optim_state_dict() for more detail. flattened_keyflattened_dictrdc Ss|d}i}|D]=\}}||sq |t|d}|d}|}|ddD]} | |vr4i|| <t|| ts=J|| }q*|||d<q |S)z Reconstructs a potentially nested value from flattened keys. For non-nested values, returns the value directly. For nested values, reconstructs the nested structure with string keys. rfNr)rrrzrsrurW) rrrnrrr remaining_keypartscurrentpartr;r;r<_reconstruct_nested_dicts    z=_unflatten_optim_state_dict.._reconstruct_nested_dictFrfTrExpected list, got rzaAll the parameters in the same parameter group should have the same saved param_group value. But z is z while other(s) is )rMrWr%rrr1rxrrPrQrur]rwrrr3r r&r)rrrr r3pg_state return_osdrrrl in_paramsr flatten_keyr2 state_nameflattened_state_keyreconstructed_valuefirst_param_fqnrr;r;r<_unflatten_optim_state_dicts    1       /   r* optimizersc s0|jsiStitgi}|D]}t|t|d}|jrm|t|||}Wdn1s2wY|s:q t |t D]}d|vrW|t ||t| dd<qB|tD]}dd|t D}||t <q\nt tdd|jD}tt|tt|} i|D]2\} } t|| } t| d krtd | d t| tt| } | | vrq| | }|| <| |<qt |t D]} | } |t | |t| <q|tD]}fd d|t D|t <q|sq tt|t|ttt|t|tq |jrtt t!|}t"||S) Nrrq _orig_mod.recSsg|]}|ddqS)r,rerrrkrr;r;r< rpz)_get_optim_state_dict..css|]}|tVqdSr5)r)rkgr;r;r<rsz(_get_optim_state_dict..rirrcsg|]}|qSr;r;)rkpidfqn_pid_mappingr;r<r/s)#rTrrrrrVrUryrr]rrrrrr from_iterabler1rWrrangerzrrrwrrr r&rr'extendrDr(rr)r`r+rrrosdrr0r2param_pid_mappingrrrrlr1groupr;r2r<_get_optim_state_dictssb           r:c Csti}g}t|t|i}i}tddtt|tDr|S|jD]}|tgi|tD]} |j| D]} | |j vrVd} tt |tD]} | tt t | tvrTd} nqCnd} | s[q3|dt} t | t sotdt| | | | jr| tt|tvrtt|t| || <n |jrtd| dtt |tD]} | tt t | tvrt|td |t| <qq3q,t|td krg}tt |tD]} ttt t | td kr|| qt|d krtd t|tt|jkrtd t|td |t| <qtt |tD])}|t|d}|dkrq|D]\}}|tkr.q#||||<q#q|S) a Extract the corresponding optim state_dict from ``optim_state_dict`` for ``optim`` and return the result optim state_dict. Args: model (nn.Module): the root model. optim (torch.optim.Optimizer): the optimizer. optim_state_dict (Dict[str, ValueType]): the superset optim state_dict that contains the optim state_dict of ``optim``. info (_StateDictInfo): state dict information. Returns: The optim state_dict of ``optim``. css|]}t|tVqdSr5)rurr.r;r;r<rrz*_split_optim_state_dict..FTrr!z'Missing optimizer state for parameter 'z' in checkpoint. The parameter requires gradients but has no saved optimizer state. To load anyway, use StateDictOptions(strict=False).rirzThere are param groups that have zero parameters. In such a case, DSD only support exactly one param group with zero parameters.But the loaded state_dict has zero or more than one param groups that have zero parameters.z`When there is a parameter group that has zero parameters, multiple optimizers are not supported.)rrallr r&r1rxrrPrQr'r]rMrurwrrrBrrzidrrr)r`rrrr3r"r# pg_mappingrrrlr$loaded_param_groupr2rpg_idxrrr;r;r<_split_optim_state_dicts        !    r@c s|jsdS|D]E}t||r*t|vrt||||}nt|ttttf||}ni}|j r| D]n\}}t ||}t ||dd} || krHq3t |dkrZt d|dt ||| |tD]} ttttf| } fdd| tD} | | t<qftt|t} t| D]}|vr| || |<qq3|t|||}Wdn1swYn|jrEd|_t||f|}d|_dfd d }ttj||}durt d t|\}}t|\}}|jrt ||d nt!||d |D]!}||vr&||vrt d |d||||<||||<qt"||}|tD]}t|vrCgttttf|t<q1t#|d|dqdS)NF)rcrizExpected 1 FQN for 'rcsg|]}|qSr;r-)rkr)rlfqn_with_compilerr;r<r/Is z*_load_optim_state_dict..Tcs4|dkrdur|j|S|jkrtd|S)NrzDevice mismatch)rrr)trr;r<_device\s  z'_load_optim_state_dict.._devicezExpected device to be setrzExpected key 'z' in osd_mappingrr)$rTrrr@r*r rWrMr%rVrrrzrwrrr rr&r]rrrrUryoptim_state_dict_to_loadr>r:r"rXrYrrCrrrr)r`r+rrrr original_fqn_rfqns_with_compilerr0valr2 osd_staterrrC flatten_osd osd_mappingflatten_local_osdlocal_osd_mapping optim_keypgr;)rrlrAr<_load_optim_state_dicts                  rQcCsVtt|dd||d}t||}t|i||WdS1s$wYdS)aH Return the model state_dict of ``model``. See ``get_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. submodules (deprecated): Optional[set[nn.Module]]: only return the model parameters that belong to the submodules. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be returned. See `StateDictOptions` for the details. Returns: The state_dict for ``model``. :rtype: typing.Dict[str, ValueType] r;FrrrN)r=rrr)r`rrrrr;r;r<r*s  $r*cCstt-t|tjjr|fnt|}t||d||d}t|||}ti|||WdS1s3wYdS)a Return the combined state_dict for optimizers. See ``get_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[None, Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. submodules (deprecated): Optional[set[nn.Module]]: only return the model parameters that belong to the submodules. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be returned. See `StateDictOptions` for the details. Returns: The state_dict for ``optimizers``. :rtype: OptimizerStateType TrRN) r=rurXr Optimizertuplerr:r)r`r+rrrrr;r;r<r+s    $r+cCst4t|tjjr|fnt|}t||d||d}t||}t|||}t |||||fWdS1s:wYdS)a Return the model state_dict and optimizers state_dict. ``get_state_dict`` can process any module that is parallelized by PyTorch FSDP/fully_shard, DDP/replicate, tensor_parallel/parallelize_module, and any combination of these parallelisms. The main functions of ``get_state_dict`` are: 1.) returning a model and optimizer state_dict that can be resharded with a different number of trainers and/or different parallelisms. 2.) hiding the parallelism-specific state_dict APIs. Users don't have to call these APIs. 3.) sanity checking the result state_dict. The keys of the result state dictionary are the canonical FQNs (Fully Qualified Names). A canonical FQN refers to the FQN based on a parameter's position in an nn.Module hierarchy. More specifically, a canonical FQN to a parameter is the FQN returned by ``module.named_parameters()`` or ``module.named_buffers()`` when the module is not distributed by any parallelisms. Since the optimizer internally uses parameter IDs to represent a parameter, there will be a conversion from the parameter IDs to the canonical FQNs when calling this API. ``get_state_dict`` can also process a module that is not parallelized. In such a case, ``get_state_dict`` only performs one function -- converting the optimizer parameter IDs to the canonical FQNs. Example: >>> # xdoctest: +SKIP >>> import torch >>> from torch.distributed.fsdp import FullyShardedDataParallel as FSDP >>> from torch.nn.parallel import DistributedDataParallel as DDP >>> from torch.distributed.checkpoint.state_dict import get_state_dict >>> fsdp_model = FSDP(copy.deepcopy(model)) >>> fsdp_optim = torch.optim.Adam(model.parameters(), lr=1e-3) >>> ddp_model = DDP(copy.deepcopy(model)) >>> ddp_optim = torch.optim.Adam(model.parameters(), lr=1e-3) >>> ddp_state_dict, ddp_optim_state_dict = get_state_dict(ddp_model, ddp_optim) >>> fsdp_state_dict, fsdp_optim_state_dict = get_state_dict( ... fsdp_model, fsdp_optim ... ) >>> # if we simply call ddp_model.state_dict() and fsdp_model.state_dict(), >>> # the asserts will fail. >>> assert ddp_state_dict == fsdp_state_dict >>> assert ddp_optim_state == fsdp_optim_state_dict Args: model (nn.Module): the nn.Module to the model. optimizers (Union[None, Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. submodules (deprecated): Optional[set[nn.Module]]: only return the model parameters that belong to the submodules. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be returned. See `StateDictOptions` for the details. Returns: ``Tuple`` that contain model state_dict and optimizer state_dict. :rtype: typing.Tuple[typing.Dict[str, ValueType], OptimizerStateType] FrRN) r=rurXrrSrTrrr:r)r`r+rrrrrr;r;r<r,s"H    $r,c s|siSttt|tjrgtjdtddt t tjt t t ff|}i}| D]8\}}|D]/\}}||kr=q4t||}t|dkrLtdtt|d|fdd| Dq4q,|St t t t f|S) NzPassing model_state_dict as a ``Dict[nn.Module, Dict[str, Any]]``is deprecated and will be removed in 2.5. If you need this feature, please preprocessing the model_state_dict to achieve the same functionality.rrriz/FQNs for a submodule should only have 1 elementrfcsi|] \}}||qSr;r;)rksubfqnrrmr;r< Rsz/_unflatten_model_state_dict..)rurrrr^r_rrrr rWrMr%rrrrzrwr) r`rcast_state_dictrrsub_state_dictramrr;rmr<_unflatten_model_state_dict4s4   rZ)rcCs\t||}tt|dd|d}t|i|t|||WdS1s'wYdS)a=Load the model state_dict. The counterpart of ``get_model_state_dict`` to set the state_dict to the model. See ``set_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. model_state_dict: (Dict[str, ValueType]): the model state_dict to load. If the key of the ``model_state_dict`` is nn.Module, the key is a submodule of ``model`` and the value should be the state_dict of the submodule. When loading the state_dict, the prefix of the submodule will be append to the state_dict. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: ``NamedTuple`` with ``missing_keys`` and ``unexpected_keys`` fields: * **missing_keys** is a list of str containing the missing keys * **unexpected_keys** is a list of str containing the unexpected keys :type model_state_dict: typing.Dict[str, ValueType] r;FrrN)rZr=rrr)r`rrrr;r;r<r-Ys  $r-cCsrt,t|tjjr|fnt|}t||d|d}ti||t||||WddS1s2wYdS)aLoad the optimizers state_dict. The counterpart of ``get_optimizer_state_dict`` to set the state_dict to the optimizers. See ``set_state_dict`` for the detail usage. WARN: ``set_optimizer_state_dict`` can only be called before ``backward()`` or after ``step()`` is called on the optimizers. Otherwise, the optimizer states won't be initialized correctly. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. optim_state_dict: OptimizerStateType: the optimizer state_dict to load. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None :type optim_state_dict: typing.OptimizerStateType Tr[N) r=rurXrrSrTrrrQ)r`r+rrrr;r;r<r.s  "r.cCst||}t2t|tjjr|fnt|}t||| |d}t|||t ||||t |||WdS1s=wYdS)aLoad the model state_dict and optimizers state_dict. The counterpart of ``get_state_dict`` to set the state_dict to the model and optimizers. The given ``model_state_dict`` and ``optim_state_dict`` do not have to be returned by ``get_state_dict`` but must meet the following requirements: 1) all FQNs are canonical FQNs as defined in ``get_state_dict``, 2) if a tensor is sharded, it must be either a ShardedTensor or DTensor, 3) optimizer state_dict cannot contain the parameter IDs; the keys should be the canonical FQNs. WARN: ``set_state_dict`` can only be called before ``backward()`` or after ``step()`` is called on the optimizers. Otherwise, the optimizer states won't be initialized correctly. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. model_state_dict: (Union[Dict[nn.Module, Dict[str, ValueType]], Dict[str, ValueType]]): the model state_dict to load. If the key of the ``model_state_dict`` is nn.Module, the key is a submodule of ``model`` and the value should be the state_dict of the submodule. When loading the state_dict, the prefix of the submodule will be append to the state_dict. optim_state_dict: OptimizerStateType: the optimizer state_dict to load. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: ``NamedTuple`` with ``missing_keys`` and ``unexpected_keys`` fields: * **missing_keys** is a list of str containing the missing keys of the model state_dict. * **unexpected_keys** is a list of str containing the unexpected keys of the model state_dict. :type model_state_dict: typing.Dict[str, ValueType] :type optim_state_dict: typing.OptimizerStateType r[N) rZr=rurXrrSrTrrrQr)r`r+rrrrr;r;r<r/s.    $r/csjtjt||dfdd}||_tjt||ddtttfffdd }||_t |t |dS)aPatch the ``state_dict`` and ``load_state_dict`` attributes of ``model``. Patch the ``state_dict`` and ``load_state_dict`` attributes of ``model`` to be a partial function to call ``get_state_dict`` and ``set_state_dict``. Example: from torch.distributed.fsdp import FullyShardedDataParallel as FSDP from torch.distributed.checkpoint.state_dict import patch_model_state_dict model = fsdp(model) patch_model_state_dict(model) Args: model (nn.Module): the nn.Module to the model. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None )r`rcSr5r;r;_state_dict_callr;r<state_dict_callz0_patch_model_state_dict..state_dict_callrc|ddS)N)rr;rD_load_state_dict_callr;r<load_state_dict_callz5_patch_model_state_dict..load_state_dict_callN) rrr*rr-rWrMr rr4r)r`rr_rdr;rcr^r<_patch_model_state_dicts   rgcstjt|||dfdd}tjt|||ddtttfffdd }t|t|t |t j j r9|fnt |}|D]}||_||_q?dS)aPatch the ``state_dict`` and ``load_state_dict`` attributes of ``optimizers``. Patch the ``state_dict`` and ``load_state_dict`` attributes of ``optimizers`` to be a partial function to call ``get_state_dict`` and ``set_state_dict``. Note that if there are multiple optimizers, all of the optimizers will be patched. So users only need to call one of the state_dict() to get the full result. Example: from torch.distributed.fsdp import FullyShardedDataParallel as FSDP from torch.distributed.checkpoint.state_dict import patch_model_state_dict model = fsdp(model) patch_model_state_dict(model) Args: model (nn.Module): the nn.Module to the model. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None )r`r+rcr\r5r;r;r]r;r<r_Jr`z4_patch_optimizer_state_dict..state_dict_callrcra)N)rr;rDrbr;r<rdTrez9_patch_optimizer_state_dict..load_state_dict_callN)rrr+r.rWrMr r4rrurXrrSrTrr)r`r+rr_rdrr;rfr<_patch_optimizer_state_dict$s0    rh)rETT)rE)pr[rr6rcollections.abcrrr dataclassesrrr itertoolsrtypingr r r r rXtorch.distributedrrtorch.nnr^'torch.distributed._shard.sharded_tensorr #torch.distributed._state_dict_utilsrrrrrr;torch.distributed.algorithms._checkpoint.checkpoint_wrapperrtorch.distributed.fsdprrrryrrrrr$torch.distributed.fsdp._common_utilsrrtorch.distributed.tensorrtorch.nn.modules.moduler torch.nn.parallelr!rvtorch.utils._pytreer"__all__r{rrrrZrMr#rYrrr$r]rTrWr%r&r'r(r4rLrr=r)rNr_rKrrrrrSrrrrno_gradrrrrr*r:r@rQr*r+r,rZr-r.r/rgrhr;r;r;r<s       (       / K -    .     B  D*b   G d j   )  1  [   )  , 1  B 6