o ei; @sddlZddlmZddlZddlmZeeZddZddZ Gdd d e Z d d Z d e ejeejd fBde ejeejd fBfddZ ddedededeeeffddZ ddedededeee effddZeGdddZdS)N) dataclass)fxcs&gfdd}tj||}|fS)z\ Flatten the args into a list form and detach the tensors from computational graph. cs8t|tjr||j}||S||SN) isinstancetorchTensordetachrequires_grad_ requires_gradappend)avalflat_detached_argsm/var/www/addictedbytheproject.nl/epg/venv/lib/python3.10/site-packages/torch/distributed/pipelining/_utils.pyextract_tensor_argss   z0flatten_args_detach..extract_tensor_argsrnode map_aggregate)argsrnew_argsrrrflatten_args_detachs  rcs"gfdd}tj||S)z, Flatten the args into a list form. cs||Sr)r )r  flat_argsrrr,s z)flatten_args..extract_tensor_argsr)rrrrr flatten_args&s rc@seZdZdZdS)PipeliningShapeErrorz5Shape mismatch between configured and runtime values.N)__name__ __module__ __qualname____doc__rrrrr9srcCs|j|jkst|d|jd|j|j|jks(t|d|jd|j||ks@t|d|d|dS)Nz has a shape mismatch: expected z actual z has a dtype mismatch: expected z! has a stride mismatch: expected )shaperdtypestride)descexpectedgivenrrrvalidate_tensor_metadata=s  r'expected_tensors.actual_tensorscCsft|t|krt|dt|dt|dtt|D]}t|d|||||qdS)Nz: Number of values (z") does not match expected number ()z: value )lenrranger')r$r(r)irrrvalidate_tensors_metadataLsr.looppp_size num_stagesstylereturncCsi}|dkrt|D]}||||<q |S|dkrS||dkr*td|d|dd}t|D] }|||<|d|dkr?q0||ddkrL|d7}q0|d8}q0|Std |d ) z Compute the stage id to rank mapping for either a looped or V-style schedule. Most commonly num_stages == pp_size * 2, but this function can be used to compute the mapping for any number of stages per rank. r/vrz num_stages z% must be evenly divisible by pp_size z for V scheduleszStyle z is not supported.)r, ValueError)r0r1r2mapping stage_index rank_indexrrrgenerate_stage_to_rank_mapping[s(      r;cCsZt|||}i}|D]\}}||vrg||<|||q |D]}|q$|S)a Compute the rank to stage id mapping for either a looped or V-style schedule. This function inverts the stage_to_rank_mapping to get which stages are assigned to each rank. Returns a dictionary mapping rank -> list of stage indices assigned to that rank. )r;itemsr valuessort)r0r1r2 stage_to_rankrank_to_stagesstage_idrankstagesrrrgenerate_rank_to_stage_mapping}s   rDc@s,eZdZUdZejed<eed<eed<dS)PipeInfoz> Captures information for a pipeline (`Pipe` object). graphr1has_loss_and_backwardN) rrrr rGraph__annotations__intboolrrrrrEs   rE)r/)logging dataclassesrrr getLoggerrloggerrr RuntimeErrorrr'listrtupler.rJstrdictr;rDrErrrrsH      #