o ei0@sddlZddlmZddlZddlmmZddZddZ ddZ d d Z d d Z ej d dZddZddZddZddZddZdS)N) namedtuplecstfdd}|S)Nc sdrdj}||i|SdsdrDdr#dnd}|dkr+dnd}|j}td|dd|d|d |i|S) Nautogradsave_for_backwardbackwardz We found a 'z' registration for  at z but were unable to find a 'z' registration. To use the CustomOp API to register a backward formula, please provide us both a backward function and a 'save for backward' function via `impl_backward` and `impl_save_for_backward` respectively.) _has_impl _get_implfunclocation RuntimeError)argskwargskernelmissingfoundlocautograd_fallback custom_opc/var/www/addictedbytheproject.nl/epg/venv/lib/python3.10/site-packages/torch/_custom_op/autograd.pyinners   z*autograd_kernel_indirection..inner)autograd_not_implemented)rrrrrautograd_kernel_indirectionsrcsfdd}|S)Ncs`trtdd||frtdtj|i|WdS1s)wYdS)NcSst|tjo|jSN) isinstancetorchTensor requires_gradxrrr6z:autograd_not_implemented..kernel..z.Autograd has not been implemented for operator)ris_grad_enabledpytreetree_anyr _C_AutoDispatchBelowAutograd)r r rrrr4s    $z(autograd_not_implemented..kernelr)rrrr(rr3s rc Cs|durkt|ts |f}n|}t|t|kr%tdt|dt|g}tt||D]3\}\}}t|tjrB|sA||q.t|t rO|sN| |q.|rat d|d|dt |dq.|rm|j |dSdSdS)Nz output_differentiability length z != output length zWith output_differentiability=z . At idx z , we received an object of type za that is not a Tensor, so it cannot have be marked as differentiable in output_differentiability.)rtuplelenAssertionError enumerateziprrappendlistextendr typemark_non_differentiable)ctxoutputoutput_differentiability tuple_outputnon_differentiable_tensorsidxdifferentiableoutrrrr2?sF      r2csfdd}|S)Ncstt|\}dfdd}fdd}tjd||}|j|}dur2tdtt|S)Ncs|dtt|}tj |}Wdn1s!wYttt |}t|}||}t |||ft ||t |\}t |S)NT)set_materialize_gradsr$tree_unflattenr/rr&r'namedtuple_argstree_mapr1save_pytree_for_backwardr2 tree_flattenr))r3 flat_argsr r4 args_infosave_for_backward_fn_inputsto_save flat_output) op_overloadout_specr5save_for_backward_fnschemaspecrrforwardrs      z9construct_autograd_kernel..apply..forwardcsjdurtdtt|}t|\}}t}t|ts!|f}||g|R}t||t ||S)Nout_spec is unexpectedly None) r+r$r<r/ unpack_savedobjectrr)validate_grad_inputs_dictgrad_inputs_dict_to_flat_tuple)r3flat_grad_outputgradssavedrB inner_ctxgrad_inputs_dict) backward_fnrrGrrrs    z:construct_autograd_kernel..apply..backward _customoprL)r$r@gen_autograd_function_opnameapplyr+r<r/)r rArKr generated_clsrErVrrFr5rHrI)rGrJrrZns  z(construct_autograd_kernel..applyr)rIr5rrFrHrVrZrr\rconstruct_autograd_kernelfs1r]cCs$t|tjjft|t|d}|S)N)rKr)r1rrFunction staticmethod)namerKrr[rrrrXsrXcCs.dd|jjD}t|jd}t||}|S)NcSsg|]}|jqSr)r`.0argrrr r"z'namedtuple_args_cls.._args) argumentsflat_allstrr`r)rIattribsr` tuple_clsrrrnamedtuple_args_clss rkcCs,t|tstdt|t|}||S)Nzexpected tuple, got )rr)r+r1rk)rIr rjrrrr=s r=c sfdd}t|ts|dt|ddjjjD}|}||kr1|d|d|d|D]\}}t||}t|t rt|t t fsW|d |d t|d t |t |krp|d |d t |d t |t t ||D]6\} \} } | durqwt| tjs|d |dt| d| t| tjs|d |d| d| d|qwq5|durq5t|tjs|dt|d|dt|tjs|d|d|d|dq5dS)Ncs&d}tdd|jd|)Nrz%In the backward function defined for rz using the CustomOp API, )rr r )whatr forward_oprrerrors z(validate_grad_inputs_dict..errorzBexpected the output of the backward function to be a dict but got cSsh|] }|jr|jqSr)r1is_tensor_liker`rarrr s z,validate_grad_inputs_dict..z3expected the returned grad_input dict to have keys z but got z. The backward function must return a gradient (can be None) for each arg to the CustomOp that may be a Tensor or Sequence[Tensor]. Args declared to be non-Tensor-like types should not appear in the grad_input dictz for input 'zR' expected the grad_input dict to hold a list of gradients but got object of type .z1' expected the grad_input dict to hold a list of z gradients but got z\' expected the grad_input dict to hold a list of None or Tensor gradients but got object of z at index z(', got a Tensor as the gradient for the z(-th value but expected None because the z(-th value was not a Tensor (it was type zgot object of type z as the gradient for input 'z:', but expected the gradient to be either None or a Tensorz(got a Tensor as the gradient for input 'z3' but expected None as the gradient because input 'z ' was not a Tensor (it was type z).)rdictr1_schemarfrgkeysitemsgetattrr/r)r*r,r-rr issubclass) rUrnrBro expected_keys actual_keysr`gradarg_infor8ginforrmrrOs         rOcCsVg}|D]\}}||vr|tdd|q|||qtt|S)NcSsdSrrrrrrr!sz0grad_inputs_dict_to_flat_tuple..)_asdictrvr.r$r>r) tree_leaves)rUrBresultr`r|rrrrPsrPc Cst|\}}t|}ddt|D}ddt|D}dd|D}dd|D}||_||_|j|||_||_||_ dS)NcSs g|] \}}t|tjr|qSrrrrrbr8thingrrrrds z,save_pytree_for_backward..cSs g|] \}}t|tjs|qSrrrrrrrd"s  cSsg|] }t|tjr|qSrrrbrrrrrd'cSsg|] }t|tjs|qSrrrrrrrd(r) r$r@r*r,rJnum_eltsr tensor_idxssaved_non_tensorsnon_tensor_idxs) r3stuff flat_stuffrJrrrtensors non_tensorsrrrr?s   r?cCs^dg|j}t|j|jD]\}}|||<q t|j|jD]\}}|||<qt||j}|Sr) rr- saved_tensorsrrrr$r<rJ)r3rtensorr8 non_tensorrrrrrM3s   rM) functools collectionsrrtorch.utils._pytreeutils_pytreer$rrr2r]rX lru_cacherkr=rOrPr?rMrrrrs     '< P