o ei@sUdZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl m Z ddl m Z mZddl mZmZddlmZmZmZmZmZmZddlmZddlmZddlZdd lmZdd lm Z dd l!m"Z"d d l#m$Z$m%Z%m&Z&m'Z'd dl(m)Z)m*Z*m+Z+d dl,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7d dl8m9Z9m:Z:d dl;mZ>m?Z?m@Z@mAZAmBZBmCZCmDZDd dlEmFZFmGZGmHZHmIZImJZJmKZKmLZLmMZMddlNmOZOmPZPmQZQmRZRmSZSddlTmUZUmVZVzddlWmXZXWn eYydZXYnwer,ddlZm[Z[ddl\m]Z]m^Z^m_Z_m`Z`ddlambZbddlcmdZdmeZeddlfmgZgmhZhddlimjZjede defdZkd Zld!Zmenhd"Zoend#hZpd$ZqeZreeefesd%<Gd&d'd'etZuejvd(ed1d;eze{eSfd?e jd(dfd@dAZd?e jdBeze{efdCe{dDeedfdBdEeedBdFeze{efdBdGeze{efdBd(e jfdHdIZhdJZd0d1dKed8e dBdCe{d(eSf dLdMZGdNdOdOeSZGdPdQdQeZGdRdSdSeZGdTdUdUeZGdVdWdWeZGdXdYdYeSZGdZd[d[eZGd\d]d]eZGd^d_d_eZGd`dadaeZGdbdcdceZGdddedeeZd0d1dKe defdCe{d3eeSd4eze{eSfd(eSf dfdgZGdhdidieZGdjdkdkeZGdldmdmeSZGdndodoeZGdpdqdqeSZGdrdsdseZd(ezeeffdtduZd0d1dKe defd(eis_leaf namespace none_is_leafrPzihttps://github.com/pytorch/pytorch/issues/new?&labels=oncall%3A+pt2&projects=&template=pt2-bug-report.yml _spec_cachec@s eZdZdS)ClosureConversionErrorN)__name__ __module__ __qualname__rXrXk/var/www/addictedbytheproject.nl/epg/venv/lib/python3.10/site-packages/torch/_dynamo/variables/functions.pyrTsrTreturncCsttttddddS)Ntorchutils_pytreeSUPPORTED_NODES)r'r-rXrXrXrY!get_pytree_SUPPORTED_NODES_sourcesr_c@s0eZdZdeddfddZdeddfddZdS) FunctionSpecfuncrZNcCs|j}|j}|j|_|j|_|j|_|d|j|_||j|j|_ |j|j |_ ||j|j|j|_ |j|j}|j t @rE||nd|_||jrNdnd7}|j t@r]|||_dSd|_dSNr8r)__code__ co_varnamesco_posonlyargcount posonly_count co_argcount arg_countco_kwonlyargcount kwonly_count posonly_namespos_or_kw_names all_pos_names kwonly_namesco_flags CO_VARARGS varargs_nameCO_VARKEYWORDS varkw_name)selfracodevnoffrXrXrY__init__s "zFunctionSpec.__init__cCsH|jpd|_|jp i|_tt|jt|j dtt|j|_ dSNrX) __defaults__defaults__kwdefaults__ kwdefaultsdictziprmlenrangepos_default_map)rtrarXrXrYupdate_defaultss  $ zFunctionSpec.update_defaults)rUrVrWrrxrrXrXrXrYr`sr`racCs&t|}|durt|}|t|<|SN)rSgetr`)raspecrXrXrY _get_specs rtxrD fn_sourceargskwargscst|}t|jkr&|js&|s&|js&|js&|js&fddt|jDS| |i}t |}|j D]#\}} d} |rNt |j| rItjsNt|| } t|j| | ||<q6t|jD]3\} }| tkrst| ||<q_||vr||jvrt||||<q_||vrtd|q_t|jd} |jrtt| ||j<n| rtdtdt|j|jD]3}||vrt||||<q||jvrd} |rt||dd} t|j|| ||<qtd||jrt|||j<|S|r td t||S) Ncs i|] \}}|t|qSrX)wrap_bound_arg).0inamerrrXrY sz$bind_args_cached..z&missing required positional argument: z#Too many positional arguments: got z , expected T)is_kwz(Missing required keyword-only argument: zUnexpected keyword arguments: )rrrhrzrqrsrn enumeratermrr~ritemsr? is_literalr{r%skip_guards_on_constant_func_defaultsr+rrkpop TypeErrortupler}list)rarrrrrbarem_kwridxdefault_sourcerextrakwdefault_sourcerXrrYbind_args_cachedst       rvalsourcecCs,t|tr|S|st||Stj||Sr) isinstancer=buildrLazyVariableTrackercreate)rrrrXrXrYr s  rresultcCs6t|D]\}}t|ttfrt||||<qdSr)rrrrr~r)rrkvrXrXrYwrap_args_kwargss rparentrucCsB|jj}|jD]}|}||vr|||||||<qdS)z Update `result` to add mapping from local name to new cells created directly by `code`, or update SideEffects in `parent` if the a local cell is already in `result` (cell argument). N)output side_effects co_cellvarstrack_cell_new store_cellr)rrrurrnew_cellrXrXrY init_cellvars!s  r f_globalsrr{closurer} annotationsc Csdddlm}||||||}||_t|tr"ddlm} t| |}|dus-t|ts-J||_|S)Nrrpairwise) typesrr|rr itertoolsrr~__annotations__) rurrr{rr}rrrarrXrXrY_create_nested_fn6s    r>__doc__rc __closure__ __globals__rzr|rfncCs|ot||}|r|dkrt|}d}zt||}Wnty)tt|Ynw|tvr3t||}|rinline_user_function_return self_argsrtrrrrXrXrY call_functions z&BaseUserFunctionVariable.call_functionrcCsLd}z t||}Wnty|dkrt|trd}Ynwtj|S)NFrUT)hasattr get_functionrrNestedUserFunctionVariablerr?rrtrrrrXrXrYcall_obj_hasattrs  z)BaseUserFunctionVariable.call_obj_hasattrcCiSrrXrtrrXrXrY closure_varsrz%BaseUserFunctionVariable.closure_varscCdSNTrXrrXrXrY should_allow_nested_graph_breaksrz9BaseUserFunctionVariable.should_allow_nested_graph_breaksN)rUrVrWstrrrr~rrrCodeTyperboolrrr=rr?rrrrXrXrXrYrvs0    rc seZdZdZddhejZehdZede de ddfdd Z dDde j e jjBded e dd ffd d Zde ffdd ZdeefddZde j fddZde jfddZdefddZdefddZdeee ffddZdefddZ dd d!e!ed eeefdeeeffd"d#Z"d$d d%edefd&d'Z#d$d d%ede$fd(d)Z%d$d d!e!ed eeefdeffd*d+ Z&d$d d!e!ed eeefded Bfd,d-Z'defd.d/Z(defd0d1Z)d$d d!e!ed eeefde*de!eeeeffd Bfd2d3Z+de j d Bfd4d5Z,d6ede*ed7fd Bfd8d9Z-de deed Bfd:d;Z.de/d<fd=d>Z0de1fd?d@Z2dAe3defdBdCZ4Z5S)EUserFunctionVariablez-Some unsupported user-defined global functionr is_constant> optree.opstorch.utils._pytreetorch.utils._cxx_pytreeoptreevaluerrZcCt|tj|||dSNrr& make_guardr% CLOSURE_MATCHclsrrrXrXrYcreate_with_sources z'UserFunctionVariable.create_with_sourceFrNc srtjd i|t|ddrd|_nd|_t|tjtjj fs-t d|ddgt j dt |d|}||_dS) N_dynamo_marked_constantFTz1can't handle functions not implemented in python z2Dynamo can only handle functions defined in pythonz9Move usage of this function out of `torch.compile` regiongb_typecontext explanationhints_torchdynamo_inlinerX)superrxrrrrrr[jitScriptFunctionr#rINFERENCE_MODErrrrtrrr __class__rXrYrxs   zUserFunctionVariable.__init__cst|tr|jStSr)r6rrras_python_constantrr rXrYr s  z'UserFunctionVariable.as_python_constantcCgSrrXrrXrXrYrrzUserFunctionVariable.self_argscC|jSrrrrXrXrYrz!UserFunctionVariable.get_functioncC|jjSr)rrcrrXrXrYrzUserFunctionVariable.get_codecCtjSrrrrrXrXrY python_typerz UserFunctionVariable.python_typecCst|jddduS)N__self__)rrrrXrXrYrszUserFunctionVariable.has_selfcCrr)rrrrXrXrYrrz UserFunctionVariable.get_globalscCs |j}|rt|tjr|j}|Sr)rrrUserMethodVariable source_fn)rtrrXrXrY get_sourceszUserFunctionVariable.get_sourcerrDrc CsZ|jrJ|j}t|tstd|jj}|}t|||||}t |||j |jj p-d}t |t |jj j ks;Jtt|jj j |D]d\} } } |jj} | | vrX| | } nN|rtt|| }t|d| d}z t|| j|}Wn tyt}Ynw| || |} nz t|| j}Wn tyt}Ynw| d| |} | || <qF|S)z Assume `args` and `kwargs` are VariableTracker arguments for a call to this function, create new bindings for initial locals. z'Only supports regular Python functions.rX cell_contents) freevar_nameN)rrrrrrroot_txrrrrcrr co_freevarsrrcountrr,r)r(r=rr ValueErrorrDeletedVariabletrack_cell_existing)rtrrrrrrrrrrcellrcell_var closure_cellclosure_cell_contents contents_varrXrXrY bind_argssN         zUserFunctionVariable.bind_argsrrcCs,|tvr t||S|}t||j||Sr)r2rGetAttrVariablerrr)rtrrrrXrXrY var_getattr>s z UserFunctionVariable.var_getattrcCst|j|}tj|Sr)rrrr?rrrXrXrYrDs  z%UserFunctionVariable.call_obj_hasattrc s|jtjjur?z dd|D}dd|D}tjj|i|j}t|WSty>}z t d|d||d}~ww|jtjj ur~z t |jj |i|}|jd} t| tsaJt| WSty}}z t d|d||d}~ww|jtjjurt |jj |i|}|jd } t| ts| } d | jd } td d |jd|d|d| gtjdt| ts| } d| d} td|| d| ddgd| j}tj|tjj j!dS|j"rt#||j|||S|j$j%j&s$|jtjj'j(ur$tjj)*|t+,|||WdS1swYt-|j$j%dddkry|j$j%j.syzd dl/m0}Wn tyId}Ynw|dury|j|j1|j2fvrytjj).|t+,|||WdS1stwY|3|||}|dur|St+,|||S)NcSg|]}|qSrXr rargrXrXrY Sz6UserFunctionVariable.call_function..cSi|] \}}||qSrXr+)rkeyrrXrXrYrTsz6UserFunctionVariable.call_function..zCannot convert patch_dynamo_config args/kwargs to constants. Please fix your call to patch_dynamo_config by using simpler inputs. args: z , kwargs: error_on_graph_breakz\Improper error_on_graph_break() call. Please fix your call to error_on_graph_break(). args: rz=`nonstrict_trace` expects a callable, but got value of type <>zTypeError from user codezcall_function(, )rz(Applying `nonstrict_trace` to function ; however, `nonstrict_trace` currently requires the function to be defined outside `torch.compile` region.zLimitation of `nonstrict_tracezmake sure definition of z is outside z`torch.compile` regionkind descriptionz!torch.utils.checkpoint.checkpoint) FSDPState)4rr[_dynamopatch_dynamo_configrchangesrDynamoConfigPatchVariabler9 RuntimeErrorr2r signaturebind argumentsr rrErrorOnGraphBreakVariable Exceptionnonstrict_tracerrrrUr#rr USER_ERRORrrTorchInGraphFunctionVariableAllowInGraphKindNONSTRICT_TRACErinvoke_and_store_as_constantrcurrent_tracer,unsafe_allow_externally_visible_side_effectsr\8_disable_side_effect_safety_checks_for_current_subtracerr2allow_externally_visible_side_effects_in_subtracerrrrallow_side_effects_in_hop/torch.distributed.fsdp._fully_shard._fsdp_stater9 _pre_forward _post_forward_maybe_call_tree_map_fastpath)rtrrr args_const kwargs_constr<erMr2fn_vartypmsgfn_namerr9tree_map_resultr rXrYrJs         "" z"UserFunctionVariable.call_functionc Cs||||}|dur|\}}}n|}|}|}t|tr,|r,h|ts,t|dks.dS|d}|d} |dd} | |||| |S)Nrrr8)_rewrite_tree_map_only_callrr_is_tree_map_function_SUPPORTED_TREE_MAP_KWARGSr call_tree_map) rtrrrrewrite tree_map_fn tree_map_argstree_map_kwargsmap_fn first_treerestrXrXrYrRs0    z2UserFunctionVariable._maybe_call_tree_map_fastpathcC&t|jdddkot|jdd|jvS)NrUtree_maprVrr_TREE_MAP_MODULESrrXrXrYr\z*UserFunctionVariable._is_tree_map_functioncCrf)NrU tree_map_onlyrVrhrrXrXrY_is_tree_map_only_functionrjz/UserFunctionVariable._is_tree_map_only_functionc Cs|sdSt|dkrdSh|trdS|\}}}||}|dur&dS|}|dur0dSt||t|ddd} t|} | | |gt |fS)Nrr) rlr_TREE_MAP_ONLY_SUPPORTED_KWARGS_extract_tree_map_only_types_lookup_tree_map_functionTreeMapOnlyFunctionVariablerrrr~) rtrrr type_selectorrctree_arg allowed_typestree_map_callablewrapped_map_fntree_map_variablerXrXrYr[s(       z0UserFunctionVariable._rewrite_tree_map_only_callcCsNt|jdd}|s dStj|}|durdSt|dd}t|tjr%|SdS)NrVrg)rrsysmodulesrrrr)rt module_namemodulergrXrXrYrps   z.UserFunctionVariable._lookup_tree_map_functionselector.cCsd|sdSz|}Wn tyYdSw||}|s dStdd|Ds+dStt|S)Ncss|]}t|tVqdSr)rtype)rrWrXrXrY !zDUserFunctionVariable._extract_tree_map_only_types..)is_python_constantr r_flatten_type_specallrr~fromkeys)rtr| raw_value flattenedrXrXrYros   z1UserFunctionVariable._extract_tree_map_only_typescCst|tr|gSt|tr'g}|D]}||}|durdS||q|Sttdd}|durQt||rQg}|jD]}||}|durIdS||q;|SdS)N UnionType)rr}rrextendrr__args__)rtr collectedentryflat union_typerXrXrYr%s(        z'UserFunctionVariable._flatten_type_specTcCrrrXrrXrXrYis_python_hashable;rz'UserFunctionVariable.is_python_hashablecC t|jSr)hashrrrXrXrYget_python_hash>rz$UserFunctionVariable.get_python_hashothercCst|tjo |j|juSr)rrrrrtrrXrXrYis_python_equalAz$UserFunctionVariable.is_python_equalF)6rUrVrWrr_nonvar_fields frozensetri classmethodrrrrr[rrrrxr rr=rrrrr}rrr~rrrrrr'r)r?rrrRr\rlrr[rprorr rintrobjectr __classcell__rXrXr rYrs     >  k  !   &  rc>eZdZdZdddeedeeefdeffdd ZZ S) InspectSignatureVariablea Variable tracker for inspect.signature with caching support. inspect.Signature is expensive to trace. When inspect.signature is called repeatedly on the same function during tracing, we cache the result to avoid retracing the signature construction each time. Although this is different from CPython behavior, it is safe to do so because inspect.signature does not change across different calls to the same function. rrDrrrZcst|dkr9|s9|d}d}t|ttfr|}|dur9||jjvr)|jj|St|||}||jj|<|St|||Srb) rrrrrrsignature_cacherr)rtrrr target_arg cache_keyrr rXrYrPs   z&InspectSignatureVariable.call_function rUrVrWrrr=r~rrrrXrXr rYrEs  rcseZdZdhejZdeedfdededdffdd Z defd d Z d ede fd d Z ddde edeeefdefddZZS)rqrt.rcrrZNc "tjdi|||_||_dSry)rrxrtrc)rtrtrcrr rXrYrxos z$TreeMapOnlyFunctionVariable.__init__cCstSrrrrXrXrYryrz'TreeMapOnlyFunctionVariable.python_typenodecs:z|Wn tyYdSwtfdd|jDS)NFc3s|]}t|VqdSr) issubclass)rallowed node_typerXrYr~rzDTreeMapOnlyFunctionVariable._matches_allowed_type..)rranyrt)rtrrXrrY_matches_allowed_type|s   z1TreeMapOnlyFunctionVariable._matches_allowed_typerrDrcCsZ|s |j|||S|d}||r|j|||St|dks#|r+|j|||S|S)Nrr8)rcrrr)rtrrrleafrXrXrYrs z)TreeMapOnlyFunctionVariable.call_function)rUrVrWrrrr}r=rrxrrrrr~rrrrXrXr rYrqis2   rqc speZdZ ddejdededdffdd Zed edefd d Z d d de e de e e fde fddZZS)BuiltinMethodVariableFrrrrZNc s,tjdi|t|tjsJ||_dSry)rrxrrBuiltinMethodTyperr r rXrYrxs zBuiltinMethodVariable.__init__objcCs0|j}|j}|tur|dkpt|tuo|dkS)N__new__ __contains__)rrUrr}r)r method_self method_namerXrXrYis_supported_builtin_methods z1BuiltinMethodVariable.is_supported_builtin_methodrrDrcCsD|jj}|jj}|jot|jd}tj|||dd}|||||S)NrT)realize)rrrUrr'r=r call_method)rtrrrrr obj_sourceobj_vtrXrXrYrs z#BuiltinMethodVariable.call_functionr)rUrVrWrrrrrx staticmethodrrr=r~rrrrXrXr rYrs,  rc seZdZdejdeeefdddeddf fdd Zdejfd d Z defd d Z defddZ de fddZ defddZdefddZdefddZeZd>ddZdeeeffddZdefddZd d!defd"d#Zd d$d%edefd&d'Zd d$defd(d)Zd d$defd*d+Zd d!deefd,d-Zd d!d.eegefddfd/d0Zde d1fd2d3Z!d d$d4eddfd5d6Z"defd7d8Z#defd9d:Z$d d$d%ed;eedeeefdef fd.closeTrX)rm CALL_INTRINSIC_1throwrmr8r__InternalThrowException)rrrrr!rr push_manyrrrr>rExceptionVariable GeneratorExitrx version_infonext_instructionopnamerr>r symbolic_resultrr}rCrrr)rtrrrrrretvalexc_typer rXrYrEsv               (  > z(LocalGeneratorObjectVariable.call_methodrrArZN)'rUrVrWrrr~rrrxrrrr rrrr__repr__rrr}rr=rr?rrrrrrrr rrrrrrrXrXr rYrs   !        rc@seZdZdZdS)4ContextlibContextManagerLocalGeneratorObjectVariablez .. note:: This is only used when the function is annotated with @contextlib.contextmanager It is a special case of a generator function as we do not allow return a context manager from a torch.compile function. N)rUrVrWrrXrXrXrYrsrc seZdZdZeddedededdffdd Zd e defd d Z de j fd dZ dee effddZdefddZdddeedee efddfddZdddeedee efdefddZZS)LocalGeneratorFunctionVariablezxfunctions that behaves like iterators .. note:: This is a wrapper around (Nested)UserFunctionVariable ) generator_clsvtrrrZNc rry)rrxrr)rtrrrr rXrYrxs z'LocalGeneratorFunctionVariable.__init__rcCs"||jjvr t||St|j|Sr)r rrr)rtrrXrXrY __getattr__s   z*LocalGeneratorFunctionVariable.__getattr__cC |jSr)rrrrXrXrYr%rz'LocalGeneratorFunctionVariable.get_codecCrr)rrrrXrXrYr(rz*LocalGeneratorFunctionVariable.get_globalscCrr)rrrrXrXrYr+rz'LocalGeneratorFunctionVariable.has_selfrrErrCcCsddlm}|||||S)Nr)rC)rrCbuild_inline_tracer)rtrrrrCrXrXrY_build_inline_tracer.s z3LocalGeneratorFunctionVariable._build_inline_tracerrDcCsft|jstdt|jdddgd||t||}|j}|j}|j||||j dS)Nz'non-generator contextlib.contextmanagerzvCannot compile function decorated with `@contextlib.contextmanager` that is not a generator, i.e. does not use `yield`z5Use `yield` in the function body instead of `return`.z2Remove the `@contextlib.contextmanager` decorator.rr) rrrr#rrrrrr)rtrrrrrurrXrXrYr=s$   z,LocalGeneratorFunctionVariable.call_function)rUrVrWrrrr}rrxrrrrrr~rrrrr=rrrrrXrXr rYr sD     rcsXeZdZdZdededdffdd Zdd d eede e efdd ffd d Z Z S)3FunctionDecoratedByContextlibContextManagerVariablezm .. note:: This is only used when the function is annotated with @contextlib.contextmanager rrrZNc stj|fdti|dS)Nr)rrxr)rtrrr rXrYrxes z  z7WrappedUserMethodVariable.reconstruct..r8F add_push_nullr extend_outputrrtrrXr(rYr  z%WrappedUserMethodVariable.reconstructr) rUrVrWrrrxrr=r~rrrrrXrXr rYr(   rcr)WrappedUserFunctionVariablerrrFrrZNc s2|ddtj|jfi|||_||_dS)Nr)rrrxrrrrr rXrYrxs  z$WrappedUserFunctionVariable.__init__rrDrcrrr r#r rXrYrr$z)WrappedUserFunctionVariable.call_functionrrAcr%)Ncr&rr'rXr(rXrYr))r*z9WrappedUserFunctionVariable.reconstruct..r8Fr+r.rXr(rYr(r/z'WrappedUserFunctionVariable.reconstructr) rUrVrWrrrxrr=r~rrrrrXrXr rYr1s(   r1cs^dtdtfddfdd|D}fdd|D}||i|}|jj||t|d S) NxrZcSs|r td|S|S)NrK) is_tensorrget_real_valuer )r2rXrXrYconvert5sz-invoke_and_store_as_constant..convertcsg|]}|qSrXrXrr2r5rXrYr.:r/z0invoke_and_store_as_constant..csi|] \}}||qSrXrXrrrr7rXrYr;z0invoke_and_store_as_constant..r)r=rrrregister_attr_or_moduler*)rrrrrresrXr7rYrI.srIcseZdZdhejZ d8dededeeefdedBdedBdedBdedBd edBd ed dffd d Z d e efddZ d e j fddZd e jfddZd ee j fddZd8deedBd e j fddZd efddZdeed e j fddZddded ed efd!d"Zddd#ed$eed eeefd ef fd%d& Zd efd'd(Zddd#ed effd)d* Zddd#ed effd+d, Zd efd-d.Z d eeeffd/d0Z!d1dd$eed eeefd eeeffd2d3Z"d9d6d7Z#Z$S):rrNrYrur{r}rr wrapped_fnrrZc  s| ddur| jtdtjdi| t|ts Jt|tj s*Jt|t s1J||_ ||_ ||_ ||_||_||_||_||_dS)N mutation_typer=rX)rupdater:rrxrr rrrr~rYrurr{r}rrr<) rtrYrurr{r}rrr<rr rXrYrxJs z#NestedUserFunctionVariable.__init__cCr rrXrrXrXrYrgrz$NestedUserFunctionVariable.self_argscCrr)rrrXrXrYr jrz-NestedUserFunctionVariable.as_python_constantcCrr)rur rrXrXrYrmrz#NestedUserFunctionVariable.get_codecCrrrrrXrXrYrprz&NestedUserFunctionVariable.python_type _convertingc Csddlm}t|}|durt}||vrtd||zz ||WW||S|y<}ztd|d}~ww||w)Nr8)r9z-cycle detected in mutually recursive closuresz1failed to convert closure cell to Python constant)baser9idsetrTadd_get_function_impldiscard)rtr@r9self_idrUrXrXrYrss*     z'NestedUserFunctionVariable.get_functionc Cs(z|WdSttfyYdSwr)r rr$rrXrXrYrs z-NestedUserFunctionVariable.is_python_constantc Csd}|jrAddlm}|}g}|jjD]'}|jj|}||ur&tdt |t r1| |}n| }| t|qt|}tj|j |j|j d|d} |jr[|j | _|jrd|j | _|jr|j } t | tr}ddlm} t| | } t | tsJ| | _| S)Nr)rDz self-referential nested function)argdefsrr)rrrD current_txrrr load_cellrTrrrr rr7rrrrurrYr{rzr}r|rrrr~r) rtr@ closure_cellsrDrcellsr#rrrarrrXrXrYrEsB           z-NestedUserFunctionVariable._get_function_implrrDname_varrcCs|jj||j|tSr)rr store_attrrr>)rtrrMrrXrXrY call_setattrsz'NestedUserFunctionVariable.call_setattrrrcs0|dkr |j|g|RSt||t||S)N __setattr__)rOrrrrtrrrrr rXrYrsz&NestedUserFunctionVariable.call_methodcCs |jduSr)rrrXrXrY has_closurerz&NestedUserFunctionVariable.has_closurecsR|dkr|S|dkr|S|dkr"t|dd}|r |SdSt||S)NrUrcrzr{)rrrr r const_getattr)rtrrdr rXrYrSs z(NestedUserFunctionVariable.const_getattrcsB|dkr tjt|dS|dkrtjt|dSt||S)Nrcrurzr{)rr?rrrrrr rXrYrs z+NestedUserFunctionVariable.call_obj_hasattrcCrrrXrrXrXrYrrz#NestedUserFunctionVariable.has_selfcCrrrrrXrXrYrrz&NestedUserFunctionVariable.get_globalsrc Cs|}t||j|j|jrt|jjndtddt t |j D}|j r1|j |_t|j|i|}|t|j}t|jj|t|||t|j D]\}} | |vscJ|jj|} | || <qY|S)Ncss|]}tdVqdSr)r7)r_rXrXrYr~rz7NestedUserFunctionVariable.bind_args..)rrrrrYr r{rrrrrr}keys_as_python_constantr|rr?r@apply_defaultsr~rArrrrrr) rtrrrrurarMrrrr"rXrXrYr's(     z$NestedUserFunctionVariable.bind_argsrrAcsfdd|j|jgt|jjj|j r+|j n  dg|j r=|j n  dg|j rO|j n  dg|j rzz|j }|gWntyy|j Yn w dgtdd|jrfdd|jtddtdtddj}|jj|r|jjj|D]\}}|td|qdSdS) Ncs tdS)Nr)load_import_fromrUrXrrXrYr) z8NestedUserFunctionVariable.reconstruct..Fc ddS)N functoolswrapsrXrXrYrXrYr);rZr8rT)r,rur-create_load_const_uncheckedrr?rrrr{create_load_constrr}rr rrr<rrrrhas_pending_mutationstore_attr_mutationsrdup_toprN)rtrrrrrrXrYrYrs^            z&NestedUserFunctionVariable.reconstructrr)%rUrVrWrrr=r~rrrxrrrrr rrr}rrCrrrrrErOrrrRrSr?rrrr'rrrXrXr rYrDs    8         rcr)!WrappedNestedUserFunctionVariablerrrFrrZNc s|dd|dd|dd|dd|dd|dd|dd|ddt|j|j|j|j|j|j|j |j ||_ ||_ dS) NrYrurr{r}rrr<) rrrxrYrurr{r}rrr<rrrr rXrYrxOs(         z*WrappedNestedUserFunctionVariable.__init__rrDrcrrr r#r rXrYrjr$z/WrappedNestedUserFunctionVariable.call_functionrrAcr%)Ncr&rr'rXr(rXrYr)vr*z?WrappedNestedUserFunctionVariable.reconstruct..r8Fr+r.rXr(rYrur/z-WrappedNestedUserFunctionVariable.reconstructr) rUrVrWrrrxrr=r~rrrrrXrXr rYreNs(  rec seZdZddhejZddededBdeddffdd Zdefdd Ze ded e ddfd d Z d dde ede eefdefddZd ddedefddZd ddedefddZdefddZdefddZdedefddZZS)SkipFunctionVariablerreasonNrrZc rry)rrxrrg)rtrrgrr rXrYrx zSkipFunctionVariable.__init__cCrrrrrXrXrYr rz'SkipFunctionVariable.as_python_constantrcCst|ddr&|}|}t|ddr|j}t|d}t|dds|tjnt|r4t |tj n t |s@t |tj|||dS)N_torchdynamo_disableF_torchdynamo_orig_callabler) rrrrkr'rr%r isbuiltinr& BUILTIN_MATCHr5)rrrguard_on_sourceguard_on_valuerXrXrYrs    z'SkipFunctionVariable.create_with_sourcerrDrc Cs t|jddr&t|jdd}tdt|jd|jd|ddgd dS|jtjjurM|d }|r8| }td d |d |dd|dgd dS|jtjj ur~|d }|r`| }nd}tdd|d |dt |j dd|dgdddS|jtjj urz tddddgd WdSty}zt|jdd}~wwtjrddlm}|||j} t| ts| |||St|jdd } t|jd!d} | durd"nt| } z&t|j} d#| d$| d%}d&| d'g}d(| vr|d)| d*d+g7}Wnbty`d,d-h}| |vrd.| d| d'}d/d0g}nA| durC| d1rCd2| d| d}d3g}tjj|d4d4|nd5| d| d6}d7d8g}tjj|d4d4|Ynw| d9krjd:}g}|j rq|j nd;}td| d?|||d dS)@NrjF_torchdynamo_disable_msgz1Skip calling `torch.compiler.disable()`d functionzSkip calling function `z>` since it was wrapped with `torch.compiler.disable` (reason: r5z(Remove the `torch.compiler.disable` callrrXz%Call to `torch._dynamo.graph_break()`z0Called `torch._dynamo.graph_break()` with args `z `, kwargs ``z$User-inserted graph break. Message: z.Remove the `torch._dynamo.graph_break()` call.rz$Call to `torch._dynamo.skip_frame()`z/Called `torch._dynamo.skip_frame()` with args `z`. Skipping frame rz#User-inserted skip frame. Message: z-Remove the `torch._dynamo.skip_frame()` call.T)rrrrrz*Call to `torch._dynamo.step_unsupported()`zUser-inserted step_unsupported.z3Remove the `torch._dynamo.step_unsupported()` call.r8SourcelessBuilderrWzrVzz?Dynamo developers have intentionally marked that the function `z ` in file `z` should not be traced.zAvoid calling the function `z`.r:z:Apply `@torch._dynamo.dont_skip_tracing` to the function `zw` to force tracing into the function. More graph breaks may occur as a result of attempting to trace into the function.z Please file an issue to PyTorch._abc _warningsz6Dynamo does not know how to trace the Python builtin `zIf you are attempting to call a logging function (e.g. `_warnings.warn`), you can try adding it to `torch._dynamo.config.reorderable_logging_functions`.zKPlease file an issue on GitHub so the PyTorch team can add support for it. rz*Dynamo cannot trace optree C/C++ function zi Consider using torch.utils._pytree - https://github.com/pytorch/pytorch/blob/main/torch/utils/_pytree.py z/Dynamo does not know how to trace the builtin `z.` This function is either a Python builtin (e.g. _warnings.warn) or a third-party C/C++ Python extension (perhaps created with pybind).zIf it is a Python builtin, please file an issue on GitHub so the PyTorch team can add support for it and see the next case for a workaround.aIf it is a third-party C/C++ Python extension, please either wrap it into a PyTorch-understood custom operator (see https://pytorch.org/tutorials/advanced/custom_ops_landing_page.html for more details) or, if it is traceable, use `torch.compiler.allow_in_graph`.allow_in_graphzFound an allow_in_graph decorator to a function which is created inside the parent function that is getting compiled. This is not supported for now.zz,Attempted to call function marked as skippedzmodule: z , qualname: z, skip reason: )!rrrr#rr[r: graph_breakrr rrf_codestep_unsupportedr$r"rXrdont_skip_tracingbuilderrsrrrfrrgetfilerrr\ warn_oncejoinrg)rtrrrrXgraph_break_msgskip_frame_msgrUrs rebuilt_fnqualname module_orrzpathrrknown_python_builtin_modulesrgrXrXrYrs                  & z"SkipFunctionVariable.call_functionrcCstjt|j|Sr)rr?rrrrrXrXrYr3sz%SkipFunctionVariable.call_obj_hasattrcCs&|tvr t||St||j|j|Sr)r2rr(rrrrrXrXrYr)8s z SkipFunctionVariable.var_getattrcCrrrXrrXrXrYr>rz'SkipFunctionVariable.is_python_hashablecCrr)rrrrXrXrYrArz$SkipFunctionVariable.get_python_hashrcCst|to ||kSr)rr=r rrXrXrYrDs z$SkipFunctionVariable.is_python_equalr)rUrVrWr=rrrrxr rrrrr~rr?rr)rrrrrrrrXrXr rYrf{s>$   rfcr)WrappedSkipFunctionVariablerrrFrrZNc sD|dd|ddtj|jfd|ji|||_||_dS)Nrrg)rrrxrrgrrrr rXrYrxLs   z$WrappedSkipFunctionVariable.__init__rrDrcrrr r#r rXrYrXr$z)WrappedSkipFunctionVariable.call_functionrrAcr%)Ncr&rr'rXr(rXrYr)dr*z9WrappedSkipFunctionVariable.reconstruct..r8Fr+r.rXr(rYrcr/z'WrappedSkipFunctionVariable.reconstructr) rUrVrWrfrrxrr=r~rrrrrXrXr rYrKr0rcseZdZdZdedededdffdd Zd d d edeffd d Zde efddZ d d de ede eefdefddZ ZS)WrapperUserFunctionVariablea3 Used to represent a wrapper object that contains the actual callable as an attribute. For example, torch.jit.script/trace have the original function at their _torchdynamo_inline attribute. Similarly, functions with __script_if_tracing_wrapper have the original attr at "__original_fn". wrapper_obj attr_to_tracerrZNc rry)rrxrr)rtrrrr rXrYrxqrhz$WrapperUserFunctionVariable.__init__rrDrcsF||jkrt|j|j}|jot|j|}t|||St||Sr) rrrrr'r=rrr))rtrrrrr rXrYr)vs z'WrapperUserFunctionVariable.var_getattrcCr rrXrrXrXrYr~rz%WrapperUserFunctionVariable.self_argsrc CsLt|jdrt|j|jd}t|ddpd}|jdddddkr|}tj|j }|j }d |d |d d t |g} tjj| tjjjd } | tjrtjj} t| } | d j | d j f} d t | } d| dd | dd}|t| 7}| ||t|}tt j!"||t#|jg||S)N cache_inforVrrr8)maxsplitrr[zGDynamo detected a call to a `functools.lru_cache`-wrapped function at ':z'. Dynamo ignores the cache wrapper and directly traces the wrapped function. Silent incorrectness is only a *potential* risk, not something we have observed. Enable TORCH_LOGS=+dynamo for a DEBUG stack trace. This call originates from: z torch._dynamoz)call to a lru_cache wrapped function at: rv)$rrrrsplit frame_summaryosrbasenamefilenamelinenor traceback format_listr[r:r\r~logging getLogger isEnabledForDEBUG_guardsTracingContext extract_stackrrdebugrrrrrgetattr_and_tracerr?)rtrrr target_fnrzrrrrX dynamo_logger user_stack frame_locuser_stack_formatteduser_stack_traceall_argsrXrXrYrsF       z)WrapperUserFunctionVariable.call_function)rUrVrWrrrrxr=r)rrrr~rrrXrXr rYris rc sHeZdZdZdededededdf fdd Zdeefd d Z Z S) WrapperUserMethodVariablez Similar to WrapperUserFunctionVariable, but for methods. The only delta is saving the vt for `self` object of the method which is then used by WrapperUserFunctionVariable in `call_function` method. rrself_objrrZNc s tj||fi|||_dSr)rrxr)rtrrrrr rXrYrxs z"WrapperUserMethodVariable.__init__cCr rr rrXrXrYrrz#WrapperUserMethodVariable.self_args) rUrVrWrrrr=rxrrrrXrXr rYrs rcCstjr ddlm}|SiS)Nrtraceable_collective_remaps)r[ distributed is_available)torch.distributed._functional_collectivesrrrXrXrY_traceable_collective_remapss  rcCs>tjs Jd|tvsJ|j}|d}t||S)NzIllegal invocation.z)torch.distributed._functional_collectives)r[rrrvaluesrU import_sourcer')rr inner_name path_sourcerXrXrY_traceable_collectives_sources   rc seZdZdZdedefdededdffdd Zed d d edefd e deddf ddZ edede fddZ ed d dedefde eeffddZd d deedeeefdefddZZS)!CollectiveFunctionRewriteVariablea Some of the torch.distributed.* collective APIs are possible to rewrite to 'traceable' collectives. This class provides both a way to check if a function is remappable, and perform the remapping. In the case that a function is 'remappable' but only for some combinations of call-time arguments, we check the args at `call_function` time and fall back to graph-breaking if needed. This is no worse than status-quo as we currently graph-break on all distributed.* collectives. r.replacement_varrrZNc s,tj|fi|t|tsJ||_dSr)rrxrrr)rtrrrr rXrYrxs z*CollectiveFunctionRewriteVariable.__init__rrDold_fnroptionscKs6t||\}}t|ft|fd|i||d|S)Nr)rr)rr_r)rrrrnew_fn new_sourcerXrXrYrsz(CollectiveFunctionRewriteVariable.createvariablecCst|o |tvSr)r isfunctionr)rrXrXrY can_rewrite sz-CollectiveFunctionRewriteVariable.can_rewritecCst|}|t||fSr)rr)rrrrXrXrYr_ s z)CollectiveFunctionRewriteVariable.rewriterc Csddlm}ddlm}t|j}t|j|i|j }d}d|vrA|d rAt d|jd|d|d|jgt j d |j|j|j|jfvrs|d }|durX|jn|jd j}||vritd |tj|||d <|j|||S) Nr)REDUCE_OP_TO_STRrXasync_opz)async_op=True for distributed collectivesz, args= , kwargs=z3`torch.compile` doesn't support `async_op=True for ropzUnsupported all_reduce op: )torch.distributedrrrrr?rr~r@rAr r#r SUPPORTABLE all_reducereduce_scatter_tensor_reduce_scatter_baserr parametersdefaultrrr?rrr) rtrrrdistrr? reduce_op_var reduce_oprXrXrYr s<      z/CollectiveFunctionRewriteVariable.call_function)rUrVrWrrrrrxrrrrrrr'r_rr=r~rrrrXrXr rYrsT        rcs:eZdZdddeedeeefdeffdd ZZS)FunctoolsWrapsVariablerrDrrrZcs@|stdkrdtdtffdd }t|St||S)Nr8rrZcs8t|tjr|jddStd|dgtjddS)Nr)r<zfunctools.wrapsz]`torch.compile` can't trace `functools.wraps` on functions defined outside the compile regionr)rrrcloner#rrrrrXrYr^F s  z3FunctoolsWrapsVariable.call_function..wraps)rrr=rLambdaVariablerr)rtrrrr^r rrYr> s z$FunctoolsWrapsVariable.call_function) rUrVrWrr=r~rrrrXrXr rYr= s rc@s@eZdZdefddZdddeedeeefdefdd Z d S) CollectionsNamedTupleFunctionrZcCrrrrrXrXrYr X rz0CollectionsNamedTupleFunction.as_python_constantrrDrrc Cst||}|rHz|jdd|Didd|D}Wn"ty?}ztt||tttj |j dWYd}~nd}~wwt j |t dStdd|d |d gtjd dS) NcSr*rXr+r6rXrXrYr.e r/z?CollectionsNamedTupleFunction.call_function..cSr0rXr+r8rXrXrYrf r9z?CollectionsNamedTupleFunction.call_function..rr>znamedtuple constructionzargs=rz?`torch.compile` only support certain input types for namedtupler)r0rrrr!r}rmapr?rrrUserDefinedClassVariabler<r#rr)rtrrr constant_argsrrrXrXrYr[ s8    z+CollectionsNamedTupleFunction.call_functionN) rUrVrWrr rr=r~rrrXrXrXrYrW s rc seZdZdhejZ d&dedeedeeefdededdf fdd Z de fd d Z d'ddZ defddZ dddeedeeefdefddZdddedefddZdddedefddZdefddZdefddZdefdd Zdefd!d"Zd#edefd$d%ZZS)(FunctoolsPartialVariableoriginal_cache_hashNrarkeywordsrrZc sVtjdi|||_t|tsJ||_t|tsJ||_t t |_ ||_ dSry) rrxrarrrr~rr]partialr3 fake_valuer)rtrarrrrr rXrYrx s  z!FunctoolsPartialVariable.__init__cCrr)r]rrrXrXrYr rz$FunctoolsPartialVariable.python_typerrAcsfdd|j|jr|j|js)tt|jdddS|jt |j } t|t|jd|ddS)Ncr\)Nr]rr_rXrYrXrYr) rZz6FunctoolsPartialVariable.reconstruct..r8F) r,rarforeachrr-rrrrkeyscreate_call_function_kw)rtrrrXrYrYr s  z$FunctoolsPartialVariable.reconstructcCrrr+rrXrXrYr rz%FunctoolsPartialVariable.get_functionrrDcCs,|jt|}i|j|}|j|||Sr)rrrrar)rtrrr merged_args merged_kwargsrXrXrYr sz&FunctoolsPartialVariable.call_functionrcCstjttt|Sr)rr?rrr]rr3rrXrXrYr sz)FunctoolsPartialVariable.call_obj_hasattrcCs|jot|j|}|dkr|jS|dkrtj|j|dS|dkr1dd|jD}tj||dS|t vr;t ||St t |dS)NrarrrcSsi|] \}}t||qSrX)r?rr8rXrXrYr sz8FunctoolsPartialVariable.var_getattr..) rr'rarrJrrrConstDictVariabler2r(r!r)rtrrrrrXrXrYr) s z$FunctoolsPartialVariable.var_getattrcCs:tj|jgdd|jDRidd|jDS)NcSr*rXr+r,rXrXrYr. r/z?FunctoolsPartialVariable.as_python_constant..cSr0rXr+r8rXrXrYr r9z?FunctoolsPartialVariable.as_python_constant..)r]rrar rrrrrXrXrYr  sz+FunctoolsPartialVariable.as_python_constantcCsPtj|jgdd|jDRidd|jD}|jdur&|j|_|S)zcSimilar to as_python_constant(), but add ID_MATCH guards to try to force things to become constantscSr*rXguard_as_python_constant)rrrXrXrYr. r/zEFunctoolsPartialVariable.guard_as_python_constant..cSr0rXrr8rXrXrYr r9zEFunctoolsPartialVariable.guard_as_python_constant..N) r]rrarrrrr cache_hashrtrrXrXrYr s z1FunctoolsPartialVariable.guard_as_python_constantcCs6|jotdd|jDotdd|jDS)Ncsrrrr,rXrXrYr~ rz>FunctoolsPartialVariable.is_python_hashable..csrrrrrrXrXrYr~ r)rarrrrrrrXrXrYr s z+FunctoolsPartialVariable.is_python_hashablecCsB|j}dd|jD}dd|jD}t|g||RS)Ncsrrrr,rXrXrYr~ rz;FunctoolsPartialVariable.get_python_hash..csrrrrrXrXrYr~ r)rarrrrr)rt func_hash args_hash values_hashrXrXrYr s z(FunctoolsPartialVariable.get_python_hashrcCsXt|to+|j|jo+tddt|j|jDo+tddt|j|jDS)Ncs|] \}}||VqdSrr)rarg_aarg_brXrXrYr~  z;FunctoolsPartialVariable.is_python_equal..csrrr)rvalue_avalue_brXrXrYr~ r) rrrarrrrrrrrXrXrYr s   z(FunctoolsPartialVariable.is_python_equalrr)rUrVrWr=rrr~rrrxr}rrrrr?rr)r rrrrrrrrrXrXr rYr} sX        rc seZdZdddhejZeejdee de fe j ffddZ ede d eddfd d Zded e dd ffdd Zede de ffddZdefddZde fddZdddeed eeefdefddZdddedeed eeefdef ddZde fddZZS) PolyfilledFunctionVariablerr< traceable_fnrZ.cCrrrX)rrXrXrY_get_polyfill_handlers sz1PolyfilledFunctionVariable._get_polyfill_handlersrrcCrrrrrXrXrYr s z-PolyfilledFunctionVariable.create_with_sourcerNc stjdi|||_|||}d}t|s$Jd|d|dD]}t||d}|r:t|s6J|}n q&td|d||_||_ dS)NzPolyfill handler z is not callable for )__torch_dynamo_polyfill____python_implementation__z# does not have a traceable functionrX) rrxrrrcallablerr>r<r)rtrrhandlerrcandidate_attr candidater rXrYrx s"    z#PolyfilledFunctionVariable.__init__cCrr)rrrXrXrY polyfill_fn, sz&PolyfilledFunctionVariable.polyfill_fncCr)N*__torch_dynamo_can_constant_fold_through__F)rr<rrXrXrYcan_constant_fold_through0 z4PolyfilledFunctionVariable.can_constant_fold_throughcCrrr+rrXrXrYr5 rz'PolyfilledFunctionVariable.get_functionrrDrc Cs|r#t||r#|jdd|Didd|D}t||S|jtjurot|dkro|sot |dt j t j frot dd|djDrot jj||jd tjtd d|djDfitd d|djDd St||j}||||S) NcSr*rXr+r6rXrXrYr.C r/z.cSr0rXr+r8rXrXrYrD r9z.r8rcss>|]}|rt|tpt|tjo|tuVqdSr)rrr rrSymNodeVariablerr6rXrXrYr~O s  z;PolyfilledFunctionVariable.call_function..rcsrras_proxy)rarXrXrYr~Z rcSs"g|] }|r |n|jqSrX)rr sym_numr6rXrXrYr.^ s  )r)rr1rrr=rbuiltinssumrrrrJ TupleVariablerrrr create_proxyr[sym_sumrrr)rtrrrrtraceable_function_variablerXrXrYr8 sH      z(PolyfilledFunctionVariable.call_functionrcCs|dkr ||||St|j|d}|s$t|s$t|d|d|ji}|jr1t|j||d<t|fi|}||||S)N__call__zCannot find callable z in r)rrrr4r;rr'r)rtrrrrmethodrpolyfilled_method_variablerXrXrYrl s z&PolyfilledFunctionVariable.call_methodcCrrrrrXrXrYr  rz-PolyfilledFunctionVariable.as_python_constant)rUrVrWr=rrr]cacher~rrrrrrrrLrxpropertyrrrrrrrrrr rrXrXr rYr sX$   4  rcsleZdZdededdffdd Zdd d Zdddefd d Zdddeede e efdefddZ Z S)SysFunctionVariablerrrZNc stjdi|||_dSry)rrxr)rtrrr rXrYrx s zSysFunctionVariable.__init__rrDvariables.TupleVariablecCsTt|jr|jd}|j}||d}t||||g}ntjtjtjg}t|S)Nr __traceback__) r exn_vt_stackrr)r=rrr>r)rtrexnrWtbrrXrXrYexc_info s    zSysFunctionVariable.exc_infocCs||jdS)Nr8)rrrrXrXrY exception szSysFunctionVariable.exceptionrcCs0|jtjur ||S|jtjusJ||Sr)rrxrrrrXrXrYr s   z!SysFunctionVariable.call_function)rrDrZr ) rUrVrWrrxrr=rrr~rrrrXrXr rYr  s  r ) create_tma_experimental_metadatacreate_tma_stable_metadataTMADescriptorMetadataTritonHOPifierc @sPeZdZdedefddZdedefddZdede fd d Z d d de e j jjd ffddZd e deee fddde fddZded e fdeedeeefdedde def ddZde dedde dedef ddZde deddee fd d!Zd"ede fd#d$Zdd%dee dd%fd&d'Zdd%d(e d)eee fdddd*f d+d,Zd-S).DynamoTritonHOPifierrXrZcCstddd|gddS)Nz!triton kernel unsupported featurerz/Encountered triton kernel unsupported feature: rr#)rtrXrXrXrYraise_unsupported s  z&DynamoTritonHOPifier.raise_unsupportedmaybe_callablecCst|ttfSr)rrr)rtrrXrXrY is_callable rz DynamoTritonHOPifier.is_callablercCs|jSrri)rtrrXrXrY get_value rzDynamoTritonHOPifier.get_valuegridrI.cCsBddlm}t||r|Stddt|dgtjddS)Nr8)rIz/unsupported grid type for triton hop check_gridz grid type = z;`torch.compile` only supports list-like grid for check_gridr)listsrIrrr#r}rr)rtrrIrXrXrY check_grid s    zDynamoTritonHOPifier.check_gridmetarrDcCs&dd|D}|||gi}|S)NcSi|] \}}tj||qSrXrr?rr8rXrXrYr sz2DynamoTritonHOPifier.call_grid..)rr)rtrr!rmeta_varrXrXrY call_grid szDynamoTritonHOPifier.call_griduser_fnrrrc Cs*ddlm}|||}||||}|S)Nr8rr)r|rsrr) rtr&rrrrrswrapped_user_functionrrXrXrYcall_user_defined_fn s  z)DynamoTritonHOPifier.call_user_defined_fnuser_objrcCs6ddlm}|dus J||t|j||}|S)Nr8)VariableBuilder)r|r*r' kernel_source_wrap)rtr)rrrr*wrapped_user_objrXrXrYwrap_user_defined_obj s  z*DynamoTritonHOPifier.wrap_user_defined_objconfigscCs||}dd|D}|S)NcSr*rXr)rrrXrXrYr. r/z=DynamoTritonHOPifier.maybe_unpack_configs..)unpack_var_sequence)rtr/rrXrXrYmaybe_unpack_configs s z)DynamoTritonHOPifier.maybe_unpack_configsrcCs|s |d|S)Nz`@triton.heuristics must return constant values because configs can only contain constant values.)rrrrrXrXrYmaybe_unpack_heuristic_result s z2DynamoTritonHOPifier.maybe_unpack_heuristic_resultTritonKernelVariablecCs>|jdus t|dkr|dt||j|j|d|jdS)Nr8z7Triton kernels should be called with only a single gridr)kernel kernel_idxrr+)rrrr}r4r5r)rtrrrXrXrY call_getitem sz!DynamoTritonHOPifier.call_getitemgrids combined_argszvariables.ConstantVariablec Csddlm}i}t|D]}||}t|ttfr'|||<|||<qdd| D} ddl m } m } dd| D} dd| D} | D]}|}|sh|sh|d t|d qO| | }|| t}|jd | d |j||||d tdS)Nr8)rcSr"rXr#r8rXrXrYr3 sz1DynamoTritonHOPifier.call_HOP..r)kernel_side_tabletriton_kernel_wrapper_mutationcSs,i|]\}}t|tr|r||qSrX)rr=rr r8rXrXrYr? scSs(i|]\}}t|tr|s||qSrX)rr=rr8rXrXrYrD sz.Unexpected argument type for a Triton kernel: rrrX)r5constant_args_idxrtma_descriptor_metadatar)dictsrrrr!TMADescriptorExperimentalVariableTMADescriptorStableVariable to_metadata get_tensorr*torch._higher_order_ops.triton_kernel_wrapr9r:rrr3is_symnode_likerrepradd_constant_argsr~rrr5rrr?)rtrr7r8rrr<rrcombined_args_vtr9r:rnon_constant_argsr;r!rXrXrYcall_HOP sV       zDynamoTritonHOPifier.call_HOPN)rUrVrWrr rr=rrrrrr[fxproxyProxyr r~r%rrr r(r.rr1r2r6rHrXrXrXrYr s            rc seZdZUded<ded<edBed<ded<dededBded ed df fd d Zd ddeed e e efd efddZ d dde de ed e e efd ef fdd Z ded efddZZS)r3rGrrHr4Nr5r'r+rrZc s4|dd|_tjdi|t||||dS)Nr+rX)rr+rrx dynamo_triton_hopifier_singleton init_variable)rtr4r5rrr rXrYrxn szTritonKernelVariable.__init__rrDrcCst||||Sr)rLcall_triton_kernelrrXrXrYru sz"TritonKernelVariable.call_functionrcs>|dkr t||S|dkrt||||St||||S)N __getitem__run)rLr6call_runrrrQr rXrYr s  z TritonKernelVariable.call_methodr-cCs4ddlm}ddlm}t||r||S|S)Nr8)r?)r)rr?tensorrrr evaluate_expr)rtr-r?rrXrXrYspecialize_symbolic s   z(TritonKernelVariable.specialize_symbolic)rUrVrWrrrrxrr=r~rrrrrTrrXrXr rYr3h sJ      r3c sdeZdZdddeedeedededdf fd d Zdefd d ZdddZdefddZ Z S)r>data_ptrzvariables.DataPtrVariabledims block_dims element_sizerrZNc s>t|tjsJtjdi|||_||_||_||_dSry) rrDataPtrVariablerrxrUrVrWrX)rtrUrVrWrXrr rXrYrx s  z*TMADescriptorExperimentalVariable.__init__cCs*tdd|jDdd|jD|jS)NcSr*rXrrdimrXrXrYr. r/zATMADescriptorExperimentalVariable.to_metadata..cSr*rXrrZrXrXrYr. r/)rrVrWrXrrrXrXrYr@ s z-TMADescriptorExperimentalVariable.to_metadatarrAcsXfddjgjjj}|t|dddS)NcsddtjdS)Nz$triton.tools.experimental_descriptorcreate_d_tma_descriptor)rXrrVrXr(rXrYr) sz?TMADescriptorExperimentalVariable.reconstruct..r8F) r,rUrrVrWrXrrr)rtrrrXr(rYr s   z-TMADescriptorExperimentalVariable.reconstructcCrr)rU from_tensorrrXrXrYrA rz,TMADescriptorExperimentalVariable.get_tensorr) rUrVrWrr=rrxr@rrArrXrXr rYr> s"  r>csTeZdZdddddeddffdd Zdefd d ZdddZdefddZZS)r?rRrK block_shaperJrrZNc s.|sJtjdi|||_||_dSry)r3rrxrRr_)rtrRr_rr rXrYrx s  z$TMADescriptorStableVariable.__init__cCst|jSr)rr_rrrXrXrYr@ rz'TMADescriptorStableVariable.to_metadatarrAcs@fddd|j|jddS)Ncr\)Nztriton.tools.tensor_descriptorTensorDescriptorr_rXrYrXrYr) sz9TMADescriptorStableVariable.reconstruct..r^r)r, load_methodrRrr_rr.rXrYrYr s    z'TMADescriptorStableVariable.reconstructcCrr)rRrrXrXrYrA rz&TMADescriptorStableVariable.get_tensorr) rUrVrWrrxr@rrArrXrXr rYr? s  r?csPeZdZdededdffdd Zddd eedee efdefd d Z Z S) 'CreateTMADescriptorExperimentalVariablerankrrZNc s(|dvsJtjdi|||_dS)N)r8rrX)rrxrc)rtrcrr rXrYrx s  z0CreateTMADescriptorExperimentalVariable.__init__rrDrcCsd|vr|dn|d}t|tjs+tdd|d|d|jdd|jd gd |jd krct|t|d krHt|d t|t|d|vrP|dn|d g}d|vr]|dn|dg}nHt|t|dkr{t|dt|t|d|vr|dn|d d|vr|dn|dg}d|vr|dn|dd|vr|dn|d g}d|vr|dn|d}t|tjsJt||||dS)Nptrrz.invalid ptr argument for create_tma_descriptorzargs = z , kwargs = z#Expected `ptr` argument of `create_z]d_tma_descriptor`to be from a `.data_ptr()` call, represented internally by `DataPtrVariable`z`torch.compile` may fail to internally represent result of `.data_ptr()` with `DataPtrVariable` due to a graph break between the `.data_ptr()` call and `create_zTd_tma_descriptor`. Please ensure there were no graph breaks between these two calls.rr8rNz6TMA metadata rank=1 requires exactly 4 arguments, got r[ block_dimrz6TMA metadata rank=2 requires exactly 6 arguments, got dim1dim0 block_dim1rm block_dim0rXr)rUrVrWrX)rrrYr#rcrr;r>)rtrrrrdrVrWrXrXrXrYr sP   z5CreateTMADescriptorExperimentalVariable.call_function) rUrVrWrrrxrr=r~rrrrXrXr rYrb s"  rbc@s2eZdZdddeedeeefdefddZdS) !CreateTMADescriptorStableVariablerrDrrrZcCs<d|vr|dn|d}d|vr|dn|d}t||dS)NrRrr_r8)rRr_)r?)rtrrrrRr_rXrXrYr( s z/CreateTMADescriptorStableVariable.call_functionN)rUrVrWrr=r~rrrXrXrXrYrk' s rkc@s6eZdZdZdddeedeeefdefddZd S) !PyTreeGetNodeTypeFunctionVariableaK `torch.utils._pytree._get_node_type` function is very hot function. We want to special case it to reduce Dynamo tracing time. def _get_node_type(tree: Any) -> Any: node_type = type(tree) # All namedtuple types are implicitly registered as pytree nodes. # XXX: Other parts of the codebase expect namedtuple types always return # `namedtuple` instead of the actual namedtuple type. Even if the type # is explicitly registered. if is_namedtuple_class(node_type): return namedtuple return node_type rrDrrrZcCst|dkrt|dt|d}|djr)t|djtjt|dj}|d}t |rAt t dd}t |t|St j |||dS)Nr8z6pytree_get_node_type requires exactly 1 argument, got r collectionsrr)rr;rr&rr% TYPE_MATCHr/rrr'r-r=rr)rtrrr type_sourcerrXrXrYrF s    z/PyTreeGetNodeTypeFunctionVariable.call_functionN) rUrVrWrrr=r~rrrXrXrXrYrl7 s rlcr) PyTreeTreeIsLeafFunctionVariablea@ `torch.utils._pytree.tree_is_leaf` function is a hot function. We want to special case it to reduce Dynamo tracing time. def tree_is_leaf( tree: PyTree, is_leaf: Callable[[PyTree], bool] | None = None, ) -> bool: if is_leaf is not None and is_leaf(tree): return True return _get_node_type(tree) not in SUPPORTED_NODES When is_leaf is None (the common case), we can optimize by not tracing into the function. When is_leaf is not None, we fall back to regular tracing since it requires executing user code. rrDrrrZc st|dks t|dkrt|dt||dt}t|dkr&|d}|s2t|||S|d}ttj j j ||gi}t j |tj j jtd}||d|gi}t|j S)Nr8rz,tree_is_leaf requires 1 or 2 arguments, got rPrrr)rr;rr>rrrrlr[r\r]_get_node_typer=rr^r_rr?rr) rtrrrrPtree node_type_varsupported_nodes_varoutr rXrYrl s.    z.PyTreeTreeIsLeafFunctionVariable.call_functionrrXrXr rYrp\ s rpcTeZdZdZdededdffdd Zdd d eedee efdefd d Z Z S) SparseTensorCreationSkipVariablezk Skip variable for sparse tensor factory functions with clear messaging regarding lack of support. rrrZNc d}tj|fd|i|dS)Nz8sparse tensor creation is not supported in torch.compilergrrxrtrrrgr rXrYrx z)SparseTensorCreationSkipVariable.__init__rrDrcCsHddlm}t|jdt|j}tdd|d|dg|jddS) Nr)rrUz$Sparse tensor creation not supportedz function: zEtorch.compile does not support sparse tensor creation functions like zZ. Sparse tensors require specialized handling that is not yet implemented in the compiler.r)rrrrrr# SPARSE_TENSOR)rtrrrrrYrXrXrYr s   z.SparseTensorCreationSkipVariable.call_function rUrVrWrrrxrr=r~rrrrXrXr rYrw  rwcrv) TritonSetAllocatorSkipVariablezm Skip variable for triton.set_allocator with a clear message to move it outside the compiled region. rrrZNc rx)Nz:triton.set_allocator is not supported inside torch.compilergryrzr rXrYrx r{z'TritonSetAllocatorSkipVariable.__init__rrDrcCstddddgddS)Nz"triton.set_allocator not supportedz2triton.set_allocator called inside compiled regionz{triton.set_allocator is not supported inside torch.compile. It modifies global Triton allocator state and cannot be traced.zgMove triton.set_allocator() outside of the torch.compile region (call it before the compiled function).rrrrXrXrYr s z,TritonSetAllocatorSkipVariable.call_functionr}rXrXr rYr r~rr)rrr]rrrrrxrrrmrcollections.abcrrrrtypingrrr r r r typing_extensionsr weakrefrr[torch._dynamo.excr torch._guardsrtorch.utils._pytreerrrrrrbytecode_transformationrrrrrrrrrrr r!r"r#r$guardsr%r&rr'r(r)r*r+r,r-r.r/r\r0r1r2r3r4r5r6r7rAr9r:r;r<r=rr>r?#torch.distributed.fsdp._fully_shardr@ModuleNotFoundErrortorch._dynamo.codegenrArrBrCrDrE#torch._dynamo.variables.ctx_managerrFrBrGrHrrIrJrRrKrLrprrrr]rnrrSrrrT lru_cacher_r`rr~rrrrrrrrrrrrrrrqrrrrrrrr1rIrrerfrrrrrrrrrrr rrrrrrLr3r>r?rbrkrlrprwrrXrXrXrYst       4 ,(           Y        5$,  L P#t    -QG    b&#70'!G%7