o ei}@sUddlmZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl m Z m Z mZmZmZddlmZmZddlmZddlmZddlmZmZmZmZmZmZmZmZmZm Z m!Z!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)dd l*m+Z+ddl,Z,dd l,m-Z-m.Z.m/Z/ddl0m1m2m3Z4ddl5m6m7Z8ddl9Z:ddl;Z:ddldd l?m@Z@dd lAmBZBdd lCmDZDddlEmFZFddlGmHZHddlImJZJddlKmLZLmMZMmNZNmOZOmPZPddlQmRZRmSZSmTZTmUZUmVZVmWZWmXZXmYZYmZZZddl[m\Z\ddl]m^Z^ddl_m`Z`ddlambZbmcZcmdZdmeZeddlfmgZgddlhmiZimjZjddlkmlZlmmZmmnZnmoZompZpddljmqZqmrZrmsZsmtZtmuZuddlvmwZwddlxmyZymzZzm{Z{m|Z|ddl}m~Z~ddlmZmZdd l7mZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZdd!lmZmZmZerdd"lmZdd#lQmZdd$l[mZdd%lmZdd&lmZdd'lmZdd(l7mZneZd)ed*<z ddlZejZd+ZWneydZd,ZYnwe'd-Ze!d.Ze!d/Ze!d0Ze"ee-fZd)ed1<e"eee-fZd)ed2<e"e:jje:jjfZd)ed3<eeZeje jd4d5Ze:jjZ e"ed6eed6fd7d8eee"eeed6fd6d7d8ffZd)ed9<dd>d?Zejd+d@GdAdBdBZĐddFdGZŐddKdLZƐddPdQZǐddRdSZȐddYdZZgd[Zgd\Z dddadbZ̐ddddeZ dddfdgZedddjdkZedddmdkZ +dddpdkZϐddtduZАddxdyZѐdd{d|ZҐdd}d~ZӐdddZԐdddZՐdddZ֐dddZאdddZddZGdd8d8Zed,d@GdddZeGdddeڃZܐdddZeGddde܃ZeGdddeރZedededededededdZded< +ddddZeGddde܃Zde.dfdddZe ee-ee-gefZd)ed<GddĄdeZGddƄdeZGddȄdeZeGddʄde܃ZeGdd̄deZeGdd΄de܃ZdddЄZddd҄Z + ,  , ddddۄZddd݄ZdddZeGdddeڃZeGdddeZeGdddeZeGdddeZeGdddeZeGdddeZeGdddeZeGdddeZGdddeZeGdddeڃZeGdddeZeGdddeZdddZdddZGdddZeGdddeZGdddeZGdddeZGdddeZGdddeZGd d d eZeGd d d eZGd ddeZed,d@GdddeemZed,d@GdddeeۃZGdddeZ Gddde Z Gddde Z eGdddeڃZ eGdddeڃZ ed,d@GdddeZGdd d eZGd!d"d"eZe"eeeeee"eeeeffZGd#d$d$ZGd%d&d&eZGd'd(d(eZGd)d*d*eZGd+d,d,eZGd-d.d.eZGd/d0d0eZdÐd4d5Zed,d@Gd6d7d7eZGd8d9d9eZGd:d;d;eZed,d@Gd<d=d=eZed,d@Gd>d?d?eZ Gd@dAdAe Z!GdBdCdCeZ"GdDdEdEeZ#GdFdGdGeZ$GdHdIdIe$Z%GdJdKdKe$Z&GdLdMdMeZ'GdNdOdOeZ(GdPdQdQeZ)GdRdSdSeZ*GdTdUdUeZ+GdVdWdWe+Z,GdXdYdYe"Z-GdZd[d[eZ.Gd\d]d]eZ/Gd^d_d_e Z0Gd`dadaeZ1GdbdcdceZ2GdddedeeZ3GdfdgdgeZ4ed,d@GdhdidiZ5Gdjdkdke"Z6ed,d@Gdldmdme6Z7Gdndodoe6Z8eGdpdqdqeZ9GdrdsdseZ:ejGdtdudueڃZ;Gdvd6d6e;ZdĐd|d}Z?ed,d@Gd~ddeZ@ed,d@GdddeZAdŐddZBed,d@GdddeZCGddde6ZDGdddeڃZEeGdddeEZFeGdddeEZGGddde6ZHGdddeHZIGdddeHZJGdddeHZKdƐddZLdƐddZMdǐddZNdS() annotationsN)Callable GeneratorIterableIteratorSequence)AbstractContextManager nullcontext)Enum)partial) AnycastClassVarLiteralOptionaloverload SupportsFloat SupportsInt TYPE_CHECKING TypeAliasTypeVarUnion) assert_neverNeveroverride ParamSpecSelfTypeIs)patch)ExprIntegerSymbol)identity)GraphModuleSerializer)can_auto_functionalize)metricsget_free_symbols)is_opaque_type)compute_required_storage_lengthis_boolean_dtypeis_float_dtypemake_channels_last_strides_for StrideType) &_remove_effect_token_unbacked_bindingscompute_unbacked_bindings free_symbolsfree_unbacked_symbols IterateExprsrebind_unbackedresolve_unbacked_bindingsShapeEnvSymTypes)Node OrderedSet)_disable_current_modes)CleanDivFloorDivModModularIndexing)SymT)config dependencies)BackendFeature CodegenSymbolget_scheduling_for_deviceindex_prevent_reorderingKernel)Depextract_free_symbols#extract_input_node_reduction_rangesextract_read_writes var_builder)LoopBody) OpCounterCSE OpCountResult ReductionType StoreMode) benchmarker)DeviceProperties ReductionHint)argsort argsort_sym cache_on_selfcache_on_self_and_argsceildivconvert_shape_to_inductorconvert_shape_to_symintdeveloper_warningdo_bench_using_profilingdtype_from_sizeget_dtype_sizeget_kernel_metadataGPU_ALIGN_BYTES ir_dataclass is_dynamicis_gpu sympy_dotsympy_index_symbolsympy_index_symbol_with_prefix sympy_product sympy_substensor_is_aligned)opsOpsValueV)FakeScriptObject) SympyBoolean)Argument)CUTLASSTemplate)PythonWrapperCodegen) GraphLowering)IndentedBufferrrqTF_P_T_U_V_IntLike_NumLike _OpOverloadsz prefix TensorBoxr!IRNode _NodeOrNodesxobjectreturnTypeIs[Union[int, Integer]]cCst|ttfSN) isinstanceintr rr\/var/www/addictedbytheproject.nl/epg/venv/lib/python3.10/site-packages/torch/_inductor/ir.py _is_staticr)frozenc@s>eZdZUded<ded<ded<ded<d ed <d ed <d S)GraphPartitionSignatureOrderedSet[sympy.Symbol] symbol_inputsz5dict[str, Union[IRNode, sympy.Expr, TorchBindObject]] input_nodes list[IRNode] output_nodeszdict[str, bool]input_deallocationboolskip_cudagraph list[str]constant_namesN__name__ __module__ __qualname____annotations__rrrrrs  r node_or_nodesOptional[_NodeOrNodes]Nonecsdfdd |dS)Nnodesrrrc s|durdSt|ttfr|D]}|qdSt|tr*|D]}|q!dSt|ttttt j j j t tttf sEJdt|ddS)NzFound zE, which is not a supported top level IR node. See [Note: Inductor IR])rlisttupledictvalues ExpandView DynamicScalar AssertScalarr~sympylogicboolalgBooleanrrEffectfulKernelShapeAsConstantBuffertype)rnode_check_tensorboxrrrs2    z%validate_ir.._check_tensorbox)rrrrr)rrrr validate_irs rnamestrCallable[..., OpsValue]cs(tts Jtdfdd }|S) Nargsrkwargsrrlcstt|i|Sr)getattrrkrrrrrfn zops_wrapper..fn)rrrrrrl)rrr)rrrrr ops_wrapper srorder Sequence[int]&Callable[[Sequence[_T]], Sequence[_T]]cs(tt|tt|dfdd }|S)Nindex Sequence[_T]rc0ttks JfddttDS)Ncg|]}|qSrr.0i)r inv_orderrr z4inverse_reorder..reindex..lenrangerrrrreindexz inverse_reorder..reindexrrrr)rziprrrrrrrinverse_reordersrcsdfdd }|S)Nrrrcr)Ncrrrr)rrrrr rz1same_reorder..reindex..rrrrrrrzsame_reorder..reindexrrrrrr same_reordersrreindex1&Callable[[Sequence[_U]], Sequence[_V]]reindex2&Callable[[Sequence[_T]], Sequence[_U]]&Callable[[Sequence[_T]], Sequence[_V]]csdfdd }|S)Nrrr Sequence[_V]c |Srrrrrrrr) z fuse_reindexing..reindex)rrrrr)rrrrrrfuse_reindexing%sr)rr@)rrrr@seq(Sequence[Union[int, torch.SymInt, Expr]] shape_envOptional[ShapeEnv]cCs4|dus tdd|Drt|}|St||}|S)z1 Convert strides to fill order (argsort) Ncs |] }t|ttjfVqdSrrrrr rsrrr 9z!get_fill_order..)allrUrV)rr sorted_idxrrrget_fill_order3s  rSequence[Union[int, Integer]]cs0ddt|Dfddtt|D}|S)z Convert stride order to fill order For channel last format, stride order = [3, 0, 2, 1] and fill order = [1, 3, 2, 0] cSi|]\}}||qSrrridxposrrr Hz+stride_order2fill_order..cg|]}|qSrrrlookuprrrIz+stride_order2fill_order..) enumeraterr)r fill_orderrrrstride_order2fill_orderAsrcCs>t||}ddtt|D}t|D]\}}|||<q|S)z) Convert strides to stride order cSg|]}dqSrrr_rrrrTz$get_stride_order..)rrrr)rrroutrelemrrrget_stride_orderMs  r guard_shapercCdSrrrr rrrir_node_to_tensorZr  torch.TensorcCr rrr rrrr ^r Optional[IRNode]Optional[torch.Tensor]cs|durdS|stjjjntfdd|D}t|r,fdd|jD}nt |}| }| }t |}t |}tjjjtj||||d}Wd|S1s_wY|S)Ncg|]}|qSrrrshape_fnrrrmrz%ir_node_to_tensor..crrrrrrrrpr)sizestridedtypedevice)rmgraphsizevars size_hintr"get_sizeis_storage_and_layout get_layoutrFlexibleLayoutcontiguous_strides get_dtype get_devicer[rsuppress_guardstorch empty_stridedzero_)rr rrrrtrrrr bs.   valueOptional[Sequence[_T]] Optional[Sequence[Optional[_T]]]cCst|tr |s dgS|Sr)rrr'rrrmay_convert_to_optionalsr+2Union[IRNode, OutputSpec, torch.device, None, str] Optional[str]cCsbt|ts |dur |St|tjr|jSt|ttfr!t|St d|dt|j ddS)Nzget_device_type(: )) rrr#rrr OutputSpecget_device_typer!rrrrrrr1s   r1&Union[IRNode, torch.device, None, str]cCstt|}|dvrtt|ddkrdSdS|dus"t|}dur$dSddlm}t|ts5Jt|t||S)N)cpucuda_backendtritonTFr@)TritonScheduling) r1rrArEcodegen.tritonr7rr issubclass)rrdevice_schedulingr7rrr is_tritons  r;cCs t|dkS)Nr3)r1rrrris_cpurr<Union[Buffer, TensorBox] alignmentrcst|tr|dust|st|rdStjfdd|ddD}tt |ddt |dd}t||}t j j |S)NFc3s"|] }tt|dVqdS)rN)rEqr=rr>rrrs z-is_aligned_realized_tensor..r@)rrmaybe_get_strider1 get_striderrAndOrr?Lermrrguard_or_false)rr>aligned_stridesaligned_last_dim is_alignedrr@ris_aligned_realized_tensors    $ rKstrides1Sequence[_IntLike]strides2shapecCst|t|krt|t|ksJt|||D]'\}}}tjj|dr&qtjj||s?tjj|tjj|kr?dSqdS)zP Returns true if the strides are equal, ignoring dimensions of size 1 . r@FT)rrrmrrstatically_known_leqstatically_known_equals symbolic_hint)rLrNrOdims1s2rrrsignificant_strides_equals$rVtensorstrides"Sequence[Union[int, torch.SymInt]]cCst|s|Stddt||Dr|St|||s"|St|\}}g|j}t|D]\}}t j j |drE||||<q3t |j|j|j||j|j}tt||dS)a Tries to match the strides of the tensor to those in the meta_strides. Strides of insignificant dimensions - size 0 or 1 - will be updated. If there are real stride differences (NHWC vs NCHW), or the tensor is not realized, then the input will be returned cs$|] \}}tjj||VqdSrrmrrrQrrTrUrrrr  z2try_match_insignificant_strides..r@datalayout)rrrrCrVras_storage_and_layoutrrrmrrrP FixedLayoutrrroffset is_pinnedr~ReinterpretView)rWrXstorage old_layout new_striderr new_layoutrrrtry_match_insignificant_stridess.     rjgmtorch.fx.GraphModulecCsD|jjddd}ddt|jD|jd<ddlm}||dS)NoutputoprcSsg|]\}}|qSrr)rrrrrrrsz.gm_original_output_strides..user_visible_output_idxs)record_original_output_strides)r find_nodesrrmetatorch._inductor.compile_fxrq)rk output_noderqrrrgm_original_output_stridess   rvinputsSequence[IRNode] list[Expr]cCs@t}|D]}|t|ddO}|t|ddO}qt|S)NF unbacked_only)r9r'rrCr)rwsym_varsinprrrget_symbolic_inputs s r~cCsHt|tr|j}t|tr|}t|tr|j}t|tr"|SdSr)rr~r_BaseView unwrap_view StorageBoxBufferget_namerrrr try_get_names   rc@seZdZUdZeZded<ejddZ ded<ejddZ ded <ejddZ d ed <ejddZ d ed <e ejdddZe dddZdddZdddZdd!d"Zdd#d$Zdd%d&Zdd(d)Zdd*d+Zddd/d0Z ,ddd4d5Zdd7d8Zdd:d;Zdd=d>Zdd@dAZddCdDZddFdGZ ddHdIZ!ddKdLZ"ddNdOZ#e$ddQdRZ%ddTdUZ&ddVdWZ'ddYdZZ(ddd^d_Z)ddadbZ*ddddeZ+ddfdgZ,ddidjZ-ddldmZ.ddodpZ/ddqdrZ0ddsdtZ1ddudvZ2ddwdxZ3ddd{d|Z4dddZ5dddZ6dddZ7dddZ8 ddddZ9dddZ:dddZ; ddddZdddZ?dddZ@ ddddZAdddZBdddZCdddZDdddZEdddZFdddZGdddZHdddZIeJrse$dddZKd[Sd[S)rzBase class for all intermediate representation (IR) nodes in TorchInductor. Note: This is an abstract base class. Most methods raise NotImplementedError and must be overridden by concrete subclasses. zClassVar[OrderedSet[Any]]_current_originsF)initOrderedSet[Any]originsOptional[list[str]] tracebackOptional[torch.fx.Node] origin_nodedict[str, Any]rOrderedSet[Node]rGenerator[None, None, None]ccs.tj}||Bt_z dVW|t_dS|t_wr)rr)roldrrrcurrent_origins.s  zIRNode.current_originsrrcCst|tttttfSr)rComputedBuffer InputsKernel InputBufferreTemplateBufferrrrris_realized_node8szIRNode.is_realized_nodeattrrr'r rcCst|||dSr)r __setattr__)selfrr'rrr_post_init_setattrEszIRNode._post_init_setattrcCsLt|j}|d||dtjrtnd|dd|didS)Nrrrr)r9rrrAdebug_ir_tracebackr format_stack)rrrrr __post_init__Ks   zIRNode.__post_init__OrderedSet[str]cCtdd|DS)Ncs|]}|jVqdSrrrdeprrrrVz(IRNode.get_read_names..r9 get_readsrrrrget_read_namesUzIRNode.get_read_namescC|jSr)rrrrr get_tracebackXzIRNode.get_tracebackcCrrrrrrrget_origin_node[rzIRNode.get_origin_nodeOptional[Operation]cCr rrrrrrget_defining_op^r zIRNode.get_defining_opcCst}|j}t|tr|}|jrt|g}|D]:}t|dr*|jr*||jqt j j j di |jg}t|ts>q|D]}t j j j |d}|rR||q@q|S)N stack_trace postToPre)r9rr ExternKernelrrhasattrraddr# _inductordebug _inductor_post_to_pre_grad_nodesgetrr#_inductor_pre_grad_node_stack_trace)r stack_tracesrrrpre_grad_nodes node_namerrrrget_stack_tracesas6       zIRNode.get_stack_tracesTshorten Sequence[str]cCsdt|dd}|rt|dkr|ddd}|s!|gSg}|D]}|d||d7}|d q'|g|S) Nzorigins=r@=z...zstack_traces = { })rrrappendsplit)rrrstack_trace_strrrrr common_reprs    zIRNode.common_reprlinesSequence[object] multilinecCsbt|t||}ttt|}|r&td|}t|jd|dSt|jd|dS)Nz, z( z )(r/)rrmaprindentjoinrr)rrrr new_linesrrr str_helpers zIRNode.str_helper torch.dtypecCrrrrrrrr rzIRNode.get_dtypeOptional[torch.dtype]cC z|WStyYdSwr)r NotImplementedErrorrrrrmaybe_get_dtype   zIRNode.maybe_get_dtypeLayoutcCtdt|d)Nz#get_layout() is not implemented by !rrrrrrrrzIRNode.get_layoutOptional[Layout]cCrr)rrrrrrmaybe_get_layoutrzIRNode.maybe_get_layoutr0cC|Sr)rrrrrget_output_speczIRNode.get_output_specOptional[OutputSpec]cCrr)rrrrrrmaybe_get_output_specrzIRNode.maybe_get_output_speccCst|tS)z4True for single tensor output (excludes MultiOutput))rrrrrrrhas_tensor_outputszIRNode.has_tensor_outputSequence[Expr]cCr)Nz!get_size() is not implemented by rrrrrrrrzIRNode.get_sizeOptional[Sequence[_IntLike]]cCrr)rrrrrrmaybe_get_sizerzIRNode.maybe_get_size.Union[_IntLike, sympy.Rel, Sequence[_IntLike]]cCrrrrrrrrOz IRNode.shapercC t|Sr)rhrrrrr get_numelrzIRNode.get_numelcCtjjt|dSNrrmrrstatically_known_truerr?rrrrris_zero_elementszIRNode.is_zero_elementsr-cCtdt|)a) If the IRNode refers to data which has not been materialized (e.g., it is a Pointwise/Reduction that could potentially have more compute fused into it), realize the IRNode into physical memory, ending the possibility of fusing into it, but allowing, e.g., multiple users to access the data without having to recompute. Check StorageBox.realize for a particularly notable implementation. TODO(ezyang): I think, in principle, every IRNode should have an implementation of this, and most of the time no-op is OK, but you really do have to audit each IRNode for this, so for now, raise an error if it's not implemented. Note that some code in graph.py will catch this thrown error and suppress it with a warning. zrealize NYI on rrrrrrealizeszIRNode.realizeNwriterOptional[IndentedBuffer]cCr)Nzcodegen_reference NYI on rrrrrrcodegen_referencezIRNode.codegen_referenceOptional[torch.device]cCr rrrrrrr!r zIRNode.get_device torch.devicecC|}|dus J|Srr!rrrrrget_device_or_error zIRNode.get_device_or_errorcCdSNFrrrrrhas_exceeded_max_readsr zIRNode.has_exceeded_max_reads$Callable[[Sequence[Expr]], OpsValue]cCtt|jrrrrrrrr make_loaderrzIRNode.make_loader Callable[[Sequence[Expr]], Expr]cCrrrrrrr make_indexerrzIRNode.make_indexerrMcCrrrrrrrrCrzIRNode.get_stridecCrr)rCrrrrrrBrzIRNode.maybe_get_stridecCrrrrrrrrrzIRNode.get_namecCrr)rrrrrrmaybe_get_namerzIRNode.maybe_get_namecCs(z |tjjvWStyYdSwr)rrmr graph_inputsrrrrris_input_buffers  zIRNode.is_input_buffer threshold Optional[int]cCrrrrrrrrhas_large_inner_fn r zIRNode.has_large_inner_fnusersrcCr rrrrrrr mark_reuser zIRNode.mark_reusecCr rrrrrr realize_hintr zIRNode.realize_hintcCrrrrrrrrrzIRNode.unwrap_viewcCrrrrrrr freeze_layoutrzIRNode.freeze_layoutrr allow_paddingcCrrrrrrrrrfreeze_layout_with_stride_orderz&IRNode.freeze_layout_with_stride_ordercCrrrrrrrrfreeze_layout_with_fill_order!rz$IRNode.freeze_layout_with_fill_orderrcCrrrrrrrrfreeze_layout_with_same_order$rz$IRNode.freeze_layout_with_same_order exact_stridescCrrrrrrrrr freeze_layout_with_exact_strides'rz'IRNode.freeze_layout_with_exact_stridesdependencies.ReadWritescCrrrrrrrget_read_writes,rzIRNode.get_read_writesOrderedSet[Dep]cC |jSrr#readsrrrrr/ zIRNode.get_readscCrr)rrrrrr num_reads2rzIRNode.num_readsrycCrrrrrrrget_storage_numel5rzIRNode.get_storage_numelr{rcCrrrrr{rrrget_free_symbol_uses8rzIRNode.get_free_symbol_usescCrrrrrrrget_reduction_type=rzIRNode.get_reduction_typecCrrrrrrrget_reduction_size@rzIRNode.get_reduction_sizecCrrrrrrr is_externCr zIRNode.is_externcCrrrrrrris_no_opFr zIRNode.is_no_oprcCrrrrrrrconstant_to_deviceIrzIRNode.constant_to_devicecCrrrrrrrget_mutation_namesLrzIRNode.get_mutation_namescCrrrrrrrget_operation_nameOrzIRNode.get_operation_namecCrrrrrrrget_inputs_that_alias_outputRrz#IRNode.get_inputs_that_alias_outputcCr rrrrrrrWr z IRNode.dtype)rrrrrrrr)rrr'r rrrrrr)rrrrrrT)rrrr)TT)rrrrrrrrrr)rrrr)rrrr0)rrrrrr)rr)rrrrrr-rrrrrrrrrrrrr rrMrrrrrrrrrrrrFrrrrrrrrrrrrMrrrrMrrrrrr"rr$rrrryr{rrrrrrrrr)Lrrr__doc__r9rr dataclassesfieldrrrr staticmethod contextlibcontextmanagerrrrrrrrrrrrr rrrrrrrrpropertyrOrrrrr!rrrr rCrBrr r rrrrrrrrr!r#rr)r*r,r-r.r/r0r1r2r3r4rrrrrrrs        !                                          c@seZdZd0ddZd1ddZd2d d Zd3d d Zd4ddZd5ddZd5ddZ d6ddZ d7ddZ d8ddZ d9d d!Z d:d#d$Zd;d&d'Z (dd-d.Zd/S)? OperationrrcCs d|_dSroperation_namerrrrr]r(zOperation.__post_init__rcCtrrrrrrr!`r zOperation.get_devicercCt|dsJ|jSNr)rrrrrrrczOperation.get_origin_nodercCrd)Nr)rrrrrr get_originsgrfzOperation.get_originsrcCs|jdusJ|jSrr`rrrrr3krfzOperation.get_operation_namercCrrrrrrrr/or zOperation.is_externcCrrrrrrrr0rr zOperation.is_no_opr"cCrbrrcrrrrr#ur zOperation.get_read_writesrcCs ||vSr)rrrrrr is_user_ofxrzOperation.is_user_ofrcCr)Ncsrrrrrrrr|rz+Operation.get_read_names..rrrrrr{rzOperation.get_read_namesr$cCr%rr&rrrrr~r(zOperation.get_reads list[Buffer]cCrbrrcrrrr get_outputsr zOperation.get_outputsrcCtSrr8rrrrget_unbacked_symbol_defsrz"Operation.get_unbacked_symbol_defsFr{cCrl)a When unbacked_only=True: Returns the unbacked symbols which are required to be in scope in order to successfully perform codegen for this buffer. For example, a buffer that corresponds to an extern kernel call that takes i0 as an argument would return {i0} here. This is used to generate necessary dependencies that ensure we actually bind i0 in codegen before you try to use it. Note that this is NOT transitive; in particular, if this buffer takes in as input another buffer with dynamic shape (e.g., (i0,)), we will not report it here, because you will already have a dependency on that buffer, which will eventually have a dependency on i0 if necessary. When unbacked_only=False: Similar to `unbacked_only=True` but including all free symbols instead of only free unbacked symbols. r8r+rrrr,szOperation.get_free_symbol_usesrcCr)z Gets extra global memory size needed by this buffer. Some algorithms (e.g. group gemm) may require extra global memory in the generated code. rrrrrrget_workspace_sizeszOperation.get_workspace_sizeNr6rCr8)rrrHr>rQ)rrrrr7rRrrjrrrLrUrS)rrrrr!rrgr3r/r0r#rirrrkrmr,rnrrrrr_[s"              r_c@s eZdZUded<ded<ded<ded<ed dMdNddZdOddZdPddZeZdQddZ dRddZ dSddZ dSd d!Z e dTd&d'ZeejfdUd*d+ZedVd-d.ZdWd0d1ZedPd2d3ZdXdYd7d8ZdMdZd:d;Zd[d=d>Zd\d@dAZd]dCdDZdSdEdFZd^dHdIZd_dKdLZd4S)`LoopsrrrrCallable[..., Any]inner_fnrMrangesFr{rrrcs,tjgfdd|jD|RS)Nc3|]}t|VqdSrr&rerzrrrz-Loops.get_free_symbol_uses..)r9unionrtinner_fn_free_symbolsr+rrzrr,s zLoops.get_free_symbol_usesnamesrrcsFdjjdtjgfdd|DdjgS)N'cs g|] }|dt|qS=)rrrrrrrs z!Loops._to_str.. origin_node=)rrrrr inner_fn_strr)rr{rrr_to_strs z Loops._to_strcC |dSNrtrrrrr__str__r(z Loops.__str__rcCrrrrrrrr!rzLoops.get_devicercCrrrrrrrrrzLoops.get_origin_nodercCrrrrrrrrrzLoops.get_sizecCrrrrrrrget_pointwise_sizerzLoops.get_pointwise_sizerr rr~cOsN|dd}|dd}||i|}|d||d|p |jt|S)Nrr)poprrr~create)clsrrrtbrrrrrs    z Loops.creater}r?csfddt|DS)Ncs*g|]\}}|dkrtjjnt|qSr@)rSZerorg)rnrr|rrrz Loops._index..)r)rtr}rr|r_indexs z Loops._indexrOc Cstt}t|2ttdd|j|| WdWdS1s0wYWddS1s@wYdSNallow_indexingT) rNrm MockHandlerset_ops_handlerrrrrs inner_fn_argsgetvalue)r opcounterrrrinner_fn_opcounts  RzLoops.inner_fn_opcountSequence[Sequence[_IntLike]]cCs||jfSr)rrtrrrrrrzLoops.inner_fn_argscCstjj|jg|RSr)rmKernelFormatterHandler ir_to_stringrsrrrrrrs zLoops.inner_fn_strNrrcCs&|durd}t|tj}|j|kSr)maxrArealize_opcount_thresholdrnum_opsrrrrrs zLoops.has_large_inner_fnOrderedSet[Symbol]cCs||j}t|j||dSNrz)rrtrIrs)rr{rrrrrzs zLoops.inner_fn_free_symbolsr$cCsvttdd*|r t|||jWdSt||jWdS1s4wYdSr) rrrr-rKrrr.r'rrrrrs$zLoops.get_readsrcCt|jSr)r9r read_buffersrrrrrrzLoops.get_read_namesrcCrr)rrrrrrrr)rzLoops.num_readscCr)Nz+get_reduction_size() is not implemented by rrrrrrr.zLoops.get_reduction_sizer-cCr)Nz+get_reduction_type() is not implemented by rrrrrrr-rzLoops.get_reduction_typercCr)Nz+constant_to_device() is not implemented by rrrrrrr1!rzLoops.constant_to_devicerLrU)r{rrrrHrCr8r?)rr rr rr~)rtrMr}r?rr)rrOrrrrIr{rrrrRr7rSrArV)rrrrrXr,rr__repr__r!rrr classmethodrr[r?INDEXrrWrrrrrzrrr)r.r-r1rrrrrqs@                   rqrUnion[Expr, Sequence[Expr]]rrrlcCs"|jr ttd|Std|S)Nnanr)is_floating_pointrkconstantfloat)rrrrr nop_loader_fn's rc@sLeZdZdddZdddZeZdd d Zdd d Zd ddZd!ddZ dS)" PointwiserrcCs|r tt|jdS|jSNr)rr rrrsrrrrr0szPointwise.make_loaderrcCrrrrrrrr7r(zPointwise.__str__Sequence[sympy.Expr]cCsgSrrrrrrr.<r zPointwise.get_reduction_sizer-cCr rrrrrrr-?r zPointwise.get_reduction_type output_nameindexer!Callable[[Sequence[Expr]], Never]varsrrcCs"|}t|p d||||SNunnamed)rrkstorerrrrloaderrrr store_outputBszPointwise.store_outputrrrcCs.|}ttd||}t||j||jdSFMove this to a given device. Requires that all reads are to constants.override_devicerrrsrt)rrrConstantBufferrrrtrrrrrrr1KszPointwise.constant_to_deviceNrErH)rrrA)rr-rrrrrrrV) rrrrrrr.r-rr1rrrrr.s      rc@s6eZdZUded<dZded<dd d ZdddZdS)Scatterr output_indexerNrQ scatter_moderrrrcCs6|}ttd||}t||j||j|j|jdS)rr)rrrsrtrr) rrrrrrrtrrrrrrr1\szScatter.constant_to_devicerr-rrrrr cCs6|}|dur d}tj|||||||jdS)Nr)mode)rrkrrrrrrrris zScatter.store_outputrV)rr-rrrrrr )rrrrrr1rrrrrrWs    r logical_ormaximumminimummulr bitwise_xor)anyrminprodsumdotxor_sumz"dict[str, Callable[..., OpsValue]]REDUCTION_COMBINE_FNreduction_typearg_break_ties_leftCallable[..., object]csRtvrtSdvrdfdd }|Sd kr"dd d }|Std )Nargmaxargminatuple[object, object]brtuple[OpsValue, OpsValue]c s|\}}|\}}dkrt||}nt||}t||}trCt||}t||} t|t|| }t|t|| }rKt||nt||} t|t|| }t|||t|||fS)Nr) rkltgteqr+ner logical_andwhere) rra_valuea_indexb_valueb_indexmaskequala_isnanb_isnantierrrrrargmax_combine_fns&       z3get_reduction_combine_fn..argmax_combine_fnwelford_combine#tuple[OpsValue, OpsValue, OpsValue]c SsR|\}}}|\}}}||}||} || } ||| |||||| | fSrr) rra_meana_m2a_weightb_meanb_m2b_weightdelta new_weight w2_over_wrrrwelford_combine_fns   z4get_reduction_combine_fn..welford_combine_fnzunknown reduction_type=)rrrrrr)rrrrrr)rr)rrrrrrrrget_reduction_combine_fns rcs\eZdZUded<ded<ded<ded<dkd d ZeZeddldmfdd ZdnddZdoddZ dpd d!Z dqd#d$Z drd&d'Z dldmd(d)Z dsd-d.Ze /dtdud:d;Zedvd>d?Zeejd/fdwdBdCZedxdFdGZedxdHdIZedydMdNZedzdPdQZe /dtd{dWdXZed|d`daZed}dedfZe /dtd~dgdhZeddidjZZS) ReductionrMreduction_rangesrPrr src_dtyperTreduction_hintrrcCr)N)rtrrrrrrrrr(zReduction.__str__Fr{rrcs(ttjfdd|jDBS)Nc3rurr&rvrzrrrrxz1Reduction.get_free_symbol_uses..)superr,r9ryrr+ __class__rzrr,szReduction.get_free_symbol_usesrcCrr)rrrrrr.rzReduction.get_reduction_sizer-cCrr)rrrrrr-rzReduction.get_reduction_typerrrrreduction_varsSequence[Symbol]rc Cs8t|j|j|j|||}t|pd|||dSr)rk reductionrrrrsstore_reduction)rrrrrr'rrrrs zReduction.store_reductionrcCt|jt|jSr)rrtrrrrr index_lengthrzReduction.index_lengthSequence[Sequence[Expr]]cCs$||j}||jtj}||fSr)rrtrr?R0_INDEX)rrrindexrrrrs zReduction.inner_fn_argscCs.||j}||jtj}t|j|||dSr)rrtrr?r rIrs)rr{rr rrrrzs  zReduction.inner_fn_free_symbolsrrrc Cs>|}ttd||}t||j||j|j|j|j t j dS)rrrrrsrtrrrr) rrrrrrrtrrrrTDEFAULTrrrrr1szReduction.constant_to_deviceN dst_dtypersCallable[_P, OpsValue]rt%Union[ReductionType, Literal['scan']]reduction_numelr input_nodertuple[ReductionHint, _IntLike]c " Cstjj|} tjjt|} |dkp#tj|tj o#|dvo#tj } t | r,t | s1t j dfS|dkr:t j dfSt |} | j} d}| r[tjtjj|dd}tjtjj|dd}ndd d}|}| dkr|| | }|dkrtt j|fS|durt|trttddt|\}}Wdn1swY|dur|durtjjt||}| |krtd|||||t jdfSt j|fS| |ks| | ddkrt j dfSt||||||dkr|nd|t j d}ddd}||\}}|r||\}}t|dkrt j dfSt |!|"\\}}}d}d}|D].}tjj#||}tjj$||t%|&} t'dd| D}!|!rE|d7}q|d7}q||krXt j|| | fSt j(|| | fS)Nscanrr@r T)inner_reductionFreduction_numel_hintr numel_hintrcSrNr@r)rrrrrinner_reduction_splits7z4Reduction.num_splits..inner_reduction_splitsrzUse previous IRNode's range and reduction_ranges instead of split. current ranges: %s, current reduction ranges: %s, current split: %d, new ranges: %s, new reduction ranges: %srArrrrrtuple[Sequence[Expr], bool]c s|}|dus Jtdt|||d|d}|}|jdus%Jdd|jD}g}d}t|jdddD]7t fd d |Drq| j j t jjvrqt jjj }t|jd d}|t|jd d|krqd }q:||fS) Nrrrrr`r_cSs&g|]}t|trt|tjs|qSr)rrrNumberrrrrrrs zBReduction.num_splits..get_read_indices..FcSrrrrrrrz@Reduction.num_splits..get_read_indices..keyc3s|] }|jjvVqdSr)rr0r"mdrrrzAReduction.num_splits..get_read_indices..rT)r!rrr rr# range_varssortedr'rrrrrmrname_to_bufferrr` decide_layout) rrcb read_writesr*indiceschangedbuforiginal_striderr'rget_read_indicesos:   z.Reduction.num_splits..get_read_indicesrcss |] }|dkp |dkVqdS)rr@Nrrrrrrrz'Reduction.num_splits..)rrrrrr)rrrr))rmrrrRrh has_featurerCREDUCE_TO_SINGLE_ELEMENTrAsplit_reductionsrrTrrSrmulti_processor_count functoolsr choicesreduction_split_factorINNERrr~rrrrJlogrrrrBindex_vars_squeezerr.simplify_with_ranges stride_hintsrkeysrOUTER)"rrrrsrtrrrrrr should_splitpropsnum_smmin_elements_per_threadrouter_reduction_splitsr new_rangesnew_reduction_rangesextracted_numel_hintrr4r0r1rrranges1 num_outer num_innerrjrXouterrrr num_splits s             #       zReduction.num_splits.fn..cSsg|]}t|qSr)rrrrrrrrz>Reduction._unroll_reduction_fn..fn..)r9reduce itertoolsproductr) combine_fnrrSrrrs  z*Reduction._unroll_reduction_fn..fnrrr rcs*dd|D}||t|tjfS)NcSg|]}t|qSrrexpandrrrrrrzDReduction._unroll_reduction_fn..value_fn..)rk index_exprr#int64rr ) flatten_indexrsrrrSsz0Reduction._unroll_reduction_fn..value_fncs |dSrrr)rrrr#s z0Reduction._unroll_reduction_fn..N)rrMrr )rrMr rMrr)rmrr guard_int_seqr_fixed_indexerrr)rsrrrr)rXr`rrsrrSr_unroll_reduction_fns   zReduction._unroll_reduction_fnrrr~c  stjjtdkrBd#fdd } | d| d| d| dd vs.Jd d$fdd } tj||| t|dSdkrcdvrRd$fdd } nd$fdd } tj|| |dStt rtjj t j krt|dks}t |jrdkrtj||||dS||||| \} }d%fdd }||}|tjkr| }|dkr| dusJttddt| \}}Wdn1swY|dusJ|dusJ||||||| S|dkr?||||||| }d}t jjrt|trd&d d!}||}|r=t|jts-Jt|j|jjd|_|_ ||_!|_"|Stt||||d"}|S)'zy Create a reduction node. May split the reduction to multiple layers to expose more parallelism. rvalrrUnion[bool, float, int]csTtjkr t|Sjrt|tsJt|t|St|ts&Jt|t|Sr) r#rrrrrrrrrdrrrpy_cnsts z!Reduction.create..py_cnstr@)rrrrz* not supported for zero-dimension tensors!rrrlcstSrrkrr)rrrtypes_to_initsrrconst_fnz"Reduction.create..const_fnrrYcs tdSrrirrgrrrrzReduction.create..fncddD}||S)NcSg|]}tjjqSrrrrrrrrr"rz0Reduction.create..fn..rrreduction_index)rsrrrr! rrcs$tr|S|dkrt|tjS|Sr)rrrA min_num_split)r)rrr_maybe_increase_splitJs  z/Reduction.create.._maybe_increase_splitrANrTcur_noder~Optional[ComputedBuffer]cSsd|}t|dkr dStt|}|tjjvrdStjj|}t|ts'dS|j dus0J|Sr) rrnextiterrmrr,rrr_r-)ru read_namesbufnamer2rrr_find_split_reductions     z/Reduction.create.._find_split_reductionr)rdrrre)rrrrl)rrrr)rur~rrv)#rmrrsimplifyrhrrrrr size_hint_or_throwrAunroll_reductions_thresholdrdrrcrPrTrrrrrJ!create_multilayer_existing_rangescreate_multilayerr6mix_order_reductionr~r_rr _split_size_original_inner_fn_original_ranges_original_reduction_ranges)rrrrrsrtrrrrrhrkrhintrrtrHrIrsplit_reductionr{r)rrsrrrrjrrs              zReduction.creater#Union[_NumLike, Sequence[_NumLike]]c Cs|dvrt|r tdSt|rdSt|jS|dvr0t|r$tdSt|r*dSt|jSt|r6dnd}t|r>dnd}||||||||f|||ftd|fd |S) N)rrz-infF)rrinfTrr@)rrrrrwelford_reduceronline_softmax_reduce)r+rr*r#iinforr)rrzeroonerrrdefault_accumulators2    zReduction.default_accumulatorcCs|dkrdSt||S)Nrr)rrrrrrr default_values zReduction.default_valuerryrcCsP|dkr|S|dkr|dkr|tjkrtjS|dkr&|dkr&|tjkr&tjS|S)NrAi)rTrB OUTER_TINY)rrrrrr_multilayer_second_step_hints z&Reduction._multilayer_second_step_hintrcCs|durdStjj||sdS|zt|Wn ty&YdSw|}t |ddD]\}}tjj|drC|Sq3dS)z If we are reducing over the full tensor, and it is non-dense in the last dimension, reindex so we reduce over the dense dimension. initially just handle complete reduction case NrAr@) rmrrrQrrrarrCr)rrrrXrrrrr$check_for_split_dense_dim_reindexings$   z.Reduction.check_for_split_dense_dim_reindexingrr block_sizedefaultrc sT||}t|g|tjjt|d d fdd } | S) Nrrrrqrrlcsl|\}|^}||dfdd }r3t}tt|t|}t||S|S)NrrlcsgSrrr)r0r new_indexrrrbody(rlzCReduction._multilayer_wrap_loader..wrapper_fn..body)rrl)r^rkrr]masked)rrqreduction_blockr index_dtyperrrr need_maskrr)r0rr wrapper_fn!s    z5Reduction._multilayer_wrap_loader..wrapper_fn)rrrqrrrl) rViewdynamic_reshape_indexerrmrrrrr?) rrrrrrrr dense_indexrrrr_multilayer_wrap_loader s z!Reduction._multilayer_wrap_loader4Callable[[Sequence[Expr], Sequence[Expr]], OpsValue]original_rangesoriginal_reduction_rangesrHSequence[Integer]rI@Callable[[Sequence[sympy.Expr], Sequence[sympy.Expr]], OpsValue]csNtddDsJdt|t|t|d fd d }|S) Ncss|]}|dkVqdS)r@Nrr"rrrr@zDReduction._multilayer_wrap_loader_existing_ranges..z8Only enabled for numel_hint == 1, found original_ranges= merged_indexrnew_reduction_indexrrlcs:|dt}|td}|t|t|Sr)rr)rr original_idxrrrrrrrGs zEReduction._multilayer_wrap_loader_existing_ranges..wrapper_fn)rrrrrrl)rrrr)rrrrrHrIrrrr'_multilayer_wrap_loader_existing_ranges7s  z1Reduction._multilayer_wrap_loader_existing_rangesrry list[Integer]c  s|tjtjfvr |ntj} t|| ||||| | } | | d fdd }tj j t |}| | || } ||dt|ksDJtt|||||t|d| || d S) a Break a large reduction up into multiple smaller reductions recursively rrMrqrrlcsg||Srrrpintermediate_loaderrrintermediate_fn|sz;Reduction.create_multilayer_helper..intermediate_fnNr)rrMrqrMrrl)r#float16bfloat16rrrrrrmrroptimization_hintrhrrr~)rrrrrrrrHrIrrrintermediate_dtype intermediaterrrrrcreate_multilayer_helperTsD z"Reduction.create_multilayer_helperc  Csdt|} t| |d|} |||} |||| || | | }|||||||g||| g||| S)rr@)rhr<rrr)rrrrrsrtrrrrrrrrrrrrrs2   zReduction.create_multilayerc Cs8||||||} ||||| ||g|||| d| S)rrA)rr) rrrrrsrrrHrIrrrrrrrs( z+Reduction.create_multilayer_existing_rangesrHrLrr?rA) rr-rrrrrrrrrSrr rVr)rrrrrrrsrrtrMrrMrrrrrrrr) rsrQrrMrrrrrrR)rrrrrrrsrrrtrrrrrPrrTrrrr~rrrrrr)rryrrrrTrrT)rryrrrr)rrrrMrryrryrryrrrrrr) rrrrrrrHrrIrrr)rrrrrrrrrrrrrrHryrIrrrPrryrrTrr~)rrrrrrrsrrrtrrrrrPrryrrTrrrr~)rrrrrrrsrrrrrrrHrrIrrrPrrTrr~) rrrrrrrXr,r.r-rr rrzr1r[rPrcrrTrrrrrrrrrrr __classcell__rrrrrs^          * + R     *  ? -rrrOptional[Sequence[int]]rcrr csdfdd }|S) 1A closure containing math to read a given elementrrrrcsddur t|tksJt|tksJ}t|D]\}}}|dkr/|||}q |Sr)rr)rresultrstszrcrrrrrs z_fixed_indexer..indexerN)rrrrr)rrrcrrrrrbs rb INNER_FN_TYcs2eZdZUded<dfdd Zd ddZZS)!MultiOutputReductionr output_indexrrrr inner_fns)Union[INNER_FN_TY, Sequence[INNER_FN_TY]]rtrrrrPrrrTc sXtrftdkrd} nd fdd } tj||| |||||d | |_dS) Nr@rrr reduction_idxrtuple[OpsValue, ...]cstfddDS)Nc3s|]}|VqdSrrrrrrrrrrxz@MultiOutputReduction.__init__..loader..)rrrrrrsz-MultiOutputReduction.__init__..loaderr)rrrrrr)callablerr__init__r) rrrrrtrrrrrrrrrrs     zMultiOutputReduction.__init__rr-rrrrrrrr c CsXt|j|j|j|||}t|ttfsJt |||j }t |p&d|||Sr) rkrrrrrsrrrrrr)rrrrrrr'rrrr(s  z$MultiOutputReduction.store_reduction)rrrrrrrtrrrrrPrrrrTrr) rr-rrrrrrrr )rrrrrrrrrrrrs %rc@s"eZdZeejdfdddZdS)OnlineSoftmaxReductionNrrrrrrsrrrtrr num_outputrrrTrrrSequence[TensorBox]c s<tfddt|D} | D]} | q| S)z> Create the reduction disregarding splitting. c 3s.|]}ttd| VqdS)rN)r~rrr output_idxrrrsrtrrrrrrKs" z0OnlineSoftmaxReduction.create..)rrr) rrrrrsrtrrrrresultsr&rrrr;s  zOnlineSoftmaxReduction.create)rrrrrrrsrrrtrrrrrrrTrrrr)rrrrrTrrrrrrr:s  rc@s<eZdZeejfdddZedddZedddZ dS)WelfordReductionrrrrrSequence[Callable[..., Any]]rtrrrrPrrTrrc s6dvsJtjjt}dfdd } |dkr-| d} | d} | d} | | | fS|d krWdfd d dkrLd| d| d fStfddDStjd|d\} }tj kro| |d kr| |Sfddt dD}|D]}| q|S)N)rrrdrrr~cs&dfdd }tj|tdS) Nrrrrlcs tSrrir)rrdrrrsqsz8WelfordReduction.create..const..inner_fnrrrrrlrrr)rdrs)rrrtrfrconstpsz&WelfordReduction.create..constrr@rrcs&dfdd }tj|tdS) Nrrrrlcrm)NcSrnrrorrrrrrzKWelfordReduction.create..copy..inner_fn..r)rrq)rrrrrsrrz7WelfordReduction.create..copy..inner_fnrrr)rrs)rrrtrrrcopysz%WelfordReduction.create..copyrc3|]}|VqdSrrr)rrrrrz*WelfordReduction.create..)rrcs*g|]}tt| qSr)r~rrr)rrrrtrrrrrrs z+WelfordReduction.create..r)rdrrr~)rrrr~) rmrrr|rhrrrPrTrrrr)rrrrrtrrrrrmeanm2weightrrrr&r)rrrrrtrrrrrasT     zWelfordReduction.createrrcCr)N)rrrrrrrrrrzWelfordReduction.default_valuerryc  s$ttjjtd } | r9|dkr9dfd d } j||dt| dd t| d d f|d|dSt d t |t fdd|Dg|g||} | D]} | qadddtjjt|} | |}t |t fdd| D|gd|S)rrrrrrr'rrrlcs t|Srri)rrr'rrrrs z4WelfordReduction.create_multilayer..constantr*r@)rrrrtrrrrc 3s&|]}j|ddVqdS)r)rN)r)rr)rrrrrrrr s  z5WelfordReduction.create_multilayer..rrqrrcSs|g||Srr)rrqrrrrintermediate_loader_fn$ szBWelfordReduction.create_multilayer..intermediate_loader_fnc3s |] }t|dVqdS)rN)r rr)rrrr2  N)rrrrr'rrrl)rrrqrrrrrl)rhrmrrrrr?rr r<rrrrrr)rrrrrtrrrrrr intermediatesrrr)rrrrrrrrrsb        z"WelfordReduction.create_multilayerN)rrrrrrrtrrrrrPrrTrrr)rrrrrrrtrrrrrPrryrrTrr) rrrrrTrrr[rrrrrrr`s x rcseZdZUded<ded<ded<ded<ded <d ed <d ed <ded<eddIdJfdd ZdKfdd ZdLd#d$ZdMd%d&ZdNd'd(Z dNd)d*Z dNd+d,Z dOd-d.Z dPd0d1Z dIdJd2d3Zeejfd4d5dQd=d>ZedRdGdHZZS)SScanr scan_rangesr=Callable[[tuple[Any, ...], tuple[Any, ...]], tuple[Any, ...]]rXzFCallable[[Sequence[_IntLike], Sequence[_IntLike]], Sequence[_IntLike]]rrTrrrtuple[torch.dtype, ...]dtypestuple[Callable[..., Any], ...]rFr{rrrcDttjfdd|jDBtjfdd|jDBS)Nc3rurr&rvrzrrrT rxz,Scan.get_free_symbol_uses..c3rurr&rvrzrrrW rx)rr,r9ryrrr+rrzrr,L s zScan.get_free_symbol_usesrc0t|jt|jt|jksJtdSr)rrtrrrrrrrrr[ "zScan.__post_init__rr-r%Callable[[Sequence[_IntLike]], Never]rr scan_varsrr csR|||tfdd|jD}t|j|j|}t|p d|||jS)Nc3|]}|VqdSrrrrsrrrrg rz'Scan.store_reduction..r) rrrrkrrrXrr)rrrrrrrrrrr_ s zScan.store_reductioncCr)Ncustomrrrrrr-m zScan.get_reduction_typecCrr)rrrrrr.q rzScan.get_reduction_sizecCrrrrrrrrt rz Scan.get_sizecCrrrrrrrrw rzScan.get_pointwise_sizecCr r)rrtrrrrrr z rzScan.index_lengthrcC.||j}||jtj}|||}|fSr)rrtrr?r rrrr rrrrr}   zScan.inner_fn_argscC8||j}||jtj}|||}t|j||dSr)rrtrr?r rrIrsrr{rr rrrrrz   zScan.inner_fn_free_symbolsT)can_fallback_to_atenrr+tuple[Callable[[Sequence[Expr]], Any], ...]axisrrSequence[Optional[TensorBox]]c sg d dd g tjtjs$dgtStdkr9tjtjs9dgtStjj} | t } ttksNJ| t | drg fddt tDS|jdd | d\} t | dkrtjjduptotdkotdk} | s|rdgtSd} nt d fd d  f ddt tD}|D]}|q|S)Nr@c&g|]}tj||dqSrrrrrrrrrrrr zScan.create..r)rrrsrpointwise_rangesrrX scan_numelz3.3.0rr scan_indexrrycHt|tks Jt|tksJg|d||dSrr)rr)rrrrrr  zScan.create..reindexcsBg|]}t d|| |d qS)) rrrrsrrrtrrXrrrr)r~rr) rXrrrrrrrr scan_typerrrr s* )rrrrrry)rmrr5rCSCANrTUPLE_REDUCTIONrr|rhrrrFrrPrr#versionhip has_tritontriton_version SplitScanr)rrrrrrrXrrrrrrPsupports_splitrrr) rrXrrrrrrrrr rrr sV         z Scan.createrrrsrrrrrc s*d fdd } tj|||| ||d|d S) Nrrrrrlcs$g|d||dSrrrrrsrrr s$z#Scan.num_splits..wrapper_fnr)rrrrsrtrrr)rrrrrrl)rrP) rrrrsrrrrXrrrrrrP s zScan.num_splitsrLrr6) rr-rrrrrrrr rAr?rSr)rrrrrrrrrrrXrrrTrrrr rr)rrrrrsrrrrrrrrXrrrrr)rrrrrXr,rrr-r.rrr rrzrrTrrrPrrrrrr> s6           brc@s eZdZdS)rN)rrrrrrrr srcseZdZUded<ded<ded<ded<ded <d ed <d ed <ded<ded<edd;d<fdd Zd=fdd Zd>d"d#Zd?d$d%Zd@d&d'Z d@d(d)Z d@d*d+Z dAd,d-Z dBd/d0Z d;d.c3rurr&rvrzrrr' rx)rr,r9ryrrr+rrzrr, s zSort.get_free_symbol_usesrcrr)rrtrrrrrrrrr+ rzSort.__post_init__rr-rr rrrr csV|||tfdd|jD}t|j||j|j}t|p"d|||j S)Nc3rrrrrrrr7 rz'Sort.store_reduction..r) rrrrksortrrrrr)rrrrrrrrrrr/ s zSort.store_reductioncCr)Nrrrrrrr-= r zSort.get_reduction_typecCrr)rrrrrr.@ rzSort.get_reduction_sizecCrrrrrrrrC rz Sort.get_sizecCrrrrrrrrF rzSort.get_pointwise_sizecCr r)rrtrrrrrr I rzSort.index_lengthr cCrr)rrtrr?r rrrrrrL rzSort.inner_fn_argscCrr)rrtrr?r rrIrsrrrrrzR rzSort.inner_fn_free_symbolsrr'tuple[Callable[[list[Expr]], Any], ...]rrrc  s*g d dd g tjtjs$dgtStjj} | t } d} t j j o=| t | | } | sGdgtSttksQJ| t | drj fddttDSd  fd d  f d dttD}|D]}|q|S)Nr@rcrrrrrrrr} rzSort.create..rr sort_indexrrycr rr )rr)rrrrrr r zSort.create..reindexcsDg|]}ttd|| | d qS)) rrrrsrrrtrrrrrrr)r~rrr) rrrrrrrrrrrrrr s,)rrrrrry)rmrr5rCSORTrrr|rhrAr6persistent_reductionsrrrFrr)rrrrrrrrrrr sort_numel max_rblockis_persistent_kernelrrr) rrrrrrrrrrrrrrX s0     z Sort.createrLrr6) rr-rr rrrrrr rAr?rSr)rrrrrrrrrrrrrrrrTrr rr)rrrrrXr,rrr-r.rrr rrzrrTrrrrrrrr s0          rcCs(z t|ddWdStyYdSw)NFfreezeT)rarrrrrr s   rcCs@zt|dd\}}|r||WStyYdSwNFr")rashould_pad_strides pad_strides is_contiguousr)r_bufferr`rrr is_contiguous_storage_and_layout s  r)r#want_contiguous stride_order'Optional[Sequence[Union[int, Integer]]]rrtuple[StorageBox, Layout]c Cst|trt|j|||||dSt|tr)t|j|||||d\}}||jfSt|trc|r[|r?||s>Jn|durK|j ||dn|durW|j ||dn| t||fSt|t rvt|j|d\}}||j fSt)z Try to simplify x into a StorageBox and a Layout. allow_padding only affect how we apply stride_order. When allow_padding is True, we have the freedom to add padding when applying the stride_order. r#r*r+rrNrr")rr~rar_rrrrr'rr!r-rer`r) rr#r*r+rrrr`bufferrrrra sR      racCs2zt|dd\}}||WStyYdSwr$)rais_stride_orderedr)rr+r(r`rrr"is_stride_order_storage_and_layout s   r2rcCsrt|ttfr t|jSt|tr*|j}tjj |j t |j t }t|jp)|St|tr7|tjjvSdSr)rr~r is_unalignedr_rer`rmrrstatically_known_multiple_ofrcr_rrarrunaligned_buffers)rr`has_unaligned_layoutrrrr3 s   r3c@seZdZUded<eddEdFdd ZdGd d ZdHddZdIddZe dJddZ dKddZ dLddZ dMddZ dNd d!ZdOd#d$ZdPd(d)ZdQd*d+ZdRd-d.ZdSd/d0ZdTd2d3ZdQd4d5ZdQd6d7ZdUd9d:ZdVdd?ZdXdBdCZdDS)Yrrr_Fr{rrrcC |j|Srr_r,r+rrrr,  zBaseView.get_free_symbol_uses*Callable[[Sequence[Expr]], Sequence[Expr]]cCstd|)Nzmake_reindexer NYI on rcrrrrmake_reindexer rzBaseView.make_reindexerr c&|j|dfdd }|S)Nrrrrcrrrrinnerrrrr# rz&BaseView.make_indexer..indexer)rrrr)r_r r;rrrr=rr   zBaseView.make_indexerrcr<)Nrrrrlcrrrrr=rrr, rz$BaseView.make_loader..loaderr)r_rr;rrrr=rr( r@zBaseView.make_loaderrcC |jSr)r_r rrrrr1  zBaseView.dtypercCrBrr_rrrrrr5 r(zBaseView.get_layoutrcCrBrr_r!rrrrr!8 r(zBaseView.get_devicercCr rrrrrrr; r zBaseView.get_origin_nodercCrBrr_rrrrrr> r(zBaseView.get_namercCrrrrrrrrA rzBaseView.get_pointwise_sizerrrcCr7rr_rrrrrrD rzBaseView.mark_reusecCrBrr_rrrrrrG r(zBaseView.has_exceeded_max_readsr-cCrBrr_rrrrrrJ r(zBaseView.realizecCs|jdSrr_rrrrrrM rzBaseView.realize_hintrycCrBrr_r*rrrrr*P r(zBaseView.get_storage_numelcCrBrr_r/rrrrr/S r(zBaseView.is_externcCs$t|jts Jt|j|jSr)rr_rris_module_bufferrrrrrMV s zBaseView.is_module_bufferrcCrBrr_rrrrrrZ r(zBaseView.get_read_namesr$cCsFttddt||jWdS1swYdSr)rrrrKrrr'rrrrr] s$zBaseView.get_readscCs"|}t|tr|j}t|ts|Sr)rrr_)rrrrrrd s   zBaseView.unwrap_viewrrcCs2|}ttd||}t||||dSr)rrrrrr rrrrrr1j szBaseView.constant_to_deviceNrLrrr:rFrEr;r<rCr8rHr?rJr>rAr6rTr7rRrKrV)rrrrrXr,r;r rr^rrr!rrrrrrrr*r/rMrrrr1rrrrr s2                    rc@sFeZdZUded<eddd Zedd d Zdd dZdddZ dS)rrrrrnew_sizerMrcCstjj}dd|D}|}dgt|t|t|}t|t|ks(Jtt|D]W}||dkrE||dus>J||||<q.||dusTtjj||rUq.||}||}|duscJ|dusiJ||}|j|dddksJd|d|d |q.|S) zReplace `-1` with correct sizescSrZrr[rrrrr~ rz.ExpandView._normalize_size..NrArfallbackzBroadcast failed in ExpandView(, z) on dimension ) rmrrrrrris_size_one_or_falser)rrProld_sizerv1v2diffrrr_normalize_sizez s6   zExpandView._normalize_sizerc Cs|||}t|rWt|\}}t|t|j}|dksJtjjg|}t|j |jD]\}}| t j j |s<|ntjjq-t|j|jt|||j|j} t|| dSt||dS)Nrr^)r_r)rYrrarrrrrrrrrmrrrTrbrrrrcrdrer) rrrPrfrgskiprhrrrirrrr s,      zExpandView.createcCrrrrrrrr rzExpandView.get_sizer:cs6|}|jt|tdfdd }|S)NrrrcsRt|d}t|tksJttD]}|dkr&tjj||<q|Sr)rrrrrr)rractualrZrrr s  z*ExpandView.make_reindexer..reindexrrrr)rr_r)rtargetrrr[rr; s   zExpandView.make_reindexerN)rrrPrMrrM)rrrPrMrrr?rO) rrrrr[rYrrrr;rrrrrv s  %  rc@sFeZdZUded<eddd Zedd d ZdddZdddZdS) PermuteViewrydimsrrrrrcs||}t|ttt|ksJt|r>t|\}tjjfdd|Dfdd|Dj j }t ||dSt ||dS)Ncg|]}j|qSrrrrgrrr rz&PermuteView.create..crarrrrbrrr rr^)r_r`) _map_neg_dimsr9rrrrarbrrrcrdrer_)rrr`rfrirrbrr s    zPermuteView.create list[int]csfddDS)Ncs$g|]}|dkr |nt|qSrr )rrSr`rrr s$z-PermuteView._map_neg_dims..r)rr`rrfrrd zPermuteView._map_neg_dimsrcsDt||jttt|jksJ|jfdd|jDS)Ncrrrrrrrr rz(PermuteView.get_size..)r9rdr`rrr_rrrrrr s   zPermuteView.get_sizer:cs`ddt|jDfddtt|jDtttt|jks'Jd fdd }|S) NcSrrr)rrrNrrrr rz.PermuteView.make_reindexer..crrrrinvrrr rz.PermuteView.make_reindexer..rrrcsfddDS)Ncrrrrrrrr rz?PermuteView.make_reindexer..reindex..rrrhrrr z+PermuteView.make_reindexer..reindexr])rr`rrr9)rrrrhrr; s zPermuteView.make_reindexerN)rrr`rrr)r`rrrer?rO) rrrrrrrdrr;rrrrr_ s    r_c@s8eZdZeddddd Zedd dZdddZdS) SqueezeViewNrSrrrSrrc sLt|rzt|\}}g}g}dur*ttsJtdkr(t|jks*Jtt|j|j D]3\}\}} durOt j j |sN|||| q3|kr^|||| q3|dksfJdq3t|j|j|||j|j} t|| dSdurt|dd|DS|dksJt|fddt|DS)Nrr@zexpected squeezed size to be 1r^cSsg|] }tjj|s|qSr)rmrrrTrrrrr* s  z&SqueezeView.create..csg|] \}}|kr|qSrrrrrrlrrr2 )rrarrrrrrrrrmrrrTrrbrrrcrdrerrr) rrrSrfrgrPrhrrrrirrlrr sF       "zSqueezeView.createrr>tuple[list[int], Callable[[Sequence[Expr]], tuple[Expr, ...]]]cs@dd|D}ddt|Dt|d fdd }||fS) NcSsg|]}|dkr|qSrrrrrrr8 rz(SqueezeView.squeezer..cSsg|] \}}|dkr|qSrrrmrrrr9 rnrrrtuple[Expr, ...]csTt|tksJ|dtjjg}t|D]\}}|||<qt|S)N )rrrrrr)rrrrlengthnot_onerrr< s " z%SqueezeView.squeezer..reindex)rrrrp)rr)rrPrrrrrsqueezer4 s zSqueezeView.squeezerr_r rcCstd)Nzuse SqueezeView.create())AssertionError)rr_rrrrE rzSqueezeView.__init__)rrrSrrr)rrrro)r_r rr)rrrrrr[rurrrrrrk s - rkc@sXeZdZUded<ded<dddZdd d Zdd d ZeZedddZ dddZ dS) GenericViewrrr:rrcCrr)rrrrrr;N szGenericView.make_reindexerrcCsBddtt|jD}t||}ddtt|d|S)NcSg|]}ttj|qSr)rgr?rrrrrrrT  z+GenericView.reindex_str..zlambda rSr.)rrrrrrrr)r index_old index_newrrr reindex_strS s  zGenericView.reindex_strcCs$||jd|jd|gS)Nsize=zreindex=)rr_rr}rrrrrZ szGenericView.__str__rrrPrcCs||t||dS)Nr_rr)r)rrrPrrrrra szGenericView.createcCrrrrrrrrj rzGenericView.get_sizeNrOrH)rrrPrrr:rrr?) rrrrr;r}rrrrrrrrrrwI s     rwc@sbeZdZdZedddZeedd d ZedddZ e dd ddZ e dd!ddZ dS)"rz This class handles tensor reshaping by computing appropriate index transformations to map the new shape back to the original storage layout. rrrrcCs<t|}t|}tjjjj}|t|dr||}|Sr)rr\rmrrr evaluate_exprLt)rrrrrrhandle_negative_indexu s   zView.handle_negative_indexrrrPrcsttts Jt|\tjjr|St t dkp.t t dkt |}dd d dfd d }dvrWdfdd }|t |dS|rb|t St|sj||St|dd\}}|j}tjj} tjj|} tjj} ddlm} | | | | d} | durdd| D}t|j|j||j|j}t||dS||S)Nrr}rrPrrhrrecSs6t|dd\}}t|j|j|||j|j}t||dS)NT)r*r^)rarbrrrcrdre)r}rPrhrfrgrirrrcreate_reinterpret_view s z,View.create..create_reinterpret_viewrcs>rt|}|tS}|t|dS)z Handle the case where view is not possible with current strides. For unbacked symbols, make contiguous; otherwise use dynamic_reshape_indexer. r)rrequire_contiguousrrrr)rrrrrPrUunbacked_symbols_in_sizesrr"handle_unbacked_or_dynamic_reshape s   z7View.create..handle_unbacked_or_dynamic_reshaperr tuple[int, ...]cstdgtSr)rrr)rUrr fake_reindex rz!View.create..fake_reindexrFr")_compute_stridesize_obliviouscSs(g|]}t|dr |jjnt|qSr)rrexprrr rrrrr szView.create..r^)r}rrPrrhrrrerrrr)rr rr)rrrresolve_negative_sizerrmrrstatically_known_list_equalsrr1r)rrrrrarto_symints_or_intstorch._subclasses.fake_implsrrbrrrcrdre)rrrPr'rrrfrg old_strideold_size_symintold_stride_symintnew_size_symintrnew_stride_symintrhrirrrr~ sZ    z View.createrUtuple[list[Expr], list[Expr]]cCsdd|D}dd|D}t|}tt|D]}||dkr3tjj||<tt|t|||<nqtj j t|t|||fS)NcSg|] }tjj|qSrrmrrr|rTrrrr z.View.resolve_negative_size..cSrrrrTrrrr rrA) rrrrrOner;rhrmrr check_equals)rUrPrrrrr s  zView.resolve_negative_sizeNrM dense_dimrrc CsZz ||||}W|Sttfy,t|g}|||}|||}t||}Y|Swr)_dynamic_reshape_indexerrv IndexErrorrhr)rrUrPrrflatrrrrrr s    zView.dynamic_reshape_indexerr:cstjjj}ddtt|Dtt|}t|}|duo,|t|dko,t|dk}|r?|dus5J||}| |g|r |r |}|\} } |dkrd t j j | | | fn| dkrn| |n|| ||kr | tjj | |n|| ||kr|| ||kr|\} } | | | } | | } || ||ks | tjj | |nL|| ||krt j j} |} t| | || |} || ||kr|} t| | || |} ||}|| ||kstjj | |nt|r |sG|r$|}tjj |d t j j |s|r8|\} } tjj | d|s'|durSt|dkrS}||ntt|ksbJd fdd }|S) zG Perform a reshape entirely by modifying indexing math cSrxr)rgr?VIEWrrrrr rzz1View._dynamic_reshape_indexer..Nr@rrrcsHt|tksJt|tftt|tfddDS)Nc3rur)rirT replacementsrrrd rxzAView._dynamic_reshape_indexer..reindex..)rrrrrr view_exprrrr_ s$z.View._dynamic_reshape_indexer..reindexr])rmrrrrrrrrrrrrrrr>rvreverseinsert)rUrPrr stack_new stack_oldreordering_dense_dimold_dimsize_oldvarsize_newvar2 size_new2divisormodulus dense_exprrrrrr s                zView._dynamic_reshape_indexer)rrrrrr)rrrPrrr)rUrrPrrrr)rUrMrPrMrrrr)rUrrPrrrrr:) rrrrXr[rrrrrrrrrrrrn s k rcseZdZUdZded<d3fdd Zd4d d ZeZd4d d Zd5ddZ d6ddZ e d7ddZ d8ddZ d8ddZd9ddZd:dd Zd;d!d"Zd3d#d$Zed %dd?d.d/Zd@d1d2ZZS)Arez*Pretend our storage has a different layoutrr`rrcs2tt|jtrt|d|jdSdS)Nr_)rrrr_rrrrrrrrro s  zReinterpretView.__post_init__rcC||j|jgSr)rr_r`rrrrrt s zReinterpretView.__str__cCrBrrFrrrrr~ r(zReinterpretView.get_namercC|jjSr)r`rrrrrr! rzReinterpretView.get_devicercCr rrrrrrr r zReinterpretView.get_origin_nodercCrr)r`rrrrrr rzReinterpretView.dtypercC t|jjSr)rr`rrrrrr rzReinterpretView.get_sizecCrr)rr`rrrrrrC rzReinterpretView.get_stridercdfdd }|S)NrrrrlcsFj}t||}jjjjkr!t|jjjS|Sr)r`r rkloadrrr_to_dtype_bitcast)rr tmp_loaderrrrr s z+ReinterpretView.make_loader..loaderrrrrlrrArrrr szReinterpretView.make_loaderr cCrBr)r`r rrrrr  r(zReinterpretView.make_indexercCrrr`rrrrr rzReinterpretView.get_layoutcCr rrrrrrr r zReinterpretView.freeze_layoutFr{rrcCs*t|jj|t|jj|Bt|jj|BSr)r'r`rrrcr+rrrr, s   z$ReinterpretView.get_free_symbol_usesNrrcCs@tjjj|j|jj|jj|jj|dur|j ntjjj |jj dSr) rmr wrapper_codecodegen_reinterpret_viewr_r`rrrc writelinerrrrrr sz!ReinterpretView.codegen_referencercCrrrrrrrr) r zReinterpretView.num_readsr6rHrCr8r;r?rErFr<rLrUrrBrS)rrrrXrrrrrr!rr^rrrCrr rrrXr,rr)rrrrrrei s,             rec@sXeZdZUdZded<edd d Zdd d ZeZe dddZ dddZ dddZ dS) DtypeViewz(Pretend our storage has a different typer target_dtyperr new_dtyperrcCsHt|rt|\}}t|j||j|j|j|j}t||dSt ||dS)Nr^)r_r) rrarbrrrrcrdrer)rrrrfrgrirrrr s   zDtypeView.creatercCrr)rr_rrrrrr rzDtypeView.__str__cCrr)rrrrrr zDtypeView.dtypercCrBrr_rrrrrr r(zDtypeView.get_sizercsjdfdd }|S)Nrrrrlcst|jjjSr)rkrrr_rrr>rrrr sz%DtypeView.make_loader..loaderrr_rrArrrr s zDtypeView.make_loaderN)rrrrrrrHr;r?rE) rrrrXrrrrrr^rrrrrrrr s     rc@s.eZdZedd d Ze ddddZdS) SliceViewrrrSrstartendrtuple[int, int]cstjj||tdd||fDrtjtjnjj dfd d dfdd }||dd}|||}||fS)zz Normalize start and end such that both are in the range [0, x.get_size()[dim]] and start <= end. cs|]}t|VqdSr)r1rTrrrr rz0SliceView.normalize_start_end..rrlowerrupperrcs<||r|n||}||r|}|S||}|Sr)statically_known_geqrP)rrr clamped_lower clamped_full)max_funcmin_funcrrrclamp s z,SliceView.normalize_start_end..clamprdUnion[int, None]rUnion[Expr, int]cs$|dur|S|}|||Sr)r)rdrrr)rrdim_sizerr clamp_wraps  z1SliceView.normalize_start_end..clamp_wraprN)rrrrrrrr) rdrrrrrrrrr) rmrrrrrMinMax evaluate_min evaluate_max)rrrSrrrr)rrrrrrrnormalize_start_end s   zSliceView.normalize_start_endr@Tsteprrc stttsdksJzdkr"|dkr"dkr"|WSWn ty,Ynwt||r?|||\}t|d<t |r}t |\}}t|j } | | <t |j |j| |j|j |j} t|| dSd fdd } t|| d S) Nrlr@r^rrrcsDt|tksJd|dt|}||<|S)Nz wrong ndim rq)rrrrSrPrrrrr@s$z!SliceView.create..reindexrr])rr\rr TypeErrorrrrr<rrarrbrrrcrdrer) rrrSrrrrrfrgrhrirrrrrs8       zSliceView.createN) rrrSrrrrrrr)r@T)rrrSrrrrrrrrrrr)rrrrrrrrrrr s )rc@sFeZdZUded<ded<dddZdd d Zdd dZdddZdS) BaseConstantrrrrrrcCrNrrrrrrrQr zBaseConstant.get_sizercCrrrrrrrr!TrzBaseConstant.get_devicercCr rrrrrrrWr zBaseConstant.get_origin_noder$cCrlrr8rrrrrZrzBaseConstant.get_readsNr?rCr8rR)rrrrrr!rrrrrrrLs    rc@sDeZdZUded<ded<ded<dd d Zdd d ZdddZdS)Constantr r'rrrrrrcr)NrrrrlctjjSr)rkrr'rrrrrrerlz$Constant.make_loader..loaderrrrArrrrdzConstant.make_loaderr-cCr rrrrrrrjr zConstant.realizercCt|j|j|dS)N)r'rr)rr'rrrrrr1mrzConstant.constant_to_deviceNrErArV)rrrrrrr1rrrrr^s   rc@s:eZdZUded<ded<ded<dd d Zdd d ZdS)IndexingConstantr rrrrrrrcr)Nrrrrlcrr)rkr]rrrrrrrxrlz,IndexingConstant.make_loader..loaderrrrArrrrwrzIndexingConstant.make_loaderrcCr)N)rrr)rrrrrrrr1}rz#IndexingConstant.constant_to_deviceNrErV)rrrrrr1rrrrrqs  rcCspd}d}ttt||D](\}}|dkrq tjj||s)tjj||s)dS|td|9}||9}q dSNr@FT) reversedrrrmrrrQrr)rrOexpected_strideexpected_stride_maxryrrris_contiguous_strides_for_shapes rcCs tj|jSr)rApadding_alignment_bytesitemsizerrrrget_align_for_dtyperrc@s2eZdZdZdddZdddZ ddd dZdS)r0zxAbstract base for Layout, MultiOutputLayout, NoneLayout. Represents the memory layout of the output of an Operation.rrcCrrrrrrrr!rzOutputSpec.get_devicercCrrrrrrr storage_sizerzOutputSpec.storage_sizeFr{rrcCrrrr+rrrr,rzOutputSpec.get_free_symbol_usesNrCrSrLrU)rrrrXr!rr,rrrrr0s   r0c@s8eZdZdZdeddfdMddZedNddZejdOddZedNddZ e jdOddZ edPddZ e jdQddZ dRd d!Z e Z dSd"d#Z dTd%d&ZdUd'd(ZedVd,d-ZdUd.d/ZdWd2d3ZdUd4d5ZedXd7d8ZdYd9d:ZdUd;d<ZdZd>d?Zd[dAdBZd\dEdFZdPdGdHZed d]d^dKdLZdS)_rzo Layout base class Carries tensor meta-information including offset and whether it is pinned. NrFrrrrrrrOptional[Sequence[Expr]]rcrrdrrrcCs|dur t|}||_||_t|t|ks!Jd|d|tdd|Ds,J||_||_||_||_ |j rE|jj dksGJddSdS)Nr~ , stride=css|] }t|ttfVqdSr)rrrrrrrrz"Layout.__init__..r3zOnly CPU tensors can be pinned) rrrrrr_size_stride_offsetrdr)rrrrrrcrdrrrrs $ zLayout.__init__cCrrrrrrrrrz Layout.sizer'cC ||_dSrrrr'rrrrrCcCrrrrrrrrrz Layout.stridecCrrrrrrrrrCcCrrrrrrrrcrz Layout.offsetcCrrrrrrrrcrCrc Csd}|jdkr d|j}|jjdurdnd|jj}d}|jr'd|j}t|jd|jj|d|jd|jd |j||d S) Nrrz , offset=:z , is_pinned=z('z', z, size=rr/) rcrrrdrrrrr)rrcdevice_index_str is_pinned_strrrrrs"   "zLayout.__str__cCrrrrrrrr!rzLayout.get_devicercCsPtjtjt|jt|j|j|j|j dWdS1s!wYdS)N)rr pin_memory) rm fake_moder#r$r[rrrrrdrrrr get_examples$zLayout.get_examplecCst|j|jSr)rrrrrrrr'rzLayout.is_contiguousrOrMrXcCsVt|}|dvs|ddkrdSt|t||D]\}}}|dkr(||kr(dSqdS)N)rr@FT)rrr,)rOrXndimleftrightrrrris_channels_last_contiguoussz"Layout.is_channels_last_contiguouscCsJt|jtttt|j|jD]\}}}|dkr"||kr"dSqdSr)rrrrrrr)rrrrrrr is_transposedszLayout.is_transposedrrcstjtks JddtjD}fdd|D}fdd|Dddd }|d gt}ttD] }||||<q=ttd D]'}||||d k}t|tsrtjj j ||||d kd d }|rwdSqPd S)NcSs*g|]\}}tjjj|dddkr|qS)rrQr@)rmrrr)rrrSrrrr#s z,Layout.is_stride_ordered..crarrcrrrrr)rcrrrrrrrr*rarrrrcst|fdd|DS)Ncg|]}|qSrr)relement sorted_arrrrr.rzDLayout.is_stride_ordered..sorted_indices..)r+)r rr rsorted_indices,sz0Layout.is_stride_ordered..sorted_indicesrAr@TrF)r rrr) rrrrrrrrmr _shape_envr)rr non_1_indicesrrstride_orderedrrr)rrrr1s*  zLayout.is_stride_orderedcCs:dgtttdt|jd}t|g|}||SNrr@)rrrrrr1rrrris_channels_last_stride_orderedBs" z&Layout.is_channels_last_stride_ordered in_stridescst|}t|dkr |Stjst||r|St}t|dr)|j ddr)|St ddt ||D }tjs>|r>|SttjdrHtjjnddfd d r`tfdd|Dr`|St|}t|}ddtt|D}d||d<d} t|ddddD]?\} } || d} || || } t| ttjfo| tjko| |dkpt| tjotj}| || <|rt| |||| <d} q| s|Stjd7_|S)z The padding does not change stride order but makes sure all strides larger than the threshold are multiple of align. rrsdislike_paddingFcsrrrrrrrrarz&Layout._pad_strides..rNrsympy.Expr | intrrcs4durdSt|tjsdStfdd|jDS)NFc3s|]}|VqdSr)is_unbacked_symintrrrrrorxzILayout._pad_strides..contains_unbacked_symints..)rrrrr0rrrrcontains_unbacked_symintsjs  z6Layout._pad_strides..contains_unbacked_symintsc3rrrr)rrrrrrcSrrrrrrrrxrz'Layout._pad_strides..r@)rT)rrrr)rrrApad_channels_lastrrrmget_current_noderrsrrrVchainpad_dynamic_shapesrrrrrrrrrrr padding_stride_thresholdrrYr%num_comprehensive_padding)rrraligncurrent_fx_nodercr+r new_stridespaddedrankrprev_idxrrequire_paddingr)rrr _pad_stridesHsZ        zLayout._pad_stridescCs>t|ts Jt||jdusJ||j|j|j|_dSr)rrrrr(rrrrrrr&szLayout.pad_stridescCstjot|tSr)rAcomprehensive_paddingrrrrrrr%rlzLayout.should_pad_stridesrbcCs<t|tr|S|r|t|j|j|j|j|j|j Sr) rrbr%r&rrrrrcrdrrrras_fixeds zLayout.as_fixedr cCs(tjsJdt|jd|S)Nzconvert z to FixedLayout first)rrrrr*r rrrrr s zLayout.make_indexerotherrcCsRt|to(|j|jko(|j|jko(|j|jko(|j|jko(|j|jko(|j|jkSr)rrrrrrrcrd)rr+rrr__eq__s       z Layout.__eq__cCt|j|j|jSr)r)rrrcrrrrrrzLayout.storage_sizer{rcCs$t|j|t|j|Bt|j|BSr)r'rrrcr+rrrr,s   zLayout.get_free_symbol_uses)rrrrrrrrrcrrdrrrr?r'rrrr@r'rrrrHrD)rrr>)rOrMrXrMrr)rrrr)rrrrrrrrr6rrbrF)r+rrrrLrU)rrrrXr rr^rsetterrrcrrr!rr'r[rrr1rr(r&r%r*r r,rrXr,rrrrrsN               #  N     rc@seZdZdZdddZdS)rbz A Tensor layout we cannot changerr cCr-)r)rbrrrcrrrrr rgzFixedLayout.make_indexerNrF)rrrrXr rrrrrbsrbcseZdZdZdZdDddZedEd d ZedFd dZedGddZ edHddZ edIddZ e dJddZ e jdKddZ e dJdd ZejdKd!d Ze dLd#d$ZejdMd%d$Z dNdOd(d)Z dNdPd+d,ZdQd-d.ZdRd/d0ZdSd2d3ZdTd7d8Z 9 dUdVfdBdC ZZS)Wrz| A Tensor layout that we are allowed to change Assumption: layout change should NOT add or remove free symbols FrrbcCst|S)z Compute what the strides would be if this layout were frozen, without actually modifying the layout. This is used for speculative stride computation during Triton template code generation. )rdeepcopyr*rrrr!get_fixed_layout_without_freezingsz0FlexibleLayout.get_fixed_layout_without_freezingsizesrrycCsNt|dkrgStjjg}t|ddD] }|||dqtt|S)Nrr@rA)rrrrrrr)r4reversed_stridesrrrrrs   z!FlexibleLayout.contiguous_stridesrcCs\ttt|t|ksJ||ftjj}dgt|}|D] }|||<|||}q|S)z Create a stride based on the order the dimensions should be filled in. In this format, channels last would be: [1, 3, 2, 0] N)r9rrrrr)r4r next_striderXrrrr fill_ordereds$zFlexibleLayout.fill_orderedrcCs0ttt|t|ksJt|}t||S)z Create a stride based on the sorted order of a permuted range. In this format, channels last would be: [3, 0, 2, 1] )r9rrrrr7)r4rrrrrrs zFlexibleLayout.stride_ordered memory_formattorch.memory_formatcCsP|tjkr t|tS|tjkrt|tS|tjkr t|St d|t )aq Create a stride based on a memory format. Memory format is translasted into a stride order, so channels_last is the same as: FlexibleLayout.stride_ordered(sizes, [3, 0, 2, 1]) This interface does not support memory_format `torch.preserve_format` which should be used to deduce a format from another source z>stride_ordered_for_memory_format, unsuppored memory_format: %s) r# channels_lastrrNHWC_STRIDE_ORDERchannels_last_3dNHWDC_STRIDE_ORDERcontiguous_formatrr=rr)r4r8rrr stride_ordered_for_memory_formats      z/FlexibleLayout.stride_ordered_for_memory_formatrrMcCsDt|t|ks Jdd|D}ttt||jd}t||S)z Create a stride that has the same stride order as given stride For example, if given stride is [1000, 1, 100, 10], the fill order should be [1, 3, 2, 0] cSrr)rmrrr}rTrrrr4rz/FlexibleLayout.same_ordered..r%)rr+r __getitem__rr7)r4rrrrr same_ordered)s  zFlexibleLayout.same_orderedcCrrrrrrrr8rzFlexibleLayout.sizer'rcC|d|||_dS)Nr)!assert_free_symbol_uses_unchangedrrrrrr<  cCrrrrrrrrArzFlexibleLayout.stridecCrB)Nr)rCrrrrrrErDrcCrrrrrrrrcJrzFlexibleLayout.offsetcCrB)Nrc)rCrrrrrrcNrDrrcCsH||j|}|r|r|||j|j}t|j|j|j||j|jSr) rrr%r(rrbrrcrd)rrrrhrrras_stride_orderSs zFlexibleLayout.as_stride_orderrcCs>|}|r|r|||j|j}t|j|j|j||j|jSr)r%r(rrrbrrcrd)rrrrhrrras_exact_stridescs zFlexibleLayout.as_exact_stridescCD||j|}|r|||j|j}t|j|j|j||j|jSr) r7rr%r(rrbrrcrd)rrrhrrr as_fill_orderszFlexibleLayout.as_fill_ordercCrGr) rArr%r(rrbrrcrd)rrrhrrr as_same_orderrIzFlexibleLayout.as_same_order$dict[tuple[str, bool], sympy.Symbol]cCs<i}dD]}dD]}||f}ttt|||||<qq|S)N)rrrcTF)r9r'r)rinitial_free_symbolsrr{r&rrrget_initial_free_symbol_usess z+FlexibleLayout.get_initial_free_symbol_usesrrr2cCsFdD]}|j||f}tt||}||ks Jd|d|qdS)NrLz)Expected free symbols unchanged, but got z vs )rMr9r')rrr'r{old_free_symbolsnew_free_symbolsrrrrCs z0FlexibleLayout.assert_free_symbol_uses_unchangedNrrrrrr+r,rdcs@|r t||}nt|}tj|||||d||_dS)Nrd)rr7rrrrNrM)rrrrr+rdrXrrrrs  zFlexibleLayout.__init__r0)r4rrry)r4rrrrry)r4rrrrr)r4rr8r9rr)r4rrrMrrr?r.r@r/rL)rrrrrrb)rrMrrrrb)rrrrb)rrMrrb)rrK)rrr'r2rrr) rrrrrrr+r,rdrrr)rrrrXrr3r[rr7rr?rAr^rr1rrcrErFrHrJrNrCrrrrrrrsJ                rcsLeZdZdZdfdd Zdd d Zdd d Zed ddddZZ S)NonOwningLayoutz,Is a view into the storage of another tensorviewUnion[BaseView, TensorBox]rrcs,|}t|j|j|j|j||_dSr)rrrrrrrrS)rrSr`rrrrs zNonOwningLayout.__init__r cC |Sr)r*r rrrrr rzNonOwningLayout.make_indexerrcCs4|jj}|dkr dSddlm}tjj||S)NrTr@) ALIGNMENT) rSrrcutilsrVrmrrr4)rrcrVrrrmaybe_guard_aligneds  z#NonOwningLayout.maybe_guard_alignedFr{rcCsVt|jtsJ|jj}t|tsJt||j}t|ts%Jt||j|Sr) rrSrer_rrrr`r,)rr{box input_bufferrrrr,s  z$NonOwningLayout.get_free_symbol_uses)rSrTrrrFr>rLrU) rrrrXrr rXrXr,rrrrrrRs rRc@eZdZdZdS)CommBufferTypesymm_memN)rrrSYMM_MEMrrrrr\r\cs4eZdZUdZded<ded<d fdd ZZS) CommBufferLayoutax A layout that signifies the buffer is a comm buffer. In terms of striding, the layout is identical to `FixedLayout`. Buffers with this layout do not participate in in-place reuse - it can be neither the source nor the target for in-place reuse. For detailed motivation and usage of this layout, see NOTE [lowering-time collective optimization]. r\comm_buffer_typer group_namer`"Union[FlexibleLayout, FixedLayout]csJt|tr |n|}tj|j|j|j|j|j |j d||_ ||_ dS)Nrrrrrcrd) rrr*rrrrrrrcrdrarb)rr`rarbfixedrrrrs zCommBufferLayout.__init__)r`rcrar\rbr)rrrrXrrrrrrrr`s  r`c@sdeZdZUded<ejdddZded<ejdddZded <dd d ZdddZ dddZ dS) NoneLayoutrrcCdgSrrrrrrr#r$zNoneLayout.default_factoryrercCrgrrrrrrr# r$rrrcCrrrrrrrr r zNoneLayout.storage_sizer0cC|Srrrrrrr*r zNoneLayout.as_fixedcCrrrrrrrr!rzNoneLayout.get_deviceNrSr=rC) rrrrrYrZrrrr*r!rrrrrfs    rfcseZdZd#fdd Zed$dd Zejd%d d Zd&ddZd'ddZd(ddZ e d)d*ddZ d+ddZ d,d!d"Z ZS)-MutationLayoutSHOULDREMOVEr^rrrcs@t|||d||_|}tj |dSr) rrrr rr^ get_bufferrrmrmark_buffer_mutated)rr^rrrrrs z#MutationLayoutSHOULDREMOVE.__init__rcCr%r) real_layoutrrrrrr!rCz!MutationLayoutSHOULDREMOVE.strider'rcCr rrrrrrr%rrcCrUr)rnrrrrrr)rz'MutationLayoutSHOULDREMOVE.storage_sizercs2dfdd |j}t|tsJt||S)Nr^r rcsBt|tr |jSt|tr|St|tr|jS|Sr)rrkr^rr MutableBoxr_)r^ unwrap_viewsrrrq-s      z;MutationLayoutSHOULDREMOVE.get_buffer..unwrap_views)r^r rr )r^rrr)rrrrprrl,s z%MutationLayoutSHOULDREMOVE.get_bufferrcCs|j}t|ts J|Sr)rlr`rr)rr`rrrrn:s z&MutationLayoutSHOULDREMOVE.real_layoutFsrcdst unsafe_aliasrc Cs|tj|t|tr|j}||sAt j | | | ddt||Dd}t|ttfs>J|j}|t|dsNJ|t|jjts]Jt|jjt||j_|jS)NcS g|] \}}tjj||qSrrmrrcheck_equals_and_simplifyrrrrrrrYz;MutationLayoutSHOULDREMOVE.realize_into..rr_)rrmrrmrrr~r_rrrr!r rrrrrorr`rrrk)rrrrsrtrrrr realize_into?s*   z'MutationLayoutSHOULDREMOVE.realize_intorcCrjrrrrrrr*gr z#MutationLayoutSHOULDREMOVE.as_fixedr cCrBr)r^r rrrrr jr(z'MutationLayoutSHOULDREMOVE.make_indexer)r^rrrr?)r'rrrr@)rrr<rL)rrrrsrrtrrr)rrrF)rrrrr^rr1rrlrnrrzr*r rrrrrrks      'rkcsleZdZUded<ded<d]fdd Zd^d d Zd_d dZd`ddZdaddZdbddZ e dcddZ ddddZ dedd Z dfd"d#Zdgd%d&Zdhd'd(Zdid*d+Zdjd-d.Zd]d/d0Z 1dkdld5d6Zdmd7d8Zdnd:d;Z 1dkdod=d>Zdjd?d@ZdpdBdCZdqdrdGdHZd]dIdJZdsdLdMZdsdNdOZdtdQdRZed 1dkdudUdVZ dvdWdXZ!dwdYdZZ"djd[d\Z#Z$S)xrr-rr0r`rrcst|dddSre)rrrrrrrrxs zBuffer.__post_init__r cCrUr)rr rrrrr |rzBuffer.make_indexerrcCs|jsJ||jSrrrrrrrrfzBuffer.get_name!Union[torch.Tensor, torch.SymInt]cCs&t|jtr |jStt|jjr)rr`rrrrrrrrrrs  zBuffer.get_examplercCrUr)rr!rrrrr!rzBuffer.get_devicercCr rrrrrrrr zBuffer.get_defining_oprcCr%r)rrrrrrrrCz Buffer.dtypercCg|jSr)rrrrrrrrzBuffer.get_sizerycCr|r)rrrrrrrCrzBuffer.get_stridercCr%r)rrcrrrr get_offsetr(zBuffer.get_offsetrcCs"t|jtr |jStt|jjr)rr`rrrrrrrrrs zBuffer.get_layoutcCrrrrrrrrrzBuffer.get_output_specrcCrr)rrrrrr*rzBuffer.get_storage_numelrcCr%r)rrdrrrr get_is_pinnedr(zBuffer.get_is_pinnedcCs0t|jtrt|jts|j|_dSdSdSr)rr`rrRr*rrrrrs zBuffer.freeze_layoutFrrrcC0t|jts Jt|j|jj||d|_dSNr/)rr`rrrErrrrrsz&Buffer.freeze_layout_with_stride_ordercC,t|jts Jt|j|j||_dSr)rr`rrrHrrrrrz$Buffer.freeze_layout_with_fill_orderrcCrr)rr`rrrJrrrrrrz$Buffer.freeze_layout_with_same_orderrcCrr)rr`rrrFr rrrr!s z'Buffer.freeze_layout_with_exact_stridescCrrrrrrrrrzBuffer.is_zero_elementsrcs*r ttdSdfdd }|S) Nrrrrrlcs}tjp d||Sr)r rkrrrrrrrrsz"Buffer.make_loader..loaderr)rr rr rArrrrszBuffer.make_loaderNrrcCrrrrrrrrrzBuffer.codegen_referencecCr rrrrrrr-r zBuffer.decide_layoutrcCt|jtr |jjgSdSr)rr`rRrSrrrrrr4 z#Buffer.get_inputs_that_alias_outputcCrr)rr`rkr^rrrrrr2rzBuffer.get_mutation_namesrcCst|gSr)r9rrrrrrrzBuffer.get_read_namesr{rcCrlrr8r+rrrr,zBuffer.get_free_symbol_usescCrlrr8rrrrrmrzBuffer.get_unbacked_symbol_defscCr rrrrrrrr zBuffer.realizecCrrrrrrrshould_allocaterzBuffer.should_allocater6rFrH)rr{rCr9r;r?)rryr@r<r=rSr>rLrMrN)rrrr)rrrrrrrErrBrWr7rUrprA)%rrrrrr rrr!rr^rrrCr}rrr*r~rrrrr!rrrr-r4r2rrXr,rmrrrrrrrrnsL                         rc@s0eZdZd ddZd ddZejZdd d Zd S)OperationBufferrrjcCs|gSrrrrrrrkrzOperationBuffer.get_outputsr_cCrjrrrrrrrr zOperationBuffer.get_defining_oprcCst|t|dSr)rrr_rrrrrs zOperationBuffer.__post_init__Nrorr_r6)rrrrkrr_r3rrrrrrs   rc@eZdZdddZdS)rrrcCrrrrrrrr) r zInputBuffer.num_readsNrS)rrrr)rrrrrrc@r[) DonatedBufferaY Represents a donated buffer which is a saved tensor that is not alias to any fwd inputs, fwd user outputs, and bwd outputs. We generally cannot inplace reuse the input tensor memory during backward since it might be used in another function. However, donated buffer can be inplace reused during backward to save memory. N)rrrrXrrrrr r_rc@s.eZdZUdZded<d ddZdd d ZdS)rNrrrrcr)Nrrrrlcs,}ttjj||Sr) rr rkrrmr constant_namerrrrrrrs z*ConstantBuffer.make_loader..loaderrrrArrrrszConstantBuffer.make_loaderrrrcCsttj|||jdSN)rr`)rrmrrrr`rrrrr1$sz!ConstantBuffer.constant_to_devicerErV)rrrrrrr1rrrrrs   rc@sLeZdZdddZed ddd d ZddddZdddZdddZd S)NoneAsConstantBufferrr$cCrlrr8rrrrr,rzNoneAsConstantBuffer.get_readsFr{rrcCrlrr8r+rrrr,/rz)NoneAsConstantBuffer.get_free_symbol_usesNrrrcCs tjjjSr)rmrrnone_strrrrrr5r(z&NoneAsConstantBuffer.codegen_referencer0cCs tddSNr)rfrrrrr8r(z$NoneAsConstantBuffer.get_output_speccCrrrrrrrr;r z&NoneAsConstantBuffer.has_tensor_outputrRrLrUrrBr=r>) rrrrrXr,rrrrrrrr*s   rc@sBeZdZUded<ed dddd ZddddZdddZd S)rrrFr{rrrcC t|j|Sr)r'rr+rrrr,C z*ShapeAsConstantBuffer.get_free_symbol_usesNrrrcCstjj|jSr)rmrrcodegen_sizevarrrrrrrIrlz'ShapeAsConstantBuffer.codegen_referencecCrrrrrrrrLr z'ShapeAsConstantBuffer.has_tensor_outputrLrUrrBr>)rrrrrXr,rrrrrrr?s  rcsheZdZUdZded<dZded<dZded <dZd ed <dZd ed <dZ d ed<e j d[ddZ e e j d[ddZd\ddZd]ddZd^ddZd_dd Zd`d"d#Zed dadbd'd(Zdcfd*d+ Zddd,d-Zded/d0Zdfd2d3Zdgd5d6Zedhd8d9Z  didjd>d?Ze dkdldIdJZdmdLdMZdmdNdOZ d\dPdQZ!dddRdSZ"dddTdUZ#dndYdZZ$Z%S)orzb Represents a buffer that is computed during kernel execution rather than being an input. rqr_FzClassVar[bool]_force_realizeNrrOptional[Callable[..., Any]]rrrrrIterator[None]c cs|jdusJ|jdusJ|jdusJ|jdusJt|jts+Jt|j|j}|j}z3t|j |j |j|j|j|j |j |j d}||_t|j |j |j|_|j|dVW||_||_dS||_||_w)Nr)rrrrrr_rrr`rrrrrrbget_default_sizes_body clear_cache)rold_datargnew_datarrrwith_original_inner_fn_s>   z%ComputedBuffer.with_original_inner_fnccs*tj}z dt_dVW|t_dS|t_wNT)rr) old_valuerrr force_realizes zComputedBuffer.force_realizer-cCs(|jdur|jSt|jdr|jjSdS)z Returns self.name if it exists, otherwise returns the name of the data node if that exists. If neither exist, returns None. Nr)rrr_rrrrget_computed_buffer_names  z'ComputedBuffer.get_computed_buffer_namercCrBrr_r)rrrrr)r(zComputedBuffer.num_readsr$cCrBrr_rrrrrrr(zComputedBuffer.get_readsrcCrBrrNrrrrrr(zComputedBuffer.get_read_namesr"cCst|jttttfstjtttdSt t dd,|j r7t ||j|jWdSt ||jWdS1sKwYdS)Nr'writes index_exprsrT)rr_rrrrrB ReadWritesr9rrrr-rKget_store_functionrr.rrrrrr#s& $zComputedBuffer.get_read_writesr{rrcCs6|j||j|B}|r|||O}|Sr)r`r,r_has_store_functionr#)rr{rrrrr,s z#ComputedBuffer.get_free_symbol_usesrcs<|s|jtjjvr|dkr|js|jSt Sr) r-rrmrmutated_buffersr)rr_rrrrrrrs   zComputedBuffer.make_loadercCst|jttttfSr)rr_rrrrrrrrrrz!ComputedBuffer.has_store_functionCallable[..., None]cCs`|}t|jtttfrt|jj |j |St|jt s'Jt |jt|jj |j |Sr)rr*r rr_rrrr rrrrrr?rrrrs z!ComputedBuffer.get_store_functionOptional[list[int]]cst|jtrYt|j|j\\}}|j }t dd|Ds&Jfdd|D}|rYt|jt t frA|j |n|fdd|D}ddlm}|||SdS) al If our layout is still flexible, try to determine the stride order based on stride orders of reads. TODO(jansel): A better algorithm here would look at downstream consumers of this value and try to do global graph-level layout optimization. This is also something just begging to be autotuned. css"|] }t|tjtjfVqdSr)rrBStarDep MemoryDepr"rrrr  z0ComputedBuffer.get_fill_order..cs.g|]}t|tjrt|jddDqS)cSsi|] }|dkr|tjjqSrro)rvrrrrz..)rrBrrirr")rrrrs  z1ComputedBuffer.get_fill_order..csg|] }tjj|qSrrmrrr@rr)r0rrrsr@pick_loop_orderN)rr`rrBr>r_rr.r#r'rrrr schedulerrr)r index_varsrr'stride_lengthsrr)r0rrrs*     zComputedBuffer.get_fill_orderrcCs6t|jtr|}|r||dS|dSdSr)rr`rrrrrrrrr- s  zComputedBuffer.decide_layoutMtuple[tuple[list[Expr], list[Expr]], LoopBody, tuple[list[Expr], list[Expr]]]c Cstj||dd\}}ttd|t| | r"|n|dd|g|R}Wdn1s8wYg}g}g}g}| D]+\}} ||dvrb|rWJ| || | qI||dvsjJ| || | qI||f|||ffS)Nqr|rr@r) rBr>rr.rrrr!rMrr-itemsr) rr var_rangesrr reduce_vars index_size reduce_sizerrrrrrs2       z%ComputedBuffer.get_default_sizes_bodyextra_indexing_constraints*Optional[tuple[dict[Any, Any], list[Any]]]recompute_sizes_body_func8tuple[tuple[list[Expr], list[Expr]], Optional[LoopBody]]cs\\}}}\}}|r|||f|||f\\}}}\}}g|j|durut|tr4t|dks6J|\}} t|tsEJt|t| tsPJt| t dd| Ds[J|j } | |kshJ| |ffdd| D} | 7g| t j tjs|dfdd } ||} tt ptj } | || || \}}}| || || \}}}tj||dd\\}}}t|||||g|||}||f|fS)an This is a main place where we do loop transformations in a backend-agnostic way. Here we: 1) Remove any 1 dimensions 2) Fuse contiguous dimensions together 3) Reorder dimensions based on stride orders Optional argument extra_indexing_constraints can be used to append additional indexing expressions to existing ones derived from buffer's body. This can be useful to fuse scheduler nodes with compatible ranges, e.g. (s0*s1*...,) and (s0, s1, s2, ...) on CPU by preventing indexing simplifications and obtaining index/reduce ranges for the scheduler node compatible with other nodes. Optional argument recompute_sizes_body_func can be used to recompute sizes and body on the default body. This can be useful to append additional loop transformations. Nrcs|]}t|tVqdSr)rr)rfrrrrcrxz6ComputedBuffer.simplify_and_reorder..csg|]}|vr|qSrrrv)index_formulasrrrkz7ComputedBuffer.simplify_and_reorder..x_varsSequence[sympy.Symbol] support_varsr4rsimplify_loopsrrdtuple[list[int], Callable[[Sequence[int]], Sequence[int]], Callable[[Sequence[int]], Sequence[int]]]c s||\}}}dkr8tdkr8ttt}||ddkr8fdd|D}t|}t|}||}|rTtjj ||t ||\}}} t ||} n|} || |fS)Nrrrcrrrrr4rrrrzUComputedBuffer.simplify_and_reorder..simplify_and_reorder..) _apply_loop_reorderingr-rrrrrrmrr_simplify_loopsrFr) rrr4rnewsizesreindex0rrr_prunerr memory_addrsrrrsimplify_and_reorderts&     zAComputedBuffer.simplify_and_reorder..simplify_and_reorderpr|) rrrrr4rrrrr)rindexing_exprsrrrrrrrrrget_write_exprsrmrr5rCPREFER_STORE_LOOP_ORDERextendget_read_exprsrdr1rAloop_ordering_after_fusionrBindex_vars_no_squeezerM)rrrrrrrrextra_indexing_rangesextra_indexing_exprexpected_var_rangesrrshould_merge_loops iter_ranges iter_reindexr reduce_rangesreduce_reindex iter_varsrrrrr5sx   6    z#ComputedBuffer.simplify_and_reorderrrrr4rrlist[sympy.Expr] priority_idxrc sddlm}|dur g}z*fdd|D}t|t|kr)t|dtks+Jtt|||}Wn tyVtjrLt dt t |tt t}Ynwfdd|Dt |t|fS) zU Shuffle the order of loops around to hopefully improve performance. r@rNcsg|] }tjj|qSrrr)rrrrrz9ComputedBuffer._apply_loop_reordering..rz%Did not simplify complex index: %s %scrrrrrrrrr)rrrrr ExceptionrArr=warningrrrrr)rrr4rrrrXrr)rr4rrrs,     z%ComputedBuffer._apply_loop_reorderingrcCrBr)r_rrrrrrr(z!ComputedBuffer.get_pointwise_sizecCrBrr_r.rrrrr.r(z!ComputedBuffer.get_reduction_sizecCrBrr_r-rrrrr-r(z!ComputedBuffer.get_reduction_typecCrBr)r_rrrrrr0r(zComputedBuffer.is_no_opcCrrrrrrrrr zComputedBuffer.should_allocaterrrcCr7)rr_r1rrrrr1r9z!ComputedBuffer.constant_to_device)rrrArSrRr7rQrLrUrEr>)rr)rrr6)rrNN)rrrrrrr) rrrrr4rrrrrrrr?rV)&rrrrXrrrrrrr\r]rr[rrr)rrr#rXr,rrrrr-rWrrrrr.r-r0rr1rrrrrrPsR       "       ' "  '    rcsfeZdZdZd#fd d Zd$d dZd%d&ddZd'ddZd(ddZd)ddZ  d*d+d!d"Z Z S),rzt Represents a Triton (in the future other type) of template operator that we can fuse an epilogue onto. r`r0rwrxmake_kernel_renderrrrcsFtjd|dt||_||_tj||_ tj |i|_ dSr) rrrunwrap_storagerwrrmrregister_bufferrregister_operationr)rr`rwrrrrrs    zTemplateBuffer.__init__r"cCs |jddS)NT normalize)rKrrrrr#rzTemplateBuffer.get_read_writesFrrc s||d fdd }tj||d|d }|jD]:ttt fs/Jt tj t s.dummyrrcs$t|dksJt|Sr)rrkrrr_)rr}rrr1s)rrr rrr )rrr rBrKrrwrrerrr`rr')rrrdepsr)rr}rrrKs"      z"TemplateBuffer.extract_read_writesrcCstjjSr)rrrrrrrr.<rz!TemplateBuffer.get_reduction_sizer-cCr rrrrrrr-?r z!TemplateBuffer.get_reduction_typecCrrrrrrrrBr zTemplateBuffer.should_allocateNrrrr)rrrrrr) rrrrXrr#rKr.r-rrrrrrrrs     rcs\eZdZ  dd fdd Zed d!d"fdd Zd#ddZd$ddZd%ddZZ S)&TritonTemplateBufferNr`rrwrxrOptional[Callable[_P, _T]]mutated_inputsOptional[Iterable[IRNode]]allowed_prologue_inpsOptional[OrderedSet[str]]rrcst||||_g_|dur:tjdts$Jtjdjdjfdd|D7_|r>|nt _ d_ d_ dS)a NOTE:[TritonTemplates with multiple outputs] We want the ability for TritonTemplates to output multiple tensors. Triton kernels have no notion of outputs and this is done by creating tensors that are then mutated by the kernel. Currently our STORE_OUTPUT codegen doesn't support creating multinode outputs for triton templates. We work around this by creating an extra input buffer during the lowering and we mark them as mutated inputs. Nrcg|] }ttd|qSrMutationOutputrfrr2rrrrrkrz1TritonTemplateBuffer.__init__..) rrroutputsrrwrrr!r9r subgraph_inps subgraph_outs)rr`rwrrrrrrrTs"  zTritonTemplateBuffer.__init__Fr{rrcst|}|jr |jng}|jr|jng}|D]%}t|tjr)|t||qt|t r7|||q|dus=Jq|D]}t|t rP|||q@|dusVJq@|Sr) rr,r rrrrupdater'r)rr{resr rr}rrrrr,ws    z)TritonTemplateBuffer.get_free_symbol_usesrjcCrrrrrrrrkrz TritonTemplateBuffer.get_outputsrcCrr)rrrrrget_allowed_prologue_inpsrz.TritonTemplateBuffer.get_allowed_prologue_inpsrcCsd|jd}|S)NzTritonTemplateBuffer(layout=r/r)rrrrrrszTritonTemplateBuffer.__str__r) r`rrwrxrrrrrrrrrLrUror7rH) rrrrrXr,rkr rrrrrrrSs#  rcs|eZdZdZd'fd d Zd(ddZd)ddZd*ddZd)ddZd)ddZ d+ddZ d,d!d"Z d)d#d$Z d-d%d&Z ZS). ChoiceCallera1 Represents a possible choice used in autotune_process.py. During autotuning, self.benchmark() is first called to get benchmark result, and if this choice is selected, self.output_node() is called to get the output_node. Children classes: TritonTemplateCaller, CUTLASSTemplateCaller. rrrrjr`r descriptionrrcs2t||_||_||_||_d|_i|_dSr)rrrr`rrfailedr)rrrr`rrrrrs  zChoiceCaller.__init__rr rrrcs6|tjrtfddStjd|iddS)NcsSrrralgorrrr#sz(ChoiceCaller.benchmark..rr) to_callablerA/profile_bandwidth_with_do_bench_using_profilingr]rR benchmark)rrrrrrrszChoiceCaller.benchmarkcCrbrrcrrrr call_namer zChoiceCaller.call_namerrcCrbrrcrrrrrr zChoiceCaller.to_callablecCr)z Hash key for the underlying kernel. By default, we assume there are no runtime params, so kernel hash key defaults to choice caller's hash key. )hash_keyrrrrkernel_hash_keyszChoiceCaller.kernel_hash_keycCrbrrcrrrrrr zChoiceCaller.hash_keyr~cCrbrrcrrrrrur zChoiceCaller.output_node.) rr_choice_timings_fn_choice_timings_choicesoriginal_inputsr_output_plannable_make_kernel_renders)rr`rwr!r#rrrrrs zMultiTemplateBuffer.__init__rcCr)z^ Are all possible choices TritonTemplates or Extern Kernels with out variants )r-rrrroutput_plannable sz$MultiTemplateBuffer.output_plannablecCrr)r+rrrrr:rzMultiTemplateBuffer.choicesN hint_overriderdict[ChoiceCaller, float]cCs$||jvr |||j|<|j|Sr)r*r))rr0rrrchoice_timingss  z"MultiTemplateBuffer.choice_timingscallerrrccsZt|tjjjsJt||j|jksJ|j}||_z dVW||_dS||_wr) rr#rr%TritonTemplateCallerrr`rr)rr3renderrrrswap_as_triton_callers  z)MultiTemplateBuffer.swap_as_triton_callercCsRt|tjjjsJt|||jjksJ| |jj ks"J| |_ dSr) rr#rr%r4rrr`rrCrrr)rr3rrrfinalize_as_triton_caller-s z-MultiTemplateBuffer.finalize_as_triton_callertuple[ChoiceCaller, float]cCs&|j|d}t||jd}|||fS)N)r0r%)r2rr)rr0timings min_choicerrrget_min_choice5s  z"MultiTemplateBuffer.get_min_choicecallers-dict[Optional[int], TritonTemplateCallerBase]cCs0|D] \}}||j|<q|jd|_dS)z;Finalize with multiple callers for different hint overridesN)rrr.r)rr<r0r3rrrfinalize_as_triton_callers<sz.MultiTemplateBuffer.finalize_as_triton_callers) r`rrwrxr!r"r#r$rrrrr>)rr$r)r0rrr1)r3rrr)r3rrr)r0rrr8)r<r=rr)rrrrXrr^r/r:r2r\r]r6r7r;r>rrrrrr s      r cs2eZdZdfdd ZdddZdddZZS)CUTLASSTemplateBufferr`rrwrxrCallable[_P, _T]workspace_sizertemplaterqsupports_epilogue_fusionrrrcs&t|||||_||_||_dSr)rrrArBrC)rr`rwrrArBrCrrrrHs  zCUTLASSTemplateBuffer.__init__cCs|jdur|jSdSrrArrrrrnWrz(CUTLASSTemplateBuffer.get_workspace_sizecCs$|D] }t|ddqdSr)rkrkrr)rrmrrremulate_store_fnZs z&CUTLASSTemplateBuffer.emulate_store_fn)r`rrwrxrr@rArrBrqrCrrrrSr6)rrrrrnrErrrrrr?Gs r?cs,eZdZdfd d Zdfdd ZZS)CppTemplateBufferr`rrwrxrr@rBrqr(r rrcs&t|||||_||_d|_dSr)rrrBr(r)rr`rwrrBr(rrrr`s zCppTemplateBuffer.__init__cspt|jtr3t|jtsJt|j|jd}t|ts#Jt||j}t|ts1Jt||St Sr) rr`MultiOutputLayoutrrrrrrr)r first_outputr`rrrrms   zCppTemplateBuffer.get_layout) r`rrwrxrr@rBrqr(r rrr<)rrrrrrrrrrrF_s rFcs0eZdZdZ ddfdd ZdddZZS)CuteDSLTemplateBufferz Buffer for CuteDSL (CUTLASS Python DSL) template kernels. Similar to other template buffers but specialized for CuteDSL operations. Nr`rrwrxrr@rBr rrrrcst||||_|_g_|dur?tjdts'Jtjdjd jfdd|D7_dSdS)Nrcrrrrrrrrrz2CuteDSLTemplateBuffer.__init__..) rrrBrrrrwrrr!)rr`rwrrBrrrrrs"zCuteDSLTemplateBuffer.__init__rjcCrrr rrrrrkrz!CuteDSLTemplateBuffer.get_outputsr) r`rrwrxrr@rBr rrrrro)rrrrXrrkrrrrrrIzs  rIcsPeZdZdZ     d"d#fdd Zd$ddZd%ddZ d&d'd d!ZZS)(NVUniversalGemmBufferz Buffer for NVIDIA Universal GEMM kernels. Unlike CuteDSL templates which use Jinja templates, this generates simpler Python code that directly calls the cutlass_api library. rNr`rrwrxkernelr accumulator_typevariantrAr scale_type_a Optional[Any] scale_type_bswizzle_type_aswizzle_type_brrc sjtj||dd||_||_|g|_||_||_||_||_| |_ | |_ |j j |j j d|_|j|_dS)N)r) kernel_namemin_cc)rrrKrLrrArMrNrPrQrRmetadatarSrTkernel_metadata_make_kernel_renderr) rr`rwrKrLrMrArNrPrQrRrrrrs zNVUniversalGemmBuffer.__init__cCr)z#Return the workspace size in bytes.rDrrrrrnrz(NVUniversalGemmBuffer.get_workspace_sizerjcCrrr rrrrrkrz!NVUniversalGemmBuffer.get_outputsout_noder0rtuple[Any, Any]c sddlm}ddlm}g}|jD]}t|tr|j}t|tr#|j}| |qt |j }|||||j |j |j|j|j|j|j|jd fdd}|fS)z Create a kernel renderer for code generation. Returns (kernel, render) tuple where: - kernel: NVUniversalGemmKernel object with call_kernel() method - render: function that returns source code string r)NVUniversalGemmKernel) Placeholder) rSrrurVrLrArMrNrPrQrRcsSr)r5r render_kernelrrr5rz9NVUniversalGemmBuffer._make_kernel_render..render)Btorch._inductor.codegen.nv_universal_gemm.nv_universal_gemm_kernelrZtorch._inductor.utilsr[rwrr~r_rrr KERNEL_NAMErVrLrArMrNrPrQrR) rrXr0rZr[rr}rSr5rr\rrWs2       z)NVUniversalGemmBuffer._make_kernel_render)rNNNN)r`rrwrxrKr rLr rMr rArrNrOrPrOrQrOrRrOrrrSror)rXr r0rrrY) rrrrXrrnrkrWrrrrrrJs ! rJr)Sequence[Union[IRNode, Sequence[IRNode]]]TypeIs[Sequence[IRNode]]cCstdd|DS)Ncsrrrrryrrrrrxz#is_node_sequence..)r)rrrris_node_sequencerjrdc@szeZdZUded<d!ddZd"d d Zd#d dZed$ddZe d%ddZ d&ddZ d'ddZ e d d(d)ddZd S)*rrarwrrrrcCs |j|}t|ts J|Srrwrrr)rrinputrrr input_names zInputsKernel.input_namer"csttj}tj|jD]#}t|tr |fdd|Dq t|tr&q | | q ttjfdd| D}tj ||tdS)Nc3|] }|VqdSrrrTrrrr r)z/InputsKernel.get_read_writes..c3rhrrrrirrr  r) r9rBrHrrwrrr rrrrkr)rr'rfrrrirr#s    zInputsKernel.get_read_writesr$cCr%rr&rrrrrr(zInputsKernel.get_readsrrcCs~t|tr|j}t|tr|j}t|trt|tst|}t|tr)||St|t r0|St|t tfs=Jt ||Sr) rr~r_rrrer realize_inputunwrap_storage_for_inputTorchBindObjectrrrrrrrrls      z%InputsKernel.unwrap_storage_for_input%list[Union[IRNode, Sequence[IRNode]]]cCs@g}|D]}t|trdd|D}nt|}||q|S)NcSrZr)rrlrrrrr8rz/InputsKernel.unwrap_storage..)rrrrlr)rw inputs_newrrrrr1s   zInputsKernel.unwrap_storagercCrrrrrrrr/>r zInputsKernel.is_externcCrrrrrrrr)Ar zInputsKernel.num_readsFr{rcCsNttj}|jD]}t|tr|||O}q |D] }|||O}qq |Sr)r9rr!rwrrr,)rr{rr} inner_inprrrr,Ds   z!InputsKernel.get_free_symbol_usesN)rrrrrQrRr)rwrarror>rSrLrU)rrrrrgr#rrrlr[rr/r)rXr,rrrrrs       rc@s eZdZd ddZd ddZdS) NopKernelrrcCrrrrrrrr0Sr zNopKernel.is_no_opr$cCrlrr8rrrrrVrzNopKernel.get_readsNr>rR)rrrr0rrrrrrrRs rrc@s^eZdZdZeddd Ze ddddZed ddddZed ddZ d!ddZ d S)" ConcatKernelzn There isn't actually a real kernel for concat, we just change the storage for the upstream data. rwrxrSrrrc Cs|d}|d}t|d}dg}||g}d|kr)t|ks,JJtdt|D]Z}||} |||t| t|ksLJ|||ksVJ|||ks`Jtt|D]} | |krw|| | | || <qftjj || | | || <qf|||q3t |} t jrt| ||dj} tt|D]!}||} t| r| } t| trt| j| jrt|} nqtdd|D}tjjjd}t|tsJt||durtdd|Drt|} tdd|D}|dusJtdt|||| |d gd }t |}g}t!|D]{\}}t|t"t#fs.Jt||$|t%j&||||||dd }t|t'sLJt|t|j(tsZJt|j(|j(|t|j)t"rm|j)*}n|j)}t|t r|+r|}durt,|jrt-|s||.qt|dkrtj/|t0j1rtj2|tj3||_4|5|j(|_(tj6||S) z6 Create the concat kernel from inputs rr@csrr)rrTrrrrrz&ConcatKernel.create..FcssB|]}d|jvo|jdjtjdp|jdjtjdVqdS)rdr8N)rsr'r#r:r<rargrrrrs  css"|] }t|o |jVqdSr)rrrdrTrrrrs N)rrrrrdrr`rwr)7r!r rrrrrrmrrrwrrrAr)rr(rrrrrbrrrr,r current_noderrrrsrrrrorzrrrrwr_rr rdrcr3r5rCFOREACHregister_operation_listrrrr)rrwrSrrrP offsets_start offsets_endr input_sizerN output_striderr`any_input_is_storage_and_layout fx_node_argsrd concat_kernelrKop_namesr}rZinput_unwrappeddevrrrr`s                 zConcatKernel.createNrrrrsrrcCst|tr ||j|St|ttfsJt|t|jtrPt|jjt r*|jj s,dS|dur2dSt | t | kr@dSt ddt| | DSt|jdoct|jjtoct|jt S)NFTcsrZrr[r\rrrrr]z=ConcatKernel.can_realize_into_without_copy..r`)rr~can_realize_into_without_copyr_rrrr r`rbr/rrCrrrrExternKernelAlloc)rrrrsrrrrs*      z*ConcatKernel.can_realize_into_without_copyFr{rcCs t||Sr)rrr,r+rrrr,rz!ConcatKernel.get_free_symbol_usesc Cst|tst|rt|\}}t||d}t|ts Jt|t|tr,||j|St|trL| t |jds=J| ||rLt ||j_ |jStj|||ddt||Dd}|||S)Nr^r`cSrurrvrxrrrrryz-ConcatKernel.realize_into..r)rrerrarr~rzr_rrrrrRr`rrr!r rrr)rrrrsrfr`pwrrrrzs,        zConcatKernel.realize_intocCrrrrrrrrr zConcatKernel.should_allocate)rwrxrSrrrr)rrrrsrrrrLrU)rrrrsrrrr>) rrrrXrrrrXr,rzrrrrrrsZs q# !rscseZdZUdZdZded<ejedZ ded<dZ d ed <dZ d ed <dZ d ed <eje dZded<dZded<dZded<ejedZded<dZded<ejedZded<eje dZded<       ddfd$d% Zdd'd(Zdd*d+Zdd,d-Zdd.d/Z ddd3d4Zdd5d6Zddd7d8Zdd9d:Zdd;d<Zdd>d?Ze ddCdDZ!e"ddJdKZ#e"ddMdNZ$e"ddOdPZ%e"ddQdRZ&e"   SdddZd[Z'e" Sddd]d^Z(e" Sddd`daZ)e"ddbdcZ*e"ddddeZ+e"ddfdgZ,e"ddhdiZ-ddjdkZ.ddldmZ/dddqdrZ0ddsdtZ1ddvdwZ2dddydzZ3dd{d|Z4dd}d~Z5dddZ6dddZ7dddZ8dddZ9e:d SddddZ;dddZS)rz A class that represents Kernels which are not directly lowered to Inductor Loop Level IR, such as custom operators, or aten operators which we fallback to. rr constant_argsrhrrNOptional[ReinterpretView] output_viewr-python_kernel_namecpp_kernel_name Iterable[str]ordered_kwargs_for_cpp_kernelOptional[_OpOverloads] op_overloadzOptional[list[dict[str, Any]]]arg_propertieszdict[str, dict[str, Any]]allarg_propertiesz#Optional[dict[str, dict[str, Any]]]kwarg_propertiesz"dict[sympy.Symbol, pytree.KeyPath]unbacked_bindingszlist[MutationOutput]mutation_outputsrr`r0rwradict[str, Any] | Nonerrc sttj|||d||_|r|ni|_||_| |_||||| |_| i|_ g|_ t j j|_i|_dSNrw)rrrrrrset_cpp_kernel_nameset_python_kernel_namercollect_arg_kwarg_propertiesrrrmrryfx_noder) rrr`rwrrrrrrrrrrr>s"     zExternKernel.__init__rjcCs |g|jSr)rrrrrrk^rzExternKernel.get_outputsrcCrlrr8rrrrrmarz%ExternKernel.get_unbacked_symbol_defscCst|jtjjrdd|jjjDn ddtt|j D|_ t|jtjjr1dd|jjjDni|_ t|jtjjrW|j sJdd|jjjD|_ dd|jjjD|_ dSg|_ dS)NcSs$g|]}|js|j|j|jdqS))rrr) kwarg_onlyr real_typerrTrrrrhsz=ExternKernel.collect_arg_kwarg_properties..cSsg|]}iqSrrrrrrrrrcSsi|] }|j|j|jdqS))rr)rrrrTrrrrurz=ExternKernel.collect_arg_kwarg_properties..cSg|]}|jr|jqSrrrrTrrrr cSsg|]}|jr|qSr)rrTrrrrs )rrr#_ops OpOverload_schema argumentsrrrwrrr schema_kwargsrrrrrds*    z)ExternKernel.collect_arg_kwarg_propertiescCs$t|jtr||dSdSr)rr`rapply_constraintrrrrrr-s  zExternKernel.decide_layoutwrapperrrrScCsZt||\}}|r|||s|}|r+ddlm}|||dd}|||dSdS)Nr@)'set_kernel_post_grad_provenance_tracingT)r/)r` make_commenttry_get_kernel_namerrwrite_provenance_debug_handle)rrrS origin_str_detailed_origin_strr debug_handlerrrcodegen_comments  zExternKernel.codegen_commentcCrbrrcrrrrrcodegenr zExternKernel.codegencCs||_tjjrt|jtjjsdS|j}|jdurB|j dkr;|j dkr+|j ddn|j dd}d|d|_dS|jj|_dSdS)Natenr.rrz at::_ops::z::call)rrmr cpp_wrapperrrr#rr namespace _overloadnamerrreplacerr)rrrKopnamerrrrs      z ExternKernel.set_cpp_kernel_namecCsd||_|dur dS|j}|durdSt|tjjr"d|j|_dS|jddd|j|_dS)Nztorch.ops.higher_order.._ops..ops.r) rrrr#rHigherOrderOperatorrrr)rrrKrrrrsz#ExternKernel.set_python_kernel_namecCsddlm}|}r|jntjj}tjjr|jStjj r=t tjj |s-Jttjj |j dur4dStjj |j |S|jS)Nr@) CppWrapperCpu)codegen.cpp_wrapper_cpurr!rrmr device_type fx_wrapperrrrrrget_c_shim_func_name)rrdrrrrrs  z ExternKernel.try_get_kernel_namercCrr)rrhrrrget_kernel_namerzExternKernel.get_kernel_namerrr~cCs:tj||||||d}||S)N)rrrsrtrr) rrr!r rrrrr)rrrrr copy_inputszExternKernel.copy_inputrKr{rr ituple[Any, list[Any], list[Any], Callable[[Any, Any], Any], Optional[dict[sympy.Symbol, pytree.KeyPath]]]cs||d}t|\}gg}g}|D].}t|to"t|t dr.||qt|tr=tjj j j |dd}||qdfd d } fd d |D}|D] } t | rbt | d dqVg} |D]n} t| ts| tjjvr| tjj| qgt| ts| tjjvr| tjj| qgt| tr| | qgt| tjjjr| jj} | jjdkr| dusJ| tjj| qg| t| d dqg| | |\} }|| i|}d}tjj }r*tjj !d}t"}tjj#tj$j%j&ur |d}t'tj}|t(|tj|Wdn 1swYt)|||}t|t*t+fs5|gn|}|D]+}t|tj,rc|j-rct.j/scd}tjjj !dd}r_|d|}|tj_0q9|||| |fS)NrrA)rnew_tensor_argsrnew_non_tensor_argsrtuple[list[_T], dict[str, _T]]csdg}t|}t|}D]}|r|t|q |t|q t|}|dg|difS)Nrr)rxrrwpytreetree_unflattenr)rrr it_tensorsit_non_tensors is_tensorr) args_spec is_arg_tensorrrunflatten_args s z3ExternKernel.process_kernel..unflatten_argscr rrkrTrrrrrz/ExternKernel.process_kernel..Tr"r4)r rdr@zEsparsity not handled. Please file issue for sparse inference weights.rz Found from : )rrrrrr)1r tree_flattenrrrGeneratorStaterrmrrrcreate_symintnoderrarr constantstorchbind_constantsrm get_valuer#rirrrrr4default_generators clone_stater rryrsrr r^_higher_order_opseffects with_effectsr.r3r/rrTensor is_sparserAgraph_partitiondisable_cudagraphs_reason)rrKrr binded_args args_flat tensor_argsnon_tensor_argsrvrr example_args device_indexnew_args new_kwargsexample_outputrr node_meta_valctxexample_out_lir&msgrr)rrrrprocess_kernels            zExternKernel.process_kernelrec Cspt|ts Jt|t|tr|S|}tj|}|dus$J| }|dur]d|j vr]t|tt t fr]t|j tr]|j djtjdsS|j djtjdr]|t|n|tj|dd\}}|d}||}tjj||}tjj||} tjj||} t|| | } || krt d| | |t!t|j"t#|$|%|| | dd d S) z In order to pass this to an extern kernel we need a ReinterpretView not a View. This allows us to avoid some unneeded copies. Nrdrtrr|rz@convert_to_reinterpret_view failed: stride=%s offset=%s index=%sFrdr^)&rrrrerrmrrlrrrsrror`rr'r#r:r<rr,rrrBr>r rr? stride_vars offset_varrer=rrr_rbrr ) rr x_unwrap_viewr2x_unwrap_view_fx_node index_argsrr*rrXrcexpectedrrrconvert_to_reinterpret_viewlsj         z(ExternKernel.convert_to_reinterpret_viewcCs6|durtSt|ttjjjtfrt|dSt|t r@t t j tj|j||dWdS1s;wYt|trG|St|trR||jSt|trct||j|dSt|tr|t|rz||WStyYnwt|tr||St|t tfr|S|!|S)Nr)rrr^)"rrrrrrrrrrr:rmradd_tensor_constantr#rWr'r r!rr~rkr_rerrrrrrrr NonTensorObjrrnrrrrks@            zExternKernel.realize_inputcCsDt|rt|dkr|S|D] }|dkr|Sq||Sr)rrrCr)rrrrrrrequire_stride1s  zExternKernel.require_stride1Frrrrrrc s|dus dus JdvrsStrttrS|rGt|o-tj }tdd|r@t t j j jn||dStddd|dStttfr}|re|srr}tjr}dur{tSSt}trt|}trtdt|tr|r||srt|jrSttr|r|sLjrtjrɈSttrtjtrtjtst}rt|drt|jtsz!|j_|r|j ||dWSr|j!|dWSWn t"yYnwd} } durOt j j fd d t#t$D} | D] } t%j&j'(| d d qA|)tdd||d|rkt|siJS| r| durxduszJt%j&j'*| tSS) N)rr@TF)r#r*r+rr.zHthe MutationLayoutSHOULDREMOVE's real layout shouldn't be FlexibleLayoutr_r/cs4g|]}|dr|dr|qS)rr)rQrrrrrrrrrdsz0ExternKernel.require_strides..rr@)+rrrrrr2r1rrarrmrrsize_hints_or_throwrbrRr1rVrrjrkrnrvrr~r_rrerrrrrequire_stride_orderrequire_exact_stridesrrrr#rloweringslice_rr\) rrrrruse_current_stride_ordermutation_layoutrnr expanded_dims orig_sizerSrrrrequire_stridess                 zExternKernel.require_stridesrMcCs|j|dd|D|dS)NcSs$g|]}t|tjr|jjn|qSr)rr#SymIntrrrrrrrsz6ExternKernel.require_exact_strides..)rrr)rrrrrrrrsz"ExternKernel.require_exact_stridesrcCs|j|||dS)N)rrr)rrrrrrrrsz!ExternKernel.require_stride_ordercC ||tSr)rr;rnrrrrequire_channels_lastr9z"ExternKernel.require_channels_lastcCrr)rr=rnrrrrequire_channels_last_3dr9z%ExternKernel.require_channels_last_3dcCs,ddd}||r |S||t|S)Nrrrrc Ss@z|}Wn ttfyYdSw|tjjvotjj|jSr)rAttributeErrorrrmrr is_mkldnn)rrrrris_mkldnn_tensors  z9ExternKernel.require_contiguous..is_mkldnn_tensorrrrrrrrr)rrr rrrrs  zExternKernel.require_contiguouscCs||t|Srrrnrrrrequire_contiguous_stridessz'ExternKernel.require_contiguous_stridescCr rrrrrrrr zExternKernel.apply_constraintcCst|ts Jt|t|tst|}|jsJdt|}t|j}||krStd|j||t ||D]}|j|d}| ||vrJ||n|j|dq7|S)Nz/ExternKernel.arg_properties should not be emptyzv%s has %d unprovided positional arguments. Will check if they are in the keyword arguments or will use default values.rr) rrrrrrr=rrrr)rrrn_args n_pos_argsrarg_namerrrfill_non_provided_argss(   z#ExternKernel.fill_non_provided_argsr{rrc Cstjjrog}d}|r"|jr"t|jt|ksJddd|jD}t|jD]E\}}|durF|dus5J|||}|rC|dnd}nt|j|}|jr_|t|jkr_|j|dnd}| tjj ||q'|Sdd|jDS)NzDnames passed to codegen_const_args does not match self.constant_argscSsi|]}|d|qSr)rrurrrrrzz3ExternKernel.codegen_const_args..rcSrrrmrrval_to_arg_str)rrrrrrrz3ExternKernel.codegen_const_args..) rmrrrrrrrrwrrr) rr{rname_to_arg_propertiesrrproptype_rrrrcodegen_const_argss2  zExternKernel.codegen_const_argscCstjjr|jdur|g|j|j|j}d}n|j}d}g}t|D]4\}}tjjrN|j r6|t |j ks:Jd|j | d}| tjj ||q$| tjj |q$|rb|||S)NFTz-Invalid access to ExternKernel.arg_propertiesr)rmrrrrrwrrrrrrrrrrr)rrwneed_codegen_constant_argsrrrrrrr codegen_argss&zExternKernel.codegen_argsrcKsT||vr ||S||jvr|j|S|j|}dur#|dSt|d)zGiven an argument name, queries for values in (in order): 1. any provided kwargs for this function. 2. the class self.kwargs member. 3. any available default arguments in self.allarg_properties.Nrz not in self.allarg_properties)rrrrv)rrrrvrrrget_kwargs_values    zExternKernel.get_kwargs_valueskip_outcCstjjrO|jdurt|jdkrgSg}|jD]5}|r |dkr q||}t|t r0| |q|j dus7J|j |i d}| tjj ||q|Sdd|jD}|S)NrrrcSs(g|]\}}|dtjj|qSr}r)rkrrrrr;sz/ExternKernel.codegen_kwargs..)rmrrrrrrrrrrrrrrrr)rrrrrrrrrcodegen_kwargs's$     zExternKernel.codegen_kwargscCsT|jdur&|jj}t|dd}|dd}|ddd}|d|}|Sd}|S) Nrunknown_namespacerrrr@r unknown_op)rr^rrrsplit)rr^ op_namespaceop_namerrr get_op_nameAs   zExternKernel.get_op_namec Cstjr=tjjs?t|dkrdStjj|}tjj| }| }| d| d|d|d|d dSdSdS)Nrzassert_size_stride(rSr/) rA size_assertsrmrrrhrrcodegen_shape_tuplerCr&rr)rrrrr%rrrcodegen_size_assertsLs z!ExternKernel.codegen_size_assertsc Csttjr6tjjs8|}|tjjv}|}|r(|d|dt d|ddS|d|d|ddSdSdS)Nzassert_alignment(rSr/z # buffer z (op: z) is assumed to be not aligned) rAalignment_assertsrmrrrr5r&rra)rrralignedr%rrrcodegen_alignment_assertsXs z&ExternKernel.codegen_alignment_assertscCs@tjjrtjjr dS||}|d|d|ddS)zc Track outputs of fallback operators if config.test_configs.track_memory_lifecycle Nz track_tensor(z, 'z')) rA test_configstrack_memory_lifecyclermrr"write_memory_track_allocation_oncerr)rrrrrrcodegen_memory_trackingfs z$ExternKernel.codegen_memory_tracking'tuple[list[Sequence[Expr]], list[Expr]]cCs|}|}|gg|fS)zD get output sizes and strides, for template_codegen )rrC)rrrrrrget_group_strideqs zExternKernel.get_group_stridetuple[Expr, Sequence[Expr]]c stjj|}|}fdd|D}ddtt|Dttt||jdd}ddt |DfddttD}fd d|D| }|}tjj ||g\}}} t d \} t t|fd d|D} tt|| }|t|fS) zC Manually get canonicalization of the output index cr r)rrT)rrrrrz-ExternKernel.canonicalize..cSsg|] }td|qS)r)rfrrrrrrT)r&rcSrrrrrrrrrz-ExternKernel.canonicalize..crrrrrrrrrcrrrr)rrrrrccrrrrT)add_varrrrr)rmrrrrCrrr+r@rr rrLrrrirr\r) rr4rX index_orderrrr new_sizesrrr replacementr)r5rrrr canonicalizezs$   zExternKernel.canonicalizer{cCsP|rtnt}t||}|jD]}|||O}q|jD]}|||O}q|Sr)maybe_free_unbacked_symbolsmaybe_free_symbolsrr,rrr)rr{maybe_get_symbolsrrvrrrr,s   z!ExternKernel.get_free_symbol_usescsPtdd}d|g}|fddtD7}|dj|S)Nrzpython_kernel_name=cs$g|]}|jdt|jqSr})rr)rrZrrrrsz(ExternKernel.__str__..r)rrYfieldsrrr)rrSrrrrrs   zExternKernel.__str__rNNNNrN)rr-r`r0rwrarrrrrrrr-rr-rrrrrrrorpr6r)rrrrSr-rrrrrrrrr-rr)rr-rrrArH)rrrr~)rKr{rr rr rr)rrrrer)NNF) rrrrrrrrrrrL)rrrrMrrrr)rrrrrrrr)rrrrrr)r{rrrrr)rrrr rr )rrrr)rr1)rr3rU)?rrrrXrrrYrZrrrrrrrrrrrrrrrkrmrr-rrrrrrr[rrrrrkrrrrr r rrrrrrrr r&r)r,r0r2r9rXr,rrrrrrrr"s           %       ~ E #  $        $ !      rcsBeZdZdddZ       dd fdd Zd!ddZZS)"ExternKernelOutrrrrrcC||dSr)generate_extern_kernel_outrrrrrrzExternKernelOut.codegenrNr`rrwrxrrrOptional[dict[str, Any]]rrrr-rrrrc s`||} t| tsJt| td|| ||pid|||| tj||_ tj |dSr) rrrrrrrmrrrr) rr`rwrrrrrrrunwrapped_inputsrrrrs  zExternKernelOut.__init__rcCrrrrrrrrr zExternKernelOut.should_allocater?r>)r`rrwrxrrrrErrrr-rr-rrrrrrr>)rrrrrrrrrrrrBs rBceZdZd fdd ZZS) RandomSeedscountrrrrrcsFttj}tjt|tj|gdg|j|j|ggddtj j ddS)Nrzaten.randint.low_outzat::_ops::randint_low_out::call)r`rwrrrr) r#rr^rrrbrrrrandintlow_out)rrIrlimitsrrrrs  zRandomSeeds.__init__)rIrrrrrrrrrrrrrrrHrHcsJeZdZdddZ      dd fdd Zd!ddZd"ddZZS)#rrrrrrcCrCr)generate_extern_kernel_allocrrrrrrzExternKernelAlloc.codegenrNr`r0rwrxrrrrErr-rrrrc sp||} tdd| DsJtd|ttt| ||pid|||| g|_tj ||_ tj |dS)Ncsrrrcrrrrrrxz-ExternKernelAlloc.__init__..) rrrrr rrrrmrrrr) rr`rwrrrrrrrFrrrrs"  zExternKernelAlloc.__init__rcCrrrrrrrrr z!ExternKernelAlloc.should_allocatecCrbrrcrrrrrr z"ExternKernelAlloc.apply_constraintr?)rNNNrN)r`r0rwrxrrrrErr-rr-rrrrrrr>r6)rrrrrrrrrrrrrs  rcsJeZdZdZdfd d Zdd d ZdddZdddZdddZZ S)rzP An output buffer that represents the mutation of a pre-existing buffer r`r0 mutated_noder mutating_noder_rrcsDtjd|d|}tj||g|_||_tj||_ dSr) rrrrmrrmmutation_namesrQrr)rr`rPrQmutated_node_namerrrr"s  zMutationOutput.__init__cCrr)rQrrrrr,rzMutationOutput.get_defining_oprcCrr)rRrrrrr2/rz!MutationOutput.get_mutation_namesrcCrrrrrrrr2r zMutationOutput.should_allocaterxcCs |}dddd|DDS)NcSg|]}|dur|qSrrrrrrr7s z7MutationOutput.get_mutation_buffers..css|] }tj|VqdSr)rmrtry_get_bufferrrrrr9r)z6MutationOutput.get_mutation_buffers..)r2)rrRrrrget_mutation_buffers5s z#MutationOutput.get_mutation_buffers)r`r0rPrrQr_rrrrWr>rrx) rrrrXrrr2rrVrrrrrrs  rcs`eZdZUdZiZded<edd d Zedd d Zdfdd Z dddZ dddZ Z S) TMADescriptorad An IR node representing a generic host-side TMA descriptor in the Triton API Mostly useful for user-defined Triton kernels relying on host-side TMA; but can, in principle, be used for Inductor's Triton templates, too. See TMADescriptorExperimental and TMADescriptorStable for the two implementations (the old API and the new API) zdict[Any, TMADescriptor]_CACHErWrtma_metatuple[str, tuple[Any, ...]]rcCsTt|dksJ|ddkrt|g|dRS|ddks Jt|g|dRS)Nrr experimentalr@r)rTMADescriptorExperimentalTMADescriptorStable)rrWrZrrr _create_implLs  zTMADescriptor._create_implcCs2t||f}||jvr||||j|<|j|Sr)idrYr_)rrWrZr&rrrrWs   zTMADescriptor.createrwrrrcsVtdtt||dttt|t|d||_ t j ||_ t j |dS)Nr^)rrrRrerr rrrrWrmrrrr)rrWrwrrrrr`s zTMADescriptor.__init__rrrcCrCr)generate_tma_descriptorrrrrrwrzTMADescriptor.codegencCrr)rWrrrr get_tensorzrzTMADescriptor.get_tensor)rWrrZr[rrX)rWrrwrrrrrr?rK) rrrrXrYrrr_rrrrbrrrrrrX>s     rXcs&eZdZdZ ddfd d ZZS)r]z the new host-side TMA Descriptor API: (the ones obtained via create_{1d,2d}_tma_descriptor calls). See also TMADescriptorStable for the new API. NrWrr`list[Union[int, torch.SymInt]] block_dims element_sizerrrcst|dvsJt|t|ksJ|dur|j}||_||_||_t|j|_|g}g|j|j|j}tj|||ddS)N)r@rrWrwr) rr rr`rdrer%rr)rrWr`rdrerwrrrrrs*   z"TMADescriptorExperimental.__init__r) rWrr`rcrdrcrerrrrrrrXrrrrrrr]~s r]cs"eZdZdZdfdd ZZS) r^z the new host-side TMA descriptor API (the ones obtained via TensorDescriptor.from_tensor). See also TMADescriptorExperimental for the old API. rWr block_shaperccs||_tj||g|ddS)Nrf)rhrr)rrWrhrrrrs  zTMADescriptorStable.__init__)rWrrhrcrgrrrrr^sr^cs(eZdZdfd d ZdddZZS)SubgraphBufferr`rrrjrkrlexample_inputs list[Any] subgraph_namerc s,td||||_||_tj||_tj|tj |j|||_ t |j s,Jt |j }|D]}||j j|j<|j j|jq3dd|D|_ddlmm}t|j 0|jdddd|j j|jWdn1swwYWddSWddS1swYdS)NcSg|]}|jqSrr)rsym_varrrrrz+SubgraphBuffer.__init__..rFATEN) max_autotunemax_autotune_gemmmax_autotune_gemm_backends)rrrkrjrmrrrr make_subgraphsubgraphrdrwr~r graph_input_namesr sym_inputstorch._inductor.configrrAset_graph_handlerrrun) rr`rrkrjrlrwsym_inpinductor_configrrrrs0  "zSubgraphBuffer.__init__rrrrrcCsRGddd}t|jsJdd|jD}|||jg|j||jgdS)Nc@r)z,SubgraphBuffer.codegen..CodegenGraphrrscSs||_|j|_dSr)rr)rrrrrrs z5SubgraphBuffer.codegen..CodegenGraph.__init__N)rrs)rrrrrrrr CodegenGraphrr}cSg|]}|qSrrrr&rrrrrz*SubgraphBuffer.codegen..)rdrw'codegen_subgraph_with_flattened_outputsrurwr)rrr} outer_inputsrrrrs zSubgraphBuffer.codegen) r`rrrjrkrlrjrkrlrr?rrrrrrrrrrris$ricsneZdZd!ddZed"dd Zed d#d$fdd Zd%ddZd&fdd Z d'ddZ d(dd Z Z S))UserDefinedTritonKernelr(tuple[Kernel, Any, list[str], list[str]]csddlm}ddlm}||jg}g}g}t|retdr0|fddj Dn tds7J|j tdrRj D] }| j j|qEn td sYJ|jj}j |||fS) Nr) Autotuner)kernel_side_table restore_idxc3s|] }jj|VqdSr)r arg_namesrrKrrrrjzBUserDefinedTritonKernel.get_kernel_and_metadata.. restore_value reset_idx reset_to_zero)triton.runtime.autotunerr*torch._higher_order_ops.triton_kernel_wrapr get_kernel kernel_idxrrrrrrrrrrconfigs)rrrrrestore_value_argsreset_to_zero_argsrrrrget_kernel_and_metadatas,           z/UserDefinedTritonKernel.get_kernel_and_metadatarrrrc sddlm}\}}}}|||j||j\}}} } fddjD} dd|jDdd|jD} tfdd | D} g}g}g}g}t | t t d | D]w\}}|| vre|reqY||||t|tr||||qYt|ttttjfr|||t|qY|| vr|d |tqY|d ur |r|d |tqY||qYtd t|d||||j||||||| djjd d S)YOverrides the parent member. See https://github.com/pytorch/pytorch/issues/151692r)triton_version_uses_attrs_dictcsi|]}||qSrrrrrrrr)rzz3UserDefinedTritonKernel.codegen..cSrmrrrrrrrr,roz3UserDefinedTritonKernel.codegen..cSrr) is_constexprnumrrrrr-rc3s|]}|VqdSrrr)rrrr.rz2UserDefinedTritonKernel.codegen..rrANzUnsupported arg type: r.T) arg_typesraw_argsraw_keys triton_meta inductor_metar6roriginal_fxnode_name) r_rr!define_user_defined_triton_kernelrgridrparamsr9rVrrrrepeatrrrrr rrrrrrrrrgenerate_kernel_callr!rr)rrrrKrrrnew_namerrextra_launch_args named_args constexprsconstexpr_namesrrraw_keys_filteredraw_args_filteredrrvr)rrrr s               zUserDefinedTritonKernel.codegenFr{rrcst|t|j|BSr)rr,r'rr+rrrr,hs z,UserDefinedTritonKernel.get_free_symbol_usescCrlrr8rrrrrmrrz0UserDefinedTritonKernel.get_unbacked_symbol_defsrrrr tma_descriptor_metadatar kernel_argscszg}i}g}D]0\}} t| tr1t| } ||vr't| ||} || | ||<q || | ||<q t |dksCJ|d _ t|t sUJt |tdtj d|t|||_|_\} } } } t| dszJfdd| jD_ddlm}t | dkr| djni}fdd|| i||D_fddjD_tjdS) Nrrrcsg|]}|vr|qSrrrurrrrrz4UserDefinedTritonKernel.__init__..)identify_mutated_tensorscrrrrr&rrrrscs g|] }ttjd|qSr)rrfrrrrrrs) rrr~rrlrkrXrrrr!rrrrrrfrrrrrrrrrr mutable_argsrrmrr)rrrrrrwrrrrr&rKrrrautotuned_kwargsr)rrrrusT           z UserDefinedTritonKernel.__init__rjcCs t|jSr)rrrrrrrkr(z#UserDefinedTritonKernel.get_outputsrcCrrrrrrrr!rz"UserDefinedTritonKernel.get_device)rrr?rLrUrp) rrrr rrrrrrrorC) rrrrrrrXr,rmrrkr!rrrrrrs  Z  ?rcsJeZdZdZdddZdd d Zdd d ZdddZdfdd ZZ S)InplaceBernoulliFallbackE This needs to be a custom class to handle mutation properly rrrrrc Cstdd|jDs Jdd|jD\}tjjr5||d|ddtt |j d|j dS||d|ddtt |j d|j dS)Ncsrrrcrrrrrrxz3InplaceBernoulliFallback.codegen..css|] }tt|VqdSr)r rrrrrrrrrrSz, NULL)r/) rrwrmrrrrrrreprrending)rrrrrrrs,,z InplaceBernoulliFallback.codegenrcCrrrrrrrrr z(InplaceBernoulliFallback.should_allocatercC |dgSrrgrrrrr2rz+InplaceBernoulliFallback.get_mutation_namesrcCrlrr8rrrrrmrz1InplaceBernoulliFallback.get_unbacked_symbol_defsrr{rrrr csVtjdt|d||g||dtj|tj ||_ tj |dS)Nrr) rrrfr!rrmrrmrrrr)rrrrrrrrs  z!InplaceBernoulliFallback.__init__r?r>rWrp)rr{rrrr rr rrrrXrrr2rmrrrrrrrs    rcs\eZdZdZd ddZd!d d Zd"d d Zd#ddZd$fdd Ze d%d&ddZ Z S)'InplaceCopyFallbackrrrrrrcCs |\}}}||||dSr)rcodegen_device_copy)rrrsrr non_blockingrrrrszInplaceCopyFallback.codegenrcCrrrrrrrrr z#InplaceCopyFallback.should_allocatercCrrrrrrrr2rz&InplaceCopyFallback.get_mutation_namesrcCrlrr8rrrrrmrz,InplaceCopyFallback.get_unbacked_symbol_defsr`r0rwrxrrcsJtjd|||dddtj|dtj||_tj|dS)Nz aten.copy_aoti_torch_copy_)rrr) rrrmrrmrrrr)rr`rwrrrrrszInplaceCopyFallback.__init__Frsrrrrcs6fdd||fD}|f}tt|d||}|S)Ncr rrrrrrrrz.InplaceCopyFallback.create..r)rrfr!)rrsrrrrwrrrrrr s zInplaceCopyFallback.creater?r>rWrp)r`r0rwrxrrrrrL)rsrrrrrrrr) rrrrXrrr2rmrrrrrrrrrs    rc@sBeZdZdZdddZdd d Zdd d ZdddZdddZdS)MutatingFirstArgExternKernelrrrrrrcCsXt|jsJgdd|jDtt|j}||dd|d|jdS)Ncss|]}|VqdSrrrrrrr"rz7MutatingFirstArgExternKernel.codegen..rrSr/) rdrwrrrrrrr)rrargrefsrrrrs z$MutatingFirstArgExternKernel.codegenrcCrrrrrrrr)r z,MutatingFirstArgExternKernel.should_allocatercCrrrrrrrr2,rz/MutatingFirstArgExternKernel.get_mutation_namesrcCrlrr8rrrrrm/rz5MutatingFirstArgExternKernel.get_unbacked_symbol_defscCrrrrrrrhas_side_effects2r z-MutatingFirstArgExternKernel.has_side_effectsNr?r>rWrp) rrrrXrrr2rmrrrrrrs   rcrG) ResizeStorageBytesvariablerrPrrrcst|ts Jdtjdt|d||g|fdtj | tj ||_ tj |d|_d|_t|tttfsFJt|tjj|j dS)NzTODO: dynamic shapesr)rz"inductor_ops.resize_storage_bytes_z&torch::inductor::resize_storage_bytes_)rrrrrfr!rrmrrmrrrrrrrrr~rnever_reuse_buffersrr_)rrrPrrrr7s   zResizeStorageBytes.__init__)rrrPrrrrMrrrrr6rNrcs(eZdZd fdd Zd d d ZZS) SetSourceTensorKernel self_tensorrstorage_tensorrrcs|tj|||gdtjjjjdt |t t t fs$Jt |tjj|jtjj|tjj||}tt|d||tt|d||g|_dS)Nz!torch.ops.aten.set_.source_Tensor)rrr)rrrrr#rkrset_ source_Tensorrrrr~rrmrrrr_rr!rrfr)rrrrrrrrIs"  zSetSourceTensorKernel.__init__rcCs|d|dgSrrrrrrr4]rz2SetSourceTensorKernel.get_inputs_that_alias_output)rrrrrrrW)rrrrr4rrrrrrHsrcsReZdZdZd!ddZd"d d Zd#d d Zd$ddZdddd%fdd ZZ S)&ScatterFallbackz This needs to be a custom class to handle mutation properly. This class handles both aten.scatter_ and aten.scatter_reduce_. It also handle the case `src` being a scalar properly. rrrrrcCrCr)generate_scatter_fallbackrrrrrhrzScatterFallback.codegenrcCrrrrrrrrkr zScatterFallback.should_allocatercCs"|jd}t|ts J|gSrre)rr}rrrr2ns  z"ScatterFallback.get_mutation_namesrcCrlrr8rrrrrmsrz(ScatterFallback.get_unbacked_symbol_defsNTrU include_selfrr{rrrSrrrrrUr-rc st|t_jrfdd|||fD}|f} nfdd||fD}||f} tjdt|d|| ||dt|ddg|dt j | t j _t j dS) Ncr rrrrrrrrz,ScatterFallback.__init__..cr rrrrrrrrrrrUr)rrr)rr~ src_is_tensorrrrfr!rrrmrrmrrrr) rrrrSrrrrUrtensorsrrrrrvs&   zScatterFallback.__init__r?r>rArp)rr{rrrSrrrrrrrUr-rrrrrrrrrras     rcsJeZdZdZdddZdd d Zdd d Zd ddZd!fdd ZZ S)"IndexPutFallbackzQ This needs to be a custom class to handle mutation and indices properly rrrrrcCrCr)generate_index_put_fallbackrrrrrrzIndexPutFallback.codegenrcCrrrrrrrrr z IndexPutFallback.should_allocatercCrrrrrrrr2rz#IndexPutFallback.get_mutation_namesrcCrlrr8rrrrrmrz)IndexPutFallback.get_unbacked_symbol_defsrtorch._ops.OpOverloadrrr0rkrr accumulater c s|_dd|D}fdd||g|D}d}tjdt|d||fd||dtj dtj _ tj dS) NcSrTrrrrrrrrz-IndexPutFallback.__init__..cr rrrTrrrrraoti_torch_index_put_outrzaten.index_put_)rrrr) r0rrrfr!rrmrrmrgrrr) rrrr0rr valid_indicesrrrrrrs   zIndexPutFallback.__init__r?r>rWrp) rrrrr0rkrrrr rrrrrrrrs    rc@s$eZdZeddd Zdd dZdS) DeviceCopyrrrrrrrc Cs(|}|dus J|s:t|tjjvr:tdd|Dr:tj j s:tjj r5tj |tj || |Stj |tj |td|f}t|}d}|r\|}t|jog|jdkog|}|jdkost|jos|}|rt|rd|_tt|||||d||g|S)Ncss|] }|tjjvVqdSr)rmrrr"rrrrr)z$DeviceCopy.create..zDeviceCopy in input programr3TrQ)r!r/rrmrrrrrA aot_inductoruse_runtime_constant_foldingradd_device_infor1r\rrrrCrdrrrrdrrbr rk) rrrrx_devicerris_destination_pinnedis_source_pinnedrrrrsL          zDeviceCopy.createrrrrcCs\|}t|dks J|jr||d|j|ddS||d||ddS)Nrrr@)rrrrr)rrrrrrrszDeviceCopy.codegenN)rrrrrrrrr?)rrrrrrrrrrrs /rcs`eZdZdZdddZd ddZd!fdd Zd"ddZed d#d$ddZ d%ddZ Z S)&DynamicSelectStorageOffseta The result of computing a dynamic selection index is determined as follows: when the index in the select operation is unbacked, the actual index calculation is ambiguous for negative indices (index + size) versus non-negative indices (just index). To resolve this, we allocate an unbacked SymInt to represent the storage offset and decompose the select operation into a call to as_strided, computing the storage offset at runtime with this node. rr$cCrlrr8rrrrr rz$DynamicSelectStorageOffset.get_readsrcCrrrrrrrrr z*DynamicSelectStorageOffset.should_allocateunbacked_offset_symbol sympy.Symbolr base_offsetUnion[sympy.Symbol, int]base_dim_striderrrcsDtdttddg||_||_||_||_||_ ||_ dSNr3r) rrrfr#rrrrrrr)rrrrrrrrrrrs  z#DynamicSelectStorageOffset.__init__rcC t|jgSr)r9rrrrrrm$rz3DynamicSelectStorageOffset.get_unbacked_symbol_defsFr{cCrr)r'rr+rrrr,'rz/DynamicSelectStorageOffset.get_free_symbol_usesrrrcCs|j||jddS)Nrx)codegen_dynamic_select_indexrrrrrr-rz"DynamicSelectStorageOffset.codegenrRr>)rrrrrrrrrrrrrrrprLrUr? rrrrXrrrrmrXr,rrrrrrrs   rcs`eZdZdZdddZdddZd fdd Zd!ddZed d"d#ddZ d$ddZ Z S)%DynamicSliceSizeac Computes the output size of a slice call, handling the correct semantics in codegen. We do this for flexible handling for unbacked indices (to not data-dependent error). Slicing has 4 semantics for indices, i.e. x[start:] could be: 1) start < -x.size(0) -> x[0:] # negative out-of-bounds 2) start in [-x.size(0), 0) -> x[x.size(0) + start:] # negative slicing 3) start in [0, x.size(0)) -> x[start:] # standard slicing 4) start >= x.size(0) -> empty slice # positive out-of-bounds If the appropriate semantics are known beforehand, the output size is computed based on the start & end indices. If not (with unbacked indices), a new unbacked symbol is created to represent the output size, and codegen handles computing the correct case. rr$cCrlrr8rrrrrArzDynamicSliceSize.get_readsrcCrrrrrrrrDr z DynamicSliceSize.should_allocateunbacked_size_symbolrrrrrrcs>tdttddg||_||_||_||_||_ dSr) rrrfr#rrrrrr)rrrrrrrrrrGs  zDynamicSliceSize.__init__rcCrr)r9rrrrrrmWrz)DynamicSliceSize.get_unbacked_symbol_defsFr{cCst|j|t|j|Sr)r'rryrr+rrrr,Zs  z%DynamicSliceSize.get_free_symbol_usesrrrrcCrCr)codegen_dynamic_slice_sizerrrrrbrzDynamicSliceSize.codegenrRr>) rrrrrrrrrrrprLrUr?rrrrrr1s   rcsJeZdZdZdddZdddZdfdd ZdddZdddZZ S)rz; The result of a call to aten._local_scalar_dense. rr$cCrlrr8rrrrrkrzDynamicScalar.get_readsrcCrrrrrrrrnr zDynamicScalar.should_allocatesymrkeypathpytree.KeyPathr_rrcs<|tdttdd||g||_||_dSr) rrrrfr#rrrr)rrrr_rrrrqs  zDynamicScalar.__init__rcCrr)r9rrrrrrm{rz&DynamicScalar.get_unbacked_symbol_defsrrrcCrCr)codegen_dynamic_scalarrrrrr~rzDynamicScalar.codegenrRr>)rrrrr_rrrrpr?) rrrrXrrrrmrrrrrrrfs   rcs`eZdZdZdddZdddZdfdd ZdddZed ddddZ d ddZ Z S)!rz5 The result of a call to aten._assert_scalar rr$cCrlrr8rrrrrrzAssertScalar.get_readsrcCrrrrrrrrr zAssertScalar.should_allocatescalarrorrrcs,tdttddg||_||_dSr)rrrfr#rrr)rrrrrrrs zAssertScalar.__init__cCrrrrrrrrr zAssertScalar.has_side_effectsFr{rcCrr)r'rr+rrrr,rz!AssertScalar.get_free_symbol_usesrrrc CstjsdStt|jdd}tjjrdStjjr;d|d}tjj j |j dd}| d|d|j d|d dStjj j|j dd}| d |d | d t|j d| |d dS)NFrzzstd::to_string(r/)r|zif (!(z()) { throw std::runtime_error("Expected z but received " + z); }zif not (z):z raise RuntimeError(z = None)rAscalar_assertsrwrxr,rmrrrrcodegen_cpp_sizevarrrrcodegen_python_sizevarrr)rrsymbol symbol_strsizevarrrrrs& zAssertScalar.codegenrRr>)rrorrrrrLrUr?) rrrrXrrrrrXr,rrrrrrrs   rc@seZdZUded<ded<dS)ExternKernelNoderrzexport_schema.NoderNrrrrrrs  rcseZdZdZ dfdd Zd?ddZd@ddZdAd!d"Ze dBd&d'Z dCd)d*Z dDd,d-Z dDd.d/Z d0d1Zed?d2d3Ze dEd7d8ZedFd:d;ZZS)GFallbackKernelz A class that represents a fallback kernel for handling operators that are not directly support by inductor. It currently supports functional ops, view ops, inplace aten ops, and mutating ops that are auto-functionalizable. Nrr`r0rKr{rrxnontensor_argsrrrrrrEr,Optional[dict[sympy.Symbol, pytree.KeyPath]]rrcsxtj|t|t||dd_|pi_t|tjjtjj fs.Jd|dt |d|_ |_ |dur:in|_ jdusDJtjjg_g_tj tjj r[dSdj vrddSj j}tjjj r|j|ddSdd d } |jrtj s| j stdj  jj\} }dfdd } tjj || |D] \} } | | | qdS)NrF#Fails to create FallbackKernel for r. not supported_c10d_functionalrrorrrcSs2tj|tjjjpt|dotjjj|jvS)N py_kernels)r#_C%_dispatch_has_kernel_for_dispatch_keyr DispatchKey Functionalizerrrnrrrhas_functionalize_impls  z7FallbackKernel.__init__..has_functionalize_implz'NYI: Can't generate FallbackKernel for infotorch._C.Argumentrvr rcstjtjrt|ttfsJt|tjr#t|ttfr#J|dur)dSjdur0dSdfdd }t jrO|durK|D]}||qDdSdStjsWJ||dS)Nr&rrrcsLj|jdusJjjr$jtt|d|dSdSr) alias_namesrr alias_infois_writerrrfr!r&)rrrr add_alias1szPFallbackKernel.__init__..handle_aliasing_and_mutation..add_alias)r&rrr) rrr#ListTyperr library_utilsis_tensor_like_typer is_tensorlist_like_type)rrvr optional_tensor_argr)rrhandle_aliasing_and_mutation#s"     z=FallbackKernel.__init__..handle_aliasing_and_mutation)rorrr)rrrvr rr)!rrruse_runtime_dispatchrrr#rrrrrrrrrmr warn_fallbackrrRrr_libraryrWmutates_and_returns_first_argrr is_mutabler$rrwr zip_schema)rr`rKrrrrrschemarrrrrvrrrrsV      zFallbackKernel.__init__r"csHt}|jtjjjur"|jD]}t|t r!| t | }q|Sr)rr#rr#_prims rng_primsgraphsafe_run_with_rng_staterrr with_readrBrr)rr/rvrrrr#Es    zFallbackKernel.get_read_writesrrrcCs|||jt|ddSNr)(codegen_unbacked_symbol_defs_for_outputsrrrrrrrcodegen_unbacked_symbol_defsQsz+FallbackKernel.codegen_unbacked_symbol_defsrcC>t|dd}rttjjj|}|dusJt|StSrrr4rmrrrr9rArrresolvedrrrrmV   z'FallbackKernel.get_unbacked_symbol_defsrcstjGdddt|jsJfdd|jD}|||j\}}tjjrDt |j t j j rD|||}ddt|j jj|D}ndd|D}|j||S)Nc@ eZdZUded<dddZdS) z)FallbackKernel.codegen_args..Shimr refrrcSrr)r'rrrrrerz2FallbackKernel.codegen_args..Shim.__repr__NrH)rrrrrrrrrShima r(csg|]}|qSrrrTr(rrrirz/FallbackKernel.codegen_args..cSs"g|] \}}tjj||jqSr)rmrrrr)rparamrrrrrmscSrrrrTrrrrrr)rY dataclassrdrwrrrmrrrrr#rrrrrrrr )rrrrrr*rr`s  zFallbackKernel.codegen_args Optional[Sequence[torch.Tensor]]rr cCs|r dd|Dnd}|r|sJdd|D}|dSt|tjr%|jSt|ttfr`tdd|D}dd|D}t|dkrF|dS|D]}t|tjsRJt|j r[|SqH|dSdS) NcSsg|] }t|ts|qSr)rrmrrrrr}rz.FallbackKernel.find_device..cSsg|] }|r|qSrrrurrrrrnrcss|] }td|VqdSr)r find_devicerTrrrrs  z-FallbackKernel.find_device..cSsg|]}|r|qSrr)rrrrrrrr@) rr#rrrrr9rrdr)rrnon_torch_bind_tensor_argsdevices device_setrrrrr.xs0   zFallbackKernel.find_devicercCsddlm}||jS)Nr) is_impure)torch._library.utilsr2r)rr2rrrrs  zFallbackKernel.has_side_effectsrcCsrt|jtjjtjjfsJd|jdt|jdt|jtjjs6d|jvr6|jjj r6t |jr6gS|j S)Nrr.rr) rrr#rrrrrrrr$rrrrrr4s   z+FallbackKernel.get_inputs_that_alias_outputcCst|jdks J|jSr)rrRrrrrr2sz!FallbackKernel.get_mutation_namesc sftdjttsJtjj \} |}fddj D}j}t j js=g||Stdg}|||}dd d t|tjjjrb||d |dj}n|jj}t|dkrjrrjnj}|d j}||g} n fddt|jD} jdusJttjj || idd} t j!"| g||S)a ProxyExecutor Design Note We export the ExternFallbackNodes (for custom ops) into a serialized file and run it with a host side proxy executor to address the ABI problem This is currently only implemented for fbcode. Eventually, we will also make this work for OSS. Detailed design doc can be found at https://docs.google.com/document/d/1wC4DOZFaYym2t1Esz0X5yxlLI3RDnSiyRbUus3bkJ64/edit?usp=sharing z4Extern kernel node added for node %s with target %s.csg|] }j|fiqSrrrrrrrrrz.N return_type6Union[torch.TensorType, torch.ListType, torch.JitType]rmUnion[IRNode, Sequence[IRNode]]rexport_schema.ArgumentcSsnt|tjtjfrE|}t|ttfrt|dksJ|d}t|tjr8t|ts+Jtj j tj | ddS|dus>Jtj j ddSt|tj rjt|tjrjt|ts^Jt|tj j dd|Dd St|tjrt|tjr|durtj j tjj ddd St|tsJtj j tjj tj | ddd St|tjrtj j |d Std t|) Nr@rr) as_tensorT)as_nonecSsg|] }tj|dqS)r) export_schemaTensorArgumentr)rrrrrrszZFallbackKernel.export_extern_kernel_node..handle_single_output..) as_tensors)as_optional_tensor)as_intzUnsupported return type )rr# TensorTypeNoneTyperrrrr;rprr<rr getElementTyperr OptionalTypeOptionalTensorArgumentIntType RuntimeError)r5rmrrrrhandle_single_outputsR     zFFallbackKernel.export_extern_kernel_node..handle_single_outputrr@csg|] \}}|j|qSr)r)r return_schemarm)rGrrr s)r^rwrrU)rr)r5r6rmr7rr8)#r=rrrrrrrrwrrrrmraot_moder#serialize_inputsr#r torchbind CallTorchBindrreturnsrrrrrrrr;r7rextern_kernel_nodesr) rrordered_kwargsr^ serializernamed_argumentsrMrr5output_argumentsrr)rGrrrexport_extern_kernel_nodesN      5     z(FallbackKernel.export_extern_kernel_nodecsj}|dus J|jdkr5t|tjjsJt|tjj r4ddl m }t ||vr4t d|d_n|jdkrHt|tjjsGJt|n tjj rS|tjjv_tjj rt|tjjrjsdfd d jj\}t|fddjD}tfddt||jjD_|jrƈ}jdusJjdusJ| !jfddj|j"rj"nj#n|$tj%t&r'|(|)|*|dS)rNrr)inductor_fallback_opszG%s is missing a c-shim implementation, using proxy executor as fallbackT _quantizedr& torch.JitTyperrcs$t|tjr |St|tjSr)rr#rCrB NumberTyper  is_numberrrrYX s   z)FallbackKernel.codegen..is_numberc3s"|] }j|fiVqdSrrrr4rrre rz)FallbackKernel.codegen..c3s(|]\}}t|to|jVqdSr)rcomplexr)rrrrXrrrj s  csgSr)rr rrrrr#x rz(FallbackKernel.codegen..)r&rVrr)+rrrr#rrrrmrrtorchgen.aoti.fallback_opsrTrr=rrrArcustom_ops_to_c_shimsrrwrrVrrrrrrrrSr,generate_fallback_kernel_with_runtime_lookuprrrgenerate_fallback_kernelr`rr)r,r0r )rrrKrTr args_iter exported_argsr)rYrrrr1 sl                zFallbackKernel.codegenrmrrbcCsJd}z|}Wn tyYnwt|j|jt|t||dS)NFrQ)rdrFrbrrrZrr)rmrdrrrtensor_to_layout s    zFallbackKernel.tensor_to_layoutrc sbtjf}||vrttdtjj}nt}|j|g|Ri|\}}}} } Wdn1s3wYt dd|D ||} | s]t |t j jjsX|t jjjur]t d} |durpt| d|||| || dn| svJdt| d|||| || ddfd d |g} t | ttfr| _| St | trt| _| S| g_| S)z9Create an instance of FallbackKernel from an _OpOverloadsNcsrr)r3rurrrr rz(FallbackKernel.create..r3rrrz"Not sure where to find device informr r0list[tuple[Any, int]]rcstttfrtfddttDSttr,fddDSttj rNt  }t j sDsDtsLtjj|j|SttrUSttjr_jjSdusmJdtddS)Nc3s,|]}|t|fgVqdSrrrgenerate_outputr0rmrrr s  zAFallbackKernel.create..generate_output..cs*i|]\}}||t|fgqSrrd)rr&rdrerrr rzBFallbackKernel.create..generate_output..zFallbackKernel output type z is not supported)rrrrrrrrr#r MultiOutputrarA assume_unaligned_fallback_outputrjrmrr5rrrrrr)rmr0r2rrfhas_unaligned_inputpacked)r0rmrrf s<      z.FallbackKernel.create..generate_output)rmr r0rcrr )r*_fused_moving_avg_obs_fq_helper_functionalr rrmrrr rrr.rr#rrKrLrk higher_orderprintrrfrGrrrr) rrKrrfake_incorrect_kernelscontextrrrrrrrrrirr sd    " zFallbackKernel.createrr`r0rKr{rrxrrrrrrrErrrrrQr?rprA)rr-rrrr r>rW)rmrrrb)rKr{rr rr rr)rrrrXrr#r rmrr[r.rr4r2rSrrrarrrrrrrrs,  w      y U rcs>eZdZdZdddZdddZd d d dfddZZS) ComplexViewz9View a complex number as two dtyped numbers or vice versarrcCrrrrrrrr r zComplexView.should_allocatercCrrrrrrrr4!r9z(ComplexView.get_inputs_that_alias_outputNrbr`r0rKr{rrxrrrrrrrrrrc stj|||||||ddS)Nrb)rrrr`rKrrrrrrrrr!s  zComplexView.__init__r>rW)r`r0rKr{rrxrrrrrrrrrrr)rrrrXrr4rrrrrrrr s   rrc@seZdZdZd ddZdS) MemoryCheckKernelz Custom kernel for memory checking that generates direct function calls TODO - the custom op was erroring with str inputs. should be able to custom op directly. rrrrrcCsl||j\}}}t|}t|}|r&|dd|d|d|d}n d|d|d}||dS)z.Override codegen to write direct function callzV# note: dont currently distinguish between buffers returned and dealloc'd in last stepzcheck_memory_step(allocated=z, freed=z, is_final_step=r/N)r/rrr)rr alive_list dead_list is_final_step alive_repr dead_reprcallrrrr"!s zMemoryCheckKernel.codegenNr?)rrrrXrrrrrrt!srtc@r&) rGrrrrcCrrrrrrrr!8!rzMultiOutputLayout.get_deviceNrC)rrrrr!rrrrrG4!r)rGcsVeZdZdddZ ddfdd Zed ddddZdddZd ddZZ S)!rgrrrrrcCs,|||js||||dSdSr)codegen_multi_output!skip_size_stride_alignment_checksr)r,rrrrr=!s  zMultiOutput.codegenFr`r0rfrr0list[tuple[Any, ...]]r|rcs>td||gdtj||_tj|||_||_dSr) rrrmrrrrr0r|)rr`rfr0r|rrrrC!s   zMultiOutput.__init__r{rcCs&|jd}t|tsJ|||Sr)rwrrr,)rr{rrrrr,P!s  z MultiOutput.get_free_symbol_usescCst|jdkot|jdtS)Nr@r)rrwrrFrrrrrX!szMultiOutput.should_allocatercCsdd|jDS)NcSs.g|]}t|trt|dkr|qSr)rrrr4rrr}rrrr^!sz.)rwrrrrr4]!sz(MultiOutput.get_inputs_that_alias_outputr?rL) r`r0rfrr0r}r|rrrrUr>rW) rrrrrrXr,rr4rrrrrrgZdd@dAZddCdDZddFdGZdvdHdIZdvdJdKZddNdOZddQdRZd{dSdTZddUdVZ ddWdXZ!e"d )ddd[d\Z#dd^d_Z$ddadbZ%d|ddedfZ&e'ddhdiZ(ddkdlZ)ddmdnZ*ddodpZ+e'ddrdsZ,d{dtduZ-e-Z.dS)rozC TensorBox / StorageBox allow in-place mutation of Tensors rr_rrcCrBrrHrrrrrp!r(z!MutableBox.has_exceeded_max_readsrcCrBrrErrrrr!s!r(zMutableBox.get_devicercCrBrrrrrrrv!r(zMutableBox.make_loaderr cCrBr)r_r rrrrr y!r(zMutableBox.make_indexerrMcCrBr)r_rCrrrrrC|!r(zMutableBox.get_stridercCrBrrFrrrrr!r(zMutableBox.get_nameNrrcCr7r)r_rrrrrr!rzMutableBox.has_large_inner_fnrrrcCr7rrGrrrrr!rzMutableBox.mark_reusecCrBrrJrrrrr!r(zMutableBox.realize_hintcCrBr)r_rrrrrr!r(zMutableBox.unwrap_viewcCrBr)r_r rrrrr !r(zMutableBox.is_input_buffercCrBr)r_rrrrrr!r(zMutableBox.freeze_layoutFrrrcC|j||Sr)r_rrrrrr!rz*MutableBox.freeze_layout_with_stride_ordercCr7r)r_rrrrrr!rz(MutableBox.freeze_layout_with_fill_orderrcCr7r)r_rrrrrr!rz(MutableBox.freeze_layout_with_same_orderrcCrr)r_r!r rrrr!!rz+MutableBox.freeze_layout_with_exact_stridesr"cCrBr)r_r#rrrrr#!r(zMutableBox.get_read_writesr$cCrBrrrrrrr!r(zMutableBox.get_readscCrBrrrrrrr)!r(zMutableBox.num_readsrycCrBrrKrrrrr*!r(zMutableBox.get_storage_numelr-cCrBrrrrrrr-!r(zMutableBox.get_reduction_typercCrBrrrrrrr.!r(zMutableBox.get_reduction_sizecCrBrrLrrrrr/!r(zMutableBox.is_externcCrBr)r_r0rrrrr0!r(zMutableBox.is_no_oprrcCr7rrrrrrr1!rzMutableBox.constant_to_devicercCrBr)r_r2rrrrr2!r(zMutableBox.get_mutation_namescCrBr)r_r3rrrrr3!r(zMutableBox.get_operation_namecCrBr)r_r4rrrrr4!r(z'MutableBox.get_inputs_that_alias_outputcCrBrrIrrrrr!r(zMutableBox.realizer{rcCr7rr8r+rrrr,!rzMutableBox.get_free_symbol_usesrcCrBrrNrrrrr!r(zMutableBox.get_read_namesrcCrBr)r_rrrrrr!r(zMutableBox.get_defining_oprrcCr7r)r_rrrrrr!rzMutableBox.codegen_referencer0cCrBrr_rrrrrr`!s zMutableBox.layoutrcCrBrrDrrrrr!r(zMutableBox.get_layoutcCrBrrrrrrr!r(zMutableBox.get_output_speccCrBrrrrrrr!r(zMutableBox.get_sizercCrr)r_rrrrrr!rzMutableBox.dtypecCsnt|jtrt|jdt|jjd}d}|jj}n t|jd}|j}d}|tt||g}d|S)Nrz))r/r)rr_rorrrrr)rline0endlr>rrrrr!s    zMutableBox.__str__r>rCrErFrGrHrrIrJr6rKrLrMrNrOrPrQrRrSrTrAr?rVrWrUr7r9rBr=r<r;)/rrrrXrrr!rr rCrrrrrr rrrrr!r#rr)r*r-r.r/r0r1r2r3r4rrXr,rrrr^r`rrrrrrrrrrroh!sb                                     roc@s>eZdZeed ddZeed ddZed ddZd S) r~r_rrcCr rrr_rrrr"rzTensorBox.creatercCr rrrrrrr"rcCst|tr|Stt|Sr)rrr~rrrrrr"s  N)r_rrr)r_rrr~)r_r)rrrrr[rrrrrr~"sc@sjeZdZdZdddZdddZdd d Zdd d ZdddZdddZ d ddZ d!ddZ d"ddZ dS)#rz7 StorageBox allow in-place mutation of Tensors rrcCs&t|jttfr|jtjjvSdSr)rr_rrerrmrr rrrrr "szStorageBox.is_input_buffercCst|jto|jtjjvSr)rr_rrrmrrrrrrrM"s zStorageBox.is_module_bufferr-cCst|jr |jSt|jttttfsJt |j|j }|j }|j }|dus1Jt dt||j|jdd|jd|_tj|j|j_tj|j|j|j_||j_||j_|jjS)NF)rrrrdr )rrr_rrrrrrrrrr!rrr rrmrrrrrrr)rrrrrrrr"s2        zStorageBox.realizercCs4t|jttfr|jjdkr|dSdSdS)zL Called on buffers we expect to be forced to realize later. r@N)rr_rrrnontrivial_read_countrrrrrr<"s  zStorageBox.realize_hintrrcs^ddlmfdd|D}|sdSt|}t|}t|}||ko.||dko.||kS)Nris_nonfreeable_bufferscs g|] }|stj|qSr)rmrget_dep_size_hintrrrrrI"s  zCStorageBox.has_accumulated_enough_reads_by_size..Fr)r_rrrrr)rr size_of_reads total_sizemax_sizemin_sizerrr$has_accumulated_enough_reads_by_sizeF"s   z/StorageBox.has_accumulated_enough_reads_by_sizecCs8t|jto|tjkp|ptjduo|tjSr) rr_rr)rArealize_acc_reads_thresholdr realize_acc_reads_size_thresholdrrrrrrY"s  z!StorageBox.has_exceeded_max_readsrcsh|dkr2t|jttfr2t|jr'|jddg}tfdd|Dr'dS|tj kp1| SdS)zj A heuristic to decide if we should realize a tensor that is used multiple times. r@expsigmoidc3s|]}|jvVqdSr)used_opsrTopcountrrro"rxz5StorageBox.should_realize_on_reuse..TF) rr_rrr<rrr)rArealize_reads_thresholdr)rr heavy_opsrrrshould_realize_on_reusee"s  z"StorageBox.should_realize_on_reusecCs||r |dSdSr)rrrrrrrw"s  zStorageBox.mark_reusecCrBrrrrrrr){"r(zStorageBox.num_readsNr>rAr6)rrrr)rrrrrJrS) rrrrXr rMrrrrrrr)rrrrr"s      rc@s*eZdZUded<ded<dZded<dS)Subgraphrrrl graph_moduleNzOptional[GraphLowering]r)rrrrrrrrrr"s rbufferscCs,dd|D}ttdd|Dt|kS)NcS"g|] }t|tr |n|qSrrrerrr0rrrr"z(_has_aliased_buffers..csrr)r`rrrrr"rz'_has_aliased_buffers..)rr9)rrrr_has_aliased_buffers"srcs`eZdZUdZdZded<dZded<dZded<dfdd Ze dddZ dddZ Z S)InvokeSubgraphz. Ir node for the invoke_subgraph HOP. NOptional[Subgraph]ruOptional[Sequence[IRNode]]operandsrrrxr`rGrrcs6tjd||d||_tj||_tj|dSr)rrrurmrrrr)rrurr`rrrr"szInvokeSubgraph.__init__rElist[Union[ShapeAsConstantBuffer, NoneAsConstantBuffer, MultiOutput]]csddlm}tjj}d}|jd}r5d}|jtj j j ur,|j dtj j j us*Jd}|d|d}n&d}|jtj j j urM|j dtj j j usKJd}|j |d}d d |D}fd d |D}g} t|D]\} } t| ttfr{| | qj| || || qj| }|jdurtjj|j||jd |_t|j|jj|Wdn1swY|jj} d} |D] } t| ts| } nq| dusJt||t| d ddfdd fdd t| D}|_|S)zFor each operand, get a realized input, force it to have the same strides as the subgraph inputs, then use an InvokeSubgraphr@)constrain_to_fake_tensorNeager_input_valsrrrrcSg|]}|jdqSrfrsrTrrrr"rz)InvokeSubgraph.create..cr rrrTrrrr"rrkrjrlr)rurr`rmrindrr?Union[ShapeAsConstantBuffer, NoneAsConstantBuffer, MultiOutput]c sdt|ttfr |S|}|dusJtt||||| j | j dt |fgddS)NrdT)r|) rrrr!rgrbr rrCrrcrdr)rmrr)invoke_subgraphrr create_output"s" z,InvokeSubgraph.create..create_outputcsg|] \}}||qSrr)rrrm)rrrr#r)rmrrrrr)rrrmrryrsrr^r#rkrmrrrrrrrrrtrrryrz graph_outputsr!rrGr)rrurrry fake_operandsrrc fx_operands new_operandsroperandrroutsr)rrrrr"sd      zInvokeSubgraph.createrrrcCrCr)codegen_invoke_subgraphrrrrr #rzInvokeSubgraph.codegen)rurrrxr`rGrr)rurrrrrr?) rrrrXrurrrrrrrrrrrrr"s     frcseZdZUdZdZded<dZded<dZded<dZded <dZ d ed <d*fdd Z e d+ddZ e d,d!d"Zd-d%d&Zd.d(d)ZZS)/ Conditionala IR node representing torch.cond Attributes: predicate: A boolean scalar tensor determining which branch to execute. operands: Input tensors passed to both true and false subgraphs. true_subgraph: Subgraph executed when predicate is True. false_subgraph: Subgraph executed when predicate is False. outputs: MultiOutput nodes representing the conditional's outputs. Nr predicaterrr true_subgraphfalse_subgraphOptional[Sequence[MultiOutput]]rrrxrr`rGrrrrc sj||_||_||_||_t|g|\}}tjd|||d|dur&||_tj ||_ tj |dSN)rr`rwr) rrrr_split_by_sym_typerrrrmrrrr) rrrrrr`rsym_argsrrrrr##s zConditional.__init__rUnion[int, torch.SymInt]Union[int, sympy.Expr]cCst|tr|S|jjSr)rrrr)rrrr _maybe_expr?#s zConditional._maybe_exprr~true_fnfalse_fnlist[TensorBox]list[MultiOutput]c s|}fdd|D}tjjjd}t|ts Jt|tdd|Ds+Jdd|D}tjjj d}ddd}||fD]8} | jdurztjj | j || j d| _t | j| jj||| jj|| j_Wdn1suwYqB|jdusJ|jdusJ|jj} |jj} d| fd| ffD]\} } t| rtd| d| qt| t| ksJ| | ftt| | D]5\}\}}||ksJ|||f||ksJ|||f|j|jksJ|||fqtdd||gDttjjjtjjj dd}dusJdt||||td|dfddtt| tjjj dD} | _ | S)zNCreate a Sequence of IRNodes from a conditional statement (see .lowering.cond)cr rrrTrrrrQ#rz&Conditional.create..rAcsrr)rr7ryrrrrU#rxz%Conditional.create..cSsg|] }tt|jdqSrf)r r7rsrTrrrrV#rnrdrrx fake_tensorsSequence[torch.Tensor]rrcSsPg}t||D]\}}t|tr||q|tjt||ddq|SNFr/)rrrrrrr~r)rrretrmrrrr_require_exact_stridesY#s  z2Conditional.create.._require_exact_stridesNrrrzVOutput aliasing is currently not supported in compiled torch.cond. The outputs of the z% subgraph of torch.cond are aliased: css"|] }t|ts|VqdSr)rrr!)rorrrr#s rzcannot determine devicer)rrrrr`rc svg|]7\}\}}tt|dur|n|dd|Ddd|D|j|jdt |fgqS)NcSrZrrrrrrrrr#rz1Conditional.create...cSrZrrrrrrr#s rd) rgrbr!r rrrrcrdr)rrrm merged_output) conditionalrrrr#s&   )rrxrrrr)!rkrmrryrrrrrrsrtrrryrzrrrvrrrr!r rrcrwr4rrrrrGr)rrrrrrr fake_outputsrru true_outputs false_outputsrrrt_of_orr)rrrrrE#s       $ zConditional.createrrrcCs*|||||jt|didSr)codegen_conditionalrrrrrrrrr#s zConditional.codegenrcCr!rr"r#rrrrm#r%z$Conditional.get_unbacked_symbol_defs)rrrrxrrrrr`rGrrrr)rrrr) rr~rrrrrrrrr?rp)rrrrXrrrrrrrr[rrrrrmrrrrrr#s        |rrrk-tuple[list[ShapeAsConstantBuffer], list[Any]]cCs<g}g}|D]}t|tr||jq||q||fSr)rrrr)r non_sym_argsrrvrrrr#s  rcseZdZUdZdZded<dZded<dZded<dZded<dZ d ed <d)fdd Z e d*ddZ e d+ddZ ed,d d!Zd-d$d%Zd.d'd(ZZS)/ WhileLoopzSThe IR node for while_loop and while_loop_stack_output. It supports input mutation.Nrcarried_inputsadditional_inputsr cond_subgraph body_subgraphrrrxrr`rGrr stack_outputrrrc sr||_||_||_||_tg||\}} tjd|| |d|dur'||_||_t j ||_ t j |dSr)rrrrrrrrrrmrrrr) rrrrrr`rrrrrrrr#s$  zWhileLoop.__init__cCsrt|s|Sdd|D}t}g}t||D]\}}t||vr*|t|q|t|||q|S)NcSrrrrrrrr$rz3WhileLoop._clone_aliased_inputs..)rr9rr`rrrr)runwrapped_buffers seen_buffersroriginal_inputunwrapped_bufferrrr_clone_aliased_inputs $s  zWhileLoop._clone_aliased_inputsrrcCsJt|tr|St|ttfrt|St|trt|Stdt|)NzNYI unsupported output type: )rr~rrergrrFr)rrrr_maybe_wrap_as_tensor_box'$s   z#WhileLoop._maybe_wrap_as_tensor_boxcond_fnbody_fnr7c& sddlm}d*d d }tjjjd }tjjjd } || } d d| D} dd|D} dd| D} fdd|D}t|}||| }fdd|D}||| }||||fD]S}|jdurt| t smJt | tjj |j | |j d|_t|j'|jj| ||urt|jjt| ksJ||jj| |j_Wdn1swYq[|jr|jsJ|jj}|jj}t|rtd|t|dksJ||d}t|ts|tjksJ|t|dksJ|tdksJdd}|dus Jt|t|ksJ||ftt||D]?\}\}}d+dd}||||||||ksOJ||||f||ks_J|||fq!|dushJttjjjtjjj !dd}t||||t"|d||d }|jdurt|jj#tj$j%sJ||jj#| d!}t&|}fd"d|D}t'|}g}g|_(g|_)|rt|dksJd#ttjjj d$D]/\} }!t*t+|!j,|!j-d%d|!.Dd&d|!/Dd'|t0| fg}"|j(1|"|1|"qnVt|D]Q\} }!| |vr/| t|ksJd(t2|}#|j)1t3|#j4|#||1|#qt*t+|!|!|!|!|!5j6d)|t0| fg}"|j(1|"|1|"qt||D]\}$}%|$7tjj8vrrtjj9:|%7q\|S),zcreate the while_loop IR node. stack_output controls whether it stack each iterations' output, which is necessary for training. r)check_input_alias_and_mutation tensor_boxesrxr,list[Union[int, torch.SymInt, torch.Tensor]]rrcSslt|t|ks Jg}t||D]"\}}t|tjr.t|}|tj || ddq||q|Sr) rrrr#rrrrrrr)rrrrfknew_tbrrrr@$s    z0WhileLoop.create.._require_exact_stridesrAcSrrfrrTrrrr`$rz$WhileLoop.create..cSrrfrrTrrrra$rcSrrfrrTrrrrb$rcr rrrTrrrrd$rcr rrrTrrrrg$rNrzOutput aliasing is currently not supported in compiled torch.while_loop. The outputs of the body_fn subgraph of torch.while_loop are aliased: r@z9torch.while_loop is assumed to have at least one operand. lhs_exprs Sequence[Union[int, sympy.Expr]] rhs_exprsrcSs<t|t|ks Jt||D] \}}tjj||qdSr)rrrmrrr)rrlhsrhsrrr_guard_list_equals$sz,WhileLoop.create.._guard_list_equalsrr)rrrrr`rrrcrrr)rr) all_inputsrrr$rz-NYI: while_loop_stack_output input mutations.rdcSrZrrrrrrr$rcSrZrr)rrrrrr$r)rrrrzonly carries can be mutated.)rrrrrc)rrxrrrr)rrrrrr);torch._higher_order_ops.utilsrrmrryrrrrrrrtrrryrzrrrrvrr r#rrr!rrrCr4rrrsrrGmodulefx GraphModuler9rxrrrgrbrrrrrrrwrr`rrcrr rr)&rrrrrrrrfx_carried_inputsfx_additional_inputs fx_all_inputsfake_all_inputsfake_carried_inputsfake_additional_inputscarried_inputs_additional_inputs_ru cond_outputs body_outputsrrrroborr while_loop mutated_idxsmutated_idx_setrmutated_inputs_iter all_outputsrrm multi_out mutated_inputr}rr)rrrr2$s           "$      zWhileLoop.createrrrcCs.|||j|||jt|didSr)codegen_while_looprrrrrrrrrr %szWhileLoop.codegenrcCr!rr"r#rrrrm%r%z"WhileLoop.get_unbacked_symbol_defs)rrxrrxrrrrr`rGrrrrrr)rrxrrx)rrrr) rrrrrrxrrxrrrr7r?rp)rrrrXrrrrrrrr[rrrrrrmrrrrrr#s"      "    Zrcs@eZdZ ddddfddZdfdd ZdddZZS)rNrr`r0rKr{rrxrrrrrrrErrrrc sbtj|||||d|dddlm}||} | dusJ| |_tjj| d|_ |tjj| <dS)Nrbr) _get_effect) rrtorch._higher_order_ops.effectsr effect_typermr effectful_opsrprev_effect_buffer) rr`rKrrrrrr r rrrr%s   zEffectfulKernel.__init__r"cs0t}|jdur|jt|j|Sr)rr#r r'rrBrr)rr/rrrr#;%s  zEffectfulKernel.get_read_writesrcCrrrrrrrrE%r z EffectfulKernel.has_side_effectsrrqrQr>)rrrrr#rrrrrrr%s  rc@s"eZdZed d d ddZdS) rFr{rrrcCrlrr8r+rrrr,J%rz!NonTensorObj.get_free_symbol_usesNrLrU)rrrrXr,rrrrrI%src@sReZdZUded<ded<dddZddd d Zdd dZdddZdddZdS)rmrr+Union[FakeScriptObject, torch.ScriptObject]r'rcCrrrrrrrrV%rzTorchBindObject.get_nameNrrcCrrrrrrrrY%rz!TorchBindObject.codegen_referencecCrrr*rrrrr\%rzTorchBindObject.get_valuetorch.ScriptObjectcCst|jtjr |jS|jjSr)rr'r# ScriptObjectreal_objrrrr get_real_obj_%szTorchBindObject.get_real_objrcCsZ|}t|r dSt|dsJt|}t|d}dd|D}tt j |dS)Nr__obj_flatten__cSs(g|]}t|tjr||qSr)rr#rrenumelrTrrrro%s  z1TorchBindObject.get_buf_bytes..) r r(rrr rrr9rUoperatorr)rreal_script_obj flat_dict flat_elems flat_sizesrrr get_buf_bytese%s zTorchBindObject.get_buf_bytesrHrrB)rr )rr rS) rrrrrrrr r rrrrrmQ%s     rmc@s4eZdZUded<ded<d ddZddd d ZdS)rrrrrrcCrrrrrrrr|%rzGeneratorState.get_nameNrrcCrrrrrrrr%rz GeneratorState.codegen_referencerHrrB)rrrrrrrrrrrw%s  rc@sHeZdZdddZdddZddd d ZedddZedddZdS)_CollectiveKernelrrcCrrrrrrrr%r z!_CollectiveKernel.should_allocatecCrrrrrrrr%r z"_CollectiveKernel.has_side_effectsNrr-rcCsRt|jtjjus Jd|j}|dur||_n|jj|_dd|jjD|_ dS)Nz,Setting cpp kernel needs a valid op_overloadcSrrrrTrrrr%rz9_CollectiveKernel.set_cpp_kernel_name..) rrr#rrrrrrr)rrrKrrrr%s  z%_CollectiveKernel.set_cpp_kernel_namerKr{rwUnion[IRNode, list[IRNode]]rr rc s$tjj|j||g|Ri|\}}}}} Wdn1s!wY| r1J|d| |D]} | tj| q3|d|td||||t |} j fdd| Dj dd| Dd|vrj ttd|dj |ddSdS)Nrqrrcrrrrrrkrrr%rz4_CollectiveKernel.create_inplace..cSr~rrr~rrrr%rr)rmrrrrrmrr!rfr tree_leavesrrrrr) rrKrwrr_example_outputrrrr tensor_arginpsrr rcreate_inplace%s@   z _CollectiveKernel.create_inplace!Union[TensorBox, list[TensorBox]]+Union[list[MultiOutput], _CollectiveKernel]csJtjjj||g|Ri|\}}}}} Wdn1s!wY| r1J|d| |D] } t| ts>| q3t|tr||} | dusPJt | d||||fddt |D_ t j |D]\} } t jsyt| stjj| jqnj S|||||t jst|stjjjg_ S)NrSrcs(g|]\}}t|t|fgqSr)rgrar)rrrWrrkrrr&sz9_CollectiveKernel.create_out_of_place..)rmrrrrrmrrr.rGrrrrArhrjr5rrra)rrKrwrrrrrrrr rr2rWrr# rcreate_out_of_place%s^      z%_CollectiveKernel.create_out_of_placer>rr@) rKr{rwr rr rr rr) rKr{rwr! rr rr rr" ) rrrrrrrr r$ rrrrr %s    Ar c2eZdZ ddddfddZdddZZS)_AllReduce_KernelNrr`r0rKr{rrxrrrrrrrErrrrc (tj|||||d|d|ddS)Nrb+aoti_torch_cpu__c10d_functional_all_reduce_rrrrsrrrr &  z_AllReduce_Kernel.__init__rrrcC2|d||t|jtr||dSdSNz+torch/csrc/inductor/aoti_torch/c/shim_cpu.hinclude_extra_headerrOrr`rr)rrrrr6&   z_AllReduce_Kernel.codegenrrqr?rrrrrr& &  r& cr% )_AllReduceKernelNrr`r0rKr{rrxrrrrrrrErrrrc r' )Nrb*aoti_torch_cpu__c10d_functional_all_reducer) rsrrrr?&r* z_AllReduceKernel.__init__rrrcCr+ r, r- rrrrrU&r/ z_AllReduceKernel.codegenrrqr?rrrrrr1 >&r0 r1 csXeZdZ d"ddd#fddZd$ddZd%ddZed&ddZd'fd d! ZZ S)( _WaitKernelNrr`r0rKr{rrxrrrrrrrErrrrc r' )Nrb+aoti_torch_cpu__c10d_functional_wait_tensorr) rsrrrr^&r* z_WaitKernel.__init__rrrcCr+ r, r- rrrrrt&r/ z_WaitKernel.codegencCs|jd}t|ts Jt|tr$|jd}t|ts!Jt||gSt|trB|jd}t|tr@|jd\}}|j|gSgSgSr)rwrrr rrgr0)rr}rcollrrrrrget_volatile_reads{&s       z_WaitKernel.get_volatile_readsr}r~c Cstjj|||\}}}}}Wdn1swY|r*J|d||t|d||||}|jtt|d||dS)Nrqr) rmrrrrfr!rrr) rrKr}r rrrrrkrrr create_wait&s(   z_WaitKernel.create_waitr"cs6t}|}|D] }|jt|q |Sr)rr#r6 r'rrBrr)rr/volatile_readsvrrrrr#&s z_WaitKernel.get_read_writesrrqr?rW)rKr{r}r~rrrQ) rrrrrr6 rr7 r#rrrrrr3 ]&s    r3 rrcCdt|ttfr t|St|ttfr%ttj}|D]}|t |O}q|St|t j r/t|StSr) rr6rr1rrr9rr!r:r#rrrr&rrrr:&  r:cCr: r) rr6rr0rrr9rr!r;r#rr; rrrr;&r< r;rr r torch.fx.NodecCst|trht|jtrjt|jjtr|jjd|dSt|jjtrl|jjd|t|jjtrEt|jjjtrE|jjjd|dSt|jjtrn|jjj spt|jjj dtrr|jjj dd|dSdSdSdSdSdSdS)Nrr) rr~r_rrqrrrrgr0rw)rrrrrassign_origin_node&s&   r> )rrrr)rrrr)rrrr)rrrr)rrrrrrr)rrrrrr)rrrrr:)rrr rrr)rrr rrr)rrr rrr)r'r(rr))rr,rr-)rr2rr)rr=r>rrr)rLrMrNrMrOrMrr)rWrrXrYrr)rkrlrr)rwrxrry)rrrrrrl)rrrrrrrr)rrrrrcrrr r)TFNFN)rrr#rr*rr+r,rrrr,rr-)rrr+rrrr5)rrMrOrMrr)rrrr)rrarrb)rrxrr)rrkrr)rrrr)rr rr= rr(O __future__rr\rrYr9rVloggingr textwraprcollections.abcrrrrrrr enumr r typingr r rrrrrrrrrrtyping_extensionsrrrrrr unittest.mockrrrr r!torch._export.serde.schema_exportserderr;r3rrWrtorch._loggingr#torch.fxtorch.utils._pytree_pytreertorch._dynamo.utilsr"torch._export.serde.serializer#*torch._higher_order_ops.auto_functionalizer$torch._inductorr%r_r'torch._library.opaque_objectr(torch._prims_commonr)r*r+r,r-%torch.fx.experimental.symbolic_shapesr.r/r0r1r2r3r4r5r6 torch.fx.noder7torch.utils._ordered_setr9torch.utils._python_dispatchr:torch.utils._sympy.functionsr;r<r=r>torch.utils._sympy.symbolr?rrArBcodegen.commonrCrDrErFrGrHrIrJrKrL loop_bodyrM ops_handlerrNrOrPrQruntime.benchmarkingrR runtime.hintsrSrTrUrVrWrXrYrZr[r\r]r^r_r`rarbrcrdrerfrgrhrirj virtualizedrkrlrm"torch._library.fake_class_registryrnrorpcodegen.cutlass.templaterqcodegen.wrapperrrrrsrtrrr6 __version__rr ImportErrorrurvrwrxrryrrzrrrr{ getLoggerrr=rrrrrrr,rrrrrrr;r=rrrr r+r1r;r<rKrVrjrvr~rrr_rqrrrrrrrbrrrrrrrrr)rar2r3rrr_rkrwrrerrrrrrrr0rrbrrRr\r`rfrkrrrrrrrrrrrrPrimitiveInfoTyperrr r?rFrIrJrdrrrrsrrBrHrrrXr]r^rirrrrrrrrrrrrrrrrrrtrGrgror~rrrrrrrrrrmrr r& r1 r3 r:r;r> rrrrs  8        ,     `        *    "           '   AK (# ?- :&_K   < a\0D${V)b  %_$"Y  :K*GF_]UI%+!@(4P-29,</5?7 ,p C=,% Y