o kiM#@s,UdZddlZddlmZddlmZmZddlmZddl Z ddl m Z edZ edZ iZee jjefed <ehd Zd ed ed ee e fdeee e fgee e fffddZ dCddddde jde jde jdee jdededede jfddZedde dCddddde jde jde jdee jdededede jfddZd eed!edefd"d#Zd$e jd%edede jfd&d'Zd(e jd)e jd*ed+ed eed,ede jfd-d.Zd*ed+eddfd/d0Z d(e jd)e jd*ed+ed eede jf d1d2Z!   dDdddddd3dd4d(e jd)e jd5e jd6ee jd7ee jd8ee jd9ed:ed;ed,ed eedd?Z#ed@de#   dDdddddd3dd4d(e jd)e jd5e jd6ee jd7ee jd8ee jd9ed:ed;ed,ed eed op_type opset_version fake_implreturncs.dtttfdtttfffdd }|S)zDDecorator to register an ONNX operator with a custom implementation.funcrcsPd}tjjdd|dd|}|ttttjj|<||S)Nopsetzonnx::.) mutates_args)torchlibrary custom_opr getattropsonnx register_fake)roverloadtorch_oprrrr^/var/www/addictedbytheproject.nl/epg/venv/lib/python3.10/site-packages/torch/onnx/ops/_impl.py decorator's  z_onnx_op..decorator)rrr)rrrr"rr r!_onnx_op"s* r#F) interleaved num_headsrotary_embedding_dimx cos_cache sin_cache position_idsr$r%r&cCs|S)zFFake implementation for RotaryEmbedding-23 for torch.compile purposes.)clone)r'r(r)r*r$r%r&rrr!_rotary_embedding_23_fake_impl5s r,RotaryEmbeddingcs|jt}dddurWtdkfddtjdkfddtjdkfd dtdkoNdkfd dntd kodd kfd d|d krwt|d}n$|d krt|dkfddd}||} || g} t|| }tt|jd kdd|jd } |dkr| }|ddddddd|f} |dddddd|df} |ddurntjdkojdkfddtjdkojdkfddtjdkfddtjdkfddtdtd|rk| dddddddddf} | dddddddddf}n tj| ddd\} }| |}| |}|rt|d}t|d}tj ||fdd}t|| j} n tj ||fdd} tj | | fdd}|d krt|St|dS)z_RotaryEmbedding-23 https://onnx.ai/onnx/operators/onnx__RotaryEmbedding.html#rotaryembedding-23rNcs djS)Nz6position_ids must be 2D when provided. Received shape shaper)r*rr!Zs z%rotary_embedding_23..csddjdS)Nz6position_ids first dim (batch) must match x.shape[0] ( ). Received rr1r) batch_sizer*rr!r3^r csddjdS)Nz;position_ids second dim (sequence) must match x.shape[-2] (r4r r1r)r*sequence_lengthrr!r3br6cdjdjS)NzWcos_cache/sin_cache must be 2D when position_ids is provided. Received cos_cache shape , sin_cache shape r1rr(r)rr!r3f cr8)Nz[cos_cache/sin_cache must be 3D when position_ids is not provided. Received cos_cache shape r9r1rr:rr!r3lr;)rr0r r<cs dS)NzKnum_heads must be provided for 3D inputs. Received input tensor with shape rr) input_shaperr!r3ys cSdS)Nzx should be a 4D tensor by nowrrrrr!r3csdjdddS)Nzcos has shape  but expected (batch=, seq=, ...)r1r)r5cosr7rr!r3csdjdddS)Nzsin has shape rArBrCr1r)r5r7sinrr!r3rEcsdjdddS)NzLast dimension of cos cache (rG') should match rotary_embedding_dim/2 ().r1r)rDrotary_embedding_dim_halfrr!r3csdjdddS)NzLast dimension of sin cache (rGrHrIr1r)rJrFrr!r3rKdim) r2lenr_checkrMpermutereshape unsqueezechunkcat)r'r(r)r*r$r%r& input_rank hidden_size head_size new_shapex_rotate x_not_rotatex1x2realimagx_rotate_concatoutputr) r5rDr(r>r*rJr7rFr)r!rotary_embedding_23Cs                  "$     rascalerWcCs|dur|Sdt|S)z/Get the scale factor for attention computation.Ng?)mathsqrt)rbrWrrr!_get_scale_factorsretensorr5cCs:|jd|jd}}||}|||||ddS)z1Reshape 3D tensor to 4D for multi-head attention.r r0)r2view transpose contiguous)rfr5r%r7rVrWrrr!_reshape_3d_to_4ds rjQKcurrent_q_num_headscurrent_kv_num_headsqk_matmul_output_modec Cs2|dkr t|||||Stt||ddS)z1Get QK output tensor based on the specified mode.rr/rG)_compute_qk_output_for_mode_0r zeros_likematmulrh)rkrlrmrnrbrorrr!_get_qk_output_for_aten_spdas   rscs"tdkfdddS)z-Validate Group Query Attention configuration.rcsdddS)Nz q_num_heads (z%) must be divisible by kv_num_heads (z ) for GQArrrnrmrr!r3sz-_validate_gqa_configuration..N)rrO)rmrnrrtr!_validate_gqa_configurations  ruc Cs`|}||kr||}|j|dd}t||jd}t|}||} ||} t| | ddS)zDHelper function to compute QK output for qk_matmul_output_mode == 0.r rLr<r/rG)repeat_interleaverer2rcrdrrrrh) rkrlrmrnrbK_for_qk repeat_factor scale_factor sqrt_scaleQ_scaledK_scaledrrr!rps  rp) is_causal kv_num_heads q_num_headsrorbsoftcapsoftmax_precisionV attn_maskpast_key past_valuer~rrrrcCsZ|jd} t|jdkrE|jd}|j}|dur,| ||jd|jd|jd|f}n| ||jd|jd|f}|}| |||df}n:|jd}|j}|duri|jd|jd|jd|jd|jdf}n|j}|}|jd|jd|jd|df}tj||j|jd}tj||j|jd}tj||j|jd}tj||j|jd}||||fS)z@Fake implementation for Attention-23 for torch.compile purposes.rr<r Nr0dtypedevice)r2rNremptyrr)rkrlrrrrr~rrrorbrrr5q_sequence_length output_shapepresent_key_shapepresent_value_shapeqk_output_shaper` present_key present_value qk_outputrrr!_attention_23_fake_implsR      r Attentionc' Csd\} }}t|j}|jd}t|jdkr;t|dko|dkdd|jd}t|||}t|||}t|||}tt|jdkoQt|jdkoQt|jdkdd|j|}t| |} |d urmtj||g|d n|}|d ur~tj||g|d n|}||}}|j| }|j| }|j|}|j|}| d ko| dko| d uo|d up|jtj k}t |||rtj j j ||||d || t ||kd }t||||| | }n||kr||}|j|| d }|j|| d }tj|||j|jd }|rt|d uddttj||tj |jd }||td}|d ur4|jtj kr0||td}n||}t| |jd}t|} || }!|| }"t|!|"dd}#|#}|#|}$| dkr`|$}| d krn| t|$| }$| dkru|$}| d ur| tvr|$j}%|$tj| }$tj|$dd }&|&|%}&ntj|$dd }&ntj|$dd }&| dkr|&}t|&|}|dkr|dd !||d}||||fS)zMAttention-23 https://onnx.ai/onnx/operators/onnx__Attention.html#attention-23)r r0r<rr<cSr?)Nz;q_num_heads and kv_num_heads must be provided for 3D inputsrrrrr!r3r@zattention_23..r r=cSr?)Nz'Q, K, and V should be 4D tensors by nowrrrrr!r3r@NrLr})r dropout_pr~rb enable_gqarcSr?)Nz'Cannot use both is_causal and attn_maskrrrrr!r3r@z-infr/rGr0)"rNr2rrOrjrerTr+rboolrunn functionalscaled_dot_product_attentionrsrvzerosrtrilones masked_fillfloatrcrdrrrhtanh-_ATTENTION_23_ALLOWED_INTERMEDIATE_PRECISIONStorONNX_DTYPE_TO_TORCH_DTYPEsoftmaxrirg)'rkrlrrrrr~rrrorbrr num_head_dim sequence_dimhead_diminput_shape_lenr5r q_head_sizerrrmrnkv_sequence_length can_use_sdpar`rrx attn_bias causal_maskryrzr{r|qk_matmul_output qk_with_biasoriginal_dtype qk_softmaxrrr! attention_23cs       (                        r)N)NNN)%__doc__rccollections.abcrtypingrrtyping_extensionsrrtorch.onnx.opsrrrr dict_ops OpOverload__annotations__ frozensetrstrintr#Tensorrr,rarrerjrsrurptuplerrrrrr!s                    T