o ki\1#@sFdZddlZddlmZddlmZmZddlZee Z ddgZ de e dBde e fd d Zed d d e defddZGdddeZejjdid   d9dejdejdejdejdejde de dededBde e dBdeejejejffddZej   d9dejdejdejdejdejde de dededBde e dBdeejejejffdd Zddd!d"dejdejdejdejdejde de d#edBdedBdee e fdejeejejfBfd$dZd%ed&eed'fd(eddfd)d*Zejjd+id  d:d,ejdejdejdejd-ejd.ejdejdejde de ded/ejdedBde e dBdeejejejffd0d1Zej  d:d,ejdejdejdejd-ejd.ejdejdejde de ded/ejdedBde e dBdeejejejffd2d3Zd%ed,ejd4ejd5ejdeejdBd'ff d6d7Zejeed8dS);z Variable-length attention implementation using Flash Attention. This module provides a high-level Python interface for variable-length attention that calls into the optimized Flash Attention kernels. N) lru_cache)Any NamedTuple varlen_attn AuxRequest window_sizereturncCs2|durddg}t|dkrtdt||S)Nz$window_size must have length 2, got )len ValueError)rr c/var/www/addictedbytheproject.nl/epg/venv/lib/python3.10/site-packages/torch/nn/attention/varlen.py_normalize_window_sizes  r)maxsize device_indexcCsdS)z;Cache device capability check to avoid repeated CUDA calls.Fr )rr r r_should_use_cudnnsrc@seZdZUdZdZeed<dS)rz Request which auxiliary outputs to compute from varlen_attn. Each field is a boolean indicating whether that auxiliary output should be computed. FlseN)__name__ __module__ __qualname____doc__rbool__annotations__r r r rr#s ztorch_attn::_varlen_attn) mutates_argsFquerykeyvaluecu_seq_qcu_seq_kmax_qmax_k is_causalscalec Cst| } |jo t|jj} | rGtd| ddks | ddkr$tdtj j j |||d||||dd|d |d } | d| d| d } } }n"td tj j j |||||||d|d || d| dd \} } }}}tj dtj|jd}| | |fS)z Private custom op for variable-length attention. This is the internal implementation. Users should use the public varlen_attn function instead. #Using cuDNN backend for varlen_attnrr TcuDNN backend does not support window attention. Please use Flash Attention backend.NTFr$-Using Flash Attention backend for varlen_attn)return_debug_maskr$window_size_leftwindow_size_rightr dtypedevice)ris_cudarr2indexloginfo RuntimeErrortorchopsaten_cudnn_attention_forward_flash_attention_forwardzerosuint64)rrrrr r!r"r#r$r use_cudnnresultoutput softmax_lse rng_state_ rng_state_r r r _varlen_attn-sX    rFc Cst| } t|} |d} |d} tjjr,|dd} tj| | |ftj|jd}n tj| | ftj|jd}tjdtj |jd}| ||fS)z Fake implementation for meta tensor computation and tracing. Based on the 3D varlen path from meta__flash_attention_forward: - query shape: (total, num_heads, head_dim) - logsumexp shape: (num_heads, total_q) rr&r0r/) rr8 empty_likesizeversionhipemptyfloatr2r>)rrrrr r!r"r#r$rrAtotal_q num_heads batch_size logsumexprCr r r_varlen_attn_fakess    rQ)r r ) return_auxr$rrRc CsL| dk} tjj|||||||| |t|  \} } } |dur$|jr$| | fS| S)au Compute variable-length attention using Flash Attention. This function is similar to scaled_dot_product_attention but optimized for variable-length sequences using cumulative sequence position tensors. Args: query (Tensor): Query tensor; shape :math:`(T_q, H, D)` key (Tensor): Key tensor; shape :math:`(T_k, H, D)` value (Tensor): Value tensor; shape :math:`(T_k, H, D)` cu_seq_q (Tensor): Cumulative sequence positions for queries; shape :math:`(N+1,)` cu_seq_k (Tensor): Cumulative sequence positions for keys/values; shape :math:`(N+1,)` max_q (int): Maximum query sequence length in the batch. max_k (int): Maximum key/value sequence length in the batch. return_aux (Optional[AuxRequest]): If not None and ``return_aux.lse`` is True, also returns the logsumexp tensor. scale (float, optional): Scaling factor for attention scores window_size (tuple[int, int], optional): Window size for sliding window attention as (left, right). Use (-1, -1) for full attention (default), (-1, 0) for causal attention, or (W, 0) for causal attention with sliding window of size W. Returns: output (Tensor): Output tensor from attention computation; shape :math:`(T_q, H, D)`. If ``return_aux`` is not None and ``return_aux.lse`` is True: lse (Tensor): Log-sum-exp of attention scores; shape :math:`(T_q, H)`. Shape legend: - :math:`N`: Batch size - :math:`T_q`: Total number of query tokens in the batch (sum of all query sequence lengths) - :math:`T_k`: Total number of key/value tokens in the batch (sum of all key/value sequence lengths) - :math:`H`: Number of attention heads - :math:`D`: Head dimension Example:: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_CUDA) >>> batch_size, max_seq_len, embed_dim, num_heads = 2, 512, 1024, 16 >>> head_dim = embed_dim // num_heads >>> seq_lengths = [] >>> for _ in range(batch_size): ... length = torch.randint(1, max_seq_len // 64 + 1, (1,)).item() * 64 ... seq_lengths.append(min(length, max_seq_len)) >>> seq_lengths = torch.tensor(seq_lengths, device="cuda") >>> total_tokens = seq_lengths.sum().item() >>> >>> # Create packed query, key, value tensors >>> query = torch.randn( ... total_tokens, num_heads, head_dim, dtype=torch.float16, device="cuda" ... ) >>> key = torch.randn( ... total_tokens, num_heads, head_dim, dtype=torch.float16, device="cuda" ... ) >>> value = torch.randn( ... total_tokens, num_heads, head_dim, dtype=torch.float16, device="cuda" ... ) >>> >>> # Build cumulative sequence tensor >>> cu_seq = torch.zeros(batch_size + 1, device="cuda", dtype=torch.int32) >>> cu_seq[1:] = seq_lengths.cumsum(0) >>> max_len = seq_lengths.max().item() >>> >>> # Call varlen_attn >>> output = varlen_attn( ... query, key, value, cu_seq, cu_seq, max_len, max_len ... ) )r rN)r8r9 torch_attnrFlistr)rrrrr r!r"rRr$rr#outrrDr r rrs N  ctxinputs.rAc Cs\|\ }}}}}}} } } } |\} }}||||||| ||||_| |_| |_| |_| |_dSN)save_for_backwardr!r"r#r$r)rVrWrArrrrr r!r"r#r$rrUrrCr r r_setup_contexts&   rZz!torch_attn::_varlen_attn_backwardgrad_outrUrrCcCst| } tjd|jd}|jot|jj}|rFtd| ddks(| ddkr,t dtj j j |||||||||| d| | || d\}}}n$td tj j j |||||||||| d| | || | d| dd \}}}|||fS) Nr)r2r%r r&r'r(r)r+)r$r-r.)rr8rKr2r3rr4r5r6r7r9r:_cudnn_attention_backward_flash_attention_backward)r[rrrrUrrr r!r"r#rCr$runusedr?dqdkdvr r r_varlen_attn_backwards^   rbcCs0t| } t|}t|}t|}|||fS)zF Fake implementation for meta tensor computation and tracing. )rr8rG)r[rrrrUrrr r!r"r#rCr$r grad_querygrad_key grad_valuer r r_varlen_attn_backward_fake\s     rfgrad_lsegrad_rngcCsz|j\}}}}}} } } |j} |j} |j}|j}|j}tjj ||||| | ||| | || ||\}}}|||dddddddf SrX) saved_tensorsr!r"r#r$rr8r9rSrb)rVr[rgrhrrrrr rUrrCr!r"r#r$rr_r`rar r r _backwardys. rj) setup_context)FNN)NN) rlogging functoolsrtypingrrr8 getLoggerrr5__all__rTintrrrrlibrary custom_opTensorrLtuplerF register_fakerQrrZrbrfrjregister_autogradr r r rsX        E    4    "`       D