o ki( @sUdZddlZddlZgZeeed<dejdefddZ dejd e dejfd d Z d e d e dBde fddZ     ddejdejdejdejdBddf ddZdS)zCDefines utilities for interacting with scaled_dot_product_attentionN__all__tensorsreturncGstdd|DS)z0Returns True if any of the tensors requires gradcss|]}|jVqdS)N) requires_grad).0trc/var/www/addictedbytheproject.nl/epg/venv/lib/python3.10/site-packages/torch/nn/attention/_utils.py sz'_input_requires_grad..)any)rrrr _input_requires_grad sr inpt_tensorog_sizecCs"|d|kr|dd|fS|S)z'Handles the unpad of the last dimension.N)size)r rrrr _postprocess_flash_outputsr head_dim_sizescalecCs|dur|Sdt|S)z For FlashAttention we pad the head dimension to be a multiple of 8 so we need to scale the output by the original head size and not the padded. Ng?)mathsqrt)rrrrr _calculate_scalesrFquerykeyvalue attn_maskcCs|s|j|jks|j|jkrtd|jd|jd|jd|j|jks+|j|jkrs.