sageattn_qk_int8_pv_fp16_cuda black output with pv_accum fp16, results in black screen in opensora

[nighting0le01]

sageattn_qk_int8_pv_fp16_cuda black output with pv_accum fp16, results in black screen in opensora help

[jason-huang03]

which version of opensora do you use?

[nighting0le01]

i use 1.2 with bf16

[asahni04]

@jason-huang03 @jt-zhang similar issues for me, any updates ?? my issue:#94

[asahni04]

? @jason-huang03 @jt-zhang any suggestions?

[jason-huang03]

Have you tried kernels that offers higher precision? like those with fp32 or fp16+fp32 accumulation?

[jason-huang03]

fp16 has a limited range and may encounter overflow error as the accumulator.

[asahni04]

hi jason i did @jason-huang03 specifically for i2i it runs into that issue for t2i it seems ok with fp16 accum. with fp32 and fp16+fp32 is does work but is barely faster than fa-2. i even tried with v_smooth and it does not solve this black/nan issue. please suggest what i can do to speedup on A100. qk_quant_gran will qk_quant_gran have a effect?

[jason-huang03]

I believe qk_quant_gran will not have an effect because your issue seems to be an overflow problem. By the way, what is the sequence length of attention in the model?

[nighting0le01]

I believe qk_quant_gran will not have an effect because your issue seems to be an overflow problem. By the way, what is the sequence length of attention in the model?

hey @jason-huang03 would qk_quant_gran atleast speedup fp16+fp32? what's the tradeoffs associated with its setting??

[jason-huang03]

I believe "per_warp" might be a little faster, "per_thread" will be a little more accurate.