Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CUDA] cudaErrorInvalidConfiguration with Conv2DTranspose #71180

Open
jwnhy opened this issue Feb 18, 2025 · 1 comment
Open

[CUDA] cudaErrorInvalidConfiguration with Conv2DTranspose #71180

jwnhy opened this issue Feb 18, 2025 · 1 comment
Assignees

Comments

@jwnhy
Copy link

jwnhy commented Feb 18, 2025

bug描述 Describe the Bug

import paddle as pdl

model = pdl.nn.Conv2DTranspose(47, 47, kernel_size=[46, 46], padding=[46, 15], groups=47, data_format="NCHW")
tensor = pdl.rand([1, 47, 47, 47])
model(tensor)

其他补充信息 Additional Supplementary Information

========= COMPUTE-SANITIZER                                                                                                         16:23:53 [83/1230]
W0218 16:23:53.068138 1700471 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 9.0, Driver API Version: 12.8, Runtime API Version
: 12.3
W0218 16:23:53.069804 1700471 gpu_resources.cc:164] device: 0, cuDNN Version: 9.7.
========= Program hit cudaErrorInvalidConfiguration (error 9) due to "invalid configuration argument" on CUDA API call to cudaLaunchKernel.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x4aa4a5]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:cudaLaunchKernel [0x7466d]
=========                in /opt/ext/jwnhy/miniconda3/envs/pdl300/lib/python3.12/site-packages/paddle/base/../../../../libcudart.so.12
=========     Host Frame:paddle::operators::math::DepthwiseConvInputGradFunctor<phi::GPUContext, float, false>::operator()(phi::GPUContext const&, phi
::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator
<int> > const&, std::vector<int, std::allocator<int> > const&, phi::DenseTensor*, common::DataLayout) [0x156ce8c]
=========                in /opt/ext/jwnhy/miniconda3/envs/pdl300/lib/python3.12/site-packages/paddle/base/../libs/libphi_kernel_gpu.so
=========     Host Frame:void phi::DepthwiseConv2dTransposeKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseT
ensor const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > con
st&, paddle::experimental::IntArrayBase<phi::DenseTensor> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > cons
t&, int, std::vector<int, std::allocator<int> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, phi::D
enseTensor*) [0x15d1470]
=========                in /opt/ext/jwnhy/miniconda3/envs/pdl300/lib/python3.12/site-packages/paddle/base/../libs/libphi_kernel_gpu.so
=========     Host Frame:paddle::experimental::depthwise_conv2d_transpose(paddle::Tensor const&, paddle::Tensor const&, std::vector<int, std::allocato
r<int> > const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, paddle::experimental::IntArrayBase<padd
le::Tensor> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::vector<int, std::allocator<int>
> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [0x281bf23]
=========                in /opt/ext/jwnhy/miniconda3/envs/pdl300/lib/python3.12/site-packages/paddle/base/../libs/libphi.so
=========     Host Frame:depthwise_conv2d_transpose_ad_func(paddle::Tensor const&, paddle::Tensor const&, std::vector<int, std::allocator<int> >, std:
:vector<int, std::allocator<int> >, std::vector<int, std::allocator<int> >, paddle::experimental::IntArrayBase<paddle::Tensor>, std::__cxx11::basic_st
ring<char, std::char_traits<char>, std::allocator<char> >, int, std::vector<int, std::allocator<int> >, std::__cxx11::basic_string<char, std::char_tra
its<char>, std::allocator<char> >) [0x6b0cea5]
=========                in /opt/ext/jwnhy/miniconda3/envs/pdl300/lib/python3.12/site-packages/paddle/base/libpaddle.so
=========     Host Frame:paddle::pybind::eager_api_depthwise_conv2d_transpose(_object*, _object*, _object*) [0x4c37b0a]
=========                in /opt/ext/jwnhy/miniconda3/envs/pdl300/lib/python3.12/site-packages/paddle/base/libpaddle.so
=========     Host Frame:cfunction_call in /usr/local/src/conda/python-3.12.9/Objects/methodobject.c:537 [0x14d2d3]
=========                in /opt/ext/jwnhy/miniconda3/envs/pdl300/bin/python3
=========     Host Frame:_PyObject_MakeTpCall in /usr/local/src/conda/python-3.12.9/Objects/call.c:240 [0x11e38a]
=========                in /opt/ext/jwnhy/miniconda3/envs/pdl300/bin/python3
W0218 16:26:02.785025 1701551 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 9.0, Driver API Version: 12.8, Runtime API Version: 12.3
W0218 16:26:02.785495 1701551 gpu_resources.cc:164] device: 0, cuDNN Version: 9.7.
Traceback (most recent call last):
  File "/home/jwnhy/gpu_fuzz/poc/poc13.py", line 5, in <module>
    model(tensor)
  File "/opt/ext/jwnhy/miniconda3/envs/pdl300/lib/python3.12/site-packages/paddle/nn/layer/layers.py", line 1426, in __call__
    return self.forward(*inputs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/ext/jwnhy/miniconda3/envs/pdl300/lib/python3.12/site-packages/paddle/nn/layer/conv.py", line 883, in forward
    out = F.conv2d_transpose(
          ^^^^^^^^^^^^^^^^^^^
  File "/opt/ext/jwnhy/miniconda3/envs/pdl300/lib/python3.12/site-packages/paddle/nn/functional/conv.py", line 1275, in conv2d_transpose
    return _add_with_axis(pre_bias, bias, axis=channel_dim)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/ext/jwnhy/miniconda3/envs/pdl300/lib/python3.12/site-packages/paddle/tensor/math.py", line 1152, in _add_with_axis
    return _elementwise_op_with_axis(x, y, axis, name, "add")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/ext/jwnhy/miniconda3/envs/pdl300/lib/python3.12/site-packages/paddle/tensor/math.py", line 1146, in _elementwise_op_with_axis
    return op(x, y)
           ^^^^^^^^
RuntimeError: (PreconditionNotMet) The meta data must be valid when call the mutable data function.
  [Hint: Expected valid() == true, but received valid():0 != true:1.] (at ../paddle/phi/core/dense_tensor.cc:111)
@LiYuRio
Copy link
Contributor

LiYuRio commented Feb 18, 2025

api参数值设置不合理,导致在计算过程中得到了0维tensor,导致报错

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants