Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Recursion Error when setting DataLoader Random Seed with Hydra configuration #8087

Open
adwaye opened this issue Sep 17, 2024 · 0 comments
Open

Recursion Error when setting DataLoader Random Seed with Hydra configuration #8087

adwaye opened this issue Sep 17, 2024 · 0 comments

Comments

@adwaye
Copy link

adwaye commented Sep 17, 2024

Describe the bug
When creating a monai dataloader inside a hydra app, I get the following recursion error

RecursionError: maximum recursion depth exceeded in comparison
    full_key: set_random_state
    object_type=<DataclassName>

I have created a minimum working example below.

To Reproduce
Steps to reproduce the behavior:

  1. create the following script monai_hydra.py
from monai.data import DataLoader
from hydra.core.config_store import ConfigStore
from dataclasses import dataclass, field
from monai.data.dataset import  Dataset
import torch
from hydra import compose, initialize
import hydra

@dataclass
class LoaderConfig:
    csv_path: str = './'
    spatial_size: list = field(default_factory=lambda: [128,128,128])

class MyDataset(Dataset):
    def __init__(self,csv_path, spatial_size):
        self.csv_path = csv_path
        self.spatial_size = spatial_size

    def __len__(self):
        return 10

    def __getitem__(self, idx):
        return torch.randn(self.spatial_size)


@hydra.main(config_name = 'loader_config')
def main(cfg) -> None:
    print(cfg)
    my_dataset = MyDataset(
        csv_path = cfg.csv_path,
        spatial_size = cfg.spatial_size
    )
    my_dataloader = DataLoader(
        my_dataset,batch_size=1,num_workers=0,shuffle=False
    )

if __name__=='__main__':
    cs = ConfigStore()
    cs.store(name='loader_config',node=LoaderConfig)
    with initialize(version_base=None):
        cfg = compose(config_name='loader_config',overrides=[])
    main(cfg)
  1. run the following in the terminal python monai_hydra.py

Note that changing the above to have the following in the excutable path makes the above run fine.

  if __name__=='__main__':
      # cs = ConfigStore()
      # cs.store(name='loader_config',node=LoaderConfig)
      # with initialize(version_base=None):
      #     cfg = compose(config_name='loader_config',overrides=[])
      cfg = LoaderConfig()
      main(cfg)

Expected behavior
The dataloader should initialise without any issues.

Environment

Ensuring you use the relevant python executable, please paste the output of:

python -c "import monai; monai.config.print_debug_info()"

/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/utils/tf32.py:66: UserWarning: torch.backends.cuda.matmul.allow_tf32 = True by default.
  This value defaults to True when PyTorch version in [1.7, 1.11] and may affect precision.
  See https://docs.monai.io/en/latest/precision_accelerating.html#precision-and-accelerating
  warnings.warn(
================================
Printing MONAI config...
================================
MONAI version: 1.3.1
Numpy version: 1.24.3
Pytorch version: 1.11.0
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: 96bfda00c6bd290297f5e3514ea227c6be4d08b4
MONAI __file__: /home/<username>/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/__init__.py

Optional dependencies:
Pytorch Ignite version: NOT INSTALLED or UNKNOWN VERSION.
ITK version: NOT INSTALLED or UNKNOWN VERSION.
Nibabel version: NOT INSTALLED or UNKNOWN VERSION.
scikit-image version: 0.20.0
scipy version: 1.9.1
Pillow version: 9.4.0
Tensorboard version: 2.13.0
gdown version: NOT INSTALLED or UNKNOWN VERSION.
TorchVision version: 0.12.0
tqdm version: 4.65.0
lmdb version: NOT INSTALLED or UNKNOWN VERSION.
psutil version: 5.9.8
pandas version: 1.5.3
einops version: 0.3.0
transformers version: 4.19.2
mlflow version: NOT INSTALLED or UNKNOWN VERSION.
pynrrd version: NOT INSTALLED or UNKNOWN VERSION.
clearml version: NOT INSTALLED or UNKNOWN VERSION.

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies


================================
Printing system config...
================================
System: Linux
Linux version: Ubuntu 22.04.4 LTS
Platform: Linux-6.8.0-40-generic-x86_64-with-glibc2.10
Processor: x86_64
Machine: x86_64
Python version: 3.8.5
Process name: python
Command: ['python', '-c', 'import monai; monai.config.print_debug_info()']
Open files: []
Num physical CPUs: 6
Num logical CPUs: 12
Num usable CPUs: 12
CPU usage (%): [8.0, 7.5, 9.3, 6.8, 6.2, 77.2, 8.6, 8.0, 8.1, 9.2, 9.9, 28.8]
CPU freq. (MHz): 3657
Load avg. in last 1, 5, 15 mins (%): [5.4, 5.6, 4.4]
Disk usage (%): 79.3
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 15.3
Available memory (GB): 6.9
Used memory (GB): 8.0

================================
Printing GPU config...
================================
Num GPUs: 1
Has CUDA: True
CUDA version: 11.3
cuDNN enabled: True
NVIDIA_TF32_OVERRIDE: None
TORCH_ALLOW_TF32_CUBLAS_OVERRIDE: None
cuDNN version: 8200
Current device: 0
Library compiled for CUDA architectures: ['sm_37', 'sm_50', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'compute_37']
GPU 0 Name: NVIDIA GeForce GTX 1650 with Max-Q Design
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 16
GPU 0 Total memory (GB): 3.8
GPU 0 CUDA capability (maj.min): 7.5

Also using hydra-core 1.3.2

Full trace

  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/hydra/main.py", line 83, in decorated_main
    return task_function(cfg_passthrough)
  File "monai_hydra.py", line 35, in main
    my_dataloader = DataLoader(
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/dataloader.py", line 87, in __init__
    set_rnd(dataset, int(_seed))
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 727, in set_rnd
    _seed = set_rnd(item, seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 727, in set_rnd
    _seed = set_rnd(item, seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 727, in set_rnd
    _seed = set_rnd(item, seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 727, in set_rnd
    _seed = set_rnd(item, seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 727, in set_rnd
    _seed = set_rnd(item, seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 727, in set_rnd
    _seed = set_rnd(item, seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 727, in set_rnd
    _seed = set_rnd(item, seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 727, in set_rnd
    _seed = set_rnd(item, seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 359, in __getattr__
    self._format_and_raise(key=key, value=None, cause=e)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/_utils.py", line 899, in format_and_raise
    _raise(ex, cause)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 351, in __getattr__
    return self._get_impl(
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 442, in _get_impl
    node = self._get_child(
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 73, in _get_child
    child = self._get_node(
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 475, in _get_node
    self._validate_get(key)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 164, in _validate_get
    self._format_and_raise(
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/_utils.py", line 843, in format_and_raise
    object_type = OmegaConf.get_type(node)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/omegaconf.py", line 650, in get_type
    return OmegaConf._get_obj_type(c)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/omegaconf.py", line 905, in _get_obj_type
    if is_structured_config(c):
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/_utils.py", line 448, in is_structured_config
    return is_attr_class(obj) or is_dataclass(obj)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/_utils.py", line 442, in is_attr_class
    if attr is None or isinstance(obj, Node):
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/abc.py", line 98, in __instancecheck__
    return _abc_instancecheck(cls, instance)
RecursionError: maximum recursion depth exceeded in comparison
    full_key: set_random_state
    object_type=LoaderConfig
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@adwaye