Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[eager_fail_to_run] opacus_cifar10 #2568

Open
IvanKobzarev opened this issue Jan 16, 2025 · 0 comments
Open

[eager_fail_to_run] opacus_cifar10 #2568

IvanKobzarev opened this issue Jan 16, 2025 · 0 comments

Comments

@IvanKobzarev
Copy link

Running from pytorch main

python benchmarks/dynamo/torchbench.py --only opacus_cifar10 --accuracy --no-translation-validation --training --amp --backend inductor --device cuda --output out.csv
cuda train opacus_cifar10
Traceback (most recent call last):
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 2744, in validate_model
    self.model_iter_fn(model, example_inputs)
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/torchbench.py", line 455, in forward_and_backward_pass
    self.grad_scaler.scale(loss).backward()
  File "/data/users/ivankobzarev/a/pytorch/torch/_tensor.py", line 648, in backward
    torch.autograd.backward(
  File "/data/users/ivankobzarev/a/pytorch/torch/autograd/__init__.py", line 353, in backward
    _engine_run_backward(
  File "/data/users/ivankobzarev/a/pytorch/torch/autograd/graph.py", line 815, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "/data/users/ivankobzarev/a/pytorch/torch/nn/modules/module.py", line 86, in __call__
    return self.hook(module, *args, **kwargs)
  File "/data/users/ivankobzarev/a/opacus/opacus/grad_sample/grad_sample_module.py", line 327, in capture_backprops_hook
    activations, backprops = self.rearrange_grad_samples(
  File "/data/users/ivankobzarev/a/opacus/opacus/grad_sample/grad_sample_module.py", line 383, in rearrange_grad_samples
    raise ValueError(
ValueError: No activations detected for <class 'torch.nn.modules.linear.Linear'>, run forward after add_hooks(model)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 4857, in run
    ) = runner.load_model(
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/torchbench.py", line 372, in load_model
    self.validate_model(model, example_inputs)
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 2746, in validate_model
    raise RuntimeError("Eager run failed") from e
RuntimeError: Eager run failed

eager_fail_to_run
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant