Skip to main content
kastan
Projects
col_ai
Run groups
gpt2_zero3
Logs
Group
Workspace
Runs
Logs
Files
Artifacts

Kastan's group workspace

Timestamps visible
2022-06-22 16:32:55
    self.model.backward(loss)
2022-06-22 16:32:55
  File "/u/kastan/.conda/envs/pytorch_1.11_colossal/lib/python3.8/site-packages/colossalai/zero/sharded_model/sharded_model_v2.py", line 222, in backward
2022-06-22 16:32:55
    loss.backward()
2022-06-22 16:32:55
  File "/u/kastan/.conda/envs/pytorch_1.11_colossal/lib/python3.8/site-packages/torch/_tensor.py", line 363, in backward
2022-06-22 16:32:55
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
2022-06-22 16:32:55
  File "/u/kastan/.conda/envs/pytorch_1.11_colossal/lib/python3.8/site-packages/torch/autograd/__init__.py", line 173, in backward
2022-06-22 16:32:55
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-06-22 16:32:55
  File "/u/kastan/.conda/envs/pytorch_1.11_colossal/lib/python3.8/site-packages/wandb/wandb_torch.py", line 264, in <lambda>
2022-06-22 16:32:55
    handle = var.register_hook(lambda grad: _callback(grad, log_track))
2022-06-22 16:32:55
  File "/u/kastan/.conda/envs/pytorch_1.11_colossal/lib/python3.8/site-packages/wandb/wandb_torch.py", line 262, in _callback
2022-06-22 16:32:55
    self.log_tensor_stats(grad.data, name)
2022-06-22 16:32:55
  File "/u/kastan/.conda/envs/pytorch_1.11_colossal/lib/python3.8/site-packages/wandb/wandb_torch.py", line 162, in log_tensor_stats
2022-06-22 16:32:55
    flat = tensor.reshape(-1)
2022-06-22 16:32:55
RuntimeError: setStorage: sizes [768], strides [1], storage offset 0, and itemsize 2 requiring a storage size of 1536 are out of bounds for storage of size 0