Skip to main content
kastan
Projects
LLM-Distributed-Quantization
Run groups
Aug-05__11:07
Logs
Group
Workspace
Runs
Logs
Files
Artifacts

Kastan's group workspace

Timestamps visible
2022-08-05 16:08:17
    output, label, loss = self._schedule.forward_backward_step(self, data_iter, **kwargs)
2022-08-05 16:08:17
  File "/u/kastanday/.conda/envs/nice_base/envs/col_ai_quant/lib/python3.9/site-packages/colossalai/engine/schedule/_pipeline_schedule.py", line 419, in forward_backward_step
2022-08-05 16:08:17
    input_obj_grad = self._backward_step(engine, input_obj, output_obj, output_obj_grad)
2022-08-05 16:08:17
  File "/u/kastanday/.conda/envs/nice_base/envs/col_ai_quant/lib/python3.9/site-packages/colossalai/engine/schedule/_pipeline_schedule.py", line 296, in _backward_step
2022-08-05 16:08:17
    engine.backward_by_grad(output_obj, output_obj_grad)
2022-08-05 16:08:17
  File "/u/kastanday/.conda/envs/nice_base/envs/col_ai_quant/lib/python3.9/site-packages/colossalai/engine/_base_engine.py", line 175, in backward_by_grad
2022-08-05 16:08:17
    ret = self.optimizer.backward_by_grad(tensor, grad)
2022-08-05 16:08:17
  File "/u/kastanday/.conda/envs/nice_base/envs/col_ai_quant/lib/python3.9/site-packages/colossalai/engine/gradient_accumulation/_gradient_accumulation.py", line 111, in backward_by_grad
2022-08-05 16:08:17
    self.optim.backward_by_grad(tensor, grad)
2022-08-05 16:08:17
  File "/u/kastanday/.conda/envs/nice_base/envs/col_ai_quant/lib/python3.9/site-packages/colossalai/nn/optimizer/colossalai_optimizer.py", line 43, in backward_by_grad
2022-08-05 16:08:17
    torch.autograd.backward(tensors=tensor, grad_tensors=grad)
2022-08-05 16:08:17
  File "/u/kastanday/.conda/envs/nice_base/envs/col_ai_quant/lib/python3.9/site-packages/torch/autograd/__init__.py", line 173, in backward
2022-08-05 16:08:17
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-08-05 16:08:17
RuntimeError: element 1 of tensors does not require grad and does not have a grad_fn