shamim

2023-06-03 12:36:58

EPOCH: 2 train_loss: 2.484 valid_loss: 2.466 train_acc: 0.3220 valid_accs: 0.3880

2023-06-03 12:37:22

TRAIN: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 32/32 [00:24<00:00, 1.92it/s]

2023-06-03 12:37:28

EPOCH: 12%|████████████████████▎ | 3/25 [01:28<10:50, 29.55s/it]

2023-06-03 12:37:28

EPOCH: 3 train_loss: 2.460 valid_loss: 2.442 train_acc: 0.3369 valid_accs: 0.3775

2023-06-03 12:37:40

EPOCH: 12%|████████████████████▎ | 3/25 [01:40<12:16, 33.50s/it]

2023-06-03 12:37:40

Traceback (most recent call last):

2023-06-03 12:37:40

File "/home/ActionNet/src/train.py", line 177, in <module>

2023-06-03 12:37:40

train_loss, train_acc = train((imgs, targets), model, optimizer, loss_fn, scaler)

2023-06-03 12:37:40

File "/home/ActionNet/src/train.py", line 45, in train

2023-06-03 12:37:40

scaler.step(optimizer)

2023-06-03 12:37:40

File "/opt/conda/lib/python3.10/site-packages/torch/cuda/amp/grad_scaler.py", line 374, in step

2023-06-03 12:37:40

retval = self._maybe_opt_step(optimizer, optimizer_state, *args, **kwargs)

2023-06-03 12:37:40

File "/opt/conda/lib/python3.10/site-packages/torch/cuda/amp/grad_scaler.py", line 289, in _maybe_opt_step

2023-06-03 12:37:40

if not sum(v.item() for v in optimizer_state["found_inf_per_device"].values()):

2023-06-03 12:37:40

File "/opt/conda/lib/python3.10/site-packages/torch/cuda/amp/grad_scaler.py", line 289, in <genexpr>

2023-06-03 12:37:40

if not sum(v.item() for v in optimizer_state["found_inf_per_device"].values()):

2023-06-03 12:37:40

KeyboardInterrupt

Shamim's group workspace