kimsehun725

Kimsehun725's group workspace

Group: 20221014_2325

247

1-1

of 1

Timestamps visible

2022-10-14 14:55:22

Epoch 170: 89%|██████████████████████████████████████████████████████▏ | 8/9 [00:04<00:00, 1.82it/s, loss=0.0254, v_num=xvuq, val/loss=0.0174, val/precision=0.841, val/recall=0.896, val/f1=0.868, train/loss=0.0251]

2022-10-14 14:55:32

Epoch 171: 89%|██████████████████████████████████████████████████████▏ | 8/9 [00:04<00:00, 1.90it/s, loss=0.0258, v_num=xvuq, val/loss=0.0176, val/precision=0.838, val/recall=0.881, val/f1=0.859, train/loss=0.0256]

2022-10-14 14:55:42

Epoch 172: 89%|███████████████████████████████████████████████████████ | 8/9 [00:04<00:00, 1.85it/s, loss=0.0259, v_num=xvuq, val/loss=0.0166, val/precision=0.840, val/recall=0.880, val/f1=0.859, train/loss=0.025]

2022-10-14 14:55:52

Epoch 173: 89%|███████████████████████████████████████████████████████ | 8/9 [00:04<00:00, 1.85it/s, loss=0.0247, v_num=xvuq, val/loss=0.017, val/precision=0.857, val/recall=0.901, val/f1=0.879, train/loss=0.0252]

2022-10-14 14:56:04

Epoch 174: 89%|████████████████████████████████████████████████████████ | 8/9 [00:04<00:00, 1.82it/s, loss=0.025, v_num=xvuq, val/loss=0.019, val/precision=0.836, val/recall=0.882, val/f1=0.858, train/loss=0.0243]

2022-10-14 14:56:12

Epoch 175: 89%|██████████████████████████████████████████████████████▏ | 8/9 [00:04<00:00, 1.89it/s, loss=0.0255, v_num=xvuq, val/loss=0.0177, val/precision=0.852, val/recall=0.901, val/f1=0.876, train/loss=0.0246]

2022-10-14 14:56:24

Epoch 176: 89%|██████████████████████████████████████████████████████▏ | 8/9 [00:04<00:00, 1.87it/s, loss=0.0254, v_num=xvuq, val/loss=0.0183, val/precision=0.839, val/recall=0.902, val/f1=0.869, train/loss=0.0257]

2022-10-14 14:56:34

Epoch 177: 89%|██████████████████████████████████████████████████████▏ | 8/9 [00:04<00:00, 1.74it/s, loss=0.0256, v_num=xvuq, val/loss=0.0175, val/precision=0.840, val/recall=0.905, val/f1=0.871, train/loss=0.0243]

2022-10-14 14:56:44

Epoch 178: 89%|██████████████████████████████████████████████████████▏ | 8/9 [00:04<00:00, 1.86it/s, loss=0.0258, v_num=xvuq, val/loss=0.0169, val/precision=0.852, val/recall=0.887, val/f1=0.869, train/loss=0.0251]

2022-10-14 14:56:54

Epoch 179: 89%|███████████████████████████████████████████████████████ | 8/9 [00:04<00:00, 1.90it/s, loss=0.0254, v_num=xvuq, val/loss=0.017, val/precision=0.852, val/recall=0.893, val/f1=0.872, train/loss=0.0253]

2022-10-14 14:57:04

Epoch 180: 89%|███████████████████████████████████████████████████████ | 8/9 [00:04<00:00, 1.85it/s, loss=0.0252, v_num=xvuq, val/loss=0.019, val/precision=0.841, val/recall=0.895, val/f1=0.867, train/loss=0.0248]

2022-10-14 14:57:16

Epoch 181: 89%|██████████████████████████████████████████████████████▏ | 8/9 [00:04<00:00, 1.86it/s, loss=0.0249, v_num=xvuq, val/loss=0.0187, val/precision=0.847, val/recall=0.889, val/f1=0.867, train/loss=0.0248]

2022-10-14 14:57:26

Epoch 182: 89%|████████████████████████████████████████████████████████ | 8/9 [00:04<00:00, 1.86it/s, loss=0.025, v_num=xvuq, val/loss=0.0178, val/precision=0.849, val/recall=0.894, val/f1=0.871, train/loss=0.025]

2022-10-14 14:57:36

Epoch 183: 89%|██████████████████████████████████████████████████████▏ | 8/9 [00:04<00:00, 1.90it/s, loss=0.0258, v_num=xvuq, val/loss=0.0168, val/precision=0.863, val/recall=0.889, val/f1=0.876, train/loss=0.0256]

2022-10-14 14:57:46

Epoch 184: 89%|██████████████████████████████████████████████████████▏ | 8/9 [00:04<00:00, 1.87it/s, loss=0.0256, v_num=xvuq, val/loss=0.0167, val/precision=0.841, val/recall=0.899, val/f1=0.869, train/loss=0.0254]

2022-10-14 14:57:56

Epoch 185: 89%|██████████████████████████████████████████████████████▏ | 8/9 [00:04<00:00, 1.80it/s, loss=0.0247, v_num=xvuq, val/loss=0.0178, val/precision=0.851, val/recall=0.891, val/f1=0.871, train/loss=0.0256]

2022-10-14 14:58:06

Epoch 186: 89%|██████████████████████████████████████████████████████▏ | 8/9 [00:04<00:00, 1.88it/s, loss=0.0249, v_num=xvuq, val/loss=0.0153, val/precision=0.860, val/recall=0.895, val/f1=0.877, train/loss=0.0243]

2022-10-14 14:58:28

Epoch 188: 89%|██████████████████████████████████████████████████████▏ | 8/9 [00:04<00:00, 1.88it/s, loss=0.0261, v_num=xvuq, val/loss=0.0168, val/precision=0.856, val/recall=0.896, val/f1=0.875, train/loss=0.0256]

2022-10-14 14:58:18

Validation: 0it [00:00, ?it/s]

2022-10-14 14:58:38

Epoch 189: 89%|██████████████████████████████████████████████████████▏ | 8/9 [00:04<00:00, 1.80it/s, loss=0.0252, v_num=xvuq, val/loss=0.0183, val/precision=0.853, val/recall=0.898, val/f1=0.875, train/loss=0.0252]

2022-10-14 14:58:48

Epoch 190: 89%|██████████████████████████████████████████████████████▏ | 8/9 [00:04<00:00, 1.87it/s, loss=0.0249, v_num=xvuq, val/loss=0.0167, val/precision=0.842, val/recall=0.878, val/f1=0.859, train/loss=0.0251]

2022-10-14 14:58:58

Epoch 191: 89%|███████████████████████████████████████████████████████ | 8/9 [00:04<00:00, 1.89it/s, loss=0.0247, v_num=xvuq, val/loss=0.0141, val/precision=0.859, val/recall=0.901, val/f1=0.879, train/loss=0.025]

2022-10-14 14:59:14

Testing DataLoader 0: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:04<00:00, 2.45s/it]

2022-10-14 14:59:04

`Trainer.fit` stopped: `max_epochs=192` reached.

2022-10-14 14:59:06

/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1386: UserWarning: `.test(ckpt_path=None)` was called without a model. The best model of the previous `fit` call will be used. You can pass `.test(ckpt_path='best')` to use the best model or `.test(ckpt_path='last')` to use the last model. If you pass a value, this warning will be silenced.

2022-10-14 14:59:06

rank_zero_warn(

2022-10-14 14:59:06

Restoring states from the checkpoint path at /data/group1/z44543r/vae_separation/trained_model/20221014_2325/test00/epoch=191.ckpt

2022-10-14 14:59:06

LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]

2022-10-14 14:59:06

Loaded model weights from checkpoint at /data/group1/z44543r/vae_separation/trained_model/20221014_2325/test00/epoch=191.ckpt

2022-10-14 14:59:06

/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:489: PossibleUserWarning: Your `test_dataloader`'s sampler has shuffling enabled, it is strongly recommended that you turn shuffling off for val/test/predict dataloaders.

2022-10-14 14:59:06

rank_zero_warn(

2022-10-14 14:59:06

/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:323: PossibleUserWarning: Using `DistributedSampler` with the dataloaders. During `trainer.test()`, it is recommended to use `Trainer(devices=1, num_nodes=1)` to ensure each sample/batch gets evaluated exactly once. Otherwise, multi-device settings use `DistributedSampler` that replicates some samples to make sure all devices have same batch size in case of uneven inputs.

2022-10-14 14:59:06

rank_zero_warn(

2022-10-14 14:59:06

Testing: 0it [00:00, ?it/s]

2022-10-14 14:59:14

Traceback (most recent call last):

2022-10-14 14:59:14

File "src/train.py", line 65, in <module>

2022-10-14 14:59:14

main(kwargs)

2022-10-14 14:59:14

File "src/train.py", line 25, in main

2022-10-14 14:59:14

step3(now, kwargs)

2022-10-14 14:59:14

File "/data/group1/z44543r/vae_separation/src/step3.py", line 209, in step3

2022-10-14 14:59:14

train(kwargs, use_pretrained_model, pretrained_time, pretrained_epoch, now, test_num, train_data_list,

2022-10-14 14:59:14

File "/data/group1/z44543r/vae_separation/src/step3.py", line 155, in train

2022-10-14 14:59:14

model.log('test/precision', model.metrics_container.avg_p, on_epoch=True,

2022-10-14 14:59:14

File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/core/module.py", line 430, in log

2022-10-14 14:59:14

raise MisconfigurationException(

2022-10-14 14:59:14

pytorch_lightning.utilities.exceptions.MisconfigurationException: You are trying to `self.log()` but the loop's result collection is not registered yet. This is most likely because you are trying to log in a `predict` hook, but it doesn't support logging

2022-10-14 14:59:14

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

2022-10-14 14:59:14

Test metric DataLoader 0

2022-10-14 14:59:14

test/loss 0.06354475766420364

2022-10-14 14:59:14