kimsehun725

Kimsehun725's group workspace

Group: 20221017_1717

1-1

of 1

Timestamps visible

2022-10-17 08:17:20

GPU available: True (cuda), used: True

2022-10-17 08:17:20

TPU available: False, using: 0 TPU cores

2022-10-17 08:17:20

IPU available: False, using: 0 IPUs

2022-10-17 08:17:20

HPU available: False, using: 0 HPUs

2022-10-17 08:17:26

Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/4

2022-10-17 08:17:30

----------------------------------------------------------------------------------------------------

2022-10-17 08:17:30

distributed_backend=nccl

2022-10-17 08:17:30

All distributed processes registered. Starting with 4 processes

2022-10-17 08:17:30

----------------------------------------------------------------------------------------------------

2022-10-17 08:17:46

LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]

2022-10-17 08:17:46

| Name | Type | Params

2022-10-17 08:17:46

--------------------------------------------

2022-10-17 08:17:46

0 | encoder | Conv1DEncoder | 1.8 M

2022-10-17 08:17:46

1 | fc1 | Linear | 23.1 K

2022-10-17 08:17:46

2 | fc2 | Linear | 23.1 K

2022-10-17 08:17:46

3 | fc3 | Linear | 23.4 K

2022-10-17 08:17:46

4 | decoder_0 | Conv1DDecoder | 1.8 M

2022-10-17 08:17:46

5 | decoder_1 | Conv1DDecoder | 1.8 M

2022-10-17 08:17:46

6 | decoder_2 | Conv1DDecoder | 1.8 M

2022-10-17 08:17:46

7 | decoder_3 | Conv1DDecoder | 1.8 M

2022-10-17 08:17:46

8 | decoder_4 | Conv1DDecoder | 1.8 M

2022-10-17 08:17:46

9 | decoder_5 | Conv1DDecoder | 1.8 M

2022-10-17 08:17:46

--------------------------------------------

2022-10-17 08:17:46

13.0 M Trainable params

2022-10-17 08:17:46

0 Non-trainable params

2022-10-17 08:17:46

13.0 M Total params

2022-10-17 08:17:46

51.971 Total estimated model params size (MB)

2022-10-17 08:17:56

Epoch 0: 89%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 8/9 [00:05<00:00, 1.52it/s, loss=3.25e+05, v_num=lici]

2022-10-17 08:17:50

/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py:535: PossibleUserWarning: It is recommended to use `self.log('val/loss', ..., sync_dist=True)` when logging on epoch level in distributed setting to accumulate the metric across devices.

2022-10-17 08:17:50

warning_cache.warn(

2022-10-17 08:17:52

/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1892: PossibleUserWarning: The number of training batches (8) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.

2022-10-17 08:17:52

rank_zero_warn(

2022-10-17 08:17:52

Epoch 0: 0%| | 0/9 [00:00<?, ?it/s]

2022-10-17 08:18:16

Epoch 1: 89%|███████████████████████████████████████████████████████████████████████████████████████████████████████ | 8/9 [00:12<00:01, 1.55s/it, loss=3.13e+05, v_num=lici, val/loss=1.4e+5, train/loss=3.29e+5]

2022-10-17 08:18:04

/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py:535: PossibleUserWarning: It is recommended to use `self.log('train/loss', ..., sync_dist=True)` when logging on epoch level in distributed setting to accumulate the metric across devices.

2022-10-17 08:18:04

warning_cache.warn(

2022-10-17 08:18:25

Epoch 2: 89%|███████████████████████████████████████████████████████████████████████████████████████████████████████ | 8/9 [00:04<00:00, 1.70it/s, loss=2.93e+05, v_num=lici, val/loss=1.4e+5, train/loss=3.03e+5]

2022-10-17 08:18:43

Epoch 3: 89%|██████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 8/9 [00:14<00:01, 1.83s/it, loss=2.77e+05, v_num=lici, val/loss=1.57e+5, train/loss=2.82e+5]

2022-10-17 08:18:53

Epoch 4: 89%|███████████████████████████████████████████████████████████████████████████████████████████████████████ | 8/9 [00:04<00:00, 1.72it/s, loss=2.6e+05, v_num=lici, val/loss=1.69e+5, train/loss=2.67e+5]

2022-10-17 08:19:01

Epoch 5: 89%|████████████████████████████████████████████████████████████████████████████████████████████████████████ | 8/9 [00:04<00:00, 1.79it/s, loss=2.4e+05, v_num=lici, val/loss=2.44e+5, train/loss=2.5e+5]

2022-10-17 08:19:19

Epoch 7: 89%|██████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 8/9 [00:04<00:00, 1.68it/s, loss=2.07e+05, v_num=lici, val/loss=6.82e+5, train/loss=2.13e+5]

2022-10-17 08:19:27

Epoch 8: 89%|██████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 8/9 [00:04<00:00, 1.71it/s, loss=1.98e+05, v_num=lici, val/loss=2.84e+5, train/loss=1.99e+5]

2022-10-17 08:19:45

Epoch 10: 89%|█████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 8/9 [00:04<00:00, 1.72it/s, loss=1.79e+05, v_num=lici, val/loss=2.86e+5, train/loss=1.81e+5]

2022-10-17 08:19:37

Validation DataLoader 0: 0%| | 0/1 [00:00<?, ?it/s]

2022-10-17 08:19:53

Epoch 11: 89%|█████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 8/9 [00:04<00:00, 1.66it/s, loss=1.67e+05, v_num=lici, val/loss=1.92e+5, train/loss=1.71e+5]

2022-10-17 08:20:03

Epoch 12: 89%|█████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 8/9 [00:04<00:00, 1.73it/s, loss=1.61e+05, v_num=lici, val/loss=1.66e+5, train/loss=1.65e+5]

2022-10-17 08:20:11

Epoch 13: 89%|██████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 8/9 [00:04<00:00, 1.75it/s, loss=1.59e+05, v_num=lici, val/loss=2.32e+5, train/loss=1.6e+5]

2022-10-17 08:20:29

Epoch 15: 89%|█████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 8/9 [00:04<00:00, 1.78it/s, loss=1.56e+05, v_num=lici, val/loss=1.76e+5, train/loss=1.58e+5]

2022-10-17 08:20:37

Epoch 16: 89%|█████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 8/9 [00:04<00:00, 1.71it/s, loss=1.55e+05, v_num=lici, val/loss=1.67e+5, train/loss=1.59e+5]

2022-10-17 08:20:55

Epoch 18: 89%|█████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 8/9 [00:04<00:00, 1.70it/s, loss=1.48e+05, v_num=lici, val/loss=1.56e+5, train/loss=1.44e+5]

2022-10-17 08:21:03

Epoch 19: 89%|█████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 8/9 [00:04<00:00, 1.63it/s, loss=1.47e+05, v_num=lici, val/loss=1.59e+5, train/loss=1.52e+5]

2022-10-17 08:21:21

Epoch 21: 89%|██████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 8/9 [00:04<00:00, 1.72it/s, loss=1.4e+05, v_num=lici, val/loss=1.37e+5, train/loss=1.41e+5]

2022-10-17 08:21:29

Epoch 22: 89%|█████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 8/9 [00:04<00:00, 1.67it/s, loss=1.43e+05, v_num=lici, val/loss=1.32e+5, train/loss=1.38e+5]

2022-10-17 08:21:33

Epoch 23: 0%| | 0/9 [00:00<?, ?it/s, loss=1.43e+05, v_num=lici, val/loss=1.28e+5, train/loss=1.54e+5]

2022-10-17 08:21:35

/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:653: UserWarning: Detected KeyboardInterrupt, attempting graceful shutdown...

2022-10-17 08:21:35

rank_zero_warn("Detected KeyboardInterrupt, attempting graceful shutdown...")