kimsehun725

Kimsehun725's group workspace

Group: 20221017_1639

1-1

of 1

Timestamps visible

2022-10-17 07:39:49

GPU available: True (cuda), used: True

2022-10-17 07:39:49

TPU available: False, using: 0 TPU cores

2022-10-17 07:39:49

IPU available: False, using: 0 IPUs

2022-10-17 07:39:49

HPU available: False, using: 0 HPUs

2022-10-17 07:40:01

Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/4

2022-10-17 07:40:05

----------------------------------------------------------------------------------------------------

2022-10-17 07:40:05

distributed_backend=nccl

2022-10-17 07:40:05

All distributed processes registered. Starting with 4 processes

2022-10-17 07:40:05

----------------------------------------------------------------------------------------------------

2022-10-17 07:40:09

LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]

2022-10-17 07:40:09

| Name | Type | Params

2022-10-17 07:40:09

--------------------------------------------

2022-10-17 07:40:09

0 | encoder | Conv1DEncoder | 1.8 M

2022-10-17 07:40:09

1 | fc1 | Linear | 23.1 K

2022-10-17 07:40:09

2 | fc2 | Linear | 23.1 K

2022-10-17 07:40:09

3 | fc3 | Linear | 23.4 K

2022-10-17 07:40:09

4 | decoder_0 | Conv1DDecoder | 1.8 M

2022-10-17 07:40:09

5 | decoder_1 | Conv1DDecoder | 1.8 M

2022-10-17 07:40:09

6 | decoder_2 | Conv1DDecoder | 1.8 M

2022-10-17 07:40:09

7 | decoder_3 | Conv1DDecoder | 1.8 M

2022-10-17 07:40:09

8 | decoder_4 | Conv1DDecoder | 1.8 M

2022-10-17 07:40:09

9 | decoder_5 | Conv1DDecoder | 1.8 M

2022-10-17 07:40:09

--------------------------------------------

2022-10-17 07:40:09

13.0 M Trainable params

2022-10-17 07:40:09

0 Non-trainable params

2022-10-17 07:40:09

13.0 M Total params

2022-10-17 07:40:09

51.971 Total estimated model params size (MB)

2022-10-17 07:40:21

Epoch 0: 89%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 8/9 [00:06<00:00, 1.32it/s, loss=6.86e+03, v_num=e39s]

2022-10-17 07:40:15

/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py:535: PossibleUserWarning: It is recommended to use `self.log('val/loss', ..., sync_dist=True)` when logging on epoch level in distributed setting to accumulate the metric across devices.

2022-10-17 07:40:15

warning_cache.warn(

2022-10-17 07:40:15

/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1892: PossibleUserWarning: The number of training batches (8) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.

2022-10-17 07:40:15

rank_zero_warn(

2022-10-17 07:40:33

Epoch 1: 89%|██��███████████████████████████████████████████████████████████████████████████████████████████████████▏ | 8/9 [00:05<00:00, 1.57it/s, loss=5.91e+03, v_num=e39s, val/loss=2.72e+3, train/loss=6.91e+3]

2022-10-17 07:40:27

/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py:535: PossibleUserWarning: It is recommended to use `self.log('train/loss', ..., sync_dist=True)` when logging on epoch level in distributed setting to accumulate the metric across devices.

2022-10-17 07:40:27

warning_cache.warn(

2022-10-17 07:40:41

Epoch 2: 89%|██████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 8/9 [00:04<00:00, 1.70it/s, loss=4.74e+03, v_num=e39s, val/loss=2.67e+3, train/loss=4.98e+3]

2022-10-17 07:40:49

Epoch 3: 89%|██████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 8/9 [00:04<00:00, 1.72it/s, loss=3.77e+03, v_num=e39s, val/loss=2.69e+3, train/loss=3.81e+3]

2022-10-17 07:41:07

Epoch 5: 89%|██████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 8/9 [00:04<00:00, 1.78it/s, loss=2.56e+03, v_num=e39s, val/loss=2.68e+3, train/loss=2.66e+3]

2022-10-17 07:41:15

Epoch 6: 89%|██████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 8/9 [00:04<00:00, 1.80it/s, loss=2.12e+03, v_num=e39s, val/loss=2.37e+3, train/loss=2.18e+3]

2022-10-17 07:41:25

Epoch 7: 89%|██████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 8/9 [00:04<00:00, 1.71it/s, loss=1.83e+03, v_num=e39s, val/loss=1.86e+3, train/loss=1.87e+3]

2022-10-17 07:41:33

Epoch 8: 89%|██████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 8/9 [00:04<00:00, 1.73it/s, loss=1.66e+03, v_num=e39s, val/loss=1.62e+3, train/loss=1.67e+3]

2022-10-17 07:41:41

Epoch 9: 89%|██████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 8/9 [00:04<00:00, 1.74it/s, loss=1.55e+03, v_num=e39s, val/loss=1.51e+3, train/loss=1.57e+3]

2022-10-17 07:41:59

Epoch 11: 89%|█████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 8/9 [00:04<00:00, 1.66it/s, loss=1.42e+03, v_num=e39s, val/loss=1.36e+3, train/loss=1.44e+3]

2022-10-17 07:42:07

Epoch 12: 89%|█████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 8/9 [00:04<00:00, 1.72it/s, loss=1.37e+03, v_num=e39s, val/loss=1.31e+3, train/loss=1.38e+3]

2022-10-17 07:42:15

Epoch 13: 11%|████████████▋ | 1/9 [00:02<00:21, 2.74s/it, loss=1.36e+03, v_num=e39s, val/loss=1.29e+3, train/loss=1.33e+3]

2022-10-17 07:42:14

/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:653: UserWarning: Detected KeyboardInterrupt, attempting graceful shutdown...

2022-10-17 07:42:14

rank_zero_warn("Detected KeyboardInterrupt, attempting graceful shutdown...")