Skip to main content
kimsehun725
Projects
my_vae
Run groups
20221017_1637
Logs
Group
Workspace
Runs
Logs
Files
Artifacts

Kimsehun725's group workspace

Timestamps visible
2022-10-17 07:37:18
GPU available: True (cuda), used: True
2022-10-17 07:37:18
TPU available: False, using: 0 TPU cores
2022-10-17 07:37:18
IPU available: False, using: 0 IPUs
2022-10-17 07:37:18
HPU available: False, using: 0 HPUs
2022-10-17 07:37:24
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/4
2022-10-17 07:37:36
----------------------------------------------------------------------------------------------------
2022-10-17 07:37:36
distributed_backend=nccl
2022-10-17 07:37:36
All distributed processes registered. Starting with 4 processes
2022-10-17 07:37:36
----------------------------------------------------------------------------------------------------
2022-10-17 07:37:40
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
2022-10-17 07:37:40
  | Name      | Type          | Params
2022-10-17 07:37:40
--------------------------------------------
2022-10-17 07:37:40
0 | encoder   | Conv1DEncoder | 1.8 M
2022-10-17 07:37:40
1 | fc1       | Linear        | 23.1 K
2022-10-17 07:37:40
2 | fc2       | Linear        | 23.1 K
2022-10-17 07:37:40
3 | fc3       | Linear        | 23.4 K
2022-10-17 07:37:40
4 | decoder_0 | Conv1DDecoder | 1.8 M
2022-10-17 07:37:40
5 | decoder_1 | Conv1DDecoder | 1.8 M
2022-10-17 07:37:40
6 | decoder_2 | Conv1DDecoder | 1.8 M
2022-10-17 07:37:40
7 | decoder_3 | Conv1DDecoder | 1.8 M
2022-10-17 07:37:40
8 | decoder_4 | Conv1DDecoder | 1.8 M
2022-10-17 07:37:40
9 | decoder_5 | Conv1DDecoder | 1.8 M
2022-10-17 07:37:40
--------------------------------------------
2022-10-17 07:37:40
13.0 M    Trainable params
2022-10-17 07:37:40
0         Non-trainable params
2022-10-17 07:37:40
13.0 M    Total params
2022-10-17 07:37:40
51.971    Total estimated model params size (MB)
2022-10-17 07:37:51
Epoch 0:  89%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████                 | 8/9 [00:05<00:00,  1.39it/s, loss=9.23e+03, v_num=x8nw]
2022-10-17 07:37:46
/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py:535: PossibleUserWarning: It is recommended to use `self.log('val/loss', ..., sync_dist=True)` when logging on epoch level in distributed setting to accumulate the metric across devices.
2022-10-17 07:37:46
  warning_cache.warn(
2022-10-17 07:37:46
/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1892: PossibleUserWarning: The number of training batches (8) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
2022-10-17 07:37:46
  rank_zero_warn(
2022-10-17 07:37:53
2022-10-17 07:37:58
Traceback (most recent call last):
2022-10-17 07:37:58
  File "src/train.py", line 65, in <module>
2022-10-17 07:37:58
    main(kwargs)
2022-10-17 07:37:58
  File "src/train.py", line 23, in main
2022-10-17 07:37:58
    step2(kwargs)
2022-10-17 07:37:58
  File "/data/group1/z44543r/vae_separation/src/step2.py", line 149, in step2
2022-10-17 07:37:58
    train(kwargs, now, test_num, train_data_list,
2022-10-17 07:37:58
  File "/data/group1/z44543r/vae_separation/src/step2.py", line 119, in train
2022-10-17 07:37:58
    trainer.fit(model, train_dataloaders=train_loader,
2022-10-17 07:37:58
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 696, in fit
2022-10-17 07:37:58
    self._call_and_handle_interrupt(
2022-10-17 07:37:58
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 648, in _call_and_handle_interrupt
2022-10-17 07:37:58
    return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, **kwargs)
2022-10-17 07:37:58
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 93, in launch
2022-10-17 07:37:58
    return function(*args, **kwargs)
2022-10-17 07:37:58
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 735, in _fit_impl
2022-10-17 07:37:58
    results = self._run(model, ckpt_path=self.ckpt_path)
2022-10-17 07:37:58
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1166, in _run
2022-10-17 07:37:58
    results = self._run_stage()
2022-10-17 07:37:58
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1252, in _run_stage
2022-10-17 07:37:58
    return self._run_train()
2022-10-17 07:37:58
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1283, in _run_train
2022-10-17 07:37:58
    self.fit_loop.run()
2022-10-17 07:37:58
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 201, in run
2022-10-17 07:37:58
    self.on_advance_end()
2022-10-17 07:37:58
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 287, in on_advance_end
2022-10-17 07:37:58
    epoch_end_outputs = self.trainer._call_lightning_module_hook("training_epoch_end", epoch_end_outputs)
2022-10-17 07:37:58
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1550, in _call_lightning_module_hook
2022-10-17 07:37:58
    output = fn(*args, **kwargs)
2022-10-17 07:37:58
  File "/data/group1/z44543r/vae_separation/src/models.py", line 410, in training_epoch_end
2022-10-17 07:37:58
    , norm=LogNorm(extend='both')
2022-10-17 07:37:58
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/matplotlib/colors.py", line 1663, in __init__
2022-10-17 07:37:58
    ba = bound_init_signature.bind(*args, **kwargs)
2022-10-17 07:37:58
  File "/usr/lib/python3.8/inspect.py", line 3037, in bind
2022-10-17 07:37:58
    return self._bind(args, kwargs)
2022-10-17 07:37:58
  File "/usr/lib/python3.8/inspect.py", line 3026, in _bind
2022-10-17 07:37:58
    raise TypeError(
2022-10-17 07:37:58
TypeError: got an unexpected keyword argument 'extend'