Skip to main content
kimsehun725
Projects
my_vae
Run groups
20221014_191830
Logs
Group
Workspace
Runs
Logs
Files
Artifacts

Kimsehun725's group workspace

Timestamps visible
2022-10-14 10:18:36
GPU available: True (cuda), used: True
2022-10-14 10:18:36
TPU available: False, using: 0 TPU cores
2022-10-14 10:18:36
IPU available: False, using: 0 IPUs
2022-10-14 10:18:36
HPU available: False, using: 0 HPUs
2022-10-14 10:18:36
wandb: logging graph, to disable use `wandb.watch(log_graph=False)`
2022-10-14 10:18:48
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
2022-10-14 10:18:48
  | Name                   | Type             | Params
2022-10-14 10:18:48
------------------------------------------------------------
2022-10-14 10:18:48
0 | conv_stack             | ConvStack        | 1.6 M
2022-10-14 10:18:48
1 | self_attention_block   | ConformerEncoder | 92.0 K
2022-10-14 10:18:48
2 | frame_tab_output_layer | Sequential       | 24.6 K
2022-10-14 10:18:48
3 | softmax_by_string      | Softmax          | 0
2022-10-14 10:18:48
------------------------------------------------------------
2022-10-14 10:18:48
1.7 M     Trainable params
2022-10-14 10:18:48
0         Non-trainable params
2022-10-14 10:18:48
1.7 M     Total params
2022-10-14 10:18:48
6.873     Total estimated model params size (MB)
2022-10-14 10:18:51
Sanity Checking DataLoader 0:   0%|                                                                                                                                                                    | 0/2 [00:00<?, ?it/s]
2022-10-14 10:18:54
Traceback (most recent call last):
2022-10-14 10:18:54
  File "src/train.py", line 65, in <module>
2022-10-14 10:18:54
    main(kwargs)
2022-10-14 10:18:54
  File "src/train.py", line 25, in main
2022-10-14 10:18:54
    step3(now, kwargs)
2022-10-14 10:18:54
  File "/data/group1/z44543r/vae_separation/src/step3.py", line 203, in step3
2022-10-14 10:18:54
    train(kwargs, use_pretrained_model, pretrained_time, pretrained_epoch, now, test_num, train_data_list,
2022-10-14 10:18:54
  File "/data/group1/z44543r/vae_separation/src/step3.py", line 154, in train
2022-10-14 10:18:54
    trainer.fit(model, train_dataloaders=train_loader,
2022-10-14 10:18:54
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 696, in fit
2022-10-14 10:18:54
    self._call_and_handle_interrupt(
2022-10-14 10:18:54
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _call_and_handle_interrupt
2022-10-14 10:18:54
    return trainer_fn(*args, **kwargs)
2022-10-14 10:18:54
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 735, in _fit_impl
2022-10-14 10:18:54
    results = self._run(model, ckpt_path=self.ckpt_path)
2022-10-14 10:18:54
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1166, in _run
2022-10-14 10:18:54
    results = self._run_stage()
2022-10-14 10:18:54
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1252, in _run_stage
2022-10-14 10:18:54
    return self._run_train()
2022-10-14 10:18:54
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1274, in _run_train
2022-10-14 10:18:54
    self._run_sanity_check()
2022-10-14 10:18:54
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1343, in _run_sanity_check
2022-10-14 10:18:54
    val_loop.run()
2022-10-14 10:18:54
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
2022-10-14 10:18:54
    self.advance(*args, **kwargs)
2022-10-14 10:18:54
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 155, in advance
2022-10-14 10:18:54
    dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)
2022-10-14 10:18:54
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
2022-10-14 10:18:54
    self.advance(*args, **kwargs)
2022-10-14 10:18:54
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 143, in advance
2022-10-14 10:18:54
    output = self._evaluation_step(**kwargs)
2022-10-14 10:18:54
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 240, in _evaluation_step
2022-10-14 10:18:54
    output = self.trainer._call_strategy_hook(hook_name, *kwargs.values())
2022-10-14 10:18:54
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1704, in _call_strategy_hook
2022-10-14 10:18:54
    output = fn(*args, **kwargs)
2022-10-14 10:18:54
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/pytorch_lightning/strategies/strategy.py", line 370, in validation_step
2022-10-14 10:18:54
    return self.model.validation_step(*args, **kwargs)
2022-10-14 10:18:54
  File "/data/group1/z44543r/vae_separation/src/models.py", line 658, in validation_step
2022-10-14 10:18:54
    precision, recall, f1 = calculate_metrics(y, binary_y_hat, ilens)
2022-10-14 10:18:54
  File "/data/group1/z44543r/vae_separation/src/utils.py", line 41, in calculate_metrics
2022-10-14 10:18:54
    masked_binary_pred = mask_by_length(binary_pred, ilens, fill=0)
2022-10-14 10:18:54
  File "/data/group1/z44543r/vae_separation/src/utils.py", line 149, in mask_by_length
2022-10-14 10:18:54
    ret = np.zeros_like(xs)
2022-10-14 10:18:54
  File "<__array_function__ internals>", line 180, in zeros_like
2022-10-14 10:18:54
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/numpy/core/numeric.py", line 138, in zeros_like
2022-10-14 10:18:54
    res = empty_like(a, dtype=dtype, order=order, subok=subok, shape=shape)
2022-10-14 10:18:54
  File "<__array_function__ internals>", line 180, in empty_like
2022-10-14 10:18:54
  File "/data/group1/z44543r/vae_separation/venv/lib/python3.8/site-packages/torch/_tensor.py", line 757, in __array__
2022-10-14 10:18:54
    return self.numpy()
2022-10-14 10:18:54
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.