Skip to main content
eleutherai
Projects
gpt-thicc
Run groups
20B_finetune
Logs
Group
Workspace
Runs
Logs
Files
Artifacts

Igoro's group workspace

Timestamps visible
2022-01-27 16:29:48
    loss = model.train_batch(data_iter=data_iterator)
2022-01-27 16:29:48
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/pipe/engine.py", line 305, in train_batch
2022-01-27 16:29:48
    self._exec_schedule(sched)
2022-01-27 16:29:48
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/pipe/engine.py", line 1308, in _exec_schedule
2022-01-27 16:29:48
    self._exec_instr(**cmd.kwargs)
2022-01-27 16:29:48
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/pipe/engine.py", line 787, in _exec_load_micro_batch
2022-01-27 16:29:48
    batch = self._next_batch()
2022-01-27 16:29:48
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/pipe/engine.py", line 646, in _next_batch
2022-01-27 16:29:48
    batch = next(self.data_iterator)
2022-01-27 16:29:48
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 521, in __next__
2022-01-27 16:29:48
    data = self._next_data()
2022-01-27 16:29:48
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 1186, in _next_data
2022-01-27 16:29:48
    idx, data = self._get_data()
2022-01-27 16:29:48
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 1142, in _get_data
2022-01-27 16:29:48
    success, data = self._try_get_data()
2022-01-27 16:29:48
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 1003, in _try_get_data
2022-01-27 16:29:48
    raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
2022-01-27 16:29:48
RuntimeError: DataLoader worker (pid(s) 75218, 75932) exited unexpectedly