Skip to main content
kastan
Projects
LLM-Distributed-Quantization
Run groups
Aug-05__11:13
Overview
Group
Workspace
Runs
Logs
Files
Artifacts

Kastan's group workspace

Aug-05__11:13

What makes this group special?
Tags

q_allFP32_gpt_8B_PP4_TP4_25d

Notes
Tags
Aug-05__11:13
BATCH_SIZE32
NUM_EPOCHS=3
NUM_MICRO_BATCHES=4
SLURM=513418
TP=4
WORLD_SIZE=32
Author
kastan
State
Failed
Start time
August 5th, 2022 4:13:43 PM
Runtime
32s
Tracked hours
25s
Run path
kastan/LLM-Distributed-Quantization/2i7o5afm
OS
Linux-4.18.0-305.49.1.el8_4.x86_64-x86_64-with-glibc2.28
Python version
3.9.12
Command
/u/kastanday/LLM-Distributed-Quantization/benchmarks/gpt/v2_train.py --config /u/kastanday/LLM-Distributed-Quantization/benchmarks/gpt/configs/q_allFP32_gpt_8B_PP4_TP4_25d.py --host gpub007 --port 29500 --world_size 32 --rank 18
System Hardware
CPU count64
GPU count4
GPU typeNVIDIA A40
W&B CLI Version
0.13.0
Group
Aug-05__11:13
Config

Config parameters are your model's inputs. Learn more

  • ▶
    {} 26 keys
    • 32
    • 1
    • "col_ai_quant"
    • "/u/kastanday/LLM-Distributed-Quantization/datasets/small-gpt-dataset.json"
    • ▶
      {} 1 key
      • "AMP_TYPE.NAIVE"
    • 4
    • 0.00015
    • "./quant_gpt2_2.5d_tp4_bs32_lr0.00015/"
    • ▶
      {} 1 key
      • "titans.loss.lm_loss.gpt_lmloss.GPTLMLoss"
    • ▶
      {} 7 keys
      • ▶
        {} 4 keys
        • "torch.float32"
        • "torch.float32"
        • "torch.bfloat16"
        • "torch.float32"
      • 3
      • "4"
      • 4
      • ▶
        {} 2 keys
        • 0.00015
        • 0.01
      • ▶
        {} 2 keys
        • 4
        • ▶
          {} 3 keys
          • 1
          • "2.5d"
          • 4
      • "titans.model.quant_gpt.quant_gpt.quant_gpt2_8B"
      • "titans.model.quant_gpt.quant_gpt.quant_gpt2_xl"
      • 1,024
      • "2.5d"
      • 4
      • 128
      • "32"
      • 50,304
      • 1
      • 0.01
    Summary

    Summary metrics are your model's outputs. Learn more

    • {} 0 keys