Tiewa_enguin's workspace
Runs
39
Name
7 visualized
State
Notes
User
Tags
Created
Runtime
Sweep
noise_loss
Crashed
train3 but using Adam instead of AdamW to isolate whether improved image quality is from optimizer or sinusoidal embeddings (it's because of the embeddings)
tiewa_enguin
7h 43m 4s
-
0.29736
Crashed
-
tiewa_enguin
8h 45m 3s
-
0.27344
Crashed
-
tiewa_enguin
5h 24m 3s
-
0.26526
Crashed
-
tiewa_enguin
8h 53m 18s
-
0.24841
Crashed
-
tiewa_enguin
4h 30m 41s
-
0.28149
Crashed
-
tiewa_enguin
8h 7m 4s
-
0.26184
Crashed
-
tiewa_enguin
3h 22m 32s
-
0.27124
Crashed
rerun of train3a-g since pretraining from that run didn't improve loss for some reason
tiewa_enguin
7h 54m 5s
-
0.29395
Crashed
train with fp32 GN as well as modified sinusoidal pos embedding (casted to fp32 until after sin and cos ops, then cast to bf16 for most accurate embeddings)
tiewa_enguin
1h 47m 31s
-
0.33643
Crashed
training with fp32 GN
tiewa_enguin
1h 57m 31s
-
0.33813
Crashed
lr=1e-4, GN num groups=8, with updated VAE
tiewa_enguin
6h 2m 33s
-
1
Crashed
added shuffle queue size=8192 to tfrecorddataset (in case the dying run 56 came from batches that all had similar data e.g. the same caption), also used AdamW instead of Adam
tiewa_enguin
8h 37m 3s
-
1
Crashed
added shuffle queue size=8192 to tfrecorddataset (in case the dying run 56 came from batches that all had similar data e.g. the same caption), also used AdamW instead of Adam
tiewa_enguin
1h 19m 52s
-
0.36304
Crashed
-
tiewa_enguin
8h 51m 4s
-
0.99854
Crashed
better rng and zero_module (ResBlock, MHA, FFN), note that run was ended early since it didn't improve very quickly after initial loss drop (probably because the loss function rewards returning the identity for high timesteps T which zero_module allows for)
tiewa_enguin
2h 2m 32s
-
0.31885
Crashed
naming convention: 3a = 3rd training run from scratch, part a, -a = rerun a; added zero modules on resblocks, MHA, and FF layers, added back sinusoidal timestep embedding, using updated VAE
tiewa_enguin
31m 57s
-
0.50781
Crashed
rerun with better rng
tiewa_enguin
8h 48m 3s
-
0.26428
Crashed
rerun of train2_pt14
tiewa_enguin
8h 54m 33s
-
0.27295
Crashed
-
tiewa_enguin
8h 54m 3s
-
0.27954
Crashed
-
tiewa_enguin
8h 50m 34s
-
0.25354
1-20
of 39