H/14
Created on July 31|Last edited on September 12
Comment
one step is 1/256 of total sample seen (16x laion2B-en)
this is many models on laion400m https://docs.google.com/spreadsheets/d/1xQwPa9Dd-lo26UC6Elwop2XuWbQNwi2cOPm1VZ9D2Es/edit#gid=0
this is B/32 on laion2B https://docs.google.com/spreadsheets/d/1BdPwhhj4sptJ8MGMBLQ-PMeygCryYrqa4b28lvte6_4/edit#gid=0
Difference of this run compared to previous:
- batch size 79104, bigger one previously was about 40k, BASIC paper shows bigger batch = better results (same number of sample seen, so fewer steps)
- laion2B-en and not laion400m
- H/14 is 1B params, which is 2x L/14 ; text transformer is 3x the L/14 one, visual is 2x
Experiment log: https://docs.google.com/document/d/1FXosAZ3wMrzThgnWR6KSkXIz4IMItq3umDGos38pJps/edit?usp=sharing
Eval metrics
Run: eval run of clip h/14 ; one step = 135M samples seen
3
Training metrics
Run set 2
18
Full eval
Add a comment