model size vs. results

Scaling the transformer and it trains BPE.

A trail to despair

Some more straightforward tokenization compare.

I think the models I've been using is indeed a bit too small.

Homogeneous graphs also learns, around 15% final acc less than the heterogeneous ones. But in terms of the size it's way smaller, 0.074MB of parameter size while the hetero ones should be > 0.2MB (both with SAGE layers, and level 0 basic features)

Run set 1: Grey: Old GAT Purple: GAT with the augmentation (mean pool the attentions at each layer, mean aggregate edges) - but idk why they have more parameters! Green: GraphSAGE Run et 2: composer task, as the data is better we have better comparison SAGE learns faster and have less overfitting. Conclusion: maybe GAT is not making things better.