Text2Video with Diffusion Models, Alpaca LoRA & Erasing Concepts
Created on March 19|Last edited on March 20
Comment
Text2Video
The HuggingFace space demo is available here. Just input a prompt and toy with the random seed and that's it! The results aren't perfect. Above, I used one of their sample prompts "Spiderman is surfing". There also seems to be a problem with the Shutterstock watermark (perhaps in the training data and thus the model is generating it).
Alpaca LoRA
The Alpaca LoRA repository can be found here. In short, they provide the necessary code for anyone to reproduce the Alpaca paper on their own hardware! Apparently, training this model only takes 5 hours with an RTX 4090! That's incredible! In just a few days, the AI community has both witnessed new heights in LLM capability and been given the chance to reproduce those capabilities for as little as 5 hours on a consumer-grade graphics card.
Erasing Concepts in Diffusion Models
Their paper concerns how to fine-tune model weights to erase certain concepts.
Though imperfect (e.g. other unexpected aspects of the image may change like texture, and color), it still has amazing results. They do this by introducing a special objective function for fine-tuning an already pre-trained diffusion model.
Note, erasing concepts here means erasing from the model weights as said in the image above. This differs from Stable Diffusion's negative prompting as the model will not have access to certain concepts.
A lot of machine learning systems need to be monitored for bias and unfairness. One can imagine an idea like concept erasure would be extremely useful in mitigating bias or regulating a model when it has been deployed. There's less overhead as regulation is built-in to the model.
References
Add a comment
Tags: ML News
Iterate on AI agents and models faster. Try Weights & Biases today.