Community Spotlight: Pysentimiento
Pysentimiento is a python library bringing state-of-the-art models for Spanish and English for sentiment analysis and other social NLP tasks. Here's what you need to know
Created on April 21|Last edited on May 2
Comment
This post is a piece in our series highlighting some of our favorite community repos that have instrumented W&B. If you'd like your repo to be featured, please reach out to editor@wandb.com
💡
Introduction
You are what you eat, but are you what you tweet?
With over 500 million tweets generated daily, Twitter is full of opinions, jokes, mentions, and more from people all over the world. In the past, analyzing the sentiment of those tweets—aka sentiment analysis—was limited to ML engineers or data scientists with experience in natural language processing. But, in recent years, many open-source projects have closed that skills gap enabling a larger audience to use ML to decode the complex emotions behind what people say.
What is Pysentimiento?
One of the popular repos for sentiment analysis is Pysentimiento, a Transformer-based library created by Juan Manuel Pérez, Postdoctoral Researcher at Instituto de Ciencias de la Computación. In addition to sentiment analysis, the library can perform other SocialNLP tasks like hate speech detection and irony detection. Originally intended for the Spanish language, the library has expanded to support English, Italian, and Portuguese too.
What's great about using Pysentimiento is that it's easy to use. Whether researchers are working in sociology, psychology, political science, or any other field, they can simply do pip install pysentimiento and start using the library. "Even if you don't have the skills or knowledge to deal with large language models, in one line, you can make Pysentimiento work for your use case," said Juan.
A small snippet of code using pysentimiento
Why was Pysentimiento built?
Textual information is all around us. And perhaps no source is more high-volume and real-time than social media. But deriving information from such large volumes of text data is challenging. While a good handful of repos are available for sentiment analysis, many are intended for English only and not any other language.
Past that, even if those projects existed, they were often difficult to use and would require some degree of ML experience. Filling that void in the open-source world led to the creation of Pysentimiento, allowing anyone to use those techniques for a variety of research purposes.
How does Pysentimiento work?
To build Pysentimiento, Juan first had to look for high-quality datasets to train their models on. Some languages were more challenging than others to find resources for like Spanish, while for English, there was a lot of available data to draw on. The next step was to create a custom BERT model for social media text in Spanish. "The library itself is really just a small wrapper that makes it easy for the end user to deal with these models," said Juan.
Why Pysentimiento uses Weights & Biases
Most practitioners want to eliminate the manual labor behind ML—hyperparameter tuning, and experiment tracking, to name a few. And that's a large part of why Juan decided to integrate Pysentimiento with W&B. As more languages were added to the repo, it became important to find an efficient way to optimize his ML workflow.
Leveraging W&B's automated experiment tracking capabilities, Juan could quickly iterate on his ML pipeline with the confidence that his datasets and models are tracked and versioned in a reliable system of record. Past that, W&B Sweeps can handle massive scale, saving him valuable time by quickly optimizing over thousands of hyperparameter combinations.
And best of all, because W&B is directly integrated with HuggingFace Transformers, it was a seamless implementation with the added benefits of scaling the volume and richness of Juan's experiments.
Juan's W&B dashboard for benchmarking several tasks and languages simultaneously (sentiment analysis task in Spanish is on the right).
How can you get involved with Pysentimiento?
For more people to take advantage of pysentimiento, adding more languages for the repo to support is key. If you're interested in contributing to the library and are fluent in a particular language currently unsupported, start a conversation on the project here.
Add a comment
Iterate on AI agents and models faster. Try Weights & Biases today.