Brasilidade: Measuring Brazilian Identity for Large Language Models

Brasilidade is a new benchmark for evaluating large language models in Brazilian Portuguese, comprising standardized tasks across four domains, with the first domain addressing language nuances, grammar, and variations of Brazilian Portuguese now available. Results indicate that model size alone does not determine performance, as several medium-sized models outperformed larger models, and Portuguese-specific models did not perform above average despite targeted training. The benchmark reveals significant task-level variations, with some tasks easy for most models, others extremely challenging, and some yielding mixed results.