A fine-grained evaluation framework for language models: Combining pointwise grading and pairwise comparison

Abstract

Automated evaluation of Large Language Models (LLMs) responses face fundamental challenges: evaluation bias, protocol inflexibility, and the trade-off between quality and accessibility. Current paradigms either rely heavily on expensive proprietary models or suffer from systematic biases and limited evaluation modes. We introduce MELD, an 8B-parameter evaluation model designed to overcome these limitations via systematic bias mitigation and multi-protocol adaptability. MELD is trained on a comprehensive dataset covering eight categories and 50 subcategories, each with tailored evaluation criteria. It supports both pointwise grading and pairwise comparison through model merging, achieving robust performance across protocols. Experiments show MELD consistently outperforms open-source baselines and matches or surpasses GPT-4 in human alignment. Notably, MELD reduces bias in position, length, and content. The framework includes a lightweight quantized deployment option, enabling high-quality evaluation in resource-constrained settings. This work provides a practical, cost-effective solution for LLM evaluation. Resources are available at: https://github.com/Bound2-2/MELD-Eval.