Introduction
In today’s digital content boom of limitless proportions, long documents, articles, and reports must be condensed into brief forms. Content Management Systems are where large language models (LLMs) find applications. LLMs provide a new way to process natural language by making it possible to create, understand, and summarize writing. Trained on large datasets, these models are employed in different cases because they can understand the subtleties of language, context, and even the intent behind a text. When you consider what a decent summary for a 10-page research paper looks like, you probably picture a brief overview that faithfully reflects the major conclusions and data from the original work, presented in an accessible and clear style. For LLMs such as GPT-4 or Gemini, understanding the basic concept of what constitutes a good summary, including coherence, conciseness, and accuracy, remains a major difficulty, even if this would sound quite clear to us (I mean, who doesn’t know what a decent summary looks like?).
By using such models to automatically summarize documents, you can quickly go into the content without having to read through lengthy text blocks, thus saving time and improving your understanding. Industries that depend on data-driven decision-making, such as journalism, research, and business intelligence, find this especially useful.
Two fundamental text summarizing strategies for NLP
To understand the advanced techniques for evaluation, let’s first examine two fundamental approaches to text summarization in NLP.
- Extractive summarization involves identifying and collecting significant key phrases, sentences, or sections from the original text to create a summary. This approach preserves syntactic and semantic relations while choosing passages of the original text to produce a condensed form. It does not change the text itself. Extractive summarizing is mostly used for maintaining factual correctness since it keeps the original language and context, so it is rather dependable.
- Abstractive summarization involves understanding the key ideas and concepts of the text and creating new content that distills the essence of the original material. This approach can paraphrase, interpret, and create coherent summaries that might not employ precise language from the source content. Because abstractive summarizing calls for deep semantic and subjective knowledge and generating ability, it is more difficult than extractive summarizing.
Methods for text summarizing evaluation
Evaluating tasks in NLP, specifically Natural Language Generation (NLG), has evolved significantly over time. NLG focuses on creating human-like text by machines and is an important subset of NLP, encompassing a broader range of language-related tasks. Early days saw evaluations mostly depending on human judgment and experience, with an emphasis on fluency, coherence, and relevance, which are subjective and not easily scalable. Automated metrics such as BLEU, ROUGE, and METEOR were developed to offer objective, repeatable, and quicker methods to evaluate text-generation quality depending on n-gram overlap with reference texts. However, these metrics sometimes miss the subtleties of semantic meaning and the complexity of genuine human language.
Since 2017, advanced models like BERT have cleared the way for the development of more sophisticated evaluation methods, such as BERTScore, which was introduced around 2020. This approach provides a deeper understanding of the semantic alignment by comparing the contextual embeddings of reference texts and produced ones. Aiming for a fairer assessment of tasks in capturing the complexity of human language, techniques have lately turned towards more comprehensive and considerably developed LLM evaluation frameworks, including both human and machine assessments.
Here, we should highlight the intrinsic assessment strategies. Intrinsic evaluation ignores the effect of the summary on any particular outside work and concentrates on its own excellence. This can require human judgment as well as automatic metrics. Among the most often used measures for assessing the quality of summaries are ROUGE (Recall-Oriented Understudy for Gisting Evaluation) and BLEU. Contrary to intrinsic evaluation, extrinsic methods assess the effectiveness of a summary within the context of a specific application or task. This involves determining how well a summary aids in performing certain tasks, such as improving the efficiency of information retrieval systems.
Intrinsic Evaluation Methods
Evaluation with ROUGE
ROUGE compares the generated and reference summaries’ N-gram (sequences of N words) overlaps. Often-used variants include ROUGE-N (for N-grams), ROUGE-L (for the longest common subsequence), and ROUGE-S (for skip-bigram co-occurrence statistics). Since extractive summaries come straight from the source text, ROUGE can fairly assess how well the chosen sentences cover the material of the reference summaries. High ROUGE values show that the extractive model efficiently identifies important textual material.
Let’s consider the following example, where we have a source text, a candidate summary, and a reference summary, and then list them in the following order:
- Source text
- Candidate summary
- Reference summary
Source Text: “Renewable energy sources, such as wind and solar power, are essential for reducing carbon emissions and combating climate change. These sources of energy are not only sustainable but also increasingly cost-effective. Advances in technology have significantly lowered the costs associated with renewable energy production, making it a viable alternative to fossil fuels. Governments and organizations worldwide are investing heavily in renewable energy infrastructure to ensure a cleaner, greener future.”
Candidate Summary: “Renewable energy sources like wind and solar power are crucial for reducing carbon emissions. Advances in technology have lowered the costs, making renewable energy a viable alternative to fossil fuels. Governments worldwide are investing in renewable energy infrastructure.”
Reference Summary: “Renewable energy, including wind and solar power, helps reduce carbon emissions and combat climate change. Technological advances have made it more cost-effective, leading to increased investments in renewable energy infrastructure globally.”
ROUGE IN ACTION
ROUGE-1 (unigram overlap)
The ROUGE-1 metric evaluates the overlap of unigrams (single words) between the candidate summary and the reference summary. This overlap helps assess how well the candidate summary captures key information from the reference. Below, we list the unigrams in both the candidate and reference summaries to calculate the ROUGE-1 score, which measures recall, precision, and the F1-score of the unigram overlap.
List unigrams in both texts:
- Candidate: [“Renewable”, “energy”, “sources”, “like”, “wind”, “and”, “solar”, “power”, “are”, “crucial”, “for”, “reducing”, “carbon”, “emissions”, “Advances”, “in”, “technology”, “have”, “lowered”, “the”, “costs”, “making”, “renewable”, “energy”, “a”, “viable”, “alternative”, “to”, “fossil”, “fuels”, “Governments”, “worldwide”, “are”, “investing”, “in”, “renewable”, “energy”, “infrastructure”]
- Reference: [“Renewable”, “energy”, “including”, “wind”, “and”, “solar”, “power”, “helps”, “reduce”, “carbon”, “emissions”, “and”, “combat”, “climate”, “change”, “Technological”, “advances”, “have”, “made”, “it”, “more”, “cost-effective”, “leading”, “to”, “increased”, “investments”, “in”, “renewable”, “energy”, “infrastructure”, “globally”]
- Matches (The following words are present in both the candidate summary and the reference summary, indicating shared content): “Renewable”, “energy”, “wind”, “and”, “solar”, “power”, “carbon”, “emissions”, “Advances”, “in”, “technology”, “have”, “renewable”, “energy”, “Governments”, “in”, “renewable”, “energy”, “infrastructure” = 19 matches
- Total unigrams: Unigrams are individual words from the text. In this example, there are 29 unigrams in the reference summary and 37 unigrams in the candidate summary. The total number of unigrams is calculated by counting each unique word in the summaries.
Calculate ROUGE-1 Score:
To evaluate the quality of the candidate summary, we calculate the ROUGE-1 score using the following metrics:
- Recall measures how much of the reference summary’s content is captured by the candidate summary. It is calculated as the number of matching unigrams divided by the total unigrams in the reference summary: 19/29 ≈ 0.655
- Precision measures how much of the candidate summary’s content matches the reference summary. It is calculated as the number of matching unigrams divided by the total unigrams in the candidate summary: 19/37 ≈ 0.514
- F1-Score represents the harmonic mean of precision and recall, balancing the two. It is calculated using the formula: 2 * (0.655 * 0.514) / (0.655 + 0.514) ≈ 0.577
ROUGE-2 (bigram overlap)
ROUGE-2 evaluates the overlap of bigrams (pairs of consecutive words) between the candidate summary and the reference summary. This differs from ROUGE-1, which focuses on unigrams (individual words). While ROUGE-1 measures how much of the content is captured in terms of single-word matches, ROUGE-2 provides insights into preserving the sequence and flow of information, assessing the structural similarity between the summaries. Below, we list the bigrams in both the candidate and reference summaries and calculate the ROUGE-2 score, which evaluates the alignment of phrase structures and content.
List bigrams in both texts:
- Candidate bigrams: [“Renewable energy”, “energy sources”, “sources like”, “like wind”, “wind and”, “and solar”, “solar power”, “power are”, “are crucial”, “crucial for”, “for reducing”, “reducing carbon”, “carbon emissions”, “Advances in”, “in technology”, “technology have”, “have lowered”, “lowered the”, “the costs”, “costs making”, “making renewable”, “renewable energy”, “energy a”, “a viable”, “viable alternative”, “alternative to”, “to fossil”, “fossil fuels”, “Governments worldwide”, “worldwide are”, “are investing”, “investing in”, “in renewable”, “renewable energy”, “energy infrastructure”]
- Reference bigrams: [“Renewable energy”, “energy including”, “including wind”, “wind and”, “and solar”, “solar power”, “power helps”, “helps reduce”, “reduce carbon”, “carbon emissions”, “emissions and”, “and combat”, “combat climate”, “climate change”, “Technological advances”, “advances have”, “have made”, “made it”, “it more”, “more cost-effective”, “cost-effective leading”, “leading to”, “to increased”, “increased investments”, “investments in”, “in renewable”, “renewable energy”, “energy infrastructure”, “infrastructure globally”]
- Matches (The following bigrams, pairs of consecutive words, appear in both the candidate summary and the reference summary): “Renewable energy”, “wind and”, “and solar”, “solar power”, “carbon emissions”, “Advances in”, “in renewable”, “renewable energy”, “energy infrastructure” = 9 matches
These nine matches indicate that the candidate summary captures some of the same phrases and structures as the reference summary, reflecting a level of alignment in how the content is expressed.
- Total bigrams: The reference summary has 29 bigrams, which are calculated by identifying all pairs of consecutive words in the reference text. The candidate summary has 37 bigrams, which are determined by counting all pairs of consecutive words in the candidate text.
Calculate ROUGE-2 score:
- Recall: 9/29 ≈ 0.310
This score indicates that approximately 31% of the bigrams in the reference summary are captured in the candidate summary. A higher recall suggests better coverage of the reference content.
- Precision: 9/37 ≈ 0.243
This score shows that about 24.3% of the bigrams in the candidate summary match those in the reference summary. Higher precision indicates that the candidate summary’s content closely matches the reference.
- F1-Score: 2 * (0.310 * 0.243) / (0.310 + 0.243) ≈ 0.273
The F1-score represents the harmonic mean of recall and precision, balancing both metrics to provide a single measure of summary quality. An F1-score of 0.273 suggests moderate alignment between the candidate and reference summaries in terms of phrasing.
ROUGE has limits, even if it has been widely used and offers a numerical assessment of general quality. It does not specifically assess the factual accuracy, readability, or coherence of the produced summaries. To provide a more complete evaluation of model performance, it is often used in combination with other evaluation techniques, both intrinsic (evaluating the summary itself) and extrinsic (evaluating how the summary influences the performance of a task, such as information retrieval or question answering).
Evaluation with BLEU
Here, we introduce one more example for BLEU summarization evaluation. Originally intended for the assessment of machine translation, BLEU (Bilingual Evaluation Understudy) evaluates text quality by comparing the n-grams of a candidate summary with those of a reference summary, calculating precision scores that are combined into an overall score via a geometric mean. Using n-grams of the reference text and the summary, this specific method computes precision scores that are then combined into an overall score via geometric mean.
Source Text: “Artificial intelligence (AI) has transformed numerous industries by automating processes, improving decision-making, and enhancing customer experiences. Companies are leveraging AI to analyze large datasets, uncover insights, and drive business strategies. The integration of AI technologies in healthcare, finance, and retail is particularly noteworthy, as it leads to improved efficiency and innovation.”
Candidate summary: “AI has transformed industries by automating processes and enhancing customer experiences. Companies use AI to analyze datasets and drive strategies. AI integration in healthcare, finance, and retail improves efficiency and innovation.”
Reference summary: “AI has revolutionized industries by automating tasks, improving decision-making, and enhancing customer experiences. Companies utilize AI to analyze large datasets and develop business strategies. The adoption of AI in healthcare, finance, and retail leads to greater efficiency and innovation.”
The step-by-step calculation is presented here.
- N-gram precision calculation
BLEU score usually employs n = 1 to 4 n-gram precision. We shall first show unigram and bigram precision for simplicity.
Unigram precision:
- Candidate unigrams: [“AI”, “has”, “transformed”, “industries”, “by”, “automating”, “processes”, “and”, “enhancing”, “customer”, “experiences”, “Companies”, “use”, “AI”, “to”, “analyze”, “datasets”, “and”, “drive”, “strategies”, “AI”, “integration”, “in”, “healthcare”, “finance”, “and”, “retail”, “improves”, “efficiency”, “and”, “innovation”]
- Reference unigrams: [“AI”, “has”, “revolutionized”, “industries”, “by”, “automating”, “tasks”, “improving”, “decision-making”, “and”, “enhancing”, “customer”, “experiences”, “Companies”, “utilize”, “AI”, “to”, “analyze”, “large”, “datasets”, “and”, “develop”, “business”, “strategies”, “The”, “adoption”, “of”, “AI”, “in”, “healthcare”, “finance”, “and”, “retail”, “leads”, “to”, “greater”, “efficiency”, “and”, “innovation”]
- Matches: “AI”, “has”, “industries”, “by”, “automating”, “and”, “enhancing”, “customer”, “experiences”, “Companies”, “AI”, “to”, “analyze”, “datasets”, “and”, “strategies”, “in”, “healthcare”, “finance”, “and”, “retail”, “efficiency”, “and”, “innovation” = 24 matches
- Unigram precision: Unigram precision measures how many unigrams (individual words) in the candidate summary match those in the reference summary. It is a measure of accuracy, showing the proportion of words in the candidate summary that are also found in the reference summary, where matching unigrams are the words that appear in both the candidate and reference summaries. In our example, there are 24 matching unigrams. And total candidate unigrams are the total number of individual words in the candidate summary, which is 30 in this case.
![]()
![]()
This means that 80% of the words in the candidate summary also appear in the reference summary, indicating a high level of content alignment between the two texts.
Bigram precision:
- Candidate bigrams: [“AI has”, “has transformed”, “transformed industries”, “industries by”, “by automating”, “automating processes”, “processes and”, “and enhancing”, “enhancing customer”, “customer experiences”, “Companies use”, “use AI”, “AI to”, “to analyze”, “analyze datasets”, “datasets and”, “and drive”, “drive strategies”, “AI integration”, “integration in”, “in healthcare”, “healthcare finance”, “finance and”, “and retail”, “retail improves”, “improves efficiency”, “efficiency and”, “and innovation”]
- Reference bigrams: [“AI has”, “has revolutionized”, “revolutionized industries”, “industries by”, “by automating”, “automating tasks”, “tasks improving”, “improving decision-making”, “decision-making and”, “and enhancing”, “enhancing customer”, “customer experiences”, “Companies utilize”, “utilize AI”, “AI to”, “to analyze”, “analyze large”, “large datasets”, “datasets and”, “and develop”, “develop business”, “business strategies”, “strategies The”, “The adoption”, “adoption of”, “of AI”, “AI in”, “in healthcare”, “healthcare finance”, “finance and”, “and retail”, “retail leads”, “leads to”, “to greater”, “greater efficiency”, “efficiency and”, “and innovation”]
- Matches: “AI has”, “industries by”, “by automating”, “and enhancing”, “enhancing customer”, “customer experiences”, “AI to”, “to analyze”, “datasets and”, “and retail”, “efficiency and”, “and innovation” = 12 matches
- Bigram precision is calculated as:


The bigram precision score of 0.414 means that about 41.4% of the bigram sequences in the candidate summary are also found in the reference summary. This indicates that while the candidate summary captures some of the phrasing and structural elements of the reference, there is still room for improvement in maintaining the exact word sequences found in the reference summary.
Brevity penalty (BP)
The brevity penalty is used in BLEU to discourage overly short summaries. It penalizes candidate summaries that are shorter than the reference summaries to ensure they capture enough content.
- Candidate length: 30
- Reference length: 37

Since candidate length (30) < Reference length (37):
![]()
The BP value of 0.792 indicates a moderate penalty for brevity, reflecting that the candidate summary is shorter than the reference summary. This penalty ensures that shorter summaries, which may omit important information, do not receive artificially high BLEU scores simply due to brevity.
Combined BLEU score
Multiplied by the brevity penalty, the geometric mean of the precision scores determines the BLEU score. We shall apply the equal-weight unigram (P1) and bigram precision (P2) for simplicity.
The geometric mean of precision:
![]()
Where P1 represents the unigram precision, i.e., the proportion of individual words in the candidate summary that match the reference summary, and P2 is the bigram precision, representing the proportion of word pairs in the candidate summary that match the reference summary.
![]()
BLEU Score:
![]()
![]()
The BLEU score for this candidate summary is approximately 0.456. This score suggests a moderate level of similarity. It indicates that the candidate summary captures some of the key phrases and structure of the reference summary, but there is still room for improvement in achieving a closer match. Generally, the BLEU score ranges from 0 to 1, where 1 indicates a perfect match between the candidate and reference summaries.
BLEU can be effectively used in the context of text summarizing to assess the language and main points of view in the produced summaries. High BLEU ratings imply that the produced summary is relatively similar to the reference summaries in terms of the existence and correctness of important phrases and information. However, BLEU computes the precision of n-grams in the candidate summary concerning the reference summaries, producing a bias toward shorter summaries. This occurs because, presuming that the shorter text unintentionally avoids mistakes, BLEU naturally encourages brevity by not penalizing short outputs, including valid sequences. This approach ignores the semantic meaning as well as the word and phrase context. Though they faithfully capture the substance and intent of the original text, summaries paraphrased or using different language with the same underlying meaning may not score well under BLEU.
Extrinsic evaluation method
We should also mention extrinsic evaluation methods. Extrinsic evaluation analyzes the value of a summary of a particular application or task, such as information retrieval, question answering, or any other downstream activity that might profit from summarizing. These include:
- Task-based evaluation involves integrating summaries with an external task, meaning a task outside the immediate context of generating the summary itself, to assess performance improvement. For example, summaries might be used to enhance the effectiveness of information retrieval systems, helping users find information more efficiently.
Example: Using a summary to refine search engine results, thereby reducing the time users spend finding specific documents. - User-focused evaluation, where studies involving users completing activities both with and without summaries, might help to evaluate how well the summaries support knowledge or decision-making processes.
Example: Giving lawyers a summary of a legal document to speed up case preparation.
Real-Time Agent Evaluation in Production
Agentic systems do more than generate text. They decide what to do next, call tools, handle partial failures, and maintain state across turns. This means evaluation must cover both the text and the execution path, especially under real user traffic.
In production, add metrics that reflect multi-step execution, such as:
- Tool usage accuracy: ensure the agent selects the right tool, provides correct arguments, and submits valid payloads.
- Decision-making coherence: the logic remains consistent at each step and aligns with available evidence.
- Error recovery and self-correction: identifies failures, retries safely, and exits loops.
- Multi-turn planning quality: creates a plan, implements it, and revises it when inputs change.
Make sure you clearly explain each score. Distinguish between model errors and tool errors. Record the tool’s output as the ground truth. Re-run the evaluation using the cached response. Delta metrics should be alerted on canary rollouts. For comparison of different tool versions within the same session, use summarized text, and ensure it does not contradict any critical references that must be included in the summary and are unaltered in the summary provided to the agent.
Incorporate workflow mechanisms to ensure accurate state transitions. Log the intent for each step, including the tools applied, the outcome, and the memory. Then, calculate a score based on how well the outcome aligned with the policy or the user’s intent.
The summarization systems serve as typical stress tests. A routing agent will select a long ticket or policy, or call a summary model to determine priority, ownership, or subsequent action. Evaluate context selection and whether the summary and subsequent actions are appropriately aligned. Monitor the step outcome rate, number of retries, latencies, and total cost from start to finish, since small costs from normal actions can accumulate over multiple interactions.
Conclusion
ROUGE and BLEU offer quantitative analysis of the overlap between n-grams and basic textual content between candidate and reference summaries. At the same time, these metrics struggle to convey deeper semantic specifics in the language and contextual understanding. ROUGE and BLEU remain useful tools for assessing text summarizing quality despite limitations. Extrinsic evaluation techniques improve these measures by determining the applicable value of summaries in practical applications. While user-focused evaluations assess how summaries support decision-making and improve user experiences, task-based evaluations evaluate how summaries improve the performance of particular activities. Combining extrinsic evaluations with intrinsic metrics like ROUGE and BLEU will help us to have a more complete knowledge of the efficacy and quality of a summary.
FAQs
How often should I evaluate my production LLM system?
Always evaluate after each release using a fixed test set, and continue evaluation once it goes live. Schedule weekly evaluations, audits, and set up event-based triggers for any changes to prompts, tools, or source data. For summarization, assess the LLM on real documents, monitor summary drift in coverage, and check for drift from accurate fact-based reality.
What’s the difference between pre-deployment testing and production monitoring?
Pre-deployment testing is controlled and repeatable. You run the same dataset, compare different versions, and gate a release with stable metrics. Production monitoring watches real inputs for regressions, latency, cost, and new failure modes that lab data might miss.
Can evaluation tools detect emerging failure patterns before they impact users?
Yes, especially if you gather feedback, traces, and specific sub-samples from stratified document types. These patterns reveal a spike in failure rates, identify distributional changes, cluster errors, and ultimately trigger an alert to users. Signs such as entity pulls, tool misuse, and drift before an abuse report are communicated to the system.
Yaron Friedman

Amos Rimon