# Gen AI Evaluation Service - Computation-Based Metrics Author: [Mete Atamel](https://atamel.dev/) | Published: 2025-07-02 | Topics: GenAI, VertexAI, Gemini, Google Cloud Platform | Canonical: https://atamel.dev/posts/2025/07-02_genai_eval_service_comp_metrics/ In my [Gen AI Evaluation Service \- An Overview](https://atamel.dev/posts/2025/06-30_genai_eval_service_overview/) post, I introduced Vertex AI’s [Gen AI evaluation service](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview) and talked about the various classes of metrics it supports. In today’s post, I want to dive into [computation-based metrics](https://cloud.google.com/vertex-ai/generative-ai/docs/models/determine-eval#computation-based-metrics), what they provide, and discuss their limitations. [Computation-based metrics](https://cloud.google.com/vertex-ai/generative-ai/docs/models/determine-eval#computation-based-metrics) are metrics that can be calculated using a mathematical formula. They’re deterministic – the same input produces the same score, unlike model-based metrics where you might get slightly different scores for the same input. They require a reference (ground truth) to compare against when calculating the score. That can be a drawback if you don’t have such ground truth. Also, they tend to be quite rigid and do not capture the nuances of language. We’ll talk more about this later. ## List of metrics These are the [computation-based](https://cloud.google.com/vertex-ai/generative-ai/docs/models/determine-eval#computation-based-metrics) metrics supported by Gen AI evaluation service out of the box: * `exact_match` * `bleu` * `rouge` * `rouge_1` * `rouge_2` * `rouge_l` * `rouge_l_sum` **Note**: There are metrics related to tool use and agents, which can also be considered computation-based. We’ll cover them separately in a future blog post. `EXACT_MATCH` metric computes whether a generated text matches a reference text exactly. 0: No match, 1: Match. `BLEU` ([BiLingual Evaluation Understudy](https://en.wikipedia.org/wiki/BLEU)) metric evaluates the similarity between generated text and reference text. It measures precision (accuracy). A float in the range of 0-1. 0: Poor similarity, 1: Perfect match to reference. `ROUGE` ([Recall-Oriented Understudy for Gisting Evaluation](https://en.wikipedia.org/wiki/ROUGE_\(metric\))) metric evaluates text summaries. It measures recall (completeness). A float in the range of 0-1. 0: Poor similarity, 1: Strong similarity to reference. Additionally, you can define your own custom metric with the [`CustomMetric`](https://github.com/googleapis/python-aiplatform/blob/main/vertexai/evaluation/metrics/_base.py#L75) class. ## Standard metrics Let’s see how we can run an evaluation with the standard metrics. First, define an evaluation dataset with responses from the model and the reference you want to compare against: ```python responses = [ "Hello, how are you?", "I'm good", "The cat lay on the mat." ] references = [ "Hello, how are you?", "I am good", "The cat sat on the mat." ] eval_dataset = pandas.DataFrame( { "response": responses, "reference": references, } ) ``` Then, define the metrics. Here, we’ll go with `EXACT_MATCH`, `BLEU` and the standard `ROUGE` metric: ```python metrics = [ Metric.EXACT_MATCH, Metric.BLEU, Metric.ROUGE, ] ``` Create an evaluation task, run it, and then print the results: ```python eval_task = EvalTask( dataset=eval_dataset, metrics=metrics, experiment=get_experiment_name(__file__) ) eval_result = eval_task.evaluate() print_eval_result(eval_result) ``` Run the evaluation: ```sh python computation.py ==Summary metrics== row_count: 3 exact_match/mean: 0.3333333333333333 exact_match/std: 0.5773502691896258 bleu/mean: 0.5596273066666667 bleu/std: 0.40962295108770713 rouge/mean: 0.8333333333333334 rouge/std: 0.16666665000000253 ==Metrics table== response reference exact_match/score bleu/score rouge/score 0 Hello, how are you? Hello, how are you? 1.0 1.000000 1.000000 1 I'm good I am good 0.0 0.189959 0.666667 2 The cat lay on the mat. The cat sat on the mat. 0.0 0.488923 0.833333 ``` Note that I’m simply printing the results in a utility function but you can also view the results in Vertex AI’s Experiments section: ![Vertex AI Experiments](https://github.com/meteatamel/genai-beyond-basics/blob/main/samples/evaluation/vertexai_genai_eval/images/vertexai_exp2.png?raw=true) You can see [computation.py](https://github.com/meteatamel/genai-beyond-basics/blob/main/samples/evaluation/vertexai_genai_eval/computation_based/computation.py) for the full sample. ## Why do computation-based metrics fall short? Let’s now talk about why computation-based metrics fall short in capturing the semantic meaning. These are the results we had: ```sh ==Metrics table== response reference exact_match/score bleu/score rouge/score 0 Hello, how are you? Hello, how are you? 1.0 1.000000 1.000000 1 I'm good I am good 0.0 0.189959 0.666667 2 The cat lay on the mat. The cat sat on the mat. 0.0 0.488923 0.833333 ``` As you can see, when there’s a perfect match, all the scores are good. However, when there’s a slight change in wording, the scores drop significantly. As humans, we understand that “I’m good” and “I am good” have the same meaning but BLEU score is already at 0.18 and ROUGE is not that much better with 0.66. Because of the rigid nature of these metrics, you probably need to use model-based metrics in your evaluations. We’ll cover this in a future blog post. ## Custom metrics You can also define your own metrics and run them as part of your evaluation. Note that these custom metrics run on the client side without going through the Gen AI evaluation service but they can still be useful as part of your overall evaluation. For example, let’s say you want to use “word count” as a custom metric. First, you define your metric with the CustomMetric class: ```python custom_word_count_metric = CustomMetric( name="word_count", metric_function=word_count) ``` Then, you define the actual function that will do the word counting and return the result: ```python def word_count(instance: dict[str, str]) -> dict[str, float]: """Count the number of words in the response.""" response = instance["response"] score = len(response.split(" ")) return { "word_count": score, } ``` Run the evaluation: ```python python computation_custom.py ==Metrics table== response word_count/score 0 Hello, how are you? 4 1 I'm good 2 2 The cat lay on the mat. 6 ``` Take a look at [computation\_custom.py](https://github.com/meteatamel/genai-beyond-basics/blob/main/samples/evaluation/vertexai_genai_eval/computation_based/computation_custom.py) for the full sample. ## Conclusion This wraps up our discussion about computation-based metrics. While they’re useful baseline in your evaluations, they fall short in capturing the nuances of language. In the next post, we’ll take a look at model-based metrics in more detail. References: * [Blog: Gen AI Evaluation Service \- An Overview](https://atamel.dev/posts/2025/06-30_genai_eval_service_overview/) * [Tutorial: Gen AI evaluation service \- computation-based metrics](https://github.com/meteatamel/genai-beyond-basics/tree/main/samples/evaluation/vertexai_genai_eval/computation_based) * [Documentation: Gen AI evaluation service overview](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview?utm_campaign=CDR_0xe875a906_default&utm_medium=external&utm_source=blog) * [Notebooks: GenAI evaluation service samples](https://github.com/GoogleCloudPlatform/generative-ai/tree/main/gemini/evaluation)