# Gen AI Evaluation Service - Model-Based Metrics Author: [Mete Atamel](https://atamel.dev/) | Published: 2025-07-08 | Topics: GenAI, VertexAI, Gemini, Google Cloud Platform | Canonical: https://atamel.dev/posts/2025/07-08_genai_eval_service_model_metrics/ ![Model-based metrics](https://storage.googleapis.com/gweb-cloudblog-publish/images/1_s3RjOGV.max-1700x1700.png) In the [Gen AI Evaluation Service \- An Overview](https://atamel.dev/posts/2025/06-30_genai_eval_service_overview/) post, I introduced Vertex AI’s [Gen AI evaluation service](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview?utm_campaign=CDR_0xe875a906_default&utm_medium=external&utm_source=blog) and talked about the various classes of metrics it supports. In the [Gen AI Evaluation Service \- Computation-Based Metrics](https://atamel.dev/posts/2025/07-02_genai_eval_service_comp_metrics/) post, we delved into [computation-based metrics](https://cloud.google.com/vertex-ai/generative-ai/docs/models/determine-eval#computation-based-metrics?utm_campaign=CDR_0xe875a906_default&utm_medium=external&utm_source=blog), what they provide, and discussed their limitations. In today’s third post of the series, we’ll dive into [model-based metrics](https://cloud.google.com/vertex-ai/generative-ai/docs/models/determine-eval#model-based-metrics?utm_campaign=CDR_0xe875a906_default&utm_medium=external&utm_source=blog). The idea of [model-based metrics](https://cloud.google.com/vertex-ai/generative-ai/docs/models/determine-eval#model-based-metrics?utm_campaign=CDR_0xe875a906_default&utm_medium=external&utm_source=blog) is to use a judge model to evaluate the output of a candidate model. Using an LLM as a judge allows more flexible and rich evaluations that the computational/statistical metrics fail to do. While you might question using an LLM as a judge, according to the [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://arxiv.org/abs/2306.05685) paper, LLMs are pretty good evaluators when used with good evaluation prompts and rubrics, and have 80% agreement with human evaluations—the same level of agreement seen between humans. Let’s take a look at what kind of model-based metrics the Gen AI evaluation service provides. ## List of metrics There are two classes of [model-based metrics](https://cloud.google.com/vertex-ai/generative-ai/docs/models/determine-eval#model-based-metrics?utm_campaign=CDR_0xe875a906_default&utm_medium=external&utm_source=blog) in Gen AI evaluation service. First, metrics used specifically for translation related tasks. [Comet](https://huggingface.co/Unbabel/wmt22-comet-da) and [MetricX](https://github.com/google-research/metricx) are the supported translation metrics: * `comet` * `metricx` Second, more generic metrics where the judge model evaluates a candidate model on various metrics such as fluency, safety, and so on. These are the metrics that come out of the box with the [pre-built metric prompt templates](https://cloud.google.com/vertex-ai/generative-ai/docs/models/metrics-templates?utm_campaign=CDR_0xe875a906_default&utm_medium=external&utm_source=blog): * `coherence` and `pairwise_coherence` * `fluency` and `pairwise_fluency` * `safety`and `pairwise_safety` * `groundedness` and `pairwise_groundedness` * `instruction_following` and `pairwise_instruction_following` * `verbosity` and `pairwise_verbosity` * `text_quality` and `pairwise_text_quality` * `summarization_quality` and `pairwise_summarization_quality` * `question_answering_quality` and `pairwise_question_answering_quality` * `multi_turn_chat_quality` and `pairwise_multi_turn_chat_quality` * `multi_turn_safety` and `pairwise_multi_turn_safety` Each metric has pointwise (evaluating a single model) and pairwise (comparing two models) flavors. If these metrics are not enough, you can also define your own metrics with a custom metric prompt template or even have a free-form metric prompt (more on this later). ## Translation metrics Translation metrics are pretty straightforward to use. See [`translation.py`](https://github.com/meteatamel/genai-beyond-basics/blob/main/samples/evaluation/vertexai_genai_eval/model_based/translation.py) for details. Run the evaluation: ```python python translation.py ``` After a few seconds, you should see the results: ```console ==Summary metrics== row_count: 2 comet/mean: 0.90514195 comet/std: 0.09416372713647334 metricx/mean: 3.5140458499999996 metricx/std: 0.6293740377559635 ==Metrics table== source response reference comet/score metricx/score 0 Dem Feuer konnte Einhalt geboten werden The fire could be stopped They were able to control the fire. 0.838558 3.069011 1 Schulen und Kindergärten wurden eröffnet. Schools and kindergartens were open Schools and kindergartens opened 0.971726 3.959080 ``` ## Pointwise metrics In pointwise metrics, you evaluate a single model. There are 2 ways of using pointwise metrics: 1. Bring your own response (BYOR) mode where the model responses are saved from the previous runs of the model and provided as input to the evaluation (instead of calling the model during evaluation). 2. Bring a model mode where you call the model and get responses during evaluation. If BYOR mode, you provide the responses in the evaluation dataset. Here’s an example: ```python responses = [ "Clean-up operations are continuing across the Scottish Borders and Dumfries and Galloway after flooding caused by Storm Frank.", "Two tourist buses have been destroyed by fire in a suspected arson attack in Belfast city centre.", "Lewis Hamilton stormed to pole position at the Bahrain Grand Prix ahead of Mercedes team-mate Nico Rosberg.", "Manchester City midfielder Ilkay Gundogan says it has been mentally tough to overcome a third major injury.", ] eval_dataset["response"] = responses ``` In the model mode, you define a model and use that during evaluation to get responses: ```python model=GenerativeModel("gemini-2.0-flash") eval_result = eval_task.evaluate( model=model ) ``` See [`pointwise.py`](https://github.com/meteatamel/genai-beyond-basics/blob/main/samples/evaluation/vertexai_genai_eval/model_based/pointwise.py) for details on how to define both types of pointwise metrics. Run the evaluation in byor or model mode: ```python python pointwise.py byor python pointwise.py model ``` After a few seconds, you should see the results: ```console ==Summary metrics== row_count: 4 fluency/mean: 3.25 fluency/std: 1.707825127659933 ==Metrics table== prompt response fluency/explanation fluency/score 0 Summarize the following article: The full cost... Clean-up operations are continuing across the ... The response provides very little information ... 1.0 1 Summarize the following article: A fire alarm ... Two tourist buses have been destroyed by fire ... The response is fluent, with no grammatical er... 5.0 2 Summarize the following article: Ferrari appea... Lewis Hamilton stormed to pole position at the... The response is mostly fluent, with clear word... 4.0 3 Summarize the following article: Gundogan, 26,... Manchester City midfielder Ilkay Gundogan says... The response is short and grammatical, but it ... 3.0 ``` ## Pairwise metrics If you want to compare two models, you can use pairwise metrics. You first define a baseline model with the metric: ```python baseline_model = GenerativeModel("gemini-1.5-pro") metric = PairwiseMetric( metric=Metric.PAIRWISE_FLUENCY, metric_prompt_template=MetricPromptTemplateExamples.get_prompt_template(Metric.PAIRWISE_FLUENCY), baseline_model=baseline_model ) ``` Then, you define another model to compare against and use that in the evaluation: ```python model=GenerativeModel("gemini-2.0-flash") eval_task = EvalTask( dataset=eval_dataset, metrics=[metric], experiment=get_experiment_name(__file__), ) eval_result = eval_task.evaluate( model=model ) ``` See [`pairwise.py`](https://github.com/meteatamel/genai-beyond-basics/blob/main/samples/evaluation/vertexai_genai_eval/model_based/pairwise.py) for details. Run the evaluation: ```python python pairwise.py ``` After a few seconds, you should see the results where the two models are compared and a winner model is selected: ```console ==Summary metrics== row_count: 4 pairwise_fluency/candidate_model_win_rate: 0.0 pairwise_fluency/baseline_model_win_rate: 1.0 ==Metrics table== prompt response ... pairwise_fluency/explanation pairwise_fluency/pairwise_choice 0 Summarize the following article: The full cost... Severe flooding has impacted several areas in ... ... BASELINE response is slightly more fluent due ... BASELINE 1 Summarize the following article: A fire alarm ... In the early hours of Saturday, a fire alarm a... ... Both responses are well-written and follow the... BASELINE 2 Summarize the following article: Ferrari appea... Lewis Hamilton secured pole position for the B... ... BASELINE response has slightly better writing ... BASELINE 3 Summarize the following article: Gundogan, 26,... Ilkay Gundogan is recovering from a torn cruci... ... BASELINE response has a slightly better flow a... BASELINE ``` ## Custom metrics If the metrics defined in the [pre-built metric prompt templates](https://cloud.google.com/vertex-ai/generative-ai/docs/models/metrics-templates?utm_campaign=CDR_0xe875a906_default&utm_medium=external&utm_source=blog) are not good for your use case, you can define your own prompts for custom metrics. You can do this with a metric prompt template or define a free-form metric prompt for full flexibility. Here’s an example of defining a custom metric with custom criteria with the template: ```python # Define a pointwise metric with two criteria custom_metric_prompt_template = PointwiseMetricPromptTemplate( criteria={ "one_sentence": ( "The response is one short sentence." ), "entertaining": ( "The response is entertaining." ), }, rating_rubric={ "3": "The response performs well on both criteria.", "2": "The response performs well with only one of the criteria.", "1": "The response falls short on both criteria", }, ) ``` Or you can write your own prompt in free-form: ```python custom_metric_prompt_template = """ # Instruction You are an expert evaluator. Your task is to evaluate the quality of the responses generated by AI models. We will provide you with the user prompt and an AI-generated responses. You should first read the user input carefully for analyzing the task, and then evaluate the quality of the responses based on the Criteria provided in the Evaluation section below. You will assign the response a rating following the Rating Rubric and Evaluation Steps. Give step by step explanations for your rating, and only choose ratings from the Rating Rubric. # Evaluation ## Criteria entertaining: The response is entertaining. one_sentence: The response is one short sentence. ## Rating Rubric 1: The response falls short on both criteria 2: The response performs well with only one of the criteria. 3: The response performs well on both criteria. ## Evaluation Steps Step 1: Assess the response in aspects of all criteria provided. Provide assessment according to each criterion. Step 2: Score based on the rating rubric. Give a brief rationale to explain your evaluation considering each individual criterion. # User Inputs and AI-generated Response ## User Inputs ## AI-generated Response {response} """ ``` Then, use the custom prompt template in your metric: ```python custom_metric = PointwiseMetric( metric="custom_metric", metric_prompt_template=custom_metric_prompt_template ) ``` See [`pointwise_custom_metric.py`](https://github.com/meteatamel/genai-beyond-basics/blob/main/samples/evaluation/vertexai_genai_eval/model_based/pointwise_custom_metric.py) for details. ## Implementing the RAG triad with custom metrics Let's finish up by showing an example on how flexible the Gen AI evaluation service can be. In my previous blog posts, I talked about the [RAG triad metrics](https://javapro.io/2025/05/14/evaluating-rag-pipelines-with-the-rag-triad/) (answer relevance, context relevance, groundedness) to measure effectiveness of your Retrieval Augmented Generation (RAG) pipeline. ![RAG triad](https://javapro.io/wp-content/uploads/2025/03/image-29-1024x576.png) While the Gen AI evaluation service does not have RAG triad built in, you can define these metrics with a rubric quite easily and combine them into a RAG triad metric. Here’s how you’d define the answer relevance metric: ```python answer_relevance_metric = PointwiseMetric( metric="answer_relevance", metric_prompt_template=PointwiseMetricPromptTemplate( criteria={ "answer_relevance": ( "Only check response and prompt. Ignore context. Is the response relevant to the prompt?" ), }, metric_definition="Answer relevance: Checks to see if the response is relevant to the prompt", rating_rubric={ "4": "The response is totally relevant to the prompt", "3": "The response is somewhat relevant to the prompt", "2": "The response is somewhat irrelevant to the prompt", "1": "The response is totally irrelevant to the prompt", }, input_variables=["prompt"] ) ) ``` Here’s context relevance metric: ```python context_relevance_metric = PointwiseMetric( metric="context_relevance", metric_prompt_template=PointwiseMetricPromptTemplate( criteria={ "context_relevance": ( "Only check context and prompt. Ignore response. Is the context relevant to the prompt?" ), }, metric_definition="Context relevance: Check to see if the context is relevant to the prompt", rating_rubric={ "4": "The context is totally relevant to the prompt", "3": "The context is somewhat relevant to the prompt", "2": "The context is somewhat irrelevant to the prompt", "1": "The context is totally irrelevant to the prompt", }, input_variables=["prompt", "context"] ) ) ``` Here’s groundedness metric: ```python groundedness_metric = PointwiseMetric( metric="groundedness", metric_prompt_template=PointwiseMetricPromptTemplate( criteria={ "groundedness": ( "Only check context and response. Ignore prompt. Is the response supported by the context?" ), }, metric_definition="Groundedness: Check to see if the response is supported by the context", rating_rubric={ "4": "The response is totally supported by the context", "3": "The response is somewhat supported by the context", "2": "The response is somewhat not supported by the context", "1": "The response is totally not supported by the context", }, input_variables=["prompt", "context"] ) ) ``` You combine them all into an evaluation: ```python eval_task = EvalTask( dataset=eval_dataset, metrics=[answer_relevance_metric, context_relevance_metric, groundedness_metric], experiment=get_experiment_name(__file__) ) ``` With that, you have the RAG triad implemented. See [`pointwise_rag_triad.py`](https://github.com/meteatamel/genai-beyond-basics/blob/main/samples/evaluation/vertexai_genai_eval/model_based/pointwise_rag_triad.py) for details. Run the evaluation for the RAG triad: ```python python pointwise_rag_triad.py all_metric ``` You should get detailed scores back for each metric: ```console ==Metrics table== prompt \ 0 What is Cymbal Starlight? 1 Does Cymbal have an Anti-Lock braking system? 2 Where is the cargo area located? 3 What is the cargo capacity of Cymbal Starlight? context \ 0 Cymbal Starlight is a year 2024 car model 1 Maintain tire pressure at all times 2 The cargo area is located in the trunk of\nthe vehicle. 3 Cargo\nThe Cymbal Starlight 2024 has a cargo capacity of 13.5 cubic feet. response \ 0 Sky is blue 1 The Cymbal Starlight 2024 has a cargo capacity of 13.5 cubic feet. 2 The cargo area located in the front of the vehicle 3 The Cymbal Starlight 2024 has a cargo capacity of 13.5 cubic feet. answer_relevance/explanation \ 0 The response is totally irrelevant to the prompt as it provides a statement about the sky's color instead of defining 'Cymbal Starlight'. 1 The response is completely irrelevant to the prompt, as the prompt asks about the presence of an Anti-Lock braking system while the response provi... 2 The response is totally relevant to the prompt as it answers the question of where the cargo area is located. 3 The response directly answers the question about the cargo capacity of the Cymbal Starlight. answer_relevance/score \ 0 1.0 1 1.0 2 4.0 3 4.0 context_relevance/explanation \ 0 The context defines Cymbal Starlight as a 2024 car model, which is completely relevant to the prompt asking what Cymbal Starlight is. 1 The context provided is about tire pressure and the prompt asks about an anti-lock braking system which are both related to cars but are separate ... 2 The context is totally relevant to the prompt because it provides information about the location of the cargo area in a vehicle, which is exactly ... 3 The context provides information about the cargo capacity of the Cymbal Starlight, which is exactly what the prompt asks about, so it's totally re... context_relevance/score \ 0 4.0 1 1.0 2 4.0 3 4.0 groundedness/explanation \ 0 The response is not supported by the context, as the context specifies Cymbal Starlight is a car model, while the response mentions 'Sky is blue'. 1 The response discusses the cargo capacity of a Cymbal Starlight 2024, which is not related to whether the Cymbal has an Anti-Lock braking system o... 2 The response states that the cargo area is in the front, but the context says that it is located in the trunk, so the response is not supported by... 3 The response is completely supported by the context provided, as it accurately states the cargo capacity of the Cymbal Starlight 2024. groundedness/score 0 1.0 1 1.0 2 1.0 3 4.0 ``` ## Conclusion In this post, we covered model-based metrics, showed the built-in metrics, and showed how to define custom metrics, including the RAG triad. AI applications are becoming increasingly agentic, with tools and agents playing central roles. In the next post, we’ll explore how to evaluate tools and agents effectively. References: * [Part 1: Gen AI Evaluation Service \- An Overview](https://atamel.dev/posts/2025/06-30_genai_eval_service_overview/) * [Part 2: Gen AI Evaluation Service \- Computation-Based Metrics](https://atamel.dev/posts/2025/07-02_genai_eval_service_comp_metrics/) * [Tutorial: Gen AI evaluation service \- model-based metrics](https://github.com/meteatamel/genai-beyond-basics/tree/main/samples/evaluation/vertexai_genai_eval/model_based) * [Documentation: Gen AI evaluation service overview](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview?utm_campaign=CDR_0xe875a906_default&utm_medium=external&utm_source=blog) * [Notebooks: GenAI evaluation service samples](https://github.com/GoogleCloudPlatform/generative-ai/tree/main/gemini/evaluation)