# DeepEval adds native support for Gemini as an LLM Judge

Author: [Mete Atamel](https://atamel.dev/) | Published: 2025-04-29 | Topics: GenAI, VertexAI, Gemini, Google Cloud Platform | Canonical: https://atamel.dev/posts/2025/04-29_deepeval_native_gemini_judge/


![DeepEval and Gemini](https://github.com/meteatamel/genai-beyond-basics/raw/main/samples/evaluation/deepeval/images/deepeval_gemini.png)

In my [previous post on DeepEval and Vertex AI](https://atamel.dev/posts/2024/08-12_deepeval_vertexai/), I introduced
[DeepEval](https://www.deepeval.com/), an open-source evaluation framework for LLMs. I also demonstrated how to use
Gemini (on Vertex AI) as an LLM Judge in DeepEval, replacing the default OpenAI judge to evaluate outputs from other
LLMs. At that time, the Gemini integration with DeepEval wasn’t ideal and I had to implement my own integration.

Thanks to the excellent work by [Roy Arsan](https://www.linkedin.com/in/arsan/) in [PR
#1493](https://github.com/confident-ai/deepeval/pull/1493), DeepEval now includes **native Gemini integration**. Since
it’s built on the new unified [Google GenAI SDK](https://cloud.google.com/vertex-ai/generative-ai/docs/sdks/overview),
DeepEval supports Gemini models running both on Vertex AI and Google AI. Nice!

There are two ways to use Gemini as an LLM Judge in DeepEval: via the command line or directly in your code. Let’s
explore both.

## Gemini on Vertex AI

The easiest way to use Gemini on Vertex AI with DeepEval is through the command line using the `deepeval` tool.

For example, to use `gemini-1.5-pro` in your Google Cloud project `genai-atamel` as the evaluation model:

```bash
deepeval set-gemini --model-name="gemini-1.5-pro" \
    --project-id="genai-atamel" \
    --location="us-central1"
```

Once set, when you create a metric for evaluation, DeepEval will use this Gemini model by default instead of OpenAI:

```python
metric = AnswerRelevancyMetric(threshold=0.8)
```

Alternatively, you can set the evaluation model directly in your code.

First, create a `GeminiModel` instance with your Vertex AI parameters:

```python
from deepeval.models import GeminiModel

EVAL_MODEL = "gemini-1.5-pro"
PROJECT_ID = "genai-atamel"
LOCATION = "us-central1"

eval_model = GeminiModel(
    model_name=EVAL_MODEL,
    project=PROJECT_ID,
    location=LOCATION
)
```

Then, pass the model to your metric:

```python
metric = AnswerRelevancyMetric(
    model=eval_model,
    threshold=0.8
)
```

You can find a full example in `test_answer_relevancy.py` in [my
repository](https://github.com/meteatamel/genai-beyond-basics/blob/main/samples/evaluation/deepeval/test_answer_relevancy.py).

## Gemini on Google AI

The new integration also supports Gemini via Google AI. The main difference is that you need a **Google AI API key**
instead of a project ID and location.

You can configure it from the command line like this:

```bash
deepeval set-gemini --model-name="gemini-1.5-pro" \
    --google-api-key="your-google-ai-api-key"
```

Or directly in code:

```python
from deepeval.models import GeminiModel

EVAL_MODEL = "gemini-1.5-pro"
GOOGLEAI_API_KEY = "your-google-ai-api-key"

eval_model = GeminiModel(
    model_name=EVAL_MODEL,
    api_key=GOOGLEAI_API_KEY
)

metric = AnswerRelevancyMetric(
    model=eval_model,
    threshold=0.8
)
```

---

I'm glad to see native Gemini support in DeepEval now. For more details, check out DeepEval’s [Gemini
documentation](https://www.deepeval.com/integrations/models/gemini). For examples on other metrics such as
summarization, hallucination detection, or the RAG triad, check out my [DeepEval tutorial on
GitHub](https://github.com/meteatamel/genai-beyond-basics/tree/main/samples/evaluation/deepeval), which has been updated
with the new Gemini integration.

Happy evaluations with DeepEval and Gemini as LLM Judge!

