VertexAI

The Gemini team at Google recently announced the File Search Tool, a fully managed RAG system built directly into the Gemini API as a simple, integrated, and scalable way to ground Gemini. I gave it a try and I’m impressed how easy it is to use to ground Gemini with your own data.

In this blog post, I’ll introduce the File Search Tool and show you a concrete example.

I’ve been exploring the Agent Developer Kit (ADK) and its powerful callbacks feature. In this blog post, I want to outline what callbacks are and provide a sample agent with all the callbacks implemented for a quick reference and testing.

At its core, an agent framework like ADK gives you a sequence of steps:

receive input → invoke model → invoke tools → return output

In real-world systems, we often need to hook into these steps for logging, guarding, caching, altering prompts or results, or dynamically changing behaviour based on session state. That’s exactly where callbacks come in. Think of callbacks as “checkpoints” in the agent’s lifecycle. The ADK framework automatically calls your functions at these key stages, giving you a chance to intervene.

A few days ago, I saw Logan Kilpatrick’s Tweet about the new AI-first vibe coding experience in AI Studio:

Introducing the new AI first vibe coding experience in @GoogleAIStudio! Built to take you from prompt to production with Gemini, and optimized for AI app creation. Start building AI apps for free : )

More updates and features to come! pic.twitter.com/HpI7Dsl8Bj

— Logan Kilpatrick (@OfficialLoganK) October 21, 2025

Last time I tried AI Studio for vibe coding, it was mostly a single-page web application with all the code in a single file (which wasn’t ideal).

Last year, we announced the Google Gen AI SDK as the new unified library for Gemini on Google AI (via the Gemini Developer API) and Vertex AI (via the Vertex AI API). At the time, it was only a Python SDK. Since then, the team has been busy adding support for Go, Node.js, and Java but my favorite language, C#, was missing until now.

Today, I’m happy to announce that we now have a Google Gen AI .NET SDK! This SDK enables C#/.NET developers use Gemini from Google AI or Vertex AI with a single unified library.

Earlier this month, the Gemini 2.5 Computer Use model was announced. This model is specialized in interacting with graphical user interfaces (UI). This is useful in scenarios where a structured API does not exist for the model to interact with (via function calling). Instead, you can use the Computer Use model to directly interact with user interfaces such as filling and submitting forms.

It’s important to note that the model does not interact with the UI directly. As input, the model receives the user request, a screenshot of the environment, and a history of recent actions. As output, it generates a function call representing a UI action such as clicking or typing (see the full list of supported UI actions). It’s the client-side code’s responsibility to execute the received action and the process continues in a loop:

It’s crucial to secure inputs and outputs to and from your Large Language Model (LLM). Failure to do so can result in prompt injections, jailbreaking, sensitive information exposure, and more (as detailed in OWASP Top 10 for Large Language Model Applications).

I previously talked about LLM Guard and Vertex AI and showed how to use LLM Guard to secure LLMs. Google Cloud has its own service to secure LLMs: Model Armor. In this post, we’ll explore Model Armor and see how it can help to safeguard your LLM applications.

This is the sixth and final post in my Vertex AI Gen AI Evaluation Service blog post series. In the previous posts, we covered computation-based, model-based, tool-use, and agent metrics. These metrics measure different aspects of an LLM response in different ways but one thing they all had in common: they are all for text-based outputs.

LLMs nowadays also produce multimodal (images, videos) outputs. How do you evaluate multimodal outputs? That’s the topic of this blog post.

In my previous Gen AI Evaluation Service - Tool-Use Metrics post, we talked about LLMs calling external tools and how you can use tool-use metrics to evaluate how good those tool calls are. In today’s fifth post of my Vertex AI Gen AI Evaluation Service blog post series, we will talk about a related topic: agents and agent metrics.

What are agents?

There are many definitions of agents but an agent is essentially a piece of software that acts autonomously to achieve specific goals. They use LLMs to perform tasks, utilize external tools, coordinate with other agents, and ultimately produce a response to the user.

I’m continuing my Vertex AI Gen AI Evaluation Service blog post series. In today’s fourth post of the series, I will talk about tool-use metrics.

What is tool use?

Tool use, also known as function calling, provides the LLM with definitions of external tools (for example, a get_current_weather function). When processing a prompt, the model determines if a tool is needed and, if so, outputs structured data specifying the tool to call and its parameters (for example, get_current_weather(location='London')).

In the Gen AI Evaluation Service - An Overview post, I introduced Vertex AI’s Gen AI evaluation service and talked about the various classes of metrics it supports. In the Gen AI Evaluation Service - Computation-Based Metrics post, we delved into computation-based metrics, what they provide, and discussed their limitations. In today’s third post of the series, we’ll dive into model-based metrics.

The idea of model-based metrics is to use a judge model to evaluate the output of a candidate model. Using an LLM as a judge allows more flexible and rich evaluations that the computational/statistical metrics fail to do.