Mete Atamel

Introduction

In this blog post, we’ll deep dive into function calling in Gemini. More specifically, you’ll see how to handle multiple and parallel function call requests from generate_content and chat interfaces and take a look at the new auto function calling feature through a sample weather application.

What is function calling?

Function Calling is useful to augment LLMs with more up-to-date data via external API calls.

You can define custom functions and provide these to an LLM. While processing a prompt, the LLM can choose to delegate tasks to the functions that you identify. The model does not call the functions directly but rather makes function call requests with parameters to your application. In turn, your application code responds to function call requests by calling external APIs and providing the responses back to the model, allowing the LLM to complete its response to the prompt.

Introduction

Some large language models (LLMs), such as Gemini 1.5 Flash or Gemini 1.5 Pro, have a very large context window. This is very useful if you want to analyze a big chunk of data, such as a whole book or a long video. On the other hand, it can get quite expensive if you keep sending the same large data in your prompts. Context caching can help.

Introduction

Large language models (LLMs) are great at generating content but the output format you get back can be a hit or miss sometimes.

For example, you ask for a JSON output in certain format and you might get free-form text or a JSON wrapped in markdown string or a proper JSON but with some required fields missing. If your application requires a strict format, this can be a real problem.

Introduction

Recently, I talked about why grounding LLMs is important and how to ground LLMs with public data using Google Search (Vertex AI’s Grounding with Google Search: how to use it and why) and with private data using Vertex AI Search (Grounding LLMs with your own data using Vertex AI Search).

In today’s post, I want to talk about another more flexible and customizable way of grounding your LLMs with private data: the RAG API powered by LlamaIndex on Vertex AI.

Introduction

In my previous Vertex AI’s Grounding with Google Search: how to use it and why post, I explained why you need grounding with large language models (LLMs) and how Vertex AI’s grounding with Google Search can help to ground LLMs with public up-to-date data.

That’s great but you sometimes need to ground LLMs with your own private data. How can you do that? There are many ways but Vertex AI Search is the easiest way and that’s what I want to talk about today with a simple use case.

Introduction

It’s no secret that LLMs sometimes lie and they do so in a very confident kind of way. This might be OK for some applications but it can be a real problem if your application requires high levels of accuracy.

I remember when the first LLMs emerged back in early 2023. I tried some of the early models and it felt like they were hallucinating half of the time. More recently, it started feeling like LLMs are getting better at giving more factual answers. But it’s just a feeling and you can’t base application decisions (or any decision?) on feelings, can you?

It’s been a while since I wrote a non-work related topic. Last time, I wrote about the unique kindness I experienced in Japan (see The Butterfly effect of kindness). This time, I want to write about a dilemma that I’ve been thinking about for a while.

When I reflect on my life so far, whenever I had some progress (learning a new skill, making new lasting connections, changing to a new job, losing weight), it was always due to consistency in my life. I was not traveling, I was not thinking about where to go, what to do, where to eat, how to get from point A to point B. I was in my familiar environment with a consistent (and maybe boring) routine where the basics of my life were in place. As a result, I had time, got bored, and started exploring. This consistency fueled boredom allowed me to explore an aspect of life that I wasn’t happy about and I put the time and energy into improving it.

Introduction

Once in a while, you come across a feature that is so easy to use and so useful that you don’t know how you lived without it before. For me, Vertex AI’s Grounding with Google Search is one of those features.

In this blog post, I explain why you need grounding with large language models (LLMs) and how Vertex AI’s Grounding with Google Search can help with minimal effort on your part.

Almost one year ago, I talked about AsyncAPI 2.6 and how confusing its publish and subscribe operations can be in my Understanding AsyncAPI’s publish & subscribe semantics with an example post.

Since then, a new 3.0 version of AsyncAPI has been released with breaking changes and a totally new send and receive operations.

In this blog post, I want to revisit the example from last year and show how to rewrite it for AsyncAPI 3.0 with the new send and receive operations.

Introduction

Back in February, Google announced Gemini 1.5 Pro with its impressive 1 million token context window.

Larger context size means that Gemini 1.5 Pro can process vast amounts of information in one go — 1 hour of video, 11 hours of audio, 30,000 lines of code or over 700,000 words and the good news is that there’s good language support.

In this blog post, I will point out some samples utilizing Gemini 1.5 Pro in Google Cloud’s Vertex AI in different use cases and languages (Python, Node.js, Java, C#, Go).