GoogleAI on Atamel.Dev

RAG with a PDF using LlamaIndex and SimpleVectorStore on Vertex AI

atamel@gmail.com (Mete Atamel) — Mon, 24 Mar 2025 00:00:00 +0000

LlamaIndex and Vertex AI

Previously, I showed how to do RAG with a PDF using LangChain and Annoy Vector Store and RAG with a PDF using LangChain and Firestore Vector Store. Both used a PDF as the RAG backend and used LangChain as the LLM framework to orchestrate RAG ingestion and retrieval.

LlamaIndex is another popular LLM framework. I wondered how to set up the same PDF based RAG pipeline with LlamaIndex and Vertex AI but I didn’t find a good sample. I put together a sample and in this short post, I show how to set up the same PDF based RAG pipeline with LlamaIndex.

Ensuring AI Code Quality with SonarQube + Gemini Code Assist

atamel@gmail.com (Mete Atamel) — Tue, 04 Mar 2025 00:00:00 +0000

In my previous Code Quality in the Age of AI-Assisted Development blog post, I talked about how generative AI is changing the way we code and its potential impact on code quality. I recommended using static code analysis tools to monitor AI-generated code, ensuring its security and quality.

In this blog post, I will explore one such static code analysis tool, SonarQube, and see how it improves the quality of AI-generated code.

Code Quality in the Age of AI-Assisted Development

atamel@gmail.com (Mete Atamel) — Tue, 28 Jan 2025 00:00:00 +0000

As developers transition from manual coding to AI-assisted coding, an increasing share of code is now being generated by AI. This shift has significantly boosted productivity and efficiency, but it raises an important question: how does AI-assisted development impact code quality? How can we ensure that AI-generated code maintains high quality, adheres to good style, and follows best practices? This question has been on my mind recently, and it is the topic of this blog post.

Improve the RAG pipeline with RAG triad metrics

atamel@gmail.com (Mete Atamel) — Tue, 21 Jan 2025 00:00:00 +0000

In my previous RAG Evaluation - A Step-by-Step Guide with DeepEval post, I showed how to evaluate a RAG pipeline with the RAG triad metrics using DeepEval and Vertex AI. As a recap, these were the results:

RAG triad with DeepEval

Answer relevancy and faithfulness metrics had perfect 1.0 scores whereas contextual relevancy was low at 0.29 because we retrieved a lot of irrelevant context:

The score is 0.29 because while the context mentions relevant information such as "The Cymbal Starlight 2024 has a cargo
capacity of 13.5 cubic feet", much of the retrieved context is irrelevant. For example, several statements discuss
towing capacity like "Your Cymbal Starlight 2024 is not equipped to tow a trailer", or describe how to access/load cargo
like "To access the cargo area, open the trunk lid using the trunk release lever located in the driver's footwell"
instead of focusing on the requested cargo capacity.

Can we improve this? Let’s take a look.

RAG Evaluation - A Step-by-Step Guide with DeepEval

atamel@gmail.com (Mete Atamel) — Tue, 14 Jan 2025 00:00:00 +0000

In my previous Evaluating RAG pipelines post, I introduced two approaches to evaluating RAG pipelines. In this post, I will show you how to implement these two approaches in detail. The implementation will naturally depend on the framework you use. In my case, I’ll be using DeepEval, an open-source evaluation framework.

Approach 1: Evaluating Retrieval and Generator separately

As a recap, in this approach, you evaluate the retriever and generator of the RAG pipeline separately with their own separate metrics. This approach allows to pinpoint issues at the retriever and the generator level:

Evaluating RAG pipelines

atamel@gmail.com (Mete Atamel) — Thu, 09 Jan 2025 00:00:00 +0000

Retrieval-Augmented Generation (RAG) emerged as a dominant framework to feed LLMs the context beyond the scope of its training data and enable LLMs to respond with more grounded answers with less hallucinations based on that context.

However, designing an effective RAG pipeline can be challenging. You need to answer certain questions such as:

How should you parse and chunk text documents for embedding? What chunk and overlay size to use?
What embedding model is best for your use case?
What retrieval method works most effectively? How many documents should you retrieve by default? Does the retriever actually manage to retrieve the relevant documents?
Does the generator actually generate content in line with the relevant context? What parameters (e.g. model, prompt template, temperature) work better?

The only way to objectively answer these questions is to measure how well the RAG pipeline works but what exactly do you measure? This is the topic of this blog post.

Gemini on Vertex AI and Google AI now unified with the new Google Gen AI SDK

atamel@gmail.com (Mete Atamel) — Tue, 17 Dec 2024 00:00:00 +0000

If you’ve been working with Gemini, you’ve likely encountered the two separate client libraries for Gemini: the Gemini library for Google AI vs. Vertex AI in Google Cloud. Even though the two libraries are quite similar, there are slight differences that make the two libraries non-interchangeable.

I usually start my experiments in Google AI and when it is time to switch to Vertex AI on Google Cloud, I couldn’t simply copy and paste my code. I had to go through updating my Google AI libraries to Vertex AI libraries. It wasn’t difficult but it was quite annoying.

Control LLM output with LangChain's structured and Pydantic output parsers

atamel@gmail.com (Mete Atamel) — Mon, 09 Dec 2024 00:00:00 +0000

In my previous Control LLM output with response type and schema post, I talked about how you can define a JSON response schema and Vertex AI makes sure the output of the Large Language Model (LLM) conforms to that schema.

In this post, I show how you can implement a similar response schema using LangChain’s structured output parser with any model. You can further get the output parsed and populated into Python classes automatically with the Pydantic output parser. This helps you to really narrow down and structure LLM outputs.

Tracing with Langtrace and Gemini

atamel@gmail.com (Mete Atamel) — Wed, 27 Nov 2024 00:00:00 +0000

Large Language Models (LLMs) feel like a totally new technology with totally new problems. It’s true to some extent but at the same time, they also have the same old problems that we had to tackle in traditional technology.

For example, how do you figure out which LLM calls are taking too long or have failed? At the bare minimum, you need logging but ideally, you use a full observability platform like OpenTelemetry with logging, tracing, metrics and more. You need the good old software engineering practices, such as observability, applied to new technologies like LLMs.

Batch prediction in Gemini

atamel@gmail.com (Mete Atamel) — Mon, 18 Nov 2024 00:00:00 +0000

LLMs are great in generating content on demand but if left unchecked, you can be left with a large bill at the end of the day. In my Control LLM costs with context caching post, I talked about how to limit costs by using context caching. Batch generation is another technique you can use to save time at a discounted price.

What’s batch generation?

Batch generation in Gemini allows you to send multiple generative AI requests in batches rather than one by one and get responses asynchronously either in a Cloud Storage bucket or a BigQuery table. This not only simplifies processing of large datasets, but it also saves time and money, as batch requests are processed in paralllel and discounted 50% from standard requests.

LLM Guard and Vertex AI

atamel@gmail.com (Mete Atamel) — Mon, 11 Nov 2024 00:00:00 +0000

LLM Guard and Vertex AI

I’ve been focusing on evaluation frameworks lately because I believe that the hardest problem while using LLMs is to make sure they behave properly. Are you getting the right outputs grounded with your data? Are outputs free of harmful or PII data? When you make a change to your RAG pipeline or to your prompts, are outputs getting better or worse? How do you know? You don’t know unless you measure. What do you measure and how? These are the sort of questions you need to answer and that’s when evaluation frameworks come into the picture.

Promptfoo and Vertex AI

atamel@gmail.com (Mete Atamel) — Mon, 04 Nov 2024 00:00:00 +0000

Promptfoo and Vertex AI

In my previous DeepEval and Vertex AI blog post, I talked about how crucial it is to have an evaluation framework in place when working with Large Language Models (LLMs) and introduced DeepEval as one of such evaluation frameworks.

Recently, I came across another LLM evaluation and security framework called Promptfoo. In this post, I will introduce Promptfoo, show what it provides for evaluations and security testing, and how it can be used with Vertex AI.

Firestore for Image Embeddings

atamel@gmail.com (Mete Atamel) — Tue, 29 Oct 2024 00:00:00 +0000

Firestore and LangChain

In my previous Firestore for Text Embedding and Similarity Search post, I talked about how Firestore and LangChain can help you to store text embeddings and do similarity searches against them. With multimodal embedding models, you can generate embeddings not only for text but for images and video as well. In this post, I will show you how to store image embeddings in Firestore and later use them for similarity search.

Firestore for Text Embedding and Similarity Search

atamel@gmail.com (Mete Atamel) — Wed, 09 Oct 2024 00:00:00 +0000

Firestore and LangChain

In my previous Persisting LLM chat history to Firestore post, I showed how to persist chat messages in Firestore for more meaningful and context-aware conversations. Another common requirement in LLM applications is to ground responses in data for more relevant answers. For that, you need embeddings. In this post, I want to talk specifically about text embeddings and how Firestore and LangChain can help you to store text embeddings and do similarity searches against them.

Persisting LLM chat history to Firestore

atamel@gmail.com (Mete Atamel) — Tue, 01 Oct 2024 00:00:00 +0000

Firestore and LangChain

Firestore has long been my go-to NoSQL backend for my serverless apps. Recently, it’s becoming my go-to backend for my LLM powered apps too. In this series of posts, I want to show you how Firestore can help for your LLM apps.

In the first post of the series, I want to talk about LLM powered chat applications. I know, not all LLM apps have to be chat based apps but a lot of them are because LLMs are simply very good at chat based communication.

Semantic Kernel and Gemini

atamel@gmail.com (Mete Atamel) — Mon, 19 Aug 2024 00:00:00 +0000

Semantic Kernel and VertexAI

Introduction

When you’re building a Large Language Model (LLMs) application, you typically start with the SDK of the LLM you’re trying to talk to. However, at some point, it might make sense to start using a higher level framework. This is especially true if you rely on multiple LLMs from different vendors. Instead of learning and using SDKs from multiple vendors, you can learn a higher level framework and use that to orchestrate your calls to multiple LLMs. These frameworks also have useful abstractions beyond simple LLM calls that accelerate LLM application development.