A tour of Gemini 1.5 Pro samples

def summarize_audio(project_id: str) -> str:

    import vertexai
    from vertexai.generative_models import GenerativeModel, Part

    vertexai.init(project=project_id, location="us-central1")

    model = GenerativeModel(model_name="gemini-1.5-pro-preview-0409")

    prompt = """
    Please provide a summary for the audio.
    Provide chapter titles with timestamps, be concise and short, no need to provide chapter summaries.
    Do not make up any information that is not part of the audio and do not be verbose.
    """

    audio_file_uri = "gs://cloud-samples-data/generative-ai/audio/pixel.mp3"
    audio_file = Part.from_uri(audio_file_uri, mime_type="audio/mpeg")

    contents = [audio_file, prompt]

    response = model.generate_content(contents)
    print(response.text)

    return response.text

This episode of the Made by Google podcast discusses the Pixel feature drops,
which are software updates that bring new features and improvements to Pixel
devices. The hosts, Aisha Sharif and DeCarlos Love, who are both product
managers for various Pixel devices, talk about the importance of feature drops
in keeping Pixel devices up-to-date and how they use user feedback to decide
which features to include in each drop. They also highlight some of their
favorite features from past feature drops, such as call screening, direct my
call, and clear calling.

Chapter Titles with Timestamps:

00:00 Intro
00:14 Made by Google Podcast Intro
00:35 Transformative Pixel Features
01:49 Why Feature Drops Are Important
02:28 January Feature Drop Highlights
02:58 March Feature Drop: Pixel Watch
03:41 March Feature Drop: Pixel Phone 
05:34 More Portfolio Updates
06:09 Pixel Superfans Question
07:32 Importance of User Feedback
08:07 Feature Drop Release Date
08:23 Favorite Feature Drop Features
10:17 Outro
10:18 Podcast Outro

async function transcript_audio(projectId = 'PROJECT_ID') {
  const vertexAI = new VertexAI({project: projectId, location: 'us-central1'});

  const generativeModel = vertexAI.getGenerativeModel({
    model: 'gemini-1.5-pro-preview-0409',
  });

  const filePart = {
    file_data: {
      file_uri: 'gs://cloud-samples-data/generative-ai/audio/pixel.mp3',
      mime_type: 'audio/mpeg',
    },
  };
  const textPart = {
    text: `
    Can you transcribe this interview, in the format of timecode, speaker, caption?
    Use speaker A, speaker B, etc. to identify speakers.`,
  };

  const request = {
    contents: [{role: 'user', parts: [filePart, textPart]}],
  };

  const resp = await generativeModel.generateContent(request);
  const contentResponse = await resp.response;
  console.log(JSON.stringify(contentResponse));
}

transcript_audio(...process.argv.slice(2)).catch(err => {
  console.error(err.message);
  process.exitCode = 1;
});

## Interview Transcription

**00:00** Speaker A: Your devices are getting better over time, and so we think
about it across the entire portfolio, from phones to watch, to buds, to tablet.
We get really excited about how we can tell a joint narrative across everything. 

**00:14** Speaker B: Welcome to the Made by Google Podcast, where we meet the
people who work on the Google products you love. Here's your host, Rasheed
Finch.

...

**10:19** Speaker C: Don’t miss out on new episodes. Subscribe now wherever you
get your podcasts to be the first to listen. 

  public static String videoAudioInput(String projectId, String location, String modelName)
      throws IOException {
    try (VertexAI vertexAI = new VertexAI(projectId, location)) {
      String videoUri = "gs://cloud-samples-data/generative-ai/video/pixel8.mp4";

      GenerativeModel model = new GenerativeModel(modelName, vertexAI);
      GenerateContentResponse response = model.generateContent(
          ContentMaker.fromMultiModalData(
              "Provide a description of the video.\n The description should also "
                  + "contain anything important which people say in the video.",
              PartMaker.fromMimeTypeAndData("video/mp4", videoUri)
          ));

      String output = ResponseHandler.getText(response);
      System.out.println(output);

      return output;
    }
  }

The video is an advertisement for the new Google Pixel phone. It features a
photographer in Tokyo who is using the phone to take pictures and  videos of the
city at night. The video highlights the phone's "Night Sight" feature, which
allows users to take clear and bright pictures and videos in low-light
conditions. The photographer also mentions that the phone's "Video Boost"
feature helps to improve the quality of videos taken in low light. The video
shows the photographer taking pictures and videos of various scenes in Tokyo,
including the city streets, a bar, and a puddle. The video ends with the
photographer saying that the new Pixel phone is "amazing" and that she "loves
it."

    public async Task<string> AnswerFromMultimodalInput(
        string projectId = "your-project-id",
        string location = "us-central1",
        string publisher = "google",
        string model = "gemini-1.5-pro-preview-0409")
    {

        var predictionServiceClient = new PredictionServiceClientBuilder
        {
            Endpoint = $"{location}-aiplatform.googleapis.com"
        }.Build();

        string prompt = "Watch each frame in the video carefully and answer the questions.\n"
                  + "Only base your answers strictly on what information is available in "
                  + "the video attached. Do not make up any information that is not part "
                  + "of the video and do not be too verbose, be to the point.\n\n"
                  + "Questions:\n"
                  + "- When is the moment in the image happening in the video? "
                  + "Provide a timestamp.\n"
                  + "- What is the context of the moment and what does the narrator say about it?";

        var content = new Content
        {
            Role = "USER"
        };
        content.Parts.AddRange(new List<Part>()
        {
            new() {
                Text = prompt
            },
            new() {
                FileData = new() {
                    MimeType = "video/mp4",
                    FileUri = "gs://cloud-samples-data/generative-ai/video/behind_the_scenes_pixel.mp4"
                }
            },
            new() {
                FileData = new() {
                    MimeType = "image/png",
                    FileUri = "gs://cloud-samples-data/generative-ai/image/a-man-and-a-dog.png"
                }
            }
        });

        var generateContentRequest = new GenerateContentRequest
        {
            Model = $"projects/{projectId}/locations/{location}/publishers/{publisher}/models/{model}"
        };
        generateContentRequest.Contents.Add(content);

        GenerateContentResponse response = await predictionServiceClient.GenerateContentAsync(generateContentRequest);

        string responseText = response.Candidates[0].Content.Parts[0].Text;
        Console.WriteLine(responseText);

        return responseText;
    }

- The timestamp of the image is 00:49.
- The context is that the narrator, a blind filmmaker, is talking about the
  story of his film. The story is about a blind man and his girlfriend, and the
  film follows them on their journey together.

type pdfPrompt struct {
	pdfPath string
	question string
}

func generateContentFromPDF(w io.Writer, prompt pdfPrompt, projectID, location, modelName string) error {
	// prompt := pdfPrompt{
	// 	pdfPath: "gs://cloud-samples-data/generative-ai/pdf/2403.05530.pdf",
	// 	question: `
	// 		You are a very professional document summarization specialist.
	// 		Please summarize the given document.
	// 	`,
	// }
	// location := "us-central1"
	// modelName := "gemini-1.5-pro-preview-0409"
	ctx := context.Background()

	client, err := genai.NewClient(ctx, projectID, location)
	if err != nil {
		return fmt.Errorf("unable to create client: %w", err)
	}
	defer client.Close()

	model := client.GenerativeModel(modelName)

	part := genai.FileData{
		MIMEType: "application/pdf",
		FileURI:  prompt.pdfPath,
	}

	res, err := model.GenerateContent(ctx, part, genai.Text(prompt.question))
	if err != nil {
		return fmt.Errorf("unable to generate contents: %w", err)
	}

	if len(res.Candidates) == 0 ||
		len(res.Candidates[0].Content.Parts) == 0 {
		return errors.New("empty response from model")
	}

	fmt.Fprintf(w, "generated response: %s\n", res.Candidates[0].Content.Parts[0])
	return nil
}

## Gemini 1.5 Pro: A Summary of its Multimodal, Long-Context Capabilities

**Gemini 1.5 Pro** is a cutting-edge multimodal, large language model (LLM)
developed by Google DeepMind. Its most significant advancement lies in its
ability to process and understand extremely long contexts of information,
spanning millions of tokens across various modalities like text, audio, and
video. This represents a substantial leap forward from previous LLMs, which were
typically limited to processing hundreds of thousands of tokens.

Here are the key takeaways from the document:
...

def set_system_instruction(project_id: str) -> str:
    import vertexai

    from vertexai.generative_models import GenerativeModel

    vertexai.init(project=project_id, location="us-central1")

    model = GenerativeModel(
        model_name="gemini-1.5-pro-preview-0409",
        system_instruction=[
            "You are a helpful language translator.",
            "Your mission is to translate text in English to French.",
        ],
    )

    prompt = """
    User input: I like bagels.
    Answer:
    """

    contents = [prompt]

    response = model.generate_content(contents)
    print(response.text)

    return response.text

J'aime les bagels.