Gemini

Gemini is Google's native multimodal foundation model family, featuring native audio, image, and video comprehension alongside million-token context windows and fast tool-calling.

When to Use

• Massive Multimodal Understanding: Gemini 1.5 Pro / Flash processing up to 2 million tokens of text, hours of video, audio, and large codebases.
• Structured JSON Extraction: Enforcing rigid response schemas with Pydantic and native response_schema.
• High-Speed Real-Time Inference: Gemini 1.5 Flash for sub-second, cost-effective conversational assistants and summarization.
• Code & Repository Analysis: Analyzing entire GitHub repositories in a single context window.

Quick Start

from google import genai

client = genai.Client()

response = client.models.generate_content(
    model="gemini-2.0-flash",
    contents="Explain quantum computing in one short paragraph.",
)

print(response.text)

Core Concepts

Structured Output with Pydantic & Google GenAI SDK

Enforcing type-safe JSON schema responses:

from google import genai
from google.genai import types
from pydantic import BaseModel, Field
import os

client = genai.Client(api_key=os.environ.get("GEMINI_API_KEY"))

class InvoiceExtraction(BaseModel):
    invoice_number: str
    vendor: str
    total_amount: float = Field(description="Total invoice amount in USD")
    items: list[str]
    is_paid: bool

response = client.models.generate_content(
    model='gemini-1.5-flash',
    contents='Invoice #INV-2026-908 from CloudCorp for $1,450.00. Included 2 Cloud Servers and 1 Storage bucket. Paid via Wire.',
    config=types.GenerateContentConfig(
        response_mime_type="application/json",
        response_schema=InvoiceExtraction,
        temperature=0.1
    ),
)

extracted: InvoiceExtraction = InvoiceExtraction.model_validate_json(response.text)
print(f"Extracted Invoice: {extracted.invoice_number}, Total: ${extracted.total_amount}")

Native Video & Audio Multimodal Analysis

Processing video files directly using the File API:

from google import genai

client = genai.Client()

# Upload video file to File API
video_file = client.files.upload(file="presentation.mp4")

# Query video content directly
response = client.models.generate_content(
    model="gemini-1.5-pro",
    contents=[
        video_file,
        "Summarize the key decisions made between timestamp 04:15 and 08:30 in this meeting."
    ]
)

print(response.text)

Context Caching for Massive Contexts

Reducing cost on recurring queries against massive datasets:

from google import genai
from google.genai import types
import datetime

client = genai.Client()

# Create cached content for a massive documentation corpus
cache = client.caches.create(
    model='gemini-1.5-pro',
    config=types.CreateCachedContentConfig(
        contents=[large_documentation_file],
        ttl=datetime.timedelta(hours=2),
        display_name='api_documentation_v2'
    )
)

# Query against cache at significantly reduced per-token cost
response = client.models.generate_content(
    model='gemini-1.5-pro',
    contents='What are the rate limits for the batch endpoint?',
    config=types.GenerateContentConfig(cached_content=cache.name)
)

Common Patterns

Multimodal Document Analysis (PDF / Audio / Video)

Problem: Processing multi-page PDF documents or audio files without complex local extraction libraries.

Solution: Upload file using the Google GenAI File API:

# Upload document directly to Gemini File API
doc_file = client.files.upload(file="contract.pdf")

response = client.models.generate_content(
    model="gemini-2.0-flash",
    contents=[
        doc_file,
        "Extract the governing law, termination notice period, and liability cap."
    ]
)

print(response.text)

Best Practices

Do:

• Target the modern Google GenAI SDK (google-genai) instead of legacy deprecated libraries.
• Use gemini-1.5-flash for high-volume, low-latency tasks and gemini-1.5-pro for deep reasoning.
• Enforce structured outputs with response_schema and Pydantic models for reliable API integrations.
• Leverage Context Caching for prompts exceeding 32k tokens that are reused across multiple calls.

Don't:

• Upload raw video/audio base64 payloads inline; use the client.files.upload() API.
• Leave sensitive credentials exposed in client-side code; proxy requests through a secure backend.
• Use high temperature settings (> 0.4) when strict schema compliance or factual retrieval is required.

Troubleshooting

References

• Gemini API Documentation