agentleFS
dtunaidtunai/agent-skills-for-compute/skills/flashinfer/SKILL.md

flashinfer

Skill8 starsChanged 8 months agoMITView on GitHub

FlashInfer — High-performance kernel library for LLM inference with optimized attention, paged KV-cache, FP8/FP4 quantization

  • Installs packages

The file

We split this skill into the rules it gives an agent

1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Profile workload: Use flashinfer.bench for performance analysis

    line 383

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

FlashInfer Skill

High-performance GPU kernel library for Large Language Model inference delivering state-of-the-art performance across diverse GPU architectures with optimized attention, GEMM, and MoE operations.

Official Sources:

What is FlashInfer?

Definition:

"A library and kernel generator for Large Language Models that provides high-performance implementation of LLM GPU kernels such as FlashAttention, PageAttention and LoRA."

Key Features:

  • Unified APIs: Attention, GEMM, MoE with multiple backend implementations
  • Paged & Ragged KV-Cache: Efficient memory management for dynamic batching
  • Multi-Backend: FlashAttention-2/3, cuDNN, CUTLASS, TensorRT-LLM
  • Quantization: FP8 and FP4 for attention, GEMM, MoE operations
  • Production-Ready: CUDAGraph and torch.compile compatible
  • Wide GPU Support: SM75 (Turing) through SM121 (Blackwell)

Quick Start

Installation

# Basic installation
pip install flashinfer-python

# With pre-compiled kernels (recommended)
pip install flashinfer-python flashinfer-cubin

# With JIT cache for specific CUDA version
pip install flashinfer-jit-cache --index-url https://flashinfer.ai/whl/cu129

System Requirements:

  • Linux only
  • Python 3.10-3.14
  • CUDA 12.6, 12.8, 13.0, or 13.1
  • GPU: Turing (T4) through Blackwell

Verify Installation

flashinfer show-config

Basic Usage

import torch
import flashinfer

# Single decode with paged KV-cache
output = flashinfer.single_decode_with_kv_cache(
    q=query,                    # (num_qo_heads, head_dim)
    kv_data=kv_cache,          # (num_pages, 2, num_kv_heads, page_size, head_dim)
    kv_indices=kv_page_indices,  # (num_pages,)
    kv_indptr=kv_page_indptr,    # (batch_size + 1,)
    kv_last_page_len=last_page_lengths,  # (batch_size,)
)

Core Capabilities

Attention Operations

Decode (Token-by-Token Generation):

# Single request decode
output = flashinfer.single_decode_with_kv_cache(
    q=query,
    kv_data=kv_cache,
    # ... KV-cache parameters
)

# Batch decode with cuDNN backend
output = flashinfer.cudnn_batch_decode_with_kv_cache(
    q=queries,           # (total_num_qo_heads, head_dim)
    kv_data=kv_cache,
    qo_indptr=qo_indptr,  # Query offsets
    kv_indptr=kv_indptr,  # KV offsets
)

# Using wrapper for multi-layer inference
wrapper = flashinfer.BatchDecodeWithPagedKVCacheWrapper()
wrapper.begin_forward(...)
for layer in model.layers:
    output = wrapper.forward(query)
wrapper.end_forward()

Prefill (Prompt Processing):

# Single request prefill
output = flashinfer.single_prefill_with_kv_cache(
    q=query,              # (qo_len, num_qo_heads, head_dim)
    kv_data=kv_cache,
    causal=True,          # Causal masking
)

# Batch prefill with ragged KV-cache
wrapper = flashinfer.BatchPrefillWithRaggedKVCacheWrapper()
wrapper.begin_forward(
    qo_indptr=qo_indptr,
    kv_indptr=kv_indptr,
)
output = wrapper.forward(query, kv_cache)

Append (Speculative Decoding):

# Append new tokens to KV-cache
output = flashinfer.single_prefill_with_kv_cache(
    q=new_queries,        # (num_new_tokens, num_qo_heads, head_dim)
    kv_data=kv_cache,
    causal=True,
    append_mode=True,
)

KV-Cache Formats

1. Paged KV-Cache:

# Page-based storage (like virtual memory)
kv_cache = torch.empty(
    num_pages, 2, num_kv_heads, page_size, head_dim,
    dtype=torch.float16, device='cuda'
)

# Page table maps sequences to pages
page_indices = torch.tensor([[0, 1, 2], [3, 4, 5]], device='cuda')

2. Ragged Tensor:

# Variable-length sequences without padding
kv_data = torch.cat([seq1_kv, seq2_kv, seq3_kv], dim=0)
kv_indptr = torch.tensor([0, len(seq1), len(seq1)+len(seq2), ...])

3. Padded Tensor:

# Standard dense format
kv_cache = torch.zeros(
    batch_size, max_seq_len, 2, num_kv_heads, head_dim
)

GEMM & MoE Operations

FP8 Matrix Multiplication:

# FP8 GEMM with groupwise scaling
output = flashinfer.gemm_fp8_nt_groupwise(
    x=input_fp8,           # FP8 input
    w=weight_fp8,          # FP8 weight
    x_scale=input_scale,   # Per-group scales
    w_scale=weight_scale,
    group_size=128,
)

FP4 Quantized GEMM:

# FP4 matrix multiplication
output = flashinfer.mm_fp4(
    x=input,
    w_q=weight_fp4,        # Quantized to FP4
    scales=scales,
    group_size=64,
)

Mixture of Experts:

# Fused MoE with FP8 quantization
output = flashinfer.trtllm_fp8_block_scale_moe(
    x=hidden_states,       # (num_tokens, hidden_size)
    w1=expert_weights_w1,  # FP8 weights
    w2=expert_weights_w2,
    topk_weights=routing_weights,  # (num_tokens, topk)
    topk_ids=expert_ids,           # (num_tokens, topk)
    scales_w1=scales_w1,
    scales_w2=scales_w2,
)

Sampling Operations

Top-K and Top-P Sampling:

# Top-K sampling
next_token = flashinfer.top_k_sampling_from_probs(
    probs=probs,          # (batch_size, vocab_size)
    top_k=40,
    uniform_samples=torch.rand(batch_size, device='cuda'),
)

# Top-P (nucleus) sampling
next_token = flashinfer.top_p_sampling_from_probs(
    probs=probs,
    top_p=0.9,
    uniform_samples=torch.rand(batch_size, device='cuda'),
)

# Combined Top-K and Top-P
next_token = flashinfer.top_k_top_p_sampling_from_logits(
    logits=logits,
    top_k=40,
    top_p=0.9,
    uniform_samples=torch.rand(batch_size, device='cuda'),
)

Speculative Decoding:

# Chain speculative sampling
accepted_tokens = flashinfer.chain_speculative_sampling(
    draft_probs=draft_model_probs,    # (batch_size, num_draft_tokens, vocab)
    draft_tokens=draft_tokens,          # (batch_size, num_draft_tokens)
    target_probs=target_model_probs,   # (batch_size, num_draft_tokens+1, vocab)
    uniform_samples=torch.rand(batch_size, num_draft_tokens+1),
)

Advanced Features

Cascade Inference

Optimize shared prefix scenarios (e.g., document QA):

# Shared prefix attention (stored in SMEM)
shared_output = flashinfer.single_prefill_with_kv_cache(
    q=queries,
    kv_data=shared_prefix_kv,
    # ... shared KV parameters
)

# Unique suffix attention
suffix_output = flashinfer.batch_decode_with_kv_cache(
    q=queries,
    kv_data=suffix_kv,
    # ... suffix KV parameters
)

# Merge attention states
final_output = merge_attention_states(shared_output, suffix_output)

Performance: Up to 31x speedup vs baseline PageAttention

Multi-Head Latent Attention (MLA)

For DeepSeek models:

# MLA-specific wrapper
wrapper = flashinfer.BatchMLAPagedAttentionWrapper()
wrapper.begin_forward(...)

# MLA decode
output = flashinfer.trtllm_batch_decode_with_kv_cache_mla(
    q=latent_query,
    kv_data=latent_kv_cache,
    # ... MLA-specific parameters
)

RoPE (Rotary Position Embeddings)

# Apply RoPE in-place
flashinfer.apply_rope_inplace(
    q=query,              # (seq_len, num_heads, head_dim)
    k=key,
    indptr=indptr,
    offsets=position_offsets,
    rotary_dim=head_dim,
)

# Apply RoPE with position IDs
flashinfer.apply_rope_pos_ids(
    q=query,
    k=key,
    pos_ids=position_ids,
    rotary_dim=head_dim,
)

Normalization

# RMSNorm
output = flashinfer.rmsnorm(
    input=hidden_states,
    weight=norm_weight,
    eps=1e-6,
)

# Fused Add + RMSNorm
output = flashinfer.fused_add_rmsnorm(
    input=hidden_states,
    residual=residual,
    weight=norm_weight,
    eps=1e-6,
)

Performance Characteristics

Benchmarks (vs Compiler Backends)

  • Inter-token latency: 29-69% reduction
  • Long-context inference: 28-30% latency reduction
  • Parallel generation: 13-17% speedup

Roofline Analysis

Decode Attention: O(1) operational intensity (memory-bound) Prefill Attention: O(l_q) operational intensity (compute-bound for long sequences)

GPU Performance

GPUDecode TFLOPSPrefill TFLOPS
H100~200~800
A100~80~300
RTX 4090~60~250

Integration Examples

vLLM Integration

from flashinfer import BatchDecodeWithPagedKVCacheWrapper

class FlashInferAttention:
    def __init__(self, num_heads, head_dim):
        self.wrapper = BatchDecodeWithPagedKVCacheWrapper()
    
    def forward(self, query, kv_cache, kv_indptr, kv_indices):
        return self.wrapper.forward(query)

SGLang Integration

import flashinfer

# Initialize wrappers
decode_wrapper = flashinfer.BatchDecodeWithPagedKVCacheWrapper()
prefill_wrapper = flashinfer.BatchPrefillWithPagedKVCacheWrapper()

# Use in forward pass
if is_prefill:
    output = prefill_wrapper.forward(query, kv_cache)
else:
    output = decode_wrapper.forward(query)

Best Practices

  1. Use pre-compiled kernels: Install flashinfer-cubin for faster startup
  2. Choose appropriate KV-cache format: Paged for batching, ragged for variable lengths
  3. Enable cascade inference: For shared prefix scenarios
  4. Use FP8 quantization: 2x speedup with minimal accuracy loss
  5. Leverage wrappers: For multi-layer models to amortize planning overhead
  6. Profile workload: Use flashinfer.bench for performance analysis

References

  • Attention Kernels - Prefill, decode, append, KV-cache formats
  • Advanced Techniques - Cascade inference, MLA, optimizations
  • GEMM & MoE - Matrix operations, quantization, mixture of experts
  • Sampling - Top-k, top-p, speculative decoding
  • Integration - vLLM, SGLang, deployment patterns
  • Performance - Benchmarks, optimization strategies

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.