flashinfer
Skill8 starsChanged 8 months agoMITView on GitHub
FlashInfer — High-performance kernel library for LLM inference with optimized attention, paged KV-cache, FP8/FP4 quantization
- Installs packages
In this repository23 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- validate-skillsSkill
- alchemi-toolkit-opsSkill
- autoresearch-setupSkill
- bionemoSkill
- biosimspaceSkill
- cuda-quantumSkill
- cutileSkill
- daskSkill
- materials-projectSkill
- mpiSkill
- nemotronSkill
- omniverse-simreadySkill
- physicsnemoSkill
- polarsSkill
- prime-verifiersSkill
- qiskitSkill
- raySkill
- rdkitSkill
- slurmSkill
- tinkerSkill
- tritonSkill
The file
We split this skill into the rules it gives an agent
1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 383
Profile workload: Use flashinfer.bench for performance analysis
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
FlashInfer Skill
High-performance GPU kernel library for Large Language Model inference delivering state-of-the-art performance across diverse GPU architectures with optimized attention, GEMM, and MoE operations.
Official Sources:
- FlashInfer Documentation
- GitHub Repository
- Official Blog
- MLSys 2025 Paper (Best Paper Award)
What is FlashInfer?
Definition:
"A library and kernel generator for Large Language Models that provides high-performance implementation of LLM GPU kernels such as FlashAttention, PageAttention and LoRA."
Key Features:
- Unified APIs: Attention, GEMM, MoE with multiple backend implementations
- Paged & Ragged KV-Cache: Efficient memory management for dynamic batching
- Multi-Backend: FlashAttention-2/3, cuDNN, CUTLASS, TensorRT-LLM
- Quantization: FP8 and FP4 for attention, GEMM, MoE operations
- Production-Ready: CUDAGraph and torch.compile compatible
- Wide GPU Support: SM75 (Turing) through SM121 (Blackwell)
Quick Start
Installation
# Basic installation
pip install flashinfer-python
# With pre-compiled kernels (recommended)
pip install flashinfer-python flashinfer-cubin
# With JIT cache for specific CUDA version
pip install flashinfer-jit-cache --index-url https://flashinfer.ai/whl/cu129System Requirements:
- Linux only
- Python 3.10-3.14
- CUDA 12.6, 12.8, 13.0, or 13.1
- GPU: Turing (T4) through Blackwell
Verify Installation
flashinfer show-configBasic Usage
import torch
import flashinfer
# Single decode with paged KV-cache
output = flashinfer.single_decode_with_kv_cache(
q=query, # (num_qo_heads, head_dim)
kv_data=kv_cache, # (num_pages, 2, num_kv_heads, page_size, head_dim)
kv_indices=kv_page_indices, # (num_pages,)
kv_indptr=kv_page_indptr, # (batch_size + 1,)
kv_last_page_len=last_page_lengths, # (batch_size,)
)Core Capabilities
Attention Operations
Decode (Token-by-Token Generation):
# Single request decode
output = flashinfer.single_decode_with_kv_cache(
q=query,
kv_data=kv_cache,
# ... KV-cache parameters
)
# Batch decode with cuDNN backend
output = flashinfer.cudnn_batch_decode_with_kv_cache(
q=queries, # (total_num_qo_heads, head_dim)
kv_data=kv_cache,
qo_indptr=qo_indptr, # Query offsets
kv_indptr=kv_indptr, # KV offsets
)
# Using wrapper for multi-layer inference
wrapper = flashinfer.BatchDecodeWithPagedKVCacheWrapper()
wrapper.begin_forward(...)
for layer in model.layers:
output = wrapper.forward(query)
wrapper.end_forward()Prefill (Prompt Processing):
# Single request prefill
output = flashinfer.single_prefill_with_kv_cache(
q=query, # (qo_len, num_qo_heads, head_dim)
kv_data=kv_cache,
causal=True, # Causal masking
)
# Batch prefill with ragged KV-cache
wrapper = flashinfer.BatchPrefillWithRaggedKVCacheWrapper()
wrapper.begin_forward(
qo_indptr=qo_indptr,
kv_indptr=kv_indptr,
)
output = wrapper.forward(query, kv_cache)Append (Speculative Decoding):
# Append new tokens to KV-cache
output = flashinfer.single_prefill_with_kv_cache(
q=new_queries, # (num_new_tokens, num_qo_heads, head_dim)
kv_data=kv_cache,
causal=True,
append_mode=True,
)KV-Cache Formats
1. Paged KV-Cache:
# Page-based storage (like virtual memory)
kv_cache = torch.empty(
num_pages, 2, num_kv_heads, page_size, head_dim,
dtype=torch.float16, device='cuda'
)
# Page table maps sequences to pages
page_indices = torch.tensor([[0, 1, 2], [3, 4, 5]], device='cuda')2. Ragged Tensor:
# Variable-length sequences without padding
kv_data = torch.cat([seq1_kv, seq2_kv, seq3_kv], dim=0)
kv_indptr = torch.tensor([0, len(seq1), len(seq1)+len(seq2), ...])3. Padded Tensor:
# Standard dense format
kv_cache = torch.zeros(
batch_size, max_seq_len, 2, num_kv_heads, head_dim
)GEMM & MoE Operations
FP8 Matrix Multiplication:
# FP8 GEMM with groupwise scaling
output = flashinfer.gemm_fp8_nt_groupwise(
x=input_fp8, # FP8 input
w=weight_fp8, # FP8 weight
x_scale=input_scale, # Per-group scales
w_scale=weight_scale,
group_size=128,
)FP4 Quantized GEMM:
# FP4 matrix multiplication
output = flashinfer.mm_fp4(
x=input,
w_q=weight_fp4, # Quantized to FP4
scales=scales,
group_size=64,
)Mixture of Experts:
# Fused MoE with FP8 quantization
output = flashinfer.trtllm_fp8_block_scale_moe(
x=hidden_states, # (num_tokens, hidden_size)
w1=expert_weights_w1, # FP8 weights
w2=expert_weights_w2,
topk_weights=routing_weights, # (num_tokens, topk)
topk_ids=expert_ids, # (num_tokens, topk)
scales_w1=scales_w1,
scales_w2=scales_w2,
)Sampling Operations
Top-K and Top-P Sampling:
# Top-K sampling
next_token = flashinfer.top_k_sampling_from_probs(
probs=probs, # (batch_size, vocab_size)
top_k=40,
uniform_samples=torch.rand(batch_size, device='cuda'),
)
# Top-P (nucleus) sampling
next_token = flashinfer.top_p_sampling_from_probs(
probs=probs,
top_p=0.9,
uniform_samples=torch.rand(batch_size, device='cuda'),
)
# Combined Top-K and Top-P
next_token = flashinfer.top_k_top_p_sampling_from_logits(
logits=logits,
top_k=40,
top_p=0.9,
uniform_samples=torch.rand(batch_size, device='cuda'),
)Speculative Decoding:
# Chain speculative sampling
accepted_tokens = flashinfer.chain_speculative_sampling(
draft_probs=draft_model_probs, # (batch_size, num_draft_tokens, vocab)
draft_tokens=draft_tokens, # (batch_size, num_draft_tokens)
target_probs=target_model_probs, # (batch_size, num_draft_tokens+1, vocab)
uniform_samples=torch.rand(batch_size, num_draft_tokens+1),
)Advanced Features
Cascade Inference
Optimize shared prefix scenarios (e.g., document QA):
# Shared prefix attention (stored in SMEM)
shared_output = flashinfer.single_prefill_with_kv_cache(
q=queries,
kv_data=shared_prefix_kv,
# ... shared KV parameters
)
# Unique suffix attention
suffix_output = flashinfer.batch_decode_with_kv_cache(
q=queries,
kv_data=suffix_kv,
# ... suffix KV parameters
)
# Merge attention states
final_output = merge_attention_states(shared_output, suffix_output)Performance: Up to 31x speedup vs baseline PageAttention
Multi-Head Latent Attention (MLA)
For DeepSeek models:
# MLA-specific wrapper
wrapper = flashinfer.BatchMLAPagedAttentionWrapper()
wrapper.begin_forward(...)
# MLA decode
output = flashinfer.trtllm_batch_decode_with_kv_cache_mla(
q=latent_query,
kv_data=latent_kv_cache,
# ... MLA-specific parameters
)RoPE (Rotary Position Embeddings)
# Apply RoPE in-place
flashinfer.apply_rope_inplace(
q=query, # (seq_len, num_heads, head_dim)
k=key,
indptr=indptr,
offsets=position_offsets,
rotary_dim=head_dim,
)
# Apply RoPE with position IDs
flashinfer.apply_rope_pos_ids(
q=query,
k=key,
pos_ids=position_ids,
rotary_dim=head_dim,
)Normalization
# RMSNorm
output = flashinfer.rmsnorm(
input=hidden_states,
weight=norm_weight,
eps=1e-6,
)
# Fused Add + RMSNorm
output = flashinfer.fused_add_rmsnorm(
input=hidden_states,
residual=residual,
weight=norm_weight,
eps=1e-6,
)Performance Characteristics
Benchmarks (vs Compiler Backends)
- Inter-token latency: 29-69% reduction
- Long-context inference: 28-30% latency reduction
- Parallel generation: 13-17% speedup
Roofline Analysis
Decode Attention: O(1) operational intensity (memory-bound) Prefill Attention: O(l_q) operational intensity (compute-bound for long sequences)
GPU Performance
| GPU | Decode TFLOPS | Prefill TFLOPS |
| H100 | ~200 | ~800 |
| A100 | ~80 | ~300 |
| RTX 4090 | ~60 | ~250 |
Integration Examples
vLLM Integration
from flashinfer import BatchDecodeWithPagedKVCacheWrapper
class FlashInferAttention:
def __init__(self, num_heads, head_dim):
self.wrapper = BatchDecodeWithPagedKVCacheWrapper()
def forward(self, query, kv_cache, kv_indptr, kv_indices):
return self.wrapper.forward(query)SGLang Integration
import flashinfer
# Initialize wrappers
decode_wrapper = flashinfer.BatchDecodeWithPagedKVCacheWrapper()
prefill_wrapper = flashinfer.BatchPrefillWithPagedKVCacheWrapper()
# Use in forward pass
if is_prefill:
output = prefill_wrapper.forward(query, kv_cache)
else:
output = decode_wrapper.forward(query)Best Practices
- Use pre-compiled kernels: Install
flashinfer-cubinfor faster startup - Choose appropriate KV-cache format: Paged for batching, ragged for variable lengths
- Enable cascade inference: For shared prefix scenarios
- Use FP8 quantization: 2x speedup with minimal accuracy loss
- Leverage wrappers: For multi-layer models to amortize planning overhead
- Profile workload: Use
flashinfer.benchfor performance analysis
References
- Attention Kernels - Prefill, decode, append, KV-cache formats
- Advanced Techniques - Cascade inference, MLA, optimizations
- GEMM & MoE - Matrix operations, quantization, mixture of experts
- Sampling - Top-k, top-p, speculative decoding
- Integration - vLLM, SGLang, deployment patterns
- Performance - Benchmarks, optimization strategies
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

