Python Profiler Skill

Overview

Comprehensive Python profiling skill that integrates multiple profiling tools to identify performance bottlenecks, memory leaks, and optimization opportunities in Python applications.

Capabilities

1. CPU Profiling

• cProfile: Built-in profiler for function-level analysis
• Pyinstrument: Visual call stack with flame graphs
• py-spy: Low-overhead sampling profiler for production
• line_profiler: Line-by-line performance analysis

2. Memory Profiling

• memory_profiler: Track memory usage line-by-line
• Scalene: Combined CPU + memory profiling
• Pyroscope: Continuous real-time profiling

3. Concurrency Profiling

• Yappi: Profile multithreading and greenlets/coroutines
• Thread-safe profiling for asyncio applications

Usage

Basic CPU Profiling

# Using cProfile
python -m cProfile -o output.prof your_script.py

# Using Pyinstrument
pyinstrument --html your_script.py

Memory Analysis

# Using memory_profiler
from memory_profiler import profile

@profile
def my_function():
    # Your code here
    pass

Production Profiling

# Using py-spy (no code changes needed)
py-spy record -o profile.svg --pid <process_id>

Integration Scripts

detect_bottleneck.py

Automatically runs multiple profilers and generates comprehensive reports:

#!/usr/bin/env python3
import cProfile
import pstats
from pyinstrument import Profiler

def profile_function(func):
    """Profile a function with multiple tools"""
    # cProfile analysis
    profiler = cProfile.Profile()
    profiler.enable()
    result = func()
    profiler.disable()

    stats = pstats.Stats(profiler)
    stats.sort_stats('cumulative')
    stats.print_stats(20)

    return result

Best Practices

1. Start with cProfile for high-level overview
2. Use Pyinstrument to visualize call hierarchies
3. Apply line_profiler to specific bottleneck functions
4. Monitor memory with memory_profiler for large datasets
5. Use py-spy in production (no overhead, no code changes)

Output Interpretation

• cumtime: Total time in function + subcalls
• tottime: Time in function excluding subcalls
• ncalls: Number of times function was called
• percall: Average time per call

Requirements

pip install cProfile pyinstrument memory-profiler py-spy line-profiler scalene yappi

Example Workflow

1. Run cProfile to identify slow functions
2. Visualize with Pyinstrument flame graph
3. Deep-dive with line_profiler on specific functions
4. Check memory usage with memory_profiler
5. Optimize and re-profile to confirm improvements

Common Bottlenecks to Look For

• Nested loops with high iteration counts
• Inefficient list comprehensions
• Repeated string concatenations
• Unoptimized pandas operations
• Missing database query optimizations (N+1 queries)
• Synchronous I/O in async code