agentleFS
anam-organam-org/metaxy/.claude/skills/narwhals/SKILL.md

narwhals

Skill126 starsChanged 9 months agoApache-2.0View on GitHub

Effectively use Narwhals to write dataframe-agnostic code that works seamlessly across multiple Python dataframe libraries. Write correct type annotations for code using Narwhals.

The file

We split this skill into the rules it gives an agent

8 rules. About 0 are common, 7 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈7

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Backend Agnostic: Write once, run on any supported dataframe library

    line 49
  2. Distinctive:

    All DataFrame methods are available on LazyFrame:

    line 196
  3. Distinctive:

    Use this if your function can accept a Narwhals DataFrame and returns a Narwhals DataFrame backed by the same backend.

    line 315
  4. Distinctive:

    Use this if your function can work with either and your function doesn't care about its backend.

    line 331
  5. Distinctive:

    Use this if your function accepts either nw.DataFrame or nw.LazyFrame and returns an object of the same kind.

    line 349
  6. Distinctive:

    Use this if your function accepts a narwhalifiable object but doesn't care about its backend.

    line 366
  7. Distinctive:

    Use this if your function accepts an object which can be converted to nw.DataFrame and returns an object of the same class.

    line 385
  8. Very specific:

    Type hint your functions: Use IntoDataFrame and FrameT for better IDE support

    line 458

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Narwhals - DataFrame Agnostic API

Narwhals is a lightweight, zero-dependency compatibility layer for dataframe libraries in Python that provides a unified interface across different backends.

Docs: https://narwhals-dev.github.io/narwhals/

What is Narwhals?

Narwhals enables writing dataframe-agnostic code that works seamlessly across multiple Python dataframe libraries:

Full API Support:

  • cuDF
  • Modin
  • pandas
  • Polars
  • PyArrow

Lazy-Only Support:

  • Dask
  • DuckDB
  • Ibis
  • PySpark
  • SQLFrame

Core Philosophy

Why Narwhals?

  • Resolves subtle differences between libraries (e.g., pandas checking index vs Polars checking values)
  • Provides unified, simple, and predictable API
  • Handles backwards compatibility internally
  • Tests against nightly builds of supported libraries
  • Maintains negligible performance overhead
  • Full static typing support
  • Zero dependencies

Target Use Case: Anyone building libraries, applications, or services that consume dataframes and need complete backend independence.

Key Features

  1. Backend Agnostic: Write once, run on any supported dataframe library
  2. Polars-Like API: Uses a subset of the Polars API for consistency
  3. Lazy & Eager Execution: Separate APIs for both execution modes
  4. Expression Support: Full expression API for complex operations
  5. Type Safety: Perfect static typing support
  6. 100% Branch Coverage: Thoroughly tested

Basic Usage Pattern

Three-Step Workflow

import narwhals as nw

# 1. Convert to Narwhals
df_nw = nw.from_native(df)  # Works with pandas, Polars, PyArrow, etc.

# 2. Perform operations using Polars-like API
result = df_nw.select(a_sum=nw.col("a").sum(), a_mean=nw.col("a").mean(), b_std=nw.col("b").std())

# 3. Convert back to original library
result_native = result.to_native()

Using the @narwhalify Decorator

Simplifies function definitions for automatic conversion:

@nw.narwhalify
def my_func(df: IntoDataFrameT):
    return df.select(nw.col("a").sum(), nw.col("b").mean()).filter(nw.col("a") > 0)


# Automatically handles conversion to/from Narwhals
result = my_func(pandas_df)  # Works!
result = my_func(polars_df)  # Also works!

Top-Level Functions

Conversion Functions

  • from_native(df, ...): Convert native DataFrame/Series to Narwhals object
    • Parameters: pass_through, backend, eager_only, allow_series
  • to_native(nw_obj): Convert Narwhals object back to native library type
  • narwhalify(): Decorator for automatic dataframe-agnostic functions

Data Creation

  • new_series(name, values, dtype): Create a new Series
  • from_dict(data): Create DataFrame from dictionary
  • from_dicts(data): Create DataFrame from sequence of dictionaries

File I/O

Eager Loading:

  • read_csv(source, **kwargs): Read CSV file into DataFrame
  • read_parquet(source, **kwargs): Read Parquet file into DataFrame

Lazy Loading:

  • scan_csv(source, **kwargs): Lazily scan CSV file
  • scan_parquet(source, **kwargs): Lazily scan Parquet file

Aggregation Functions

  • sum(), mean(), min(), max(), median()
  • sum_horizontal(), mean_horizontal(), etc.

Expression Creation

  • col(name): Reference column by name
  • lit(value): Create literal expression
  • when(condition): Create conditional expression
  • format(template, *args): Format expression as string

Utilities

  • generate_temporary_column_name(): Generate unique column names
  • get_native_namespace(obj): Get the native library of an object
  • show_versions(): Print debugging information

DataFrame Methods

Properties

  • columns: List of column names
  • schema: Ordered mapping of column names to dtypes
  • shape: Tuple of (rows, columns)
  • implementation: Name of native implementation

Column Operations

  • select(*exprs): Select columns using expressions
  • with_columns(*exprs): Add or modify columns
  • drop(*columns): Remove specified columns
  • rename(mapping): Rename columns

Row Operations

  • filter(predicate): Filter rows based on conditions
  • head(n): Get first n rows
  • tail(n): Get last n rows
  • sample(n): Randomly sample n rows
  • drop_nulls(): Drop rows with null values
  • unique(): Remove duplicate rows

Inspection

  • is_empty(): Check if DataFrame has no rows
  • is_duplicated(): Identify duplicated rows
  • is_unique(): Identify unique rows
  • null_count(): Count null values per column
  • estimated_size(): Estimate memory usage

Transformations

  • sort(*by): Sort by one or more columns
  • group_by(*by): Group by columns for aggregation
  • join(other, on, how): Perform SQL-style joins
  • pivot(on, index, values): Create pivot table
  • explode(*columns): Expand list columns to long format
  • lazy(): Convert to LazyFrame

Export

  • to_native(): Convert to original library type
  • to_numpy(): Convert to NumPy array
  • to_pandas(): Convert to pandas DataFrame
  • to_polars(): Convert to Polars DataFrame
  • clone(): Create a copy

LazyFrame Methods

LazyFrame provides the same API as DataFrame but with lazy evaluation:

Key Differences

  • Operations build an execution plan without computing
  • collect(): Materialize the LazyFrame into a DataFrame
  • collect_schema(): Get schema without collecting data
  • sink_parquet(path): Write results directly to Parquet

Common Methods

All DataFrame methods are available on LazyFrame:

  • select(), filter(), with_columns(), drop()
  • group_by(), join(), sort(), unique()
  • head(), tail(), top_k()
  • gather_every(): Select rows at regular intervals
  • unpivot(): Convert from wide to long format
  • with_row_index(): Add row index column
  • pipe(): Apply function to LazyFrame

Expression (Expr) API

Expressions are the building blocks for column operations.

Creation

nw.col("column_name")  # Reference column
nw.lit(42)  # Literal value

Filtering

  • filter(predicate): Filter elements
  • is_in(values): Check membership
  • is_between(lower, upper): Check range
  • drop_nulls(): Remove nulls

Aggregations

  • count(): Count non-null elements
  • null_count(): Count null values
  • n_unique(): Count unique values
  • sum(), mean(), median(): Statistical aggregations
  • min(), max(): Extremes
  • std(), var(): Spread measures
  • quantile(q): Quantile values

Transformations

Mathematical:

  • abs(): Absolute value
  • round(), floor(), ceil(): Rounding
  • sqrt(), log(), exp(): Mathematical functions

Type/Value Operations:

  • cast(dtype): Change data type
  • fill_null(value): Replace null values
  • replace_strict(old, new): Replace specific values

Window Operations:

  • rolling_mean(window_size): Moving average
  • rolling_sum(window_size): Moving sum
  • rolling_std(window_size): Moving standard deviation
  • shift(n): Shift values by n positions
  • over(*by): Compute expression over groups

Ranking/Uniqueness:

  • rank(): Assign ranks
  • unique(): Get unique values
  • is_duplicated(): Identify duplicates
  • is_first_distinct(): Mark first distinct occurrences

Namespace Methods

Expressions have specialized namespaces for specific data types:

String Operations (Expr.str)

  • String manipulation methods

DateTime Operations (Expr.dt)

  • Date/time manipulation methods

List Operations (Expr.list)

  • List column operations

Categorical Operations (Expr.cat)

  • Categorical data methods

Struct Operations (Expr.struct)

  • Struct/nested data methods

Name Operations (Expr.name)

  • Column name operations

Series API

Series represents a single column:

Properties

  • Same as DataFrame: shape, dtype, name

Methods

  • Similar to DataFrame but for single column operations
  • Has specialized namespaces: str, dt, list, cat, struct

Type Hints

Full docs: narwhals.typing

TLDR:

DataFrameT module-attribute

DataFrameT = TypeVar('DataFrameT', bound='DataFrame[Any]') TypeVar bound to Narwhals DataFrame.

Use this if your function can accept a Narwhals DataFrame and returns a Narwhals DataFrame backed by the same backend.

Examples:

>>> import narwhals as nw
>>> from narwhals.typing import DataFrameT
>>> @nw.narwhalify
>>> def func(df: DataFrameT) -> DataFrameT:
...     return df.with_columns(c=df["a"] + 1)
Frame module-attribute

Frame: TypeAlias = Union["DataFrame[Any]", "LazyFrame[Any]"] Narwhals DataFrame or Narwhals LazyFrame.

Use this if your function can work with either and your function doesn't care about its backend.

Examples:

>>> import narwhals as nw
>>> from narwhals.typing import Frame
>>> @nw.narwhalify
... def agnostic_columns(df: Frame) -> list[str]:
...     return df.columns
FrameT module-attribute

FrameT = TypeVar( "FrameT", "DataFrame[Any]", "LazyFrame[Any]" ) TypeVar bound to Narwhals DataFrame or Narwhals LazyFrame.

Use this if your function accepts either nw.DataFrame or nw.LazyFrame and returns an object of the same kind.

Examples:

>>> import narwhals as nw
>>> from narwhals.typing import FrameT
>>> @nw.narwhalify
... def agnostic_func(df: FrameT) -> FrameT:
...     return df.with_columns(c=nw.col("a") + 1)

IntoDataFrame module-attribute

IntoDataFrame: TypeAlias = NativeDataFrame Anything which can be converted to a Narwhals DataFrame.

Use this if your function accepts a narwhalifiable object but doesn't care about its backend.

Examples:

>>> import narwhals as nw
>>> from narwhals.typing import IntoDataFrame
>>> def agnostic_shape(df_native: IntoDataFrame) -> tuple[int, int]:
...     df = nw.from_native(df_native, eager_only=True)
...     return df.shape

IntoDataFrameT module-attribute

IntoDataFrameT = TypeVar( "IntoDataFrameT", bound=IntoDataFrame ) TypeVar bound to object convertible to Narwhals DataFrame.

Use this if your function accepts an object which can be converted to nw.DataFrame and returns an object of the same class.

Examples:

>>> import narwhals as nw
>>> from narwhals.typing import IntoDataFrameT
>>> def agnostic_func(df_native: IntoDataFrameT) -> IntoDataFrameT:
...     df = nw.from_native(df_native, eager_only=True)
...     return df.with_columns(c=df["a"] + 1).to_native()

Common Patterns

Group By and Aggregate

result = df.group_by("category").agg(
    count=nw.col("id").count(),
    total=nw.col("amount").sum(),
    average=nw.col("amount").mean(),
)

Conditional Operations

result = df.with_columns(category=nw.when(nw.col("value") > 100).then(nw.lit("high")).otherwise(nw.lit("low")))

Joins

result = df1.join(
    df2,
    on="key_column",
    how="left",  # inner, left, outer, cross
)

Chain Operations

result = (
    df.filter(nw.col("status") == "active")
    .select("user_id", "amount")
    .group_by("user_id")
    .agg(total=nw.col("amount").sum())
    .sort("total", descending=True)
    .head(10)
)

Best Practices

  1. Use @nw.narwhalify for library functions: Simplifies API and handles conversion automatically
  2. Prefer expressions over method chaining: More flexible and composable
    # Good
    df.select(nw.col("a").sum(), nw.col("b").mean())
    
    # Also fine, but less composable
    df.select("a", "b")
  3. Use lazy evaluation when possible: Better performance for complex pipelines
    result = df.lazy().select(...).filter(...).collect()
  4. Always convert back to native: Remember to call .to_native() when returning from library functions (unless using @narwhalify)
  5. Type hint your functions: Use IntoDataFrame and FrameT for better IDE support
  6. Check supported backends: Some operations may not be available on all backends

Important Constraints

  1. Zero Dependencies: Narwhals has no dependencies, keeping it lightweight
  2. Polars API Subset: Uses Polars-style API but may not support all Polars features
  3. Backend Limitations: Some backends (lazy-only) have restricted functionality

Tips

Checking whether a Narwhals frame is a Polars frame:

import polars as pl
import narwhals as nw

df_native = pl.DataFrame({"a": [1, 2, 3]})
df = nw.from_native(df_native)
df.implementation.is_polars()

Asserting series equality:

import pandas as pd
import narwhals as nw
from narwhals.testing import assert_series_equal
s1 = nw.from_native(pd.Series([1, 2, 3]), series_only=True)
s2 = nw.from_native(pd.Series([1, 5, 3]), series_only=True)
assert_series_equal(s1, s2)
Traceback (most recent call last):
...
AssertionError: Series are different (exact value mismatch)
[left]:
┌───────────────┐
|Narwhals Series|
|---------------|
| 0    1        |
| 1    2        |
| 2    3        |
| dtype: int64  |
└───────────────┘
[right]:
┌───────────────┐
|Narwhals Series|
|---------------|
| 0    1        |
| 1    5        |
| 2    3        |
| dtype: int64  |
└───────────────┘

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.