AGENTS.md

This file provides guidance to coding agents (Claude Code, and other tools that read AGENTS.md) when working with code in this repository.

What this is

Kotlin DataFrame — a typesafe, immutable, hierarchical in-memory data-processing library for the JVM (org.jetbrains.kotlinx:dataframe). Every DataFrame operation returns a new instance, reusing underlying column storage where possible. Type-safe column access is provided through auto-generated extension properties (on-the-fly in notebooks, via the compiler plugin / KSP in projects).

Build & test commands

Gradle (Kotlin DSL), multi-module. Use the wrapper (./gradlew, or gradlew.bat on Windows). A JDK must be on JAVA_HOME/PATH to launch the wrapper; the build daemon JDK is provisioned from gradle/gradle-daemon-jvm.properties (version controlled by gradle-jdk in gradle/libs.versions.toml).

• ./gradlew build — full build; runs all tests and the ktlint check.
• ./gradlew <module>:test — test a single module (e.g. ./gradlew core:test); much faster during development.
• ./gradlew <module>:test --tests "org.jetbrains.kotlinx.dataframe.SomeTest" — run a single test class/method.
• Add -Pkotlin.dataframe.debug=true when validating changes. It enables extra, heavier correctness checks that are off in production. PR/CI builds run with this flag (build -Pkotlin.dataframe.debug=true), so match it locally.
• ./gradlew publishToMavenLocal -PskipKodex — publish locally for iteration. -PskipKodex skips apiCheck and KDoc preprocessing (see below); the result has "broken" KDocs, so it is for local dev only.
• ./gradlew dokkaGeneratePublicationHtml — build the multi-module Dokka API reference into build/dokka/html (what CI deploys under https://kotlin.github.io/dataframe/api-docs/). Takes several minutes, as it runs KoDEx first.

Module names for <module>: come from settings.gradle.kts (e.g. core, dataframe-csv, dataframe-jdbc, dataframe-json, dataframe-arrow, dataframe-excel, dataframe-jupyter).

Architecture

Dependency graph is rooted at core, with I/O and integration split into optional modules (see MODULES.md):

• core — base module, no deps. DataFrame/DataColumn/DataRow types, the operations API (core/.../api/), aggregations, math, schema handling, codegen, and the impl/ internals. Also holds deprecated csv/tsv and HTML integrations that now live in dedicated modules.
• I/O modules (each depends on core): dataframe-csv, dataframe-json, dataframe-excel, dataframe-arrow, dataframe-jdbc. The root dataframe artifact re-exports Arrow/Excel/JDBC/CSV/JSON as api. dataframe-jdbc ships built-in DbType support for H2, MariaDB, MySQL, MS SQL Server, PostgreSQL, SQLite, and DuckDB (any other JDBC database works via a user-supplied DbType); see dataframe-jdbc/AGENTS.md.
• Integrations: dataframe-jupyter, dataframe-geo → dataframe-geo-jupyter, dataframe-openapi → dataframe-openapi-generator.
• Compiler/build tooling:
  • The Kotlin DataFrame compiler plugin — the recommended way to get compile-time type-safe extension properties — is NOT in this repo. It is developed in the Kotlin repository at github.com/JetBrains/kotlin/tree/master/plugins/kotlin-dataframe. This repo's plugins/kotlin-dataframe is a disabled, out-of-date legacy copy (its tests are turned off; see issue #1290) — do not treat it as the live plugin.
  • dataframe-compiler-plugin-core — a shaded subset of :core bundled inside that compiler plugin (and IntelliJ) to run compile-time interpreters of operations; it is not itself a compiler plugin. See its AGENTS.md.
  • plugins/* is mostly dead code — see plugins/AGENTS.md before touching any of it. plugins/symbol-processor (KSP codegen for @DataSchema) and plugins/dataframe-gradle-plugin are disabled too (KSP1 is not compatible with Kotlin 2.3+), alongside plugins/kotlin-dataframe; none of the three is in settings.gradle.kts, so nothing there is compiled or tested. Still active: expressions-converter and public-api-modifier. The former keywords-generator module has moved into build-logic as the dfbuild.keywordsGenerator convention plugin.
• Build logic lives in build-logic/ and build-settings-logic/ as convention plugins (e.g. conventions.plugins.dfbuild.*), not inline in the module build files.

Most modules ship their own AGENTS.md (core, dataframe-csv, dataframe-json, dataframe-excel, dataframe-arrow, dataframe-jdbc, dataframe-geo, dataframe-compiler-plugin-core, samples, docs/StardustDocs) with module-specific details. Read the relevant module's AGENTS.md before working in it — it usually answers structure/pattern questions directly, so you can skip broad code exploration.

The public operations API is organized as one file per operation under core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/api/ (e.g. filter.kt, groupBy.kt, join.kt). When adding or changing an operation, find the sibling file for the closest existing operation and follow its structure.

Generated sources & KDoc preprocessing (KoDEx)

Public KDocs use KoDEx notations ({@include [X]}, @set/@get/$, @sample, @ExcludeFromSources, @ExportAsHtml). See KODEX_KDOC_PREPROCESSING.md and KDOC_GUIDELINES.md.

• KoDEx (an external KDoc-preprocessing tool, applied via the kodex convention plugin) expands the templated KDocs. core:processKDocsMain processes KDocs into generated-sources; the mainKodex compilation builds those processed sources into KoDEx jars (so @ExcludeFromSources is stripped from both sources and compiled classes). Publishing selects the kodexJava component via composeOf { from(kodexAwareComponentToPublish()) }, and apiBuild (apiDump/apiCheck) reads the KoDEx jar to validate the published API.
• Dokka documents the KoDEx-processed sources too. In KoDEx modules, dfbuild.kodex suppresses the main Dokka source set and enables mainKodex instead (display name copied from main), so the API reference shows the expanded KDocs. With -PskipKodex, Dokka falls back to the plain main sources. Dokka's HTML styling (logo, footer, homepage link) lives in dfbuild.dokka and docs/dokka/.
• processKDocsMain does not track its output files — it only checks that generated-sources/ is non-empty. After reverting or editing anything under */generated-sources, run ./gradlew processKDocsMain --rerun, or Dokka/the KoDEx jars are built from the stale files.
• Don't hand-edit generated sources, and don't grep/crawl the tree. Every module built with KoDEx (core, dataframe-csv, dataframe-json, dataframe-excel, dataframe-geo, dataframe-jdbc) has a tracked generated-sources/ copy, plus checked-in accessors under **/src/generated-dataschema-accessors/ and generated HTML under docs/StardustDocs/resources/. In generated-sources/ the implementation and public API are the same as the hand-written src/main/kotlin/… — KoDEx only expands the KDoc and strips the doc-only fragments (declarations marked @ExcludeFromSources). So searching the tree (**/generated-sources/**, **/generated-dataschema-accessors/**) just doubles every hit and bloats context — do code work from src/main/kotlin/…. The one reason to open a generated file is the KDoc: it is the fully-expanded documentation (all {@include …}/@set/@get resolved, URLs inlined). If you specifically need a symbol's final rendered docs, open that single generated file deliberately — don't explore. A CI bot runs processKDocsMain korro syncExampleFolders and auto-commits */generated-sources, docs/StardustDocs/resources/snippets/, docs/StardustDocs/topics/ and examples/projects — you don't run or commit those yourself. The authority for that list is the git add line in .github/workflows/generated-sources-master.yml, not prose — check it there before you assume a path is bot-maintained. Three things the bot does not produce, which the PR that needs them must carry itself: the iframe HTML under docs/StardustDocs/resources/ outside snippets/ (api/io/guides/modify), whose path is never staged; docs/StardustDocs/topics/_shadow_resources.md, whose path is staged but whose generating task (updateShadowResources) the bot never runs; and the data-schema accessors under **/src/generated-dataschema-accessors/, which can no longer be generated at all — the KSP/Gradle plugin that produced them is deprecated (plugins/symbol-processor, plugins/dataframe-gradle-plugin, see above), so they are frozen, hand-maintained sources that happen to be named "generated". Their path is not staged either; only the second copy inside */generated-sources/src/generated-dataschema-accessors/ rides along, because it sits under a staged generated-sources directory. See docs/StardustDocs/AGENTS.md.
• For KDocs: never write from scratch — reuse/@include an existing operation's KDoc and adapt it. Reusable KDoc fragments are written once as internal/private interfaces and typealias … = Nothing declarations and composed via @include; this pattern is used in every KoDEx module (the shared cross-module fragment library lives in core's documentation/ package).

Documentation samples

Documentation snippets are runnable tests. The samples module and core's samplesTest/korro tasks execute sample code, save outputs, and inject them into the WriterSide docs under docs/StardustDocs/. Migration of samples into the :samples module is in progress; new documentation samples belong there. Rules for website topics and samples are in DOCUMENTATION_GUIDELINES.md; for an operation page use the operation-website-documentation skill (.agents/skills/operation-website-documentation/SKILL.md).

Code style

• Kotlin official style, enforced by ktlint (ktlint_official, experimental rules enabled) via .editorconfig. IntelliJ Ktlint plugin recommended. In IDEA, delegate build actions to Gradle.
• 4-space indent, LF, UTF-8, 120-column max line length (see .editorconfig for exact per-glob overrides).
• No wildcard imports — star-import thresholds are set to effectively infinite.
• Class/function signatures with ≥4 parameters are forced multiline.

• ktlint_standard_chain-method-continuation = disabled — deliberate. Group operations on one line when possible so a chain reads like a sentence; only start a new line when a new operation begins:

  df
      .update { x and y }.where { true }.with { it + 1 }
      .split { z }.by(",").into("z1", "z2")
      .groupBy {
          cols(a, b, c)
      }.aggregate {
          count { d } into "count"
          mean { d } into "mean"
      }
      .print()

Constraints

• Base all PRs on master; the current checked-out branch may be a feature branch (git status at session start).
• Bug fixes require a reproducing test; new public APIs require docs + tests. A change in behaviour that users can observe also requires docs: the operation's website page and the KDoc of the public function a user calls, not only the KDoc of an internal function or an extension point.
• Public API is guarded by binary-compatibility-validator in every publishable module — when you change public signatures, run the apiDump/apiCheck Gradle tasks and commit the module's updated api/*.api dump.
• DB tests in dataframe-jdbc are structured to run against either Dockerized or locally installed DBMS (io/local/ per-database tests, io/db/ type tests, io/h2/). Do not point tests at production databases.