Copilot Instructions

Project Overview

This is a Logstash output plugin (logstash-output-kusto) that sends events from Logstash to Azure Data Explorer (Kusto). It is a JRuby gem that wraps the Azure Kusto Java SDK. The plugin runs on the JVM via Logstash's embedded JRuby runtime.

Architecture

The plugin has three core classes, all nested under LogStash::Outputs::Kusto:

• Kusto (kusto.rb) — Main output plugin. Extends LogStash::Outputs::Base. Handles Logstash config registration, file I/O (writing events to temp files with time-based rotation), and lifecycle (register, multi_receive_encoded, close). Uses concurrency :shared.
• Ingestor (kusto/ingestor.rb) — Manages authentication (AAD app credentials, managed identity, or CLI auth) and async upload to Kusto via the Java SDK. Runs uploads on a Concurrent::ThreadPoolExecutor. Retries indefinitely on transient upload failures.
• Interval (kusto/interval.rb) — Simple timer utility that runs a callable at a fixed interval using a background thread with Mutex/ConditionVariable.
• IOWriter — Wrapper around file descriptors at the bottom of kusto.rb, tracking active/inactive state for stale file cleanup.

Data flow: Events → codec encodes to JSON lines → written to time-rotated temp files → stale files closed → Ingestor.upload_async queues file for Kusto ingestion via Java SDK → temp file deleted on success.

Build System

This project uses a dual build system — Gradle for Java dependency management and Bundler/Gem for JRuby packaging.

Vendor Java dependencies (required before tests or gem build):

./gradlew vendor

This resolves the Kusto Java SDK and all transitive dependencies, copies JARs to vendor/jar-dependencies/, and generates lib/logstash-output-kusto_jars.rb.

Install Ruby dependencies:

jruby -S bundle install

Run all unit tests:

jruby -S bundle exec rspec

Run a single test file:

jruby -S bundle exec rspec spec/outputs/kusto_spec.rb

Run a single test by line number:

jruby -S bundle exec rspec spec/outputs/kusto/ingestor_spec.rb:25

Build the gem:

jruby -S gem build logstash-output-kusto.gemspec

Full build sequence (from scratch):

jruby -S bundle install && ./gradlew vendor && jruby -S bundle exec rspec && jruby -S gem build *.gemspec

Running Locally

See run.sh for the full local workflow. It requires a local Logstash installation and the following environment variables:

export LOGSTASH_SOURCE=1
export LOGSTASH_PATH="/path/to/logstash"        # local Logstash installation root
export JRUBY_HOME=$LOGSTASH_PATH/vendor/jruby
export JAVA_HOME=$LOGSTASH_PATH/jdk
export PATH=$JRUBY_HOME/bin:$JAVA_HOME/bin:$LOGSTASH_PATH/bin:$PATH

The script runs the full pipeline: install bundler → bundle install → vendor JARs → run unit tests → build gem → install the gem into local Logstash → run e2e tests. E2e tests also require Kusto cluster connection variables (ENGINE_URL, INGEST_URL, TEST_DATABASE) and use Azure CLI auth.

Linting

RuboCop is configured (.rubocop.yml) with max line length of 120 characters.

Key Conventions

• JRuby required — All Ruby commands must use jruby -S (not ruby). The gem platform is java and the code uses Java interop extensively.
• Version source of truth — The gem version is read from the version file in the project root (not hardcoded in the gemspec).
• Java interop pattern — Java classes are accessed via Java::com.microsoft.azure.kusto.* namespace. The _jars.rb file is auto-generated by Gradle — never edit it manually.
• Logstash plugin conventions — Config parameters use config :name, validate: :type, default: value. The plugin uses config_name 'kusto' and default :codec, 'json_lines'.
• Authentication modes — Three mutually exclusive auth paths: AAD app credentials (app_id/app_key/app_tenant), managed identity (managed_identity), or Azure CLI (cli_auth — dev/test only).
• Dynamic field references — Logstash %{field} references are allowed in path (for time-based file rotation) but explicitly forbidden in database, table, and json_mapping.
• Test style — RSpec with logstash-devutils. Tests mock the logger with spy('logger'). The spec_helpers.rb captures stdout/stderr in an around filter and sets log level to debug.