agentleFS
jskhermanjskherman/engg-skills/skills/censored-regression/SKILL.md

censored-regression

Skill7 starsChanged 6 months agoApache-2.0View on GitHub

Fit censored-normal or censored-lognormal regression models for lab data reported as below-LOD, below-LOQ, above-range, or interval-censored. Use when fitting regressions on sulfur species, trace contaminants, or any analyte that is frequently censored. Don't use for ordinary uncensored regression (use engineering-statistics), time-to-event survival analysis, or compositional response data (use compositional-data-analysis).

The file

We split this skill into the rules it gives an agent

5 rules. About 0 are common, 4 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈4

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Use bootstrap/profile-likelihood checks externally when inference quality matters.

    line 42
  2. Distinctive:

    Run the fit. Inspect coefficients, convergence status, optimizer message, and fraction censored.

    line 97
  3. Distinctive:

    If the row order is a time series, use block bootstrap or a dynamic model for uncertainty rather than i.i.d.

    line 120
  4. Very specific:

    Run the listed script with representative inputs and an --output file when a deterministic calculation is available.

    line 125
  5. Distinctive:

    Check result magnitudes against the stated assumptions, references, and a hand calculation or known operating range before reporting them.

    line 127

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Censored Regression (Censored Normal / Censored Lognormal)

Overview

Many process lab measurements (H2S, COS, mercaptans, DMS, disulfides, unknown sulfur, trace metals) are reported as below-LOD, below-LOQ, above an instrument range, or within an interval. Ordinary least squares on log(S + epsilon) is biased and understates uncertainty; the correct treatment is a likelihood-based censored regression.

This skill provides:

  • A censored-normal regression on the log scale by default (censored lognormal on the original response scale).
  • A censored-normal regression on the raw scale with --no-log.
  • Left, right, and interval censoring.
  • Maximum-likelihood fitting through SciPy.

The implementation uses a hand-rolled SciPy likelihood. It does not calculate standard errors, robust covariance estimates, or the naive substitution fit. Use bootstrap/profile-likelihood checks externally when inference quality matters.

Prerequisites

  1. uv available.
  2. The script declares numpy, scipy, and pandas in its PEP-723 header.
  3. On first use, writes LICENSE_NOTIFICATION.txt.

When to Use

  • Lab data has below-LOQ rows that you cannot drop without introducing selection bias.
  • Reported values include intervals, for example between LOD and LOQ.
  • Fitting a regression of log(species) on operating variables, where the species is sometimes censored.

Don't use for

  • Fully uncensored regression: use engineering-statistics.
  • Time-to-event survival analysis: use a survival-analysis package.
  • Compositional response data (sulfur speciation fractions): combine with compositional-data-analysis.

Utility Scripts

  • uv run scripts/censored.py fit --data data.csv --response S_total --predictors temperature,c5_c6plus_balance --lower-col LOQ --output /tmp/fit.json
  • uv run scripts/censored.py interval --data data.csv --response S_total --predictors temperature --lower-col LOD --upper-col LOQ --output /tmp/int.json

Input CSV layout:

  • One row per observation.
  • response column with the measured value when fully observed; NaN or blank when censored.
  • lower-col and/or upper-col columns with the censoring bound for censored rows.
  • Predictor columns referenced by name in --predictors.

Censoring-column semantics used by the script:

  • finite response: exact observation; bounds ignored.
  • missing response + only lower-col finite: left-censored, meaning y <= lower_col (typical below-LOQ row).
  • missing response + only upper-col finite: right-censored, meaning y >= upper_col (above-range row).
  • missing response + both bounds finite: interval-censored, meaning lower_col < y < upper_col.

Procedure

  1. Build a tidy CSV with response and predictor columns and the censoring bound columns.
  2. Decide the censoring direction:
    • left-censored at LOQ for trace sulfur below quantitation;
    • right-censored for above-range measurements;
    • interval-censored when LOD and LOQ are both reported.
  3. Run the fit. Inspect coefficients, convergence status, optimizer message, and fraction censored.
  4. For downstream decisions, use a bootstrap/profile-likelihood workflow rather than treating the point estimate as final when the censoring fraction is high.

Pitfalls

  • Using log(S + 0.5 * LOQ) and reporting normal CIs; this substitution is biased and the CIs are wrong.
  • Using a censored fit and then reporting the response as if it were not censored; predictions are for the latent uncensored response.
  • Treating "non-detect" as zero. Zero is usually a measurement convention, not the physical concentration.
  • Treating LOD and LOQ as the same. Use interval censoring if both are reported and meaningful.
  • Reporting one fit with a single censoring bound when the LOQ changed during the data window.
  • Fitting on the raw scale when the data are heavy-tailed.

Fallback Strategies

  • For extreme censoring fractions (> 80%), report only the censoring fraction and rank/order conclusions; full regression coefficients are unstable.
  • If the row order is a time series, use block bootstrap or a dynamic model for uncertainty rather than i.i.d. inference.

Verification

  • Run the listed script with representative inputs and an --output file when a deterministic calculation is available.
  • Confirm the JSON result contains ok: true, expected units, and no unhandled warnings.
  • Check result magnitudes against the stated assumptions, references, and a hand calculation or known operating range before reporting them.

References

  • references/censored_likelihood.md — likelihood derivation.
  • Greene, Econometric Analysis, chapter on truncated/censored models.
  • Helsel, Statistics for Censored Environmental Data Using MINITAB and R.

Anti-Patterns

  • Hiding the censoring fraction in the report.
  • Reporting a censored-lognormal coefficient as if it were on the original response scale; by default, the response model is on log scale.
  • Using ordinary bootstrap on autocorrelated censored data without preserving time dependence.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.