censored-regression
Skill7 starsChanged 6 months agoApache-2.0View on GitHub
Fit censored-normal or censored-lognormal regression models for lab data reported as below-LOD, below-LOQ, above-range, or interval-censored. Use when fitting regressions on sulfur species, trace contaminants, or any analyte that is frequently censored. Don't use for ordinary uncensored regression (use engineering-statistics), time-to-event survival analysis, or compositional response data (use compositional-data-analysis).
In this repository31 other files it gives its agents.
- absorption-stripping-designSkill
- bayesian-hierarchical-process-modelsSkill
- caustic-merox-extractionSkill
- compositional-data-analysisSkill
- control-valve-sizing-isa75Skill
- convective-heat-transfer-correlationsSkill
- design-of-experimentsSkill
- dimensionless-numbersSkill
- distillation-shortcut-designSkill
- distributed-lag-modelsSkill
- engg-skills-commonSkill
- engineering-skill-creatorSkill
- engineering-statisticsSkill
- equation-of-state-selectionSkill
- heat-exchanger-sizingSkill
- literature-search-engineeringSkill
- material-energy-balancesSkill
- pipe-flow-pressure-dropSkill
- process-causal-inference-dagsSkill
- process-units-conversionSkill
- reactor-sizing-and-kineticsSkill
- relief-valve-sizing-api520Skill
- separator-vessel-sizingSkill
- sour-gas-amine-treatingSkill
- statistical-process-controlSkill
- steam-tables-iapwsSkill
- thermo-process-propertiesSkill
- time-series-process-data-analysisSkill
- two-phase-flowSkill
- uvSkill
- vle-flash-calculationsSkill
The file
We split this skill into the rules it gives an agent
5 rules. About 0 are common, 4 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈4
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 42
Use bootstrap/profile-likelihood checks externally when inference quality matters.
- Distinctive: line 97
Run the fit. Inspect coefficients, convergence status, optimizer message, and fraction censored.
- Distinctive: line 120
If the row order is a time series, use block bootstrap or a dynamic model for uncertainty rather than i.i.d.
- Very specific: line 125
Run the listed script with representative inputs and an --output file when a deterministic calculation is available.
- Distinctive: line 127
Check result magnitudes against the stated assumptions, references, and a hand calculation or known operating range before reporting them.
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Censored Regression (Censored Normal / Censored Lognormal)
Overview
Many process lab measurements (H2S, COS, mercaptans, DMS, disulfides,
unknown sulfur, trace metals) are reported as below-LOD, below-LOQ, above an
instrument range, or within an interval. Ordinary least squares on
log(S + epsilon) is biased and understates uncertainty; the correct treatment
is a likelihood-based censored regression.
This skill provides:
- A censored-normal regression on the log scale by default (censored lognormal on the original response scale).
- A censored-normal regression on the raw scale with
--no-log. - Left, right, and interval censoring.
- Maximum-likelihood fitting through SciPy.
The implementation uses a hand-rolled SciPy likelihood. It does not calculate standard errors, robust covariance estimates, or the naive substitution fit. Use bootstrap/profile-likelihood checks externally when inference quality matters.
Prerequisites
uvavailable.- The script declares
numpy,scipy, andpandasin its PEP-723 header. - On first use, writes
LICENSE_NOTIFICATION.txt.
When to Use
- Lab data has below-LOQ rows that you cannot drop without introducing selection bias.
- Reported values include intervals, for example between LOD and LOQ.
- Fitting a regression of
log(species)on operating variables, where the species is sometimes censored.
Don't use for
- Fully uncensored regression: use
engineering-statistics. - Time-to-event survival analysis: use a survival-analysis package.
- Compositional response data (sulfur speciation fractions): combine with
compositional-data-analysis.
Utility Scripts
uv run scripts/censored.py fit --data data.csv --response S_total --predictors temperature,c5_c6plus_balance --lower-col LOQ --output /tmp/fit.jsonuv run scripts/censored.py interval --data data.csv --response S_total --predictors temperature --lower-col LOD --upper-col LOQ --output /tmp/int.json
Input CSV layout:
- One row per observation.
responsecolumn with the measured value when fully observed; NaN or blank when censored.lower-coland/orupper-colcolumns with the censoring bound for censored rows.- Predictor columns referenced by name in
--predictors.
Censoring-column semantics used by the script:
- finite
response: exact observation; bounds ignored. - missing
response+ onlylower-colfinite: left-censored, meaningy <= lower_col(typical below-LOQ row). - missing
response+ onlyupper-colfinite: right-censored, meaningy >= upper_col(above-range row). - missing
response+ both bounds finite: interval-censored, meaninglower_col < y < upper_col.
Procedure
- Build a tidy CSV with response and predictor columns and the censoring bound columns.
- Decide the censoring direction:
- left-censored at LOQ for trace sulfur below quantitation;
- right-censored for above-range measurements;
- interval-censored when LOD and LOQ are both reported.
- Run the fit. Inspect coefficients, convergence status, optimizer message, and fraction censored.
- For downstream decisions, use a bootstrap/profile-likelihood workflow rather than treating the point estimate as final when the censoring fraction is high.
Pitfalls
- Using
log(S + 0.5 * LOQ)and reporting normal CIs; this substitution is biased and the CIs are wrong. - Using a censored fit and then reporting the response as if it were not censored; predictions are for the latent uncensored response.
- Treating "non-detect" as zero. Zero is usually a measurement convention, not the physical concentration.
- Treating LOD and LOQ as the same. Use interval censoring if both are reported and meaningful.
- Reporting one fit with a single censoring bound when the LOQ changed during the data window.
- Fitting on the raw scale when the data are heavy-tailed.
Fallback Strategies
- For extreme censoring fractions (> 80%), report only the censoring fraction and rank/order conclusions; full regression coefficients are unstable.
- If the row order is a time series, use block bootstrap or a dynamic model for uncertainty rather than i.i.d. inference.
Verification
- Run the listed script with representative inputs and an
--outputfile when a deterministic calculation is available. - Confirm the JSON result contains
ok: true, expected units, and no unhandled warnings. - Check result magnitudes against the stated assumptions, references, and a hand calculation or known operating range before reporting them.
References
references/censored_likelihood.md— likelihood derivation.- Greene, Econometric Analysis, chapter on truncated/censored models.
- Helsel, Statistics for Censored Environmental Data Using MINITAB and R.
Anti-Patterns
- Hiding the censoring fraction in the report.
- Reporting a censored-lognormal coefficient as if it were on the original response scale; by default, the response model is on log scale.
- Using ordinary bootstrap on autocorrelated censored data without preserving time dependence.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

