Exploratory Data Analysis

Turn an unfamiliar table into an understanding of what it contains and what deserves closer examination. Exploration generates observations and hypotheses, not conclusions. An unusual value or relationship is a reason to look closer.

Scope and files

Analyze one table from one CSV or XLSX file. CSV files must be comma-delimited with field names in the first record. For XLSX, read XLSX.md before selecting or loading data; it defines selection, formula restrictions, and cell interpretation. If several files or plausible tables are offered, establish which one to explore. For an unsupported structure, explain what is needed rather than silently choosing a table or changing the source.

Read the source in place and leave its bytes unchanged. The report is throwaway: the driver, the report, and every intermediate file go in the OS temporary directory, and each run writes a fresh report.

Read the workspace glossary and the relevant methodology and organization context, if present. Use shared-understanding for missing, ambiguous, or contested terms. If documented methodology conflicts with this skill's exploratory scope or temporary outputs, surface it before proceeding with affected work.

1. Orient and profile

Run uv run "<skill directory>/scripts/profile_table.py" "<source path>" with any XLSX selection options, and read the printed summary and the temporary JSON profile. Without uv, the scripts need Python 3.10+ with the dependencies pinned in the script headers of profile_table.py and, for rendering in step 5, render_report.py. Use an available interpreter when python -c "import pandas, openpyxl, PIL; print(pandas.__version__, openpyxl.__version__, PIL.__version__)" shows versions within those pins; otherwise create a virtual environment in the OS temporary directory and install them there. Run every script with that one interpreter. Script paths are relative to the skill directory, not the working directory; --help describes selection and parsing overrides.

The summary also names an exploration cache: pandas.read_pickle on it returns (raw, data, profile). Load it in ad-hoc exploration scripts rather than re-reading a large source; the driver still profiles the source itself.

Review every field's apparent role against its name and raw values, and inspect the failure and source-error examples before using parsed values. A date_format_hint lists the non-ISO date formats a field's values fit; confirm one against the raw values, and treat a hint that fits both month-first and day-first as an unresolved question about the field. Correct roles and date formats by rerunning the profiler with overrides. The summary ends with a choices line holding the selection and overrides it applied; the driver pastes that line unchanged.

The profile's entity_fields names each identifier whose keys repeat, with the fields that never vary within a key. It is evidence for the grain: those fields describe the entity rather than the record, so count or total them once per entity. When the file spans several periods, a constant field that could plausibly change over time, such as a department, location, or name, is likely the entity's current value rather than its value in each period; breakdowns by it then describe the present, which belongs in the limitations. A status, flag, or amount can also belong to a grain above the record that no single identifier names, such as one report covering several projects. When such a field repeats across records, test the plausible composite grouping (for example, person and period) for fields that stay constant within it, and state that grain beside the record's.

When an unresolved question about a field's meaning or values would materially change a calculation or interpretation, call the Skill tool with shared-understanding, giving it the specific ambiguity and the context already established. The user resolves it before you continue; if they cannot, flag the field as unresolved and continue with the rest of the dataset. An interpretation the file's own evidence settles, such as a fiscal-year period label whose only consistent reading makes the periods consecutive and none later than the file date, proceeds without the question: record it in the driver and state it, with its basis, as a limitation.

This step is complete when row and field counts, blanks, and appropriate basic field summaries are established; every field's role has been reviewed and its apparent meaning written in plain words; and the dataset's grain is stated with its basis or flagged as unresolved. An empty table can have a complete profile without supporting further analysis.

2. Assess data quality

Record the observable conditions in the file that could affect how its data is interpreted or analyzed, one condition each, in five areas:

• Completeness: blank cells by field, entirely blank records, fields that are entirely blank, and fields far sparser than the rest.
• Validity: values that fail to parse as their apparent type, typed Excel errors, dates or numbers outside a possible range, malformed values, and placeholders standing in for a missing value. The profile's possible_placeholders lists candidates such as R-000000, 99999, --, and UNKNOWN; confirm each against the field's meaning, since a value like 000 can be a real code.
• Uniqueness: exact duplicate records, and repeated values in apparent identifiers or composite keys.
• Consistency: fields that contradict each other within a record, values that vary within an apparent entity where they should stay constant, and mixed formats. The profile's value_shapes shows an identifier mixing character patterns, such as 8-digit and 5-digit IDs, and concentrated_in names a period, category, or year where most of a shape's records fall. A shape confined to one period or source often marks a scheme change that splits one entity across keys, while a shape spread across the file may be a legitimate second format. Confirm each against the records, for example by finding the same name under both shapes, and record each confirmed scheme as its own condition, counting its records only. The profile's case_variants lists category values that differ only in case or spacing and so split counts.
• Coverage: earliest and latest dates or periods and obvious gaps between them. Whether the data are current depends on the expected refresh cycle, which the file does not give; an old latest date is an open question for the user.

State each observation as a short plain-language phrase ("Blank", "Placeholder number instead of a resolution"), and say why it matters: what it changes for a reader who uses the data, such as a denominator that shrinks, a join that fails, or a total that needs affected records excluded.

The Report derives the mechanical conditions and each date field's coverage span from the profile; explain every one with Report.explain(), including a condition expected for its field. Record each judged condition, such as a confirmed placeholder or a second identifier scheme, with Report.quality(), and the span of a period stored as a label with Report.coverage(). For a condition whose impact merits attention despite affecting few records, set always_show=True and explain that impact in why_it_matters; the report otherwise collapses conditions below its display threshold. Record a check that found nothing, such as a composite key tested for repeats, with Report.checked().

This step is complete when every area has been checked against the profile and the records, every mechanical condition and span is explained, each judged condition is recorded with Report.quality(), and each check that found nothing with Report.checked().

3. Build the data overview

Help the reader understand their dataset through useful descriptive questions, using important fields as the starting point. Use judgment to select questions about composition, distributions, changes over time, or relationships that the data can meaningfully answer. Choose measures at the appropriate grain: counts of records and counts of distinct entities answer different questions, and repeated entity-level values may not be additive.

Each retained question gets one chart suited to it, a descriptive takeaway, and the context needed to read the chart correctly: the measure, population, period, units, denominators, or a limitation that applies to it, without inventing a caveat. Ordinary descriptive patterns, such as peaks over time or a category's share, belong here beside their charts. Include only questions the data supports answering with a meaningful chart.

Choose the chart type from the question it answers:

• Bar chart: compares distinct items or groups, such as departments or categories: which is bigger or smaller.
• Line chart: shows change, flow, or progression over time, such as daily traffic or monthly revenue over a year.
• Pie chart: shows the parts of one whole.

Bar and pie charts fold a long tail of groups into Other themselves and report what they folded; say so in the chart's context, and when Other outweighs the largest group shown, say in the takeaway that the field is spread thinly. Record exclusions and transformations in the driver, and apply unresolved limitations to the choice of measures and calculations.

This step is complete when the important fields have been considered for useful descriptive questions, each retained question has a rendered chart supported by calculations, and its chart and prose agree. If the file supports no meaningful overview, say why instead.

4. Develop observations when warranted

Follow patterns from the profile, data quality assessment, and overview that raise meaningful questions an auditor could take to the data provider. These may concern business activity, unusual relationships, or the meaning and reliability of the data. Inspect the contributing records and useful comparisons before retaining an observation.

• What we noticed: the supported pattern, with its denominator, population, period, or comparison group as needed. A supporting chart or table lets the reader check the claim.
• Why it matters: how the pattern could affect interpretation or what it prompts the auditor to explore. Include possible explanations only when useful and clearly label them as hypotheses.
• Question for the data provider: a specific question that would help explain or resolve the pattern.

For example, the distribution of service-request resolution times belongs in the overview; requests marked resolved before their recorded opening dates may warrant asking how those dates are defined or populated. Retain an observation only when its evidence supports a meaningful stakeholder question.

When an observation also limits analysis, apply the limitation to all affected calculations and visuals, including those earlier in the report, and keep the full evidence and stakeholder question in the observation. Describe an uncertain meaning as unresolved, without treating a suspected explanation as an established error.

This step is complete when useful follow-up signals have been examined, each retained observation is supported by calculations and inspected records and ends with a concrete stakeholder question, and any resulting limitations are reflected throughout the report.

5. Present and verify

Read HTML-REPORT.md for the driver and the Report module. The driver holds the source path, skill location, interpretation choices, and every calculation behind the report, so each number in the report derives from a calculated value.

Add a limitation for each material constraint on interpretation, referring to an observation by title when it carries the full investigation. Routine calculation choices, such as counting distinct sites, belong in chart context rather than in limitations.

Write for an auditor who has not opened the file: name the dataset in plain words inferred from the filename and contents, describe it in a sentence or two including what one record represents, and give every field an apparent meaning in plain words.

Run the driver and check:

• Counts reconcile to the selected source table; blanks, source errors, parsing failures, and exclusions explain the denominators used.
• Each reported number, ranking, comparison, and named example matches its calculation, and every interpretation remains distinct from what the data establishes. Read each takeaway against its rendered chart or table.
• Each chart's type fits its question: bars compare distinct items or groups, lines show change over time, and pies show parts of one whole.
• The report renders with readable labels, tables, and charts. Run uv run "<skill directory>/scripts/render_report.py" "<report.html>": it captures the whole page with headless Chrome or Edge and prints numbered crops from top to bottom. Read every crop. The Chrome extension opens only web URLs, not local files. If the script finds no browser or cannot capture the complete page, disclose that visual verification remains outstanding.

Open the report in the user's default browser: Start-Process on Windows, open on macOS, xdg-open on Linux. Return the report's absolute path and a short account of the dimensions explored, any stakeholder questions, and limitations. The report is temporary; rerun the skill to regenerate it.

This step is complete when the checks pass, the report is open in the browser, and the title, description, and field meanings read as plain language. Formal audit-trail packaging is outside this version's scope.