browser4-web-miner

---
name: browser4-web-miner
title: "WebMiner — Convert Detail Web Pages into Interactive Views"
description: "Groups similar web pages together and produces an interactive HTML report with clusters of related pages, plus Excel spreadsheets for analysis. Use when the user wants to cluster downloaded HTML files, convert detail web pages into interactive views, or analyze a folder of web pages locally."
tier: procedure
---

# WebMiner — Convert Detail Web Pages into Interactive Views

## Quick Start

```bash
browser4-cli webminer install            # one-time install (Java 17+ auto-detected)
browser4-cli webminer all <html-dir>     # full pipeline: encode → cluster → views
```

WebMiner groups similar web pages together and produces an interactive HTML
report with clusters of related pages — plus Excel spreadsheets for further
analysis. Give it a folder of downloaded HTML files, and it handles the rest.
Everything runs locally; no data leaves your machine.

## When to Use

Use WebMiner when you have a **folder of downloaded HTML pages** and want to cluster them into interactive views and Excel reports — fully local, no LLM tokens. It complements rather than replaces `browser4-cli crawl`/`swarm` (which acquire pages): WebMiner analyzes pages you already have. Not for single-page extraction — use `htmlsnapshot` for that.

## How It Works

WebMiner runs a three-stage local pipeline: **encode** converts each HTML page into a 69-dimension feature vector, **cluster** groups similar pages with SMILE KMeans (k auto-detected), and **views** renders an interactive HTML report plus Excel spreadsheets. Everything runs locally on your machine — no data leaves it, and no LLM tokens are consumed.

## Patterns

### 1. Full pipeline on a folder of pages

```bash
browser4-cli webminer all <html-dir>
```

### 2. Rebuild views from an existing run

```bash
browser4-cli webminer views <result-dir>
```

### 3. Try it on the sample dataset

```bash
browser4-cli webminer run-example
```

## Flags

| Flag | Applies to | Description |
|------|-----------|-------------|
| `--max-files <n>` | `webminer all` | Limit the number of HTML files processed (default 40) |
| `--output <dir>` | `webminer all` | Override the output directory |
| `--resume [<project-id>]` | `webminer all` | Resume a previous run |

## Errors & Recovery

| Symptom | Cause | Fix |
|---------|-------|-----|
| `webminer install` fails | No Java 17+ on PATH | Install JDK 17+ or point `JAVA_HOME` at it |
| `webminer all` finds no pages | Directory has no `.html` files | Check the input directory path and file extensions |
| Pipeline crashes on large corpora | Free tier limit (< 1,000 pages) | Reduce the corpus or use `--max-files`; see the commercial Spark tier for scale |
| Views land in an unexpected temp dir | The views stage uses the app task-output root | Use `webminer views <result-dir>` to rebuild beside the result dir |

## Using from the Browser4 CLI

WebMiner is a first-class Browser4 citizen: the `browser4-cli webminer`
command installs, updates, and runs the tool natively (no PowerShell needed —
the CLI locates a Java 17+ installation, preferring the JRE bundled with the
Browser4 runtime, and launches `scent-miner.jar` directly). The JAR and its
release metadata are installed to `~/.scent/webminer/`.

```bash
browser4-cli webminer install            # Download and install the latest release
browser4-cli webminer update             # Check for and install the latest release
browser4-cli webminer version            # Show installed and latest available versions
browser4-cli webminer uninstall          # Remove the installed release
browser4-cli webminer run-example        # Sample dataset + full pipeline (needs 7-Zip)
browser4-cli webminer all <html-dir>     # Full pipeline (encode → cluster → views)
browser4-cli webminer views <result-dir> # Rebuild views from an existing run
```

- `webminer all <dir>` accepts the pipeline options directly
  (`--max-files <n>`, `--output <dir>`, `--resume [<project-id>]`).
- Any other command is forwarded verbatim to `scent-miner.jar`, e.g.
  `browser4-cli webminer encode <dir>`.
- Runs started through the CLI set `-Dapp.name=webminer`, so the views
  task-output root is `%TEMP%\webminer-<user>\ml\tasks\...` (`<user>` is the OS
  user name; see [Output](#output)).
- The bare `webminer` panel and `webminer version` keep the update check quiet:
  the GitHub → OSS-mirror fallback notices (rate limit, HTTP status, unreachable)
  are suppressed, and the `Published` line is omitted entirely when the release
  carries no `published_at`.  `webminer install` / `update` still report the
  fallback.

## Installing WebMiner

`browser4-cli webminer install` downloads, verifies, and installs the latest
release (GitHub Releases with an Aliyun OSS mirror fallback; works on
Windows, Linux, and macOS — no PowerShell needed):

```bash
browser4-cli webminer install            # Download and install the latest release
browser4-cli webminer update             # Check for and install the latest release
browser4-cli webminer version            # Show installed and latest available versions
browser4-cli webminer uninstall          # Remove the installed release
```

Releases are installed to `~/.scent/webminer/` and checked against
`https://github.com/platonai/web-miner/releases`. SHA-256 checksums are
verified automatically on download.

You can also use the JAR directly if it's already available:

```bash
java -jar scent-miner.jar <command> <args>
```

## Converting Pages to Views

### Running the Example

The `run-example` command downloads a pre-uploaded test dataset of real web
pages, extracts it, and runs the full pipeline — no manual setup required
beyond Java 17 and 7-Zip:

```bash
browser4-cli webminer run-example
```

The dataset is cached at `~/.scent/test-data/amazon.com/` so subsequent runs
skip the download.

### Running on Your Own Pages

```bash
# Full pipeline (one-shot)
browser4-cli webminer all /path/to/html/files

# Or with the JAR directly
java -jar scent-miner.jar all /path/to/html/files
```

The cluster count is always auto-detected from the data — this produces better
results than guessing a number.

### Options

| Flag | Default | Purpose |
|------|---------|---------|
| `--max-files <n>` | `40` | Maximum number of HTML files to process |
| `--output <dir>` | `<html-dir>-ml-output` | Where to write the clustered results (CSV + clustering info; the views stage uses the app temp root — see [Output](#output)) |
| `--resume [<project-id>]` | — | Pick up where a previous run left off. If no project ID is given, the most recent project is used. |

### Building Views from an Existing Run

If clustering has already completed and you just need to (re)build the views:

```bash
java -jar scent-miner.jar views <html-dir>-ml-output/kmeans-result/p<timestamp>
```

## Output

`all` produces two kinds of artifacts in **two different places**:

1. **Clustered results** — written to `<html-dir>-ml-output/kmeans-result/p<timestamp>/`
   (or wherever `--output` points): one `result.csv` per feature view
   (`predictionAnd{Final,Minimal,Original}Features/result.csv`) plus
   `clusteringInfo.txt`.
2. **Views** (interactive HTML report + Excel + JSON) — the `views` stage of
   `all` writes them to the application's **temp task-output root**, NOT under
   `<html-dir>-ml-output`:
   `%TEMP%\<app>-<user>\ml\tasks\unsupervised\result\p<timestamp>\predictionAndMinimalFeatures.views\`
   on Windows, and `<java.io.tmpdir>/<app>-<user>/ml/tasks/unsupervised/result/p<timestamp>/predictionAndMinimalFeatures.views/`
   on Linux/macOS (`/tmp/...` on Linux, `$TMPDIR` on macOS) — the `<app>` prefix
   follows `-Dapp.name` (`webminer` when launched through `browser4-cli webminer`,
   `pulsar` for a direct `java -jar` run) and `<user>` is the OS user name.  The
   end of the run prints the resolved absolute views path.

So after `java -jar scent-miner.jar all ./html-pages/` the clustered results
look like:

```
html-pages-ml-output/
  └── kmeans-result/
      └── p<timestamp>/
          ├── predictionAndFinalFeatures/result.csv
          ├── predictionAndMinimalFeatures/result.csv
          ├── predictionAndOriginalFeatures/result.csv
          └── clusteringInfo.txt
```

and the views (`<project>.html`, `*.xlsx`, `*.json`) live in the temp
task-output directory printed by the run.

> **The real report is `<project>.html` (e.g. `p<timestamp>.html`), not
> `index.html`.**  The `index.html` inside the views directory is an
> auto-generated directory listing ("Index of predictionAndMinimalFeatures.views")
> — opening it shows a file list, not the interactive clustering report.
> Open `<project>.html` instead.

To place the views **beside the clustered results** (e.g. to archive them with
the project), rebuild them from the result directory:

```bash
browser4-cli webminer views <html-dir>-ml-output/kmeans-result/p<timestamp>
# (equivalent to: java -jar scent-miner.jar views <html-dir>-ml-output/kmeans-result/p<timestamp>)
```

This writes `predictionAndMinimalFeatures.views/` inside the given result
directory — the recommended way to locate artifacts, since the output path is
explicit instead of an opaque temp path.  Open the generated `<project>.html`
in a browser to explore the clustering results. The `.xlsx` files can be
opened in Excel for sorting, filtering, or further analysis.

## Tips

- **Input files** — only `*.html` and `*.htm` files are processed. Other files
  in the directory are ignored.
- **Resume interrupted runs** — if a pipeline stops partway through, use
  `--resume` to continue from the last completed stage instead of starting over.
- **Offline only** — WebMiner works with pre-downloaded HTML files. Use a
  browser, wget, or a crawler to fetch pages first.
- **Java 17** is required. Make sure `java` is on your PATH.

More agent context in platonai/Browser4

9 other files this repository gives its agents.