Scientific Data Extraction Skill

Overview

This skill provides comprehensive guidance for extracting structured data from scientific literature across multiple input formats (PDF, HTML, images, plain text). It auto-detects the scientific domain to recommend specialized tools when appropriate (particularly for chemistry and materials science) and employs a hierarchical extraction approach with multi-method validation for high-confidence results.

When to Use This Skill

Use this skill when you need to:

• Extract numerical data from scientific papers, reports, or documents
• Digitize graphs and plots to recover underlying data points
• Parse tables from PDFs or images into structured formats (CSV, DataFrame, JSON)
• Extract chemical/materials data including properties, reactions, compounds, and structures
• Convert unstructured text to structured JSON or tabular formats
• Validate extracted data through multi-method cross-checking
• Process document batches with consistent extraction methodology

Input Format Detection

The first step is identifying the input format and routing to appropriate tools:

Plain Text (.txt, .md)

• Domain detection via keyword analysis
• NLP-based entity extraction (spaCy, Stanza)
• Regex patterns for structured data (numbers with units, chemical formulas)
• LLM-based structured extraction

HTML (.html, web pages)

• HTML parsing with BeautifulSoup + lxml
• Table detection and extraction
• Text content extraction with structure preservation
• Domain-specific processing after text extraction

PDF (.pdf)

Images (.png, .jpg, .tiff)

Domain Detection

The skill automatically detects scientific domain to apply specialized tools:

Chemistry/Materials Indicators

• Chemical formulas (H2O, NaCl, TiO2)
• SMILES strings, InChI identifiers
• Reaction arrows (→, ⟶, ⇌)
• Property keywords: melting point, bandgap, conductivity, yield, purity
• Material names and IUPAC nomenclature
• Spectroscopic data patterns (NMR shifts, IR peaks)

When Chemistry/Materials Detected

Apply specialized tools:

• ChemDataExtractor v2: Property extraction, entity recognition, table parsing
• OpenChemIE: Reaction extraction from text, tables, and figures
• Domain-specific NER: Chemical named entity recognition

General Scientific Domain

Use general-purpose extraction:

• Standard NLP pipelines
• LLM-based structured extraction
• Template-based parsing

Extraction Method Hierarchy

Apply methods in order of increasing complexity based on requirements:

Level 1: Quick Extraction (Speed Priority)

When to use: Initial exploration, large document batches, simple structured data

# Quick PDF to text with PyMuPDF4LLM
import pymupdf4llm
text = pymupdf4llm.to_markdown("paper.pdf")

# Quick HTML parsing
from bs4 import BeautifulSoup
soup = BeautifulSoup(html_content, 'lxml')
tables = soup.find_all('table')

Expected confidence: Lower, suitable for screening

Level 2: Standard Extraction (Balanced)

When to use: Research-grade extraction, structure preservation needed

# GROBID for structured PDF parsing
import scipdf_parser
article = scipdf_parser.parse_pdf_to_dict("paper.pdf")

# Docling for layout-aware extraction
from docling.document_converter import DocumentConverter
converter = DocumentConverter()
result = converter.convert("paper.pdf")

# Camelot for bordered tables
import camelot
tables = camelot.read_pdf("paper.pdf", flavor='lattice')
df = tables[0].df

Expected confidence: Medium-high

Level 3: Deep Extraction (Accuracy Priority)

When to use: Publication-quality data, domain-specific extraction

# ChemDataExtractor for chemistry documents
from chemdataextractor import Document
doc = Document.from_file("paper.pdf")
records = doc.records

# OpenChemIE for reaction extraction
from openchemie import OpenChemIE
model = OpenChemIE()
reactions = model.extract_reactions_from_text(text)

# Marker-PDF with OCR for scanned documents
from marker.converters.pdf import PdfConverter
converter = PdfConverter()
result = converter("scanned_paper.pdf")

Expected confidence: High

Level 4: LLM-Enhanced Extraction

When to use: Complex figures, ambiguous data, validation needed

# LLM-based structured extraction
prompt = """
Extract all numerical data from this text as JSON:
- Property name
- Value (number only)
- Unit
- Context (what material/compound)

Text: {text}
"""

# LLM vision for graph interpretation
prompt = """
Analyze this graph image and extract:
1. X-axis label and range
2. Y-axis label and range
3. All data points as (x, y) pairs
4. Any error bars or uncertainty indicators
"""

Expected confidence: Highest when combined with validation

Multi-Method Validation Pipeline

For high-confidence results, use multiple extraction methods and validate:

Step 1: Primary Extraction

Select method based on input type and domain, extract structured data.

Step 2: Secondary Extraction

Run alternative method on same source, compare results and flag discrepancies.

Step 3: LLM Verification Queries

Ask targeted questions to verify extracted data:

• "Is this value X consistent with the context Y?"
• "Does unit Z make sense for property P?"
• "Are there any missing data points in the expected range?"

Step 4: Confidence Scoring

confidence = {
    "score": 0.0,  # 0-1 scale
    "level": "HIGH|MEDIUM|LOW|REVIEW",
    "methods_agreed": [],  # List of methods that produced same result
    "discrepancies": [],   # Any disagreements between methods
    "verification_notes": ""  # LLM verification outcome
}

# Scoring rules:
# - Single method: max 0.7
# - Two methods agree: 0.8
# - Two methods + LLM verification: 0.9
# - Multiple methods + LLM + database cross-reference: 0.95+

Step 5: Database Cross-Reference (Optional)

For chemistry/materials, compare against known databases:

• Materials Project
• AFLOW
• PubChem
• NIST databases

Flag significant deviations from expected ranges.

Output Format

Structure extracted data consistently:

{
  "extraction_metadata": {
    "source": "path/to/document.pdf",
    "source_type": "pdf",
    "domain_detected": "chemistry",
    "methods_used": ["grobid", "chemdataextractor", "llm_verification"],
    "timestamp": "2025-01-18T..."
  },
  "extracted_data": [
    {
      "data_type": "material_property",
      "entity": "TiO2",
      "property": "bandgap",
      "value": 3.2,
      "unit": "eV",
      "source_location": {
        "page": 4,
        "section": "Results",
        "table_id": "Table 2",
        "row": 3
      },
      "confidence": {
        "score": 0.95,
        "level": "HIGH",
        "methods_agreed": ["chemdataextractor", "llm_extraction"],
        "verification_notes": "Value consistent with literature range 3.0-3.4 eV"
      }
    }
  ],
  "validation_summary": {
    "total_extracted": 47,
    "high_confidence": 38,
    "medium_confidence": 7,
    "needs_review": 2,
    "discrepancies": []
  }
}

Step-by-Step Instructions

For PDF Data Extraction

1. Identify document type: Scanned or text-based PDF
2. Choose extraction level: Based on accuracy requirements
3. Detect domain: Check for chemistry/materials indicators
4. Extract text/structure: Use appropriate tool from hierarchy
5. Extract tables separately: Use Camelot, Tabula, or pdfplumber
6. Apply domain tools: If chemistry detected, use ChemDataExtractor
7. Validate: Run secondary extraction or LLM verification
8. Format output: Structure as JSON with confidence scores

For Graph/Plot Digitization

1. Assess graph quality: Resolution, clarity, labeling
2. Identify graph type: Line plot, scatter, bar chart, contour
3. Choose method:
   • Simple, clear graphs: WebPlotDigitizer (manual calibration)
   • Complex or batch: LLM vision extraction
4. Calibrate axes: Define coordinate system
5. Extract data points: Manual selection or automatic detection
6. Validate: Check extracted points against visual inspection
7. Export: CSV or JSON format with uncertainty estimates

For Table Extraction

1. Identify table type: Bordered (lattice) or borderless (stream)
2. Choose tool:
   • Bordered: Camelot with flavor='lattice'
   • Borderless: Tabula or Camelot with flavor='stream'
   • Complex: pdfplumber for fine-grained control
3. Extract to DataFrame: Review structure and headers
4. Clean data: Fix merged cells, missing values, formatting
5. Apply domain parsing: Convert units, parse chemical formulas
6. Validate: Compare against source visually
7. Export: CSV, JSON, or integrate into dataset

For Chemistry/Materials Extraction

1. Confirm domain: Verify chemistry/materials content
2. Choose specialized tool:
   • Properties: ChemDataExtractor v2
   • Reactions: OpenChemIE
   • Structures from images: OSRA or DECIMER
3. Configure extraction: Set up parsers for target properties
4. Run extraction: Process document with domain tools
5. Post-process: Normalize units, standardize identifiers
6. Cross-reference: Compare against databases (Materials Project, PubChem)
7. Validate: LLM verification of unusual values
8. Export: Structured JSON with confidence scores

Best Practices

1. Always start with format detection - Correct tool selection depends on accurate format identification
2. Use the simplest method that works - Start at Level 1 and escalate only if needed
3. Preserve source location - Track page numbers, sections, table IDs for traceability
4. Validate unusual values - Any value outside expected ranges should be flagged and verified
5. Document extraction methodology - Record which tools and settings produced each data point
6. Handle uncertainty explicitly - Include error bounds when available, note when values are approximate
7. Cross-reference chemistry data - Always compare against known databases for sanity checking
8. Use LLM verification judiciously - Most valuable for complex figures and ambiguous cases

Requirements

Core Python Packages

• pymupdf4llm: Quick PDF extraction
• pdfplumber: Detailed PDF analysis
• camelot-py: Table extraction (requires ghostscript)
• beautifulsoup4, lxml: HTML parsing
• spacy: NLP processing
• pandas: Data manipulation

Domain-Specific (Chemistry)

• chemdataextractor: Chemistry NLP (v2 recommended)
• openchemie: Reaction extraction

Optional

• tabula-py: Table extraction (requires Java)
• grobid (server): Academic PDF parsing
• docling: IBM document converter
• marker-pdf: OCR-capable PDF conversion
• tesseract or surya: OCR engines

Limitations

1. Scanned documents require OCR - Quality depends on scan resolution and OCR accuracy
2. Complex table structures - Merged cells, nested headers may require manual correction
3. Graph digitization is approximate - Precision limited by image resolution and calibration
4. Domain tools are specialized - Chemistry tools won't work well on biology or physics texts
5. LLM extraction can hallucinate - Always validate with source or alternative method
6. Some PDFs are protected - May not be extractable due to DRM or image-only content

Related Skills

• literature-review: For systematic literature searching and synthesis
• scientific-reviewer: For evaluating extracted data quality
• materials-databases: For cross-referencing extracted chemistry/materials data
• python-plotting: For visualizing extracted data

References

See the references/ directory for detailed documentation on:

• pdf-tools.md: Comprehensive PDF extraction tool comparison
• table-extraction.md: Table extraction methods and code examples
• graph-digitization.md: Graph data extraction techniques
• chemistry-tools.md: ChemDataExtractor and OpenChemIE usage
• llm-extraction.md: LLM-based extraction patterns and validation

See the examples/ directory for complete workflows:

• extract-from-pdf.md: End-to-end PDF extraction example
• extract-table-data.md: Table extraction comparison
• digitize-graph.md: Graph digitization guide
• chemistry-extraction.md: Chemistry-specific extraction workflow