Virtuoso RDF Bulk Loader Skill

ld_dir + rdf_loader_run · isql · N-Triples · Turtle · RDF/XML · N-Quads · TriG · JSON-LD · Notation3 · gzip

Purpose

Load RDF archives directly into a Virtuoso quad store — no pre-extraction needed. Virtuoso's ld_dir + rdf_loader_run reads compressed files (.gz, .bz2) natively and supports every major RDF serialization. The skill provides:

• Single-file loads — one archive → one named graph
• Directory loads — every matching file in a server-side directory
• Graph management — clear before load, append, or replace
• Progress monitoring — poll DB.DBA.LOAD_LIST until completion
• Error triage — interpret load errors and suggest fixes

Supported Formats

All formats are handled by the same ld_dir + rdf_loader_run pipeline — no format-specific flags needed.

Execution Routing

Default order for data ingestion:

1. isql CLI — direct isql host:port execution (preferred; always available)
2. MCP — if a virtuoso-support-agent MCP server is connected, it can relay SQL
3. SPARQL INSERT — via SPARQL endpoint when isql path is unavailable (wraps individual INSERTs; unsuitable for bulk)
4. URIBurner upload — when the data is at a URL and URIBurner's sponger is available

If the user names a specific mode, honor that preference.

Step 0 — Connection Detection (Always Run First)

# Check isql availability
which isql

# Quick connectivity test
echo "SELECT 'ok';" | isql <host>:<port> <user> <pass> 2>&1 | head -3

If isql is not found, ask the user for:

• isql path, OR
• Virtuoso host:port credentials to try MCP, OR
• SPARQL endpoint URL for fallback loading

Step 1 — Gather Parameters

Before any load, collect:

--dir must be accessible to the Virtuoso server process — ld_dir registers server-side paths, not client-side paths. For remote Virtuoso, the files must already be on the server.

Recommended Patterns by Format

Mode Behavior

Step 2 — Register and Load

The core workflow (two SQL calls):

-- Register files (any supported format)
ld_dir ('/data/rdf/incoming/', '*.nt.gz', 'https://example.com/my-graph');

-- Start the loader (runs asynchronously in the server)
rdf_loader_run ();

Execute via isql:

isql ${HOST}:${PORT} ${USER} ${PASS} <<EOF
ld_dir ('${LOAD_DIR}', '${PATTERN}', '${GRAPH}');
rdf_loader_run ();
CHECKPOINT;
EOF

ld_dir Parameters (Full Signature)

ld_dir (
  in dir         varchar,   -- server-side directory path
  in file_pattern varchar,  -- e.g. '*.nt.gz', '*.ttl.gz', '*.nq.gz', '*'
  in graph_iri   varchar    -- target named graph
);

Key points:

• ld_dir is idempotent — calling it twice with the same files does not double-load
• Files are matched by name in DB.DBA.LOAD_LIST — renaming a file after registration re-triggers it
• rdf_loader_run() can be called multiple times safely; it processes pending entries only
• The loader auto-detects the RDF format from file content — no format flag required
• The loader runs on server threads — isql returns immediately

Step 3 — Monitor Progress

Poll DB.DBA.LOAD_LIST until no rows remain in progress:

SELECT
  ll_file,
  ll_graph,
  CASE ll_state
    WHEN 0 THEN 'PENDING'
    WHEN 1 THEN 'LOADING'
    WHEN 2 THEN 'DONE'
    WHEN 3 THEN 'FAILED'
  END AS state,
  ll_started,
  ll_done,
  ll_rows,
  ll_error
FROM DB.DBA.LOAD_LIST;

ll_state Meanings

Monitor loop:

while true; do
  STATUS=$(isql ${HOST}:${PORT} ${USER} ${PASS} <<'EOSQL' 2>/dev/null)
    SELECT ll_file, ll_state, ll_error
    FROM DB.DBA.LOAD_LIST
    WHERE ll_state IN (0, 1, 3);
EOSQL
  if echo "$STATUS" | grep -q 'No\. of rows\|0 rows'; then
    echo "All loads complete."
    break
  fi
  echo "$STATUS"
  sleep 5
done

Step 4 — Verify

After loading, check the row count in the target graph:

SPARQL
SELECT (COUNT(*) AS ?triples)
FROM <${GRAPH}>
WHERE { ?s ?p ?o };

Or via isql:

isql ${HOST}:${PORT} ${USER} ${PASS} <<EOF
SPARQL SELECT (COUNT(*) AS ?triples) FROM <${GRAPH}> WHERE { ?s ?p ?o };
EOF

For a spot check, sample 10 triples from the graph:

SPARQL SELECT * FROM <${GRAPH}> WHERE { ?s ?p ?o } LIMIT 10;

Bundled Shell Script

scripts/bulk-load-rdf.sh automates Steps 2–3 for single-file or directory loads. Invoke it from the skill directory:

./scripts/bulk-load-rdf.sh \
    --host     localhost \
    --port     1111 \
    --user     dba \
    --pass     dba \
    --graph    https://example.com/my-graph \
    --dir      /data/rdf/incoming \
    --pattern  "*.nt.gz" \
    dataset.nt.gz

Troubleshooting

Loading Without ld_dir (Single-File Interactive)

For one-offs where ld_dir isn't practical, pipe a decompressed file directly to isql. DB.DBA.TTLP() auto-detects the format:

gunzip -c data.nt.gz | isql ${HOST}:${PORT} ${USER} ${PASS} <<'EOF'
DB.DBA.TTLP (
  file_to_string_output ('/dev/stdin'),
  '',
  'https://example.com/my-graph',
  0
);
EOF

Prefer ld_dir + rdf_loader_run for everything except single small files — it's parallel, transactional, restartable, and format-agnostic.

Quick Reference

Initialization Sequence

When invoked:

⛔ PRE-BUILD CHECK: Before producing output, re-read the relevant workflow section above and re-read any checklists or verification gates defined in this skill. Confirm each checklist item before writing output. Apply the CLAUDE.md Anti-Drift Protocol: re-read spec section before build, gate-first validation, section-by-section delivery.

1. Run Step 0 — verify isql is available and Virtuoso is reachable
2. Ask for: host, port, credentials, graph IRI, server-side directory, file pattern
3. Determine the RDF format from file extensions — confirm with user
4. Confirm mode (append / clear / replace)
5. If mode is clear or replace: warn user, then execute SPARQL CLEAR GRAPH
6. Run ld_dir to register files with the correct pattern
7. Run rdf_loader_run to start loading
8. Poll DB.DBA.LOAD_LIST until all files report ll_state = 2 (done)
9. Run verification count query — report triples loaded per file
10. If any files report ll_state = 3 (failed), surface ll_error and troubleshoot

Reference Files

Safe Virtuoso ISQL Error Output

Whenever this workflow invokes OpenLink's native Virtuoso isql client, pass the bare client option COMMAND_TEXT_ON_ERROR=OFF, for example isql {host}:{port} {user} {password} COMMAND_TEXT_ON_ERROR=OFF {script.sql}. Place it before SQL input, script arguments, or -i. OFF maps to the client's internal value 0 and prevents the client from appending the full failed command to error output. The client accepts OFF/ON; literal 0 is rejected. Do not use -u for this setting (-u populates the U argument array). This does not redact secrets from server diagnostics, shell history, or process arguments, so keep credentials out of SQL text and logs. Do not apply this Virtuoso-specific option to unixODBC/iODBC isql clients.

Version

2.0.0 — Format-agnostic. Supports N-Triples, Turtle, RDF/XML, N-Quads, TriG, JSON-LD, Notation3, OWL — gzip-compressed or raw. Same ld_dir + rdf_loader_run pipeline for all.