Engineering Whitepaper

Provenance-first metadata discovery for mainframe estates.

The PDI Metadata Scanner builds a catalog and column-level lineage graph across six classic mainframe platforms — and attaches, to every fact it emits, the source member and line number that proves it. This paper describes the architecture, the provenance ledger that makes hallucinated lineage structurally impossible, and the measured performance of the extraction engine.

PDI Metadata Scanner – Mainframe
Db2 for z/OS · Db2 for i · IMS · Adabas · CA IDMS · CA Datacom/DB
July 2026
01 · THE PROBLEM

The most important data is the least documented.

The systems of record in banking, insurance, government, and manufacturing still run where they were built: on the mainframe, inside Db2 tables, IMS segments, Adabas files, IDMS records, and Datacom tables shaped by COBOL copybooks and maintained by programs whose authors have retired. Governance initiatives stall at exactly this boundary — the modern catalog covers the cloud estate in weeks, then stops at the LPAR.

The conventional fixes both fail in the same way. Manual documentation decays the day it is written. And automated discovery tools that infer structure tend to present inference as fact — a catalog that is confidently wrong is more dangerous to a steward than no catalog at all, because it removes the instinct to check.

A catalog earns trust the way an audit does: not by claiming to be right, but by showing its evidence for every line.
02 · DESIGN PRINCIPLES

Restraint as an engineering requirement.

The product applies industrial-design discipline to a data tool. These are not aesthetic preferences; each maps to a verifiable behavior.

PrincipleBehavior in the product
One decision per screenThe source picker shows six cards and nothing else. The wizard is a single linear path: pick → enter → test → save → explore.
Clarity over jargonEvery connection field carries a one-line plain-English caption ("ask your mainframe admin if unsure"). No tooltip-hidden meaning.
State shown honestlyConnection status is a chip with icon + text — never color alone (WCAG AA). Save is disabled until the test passes.
Material honestyVerified facts render solid blue; unverified facts render dashed, flagged in orange, labeled "needs manual confirmation." The visual language is the confidence model.
One accent colorBlue marks primary actions and verified state; everything else is neutral. The interface defers to the metadata.
03 · ARCHITECTURE

The mainframe extracts. The engine builds. The interface only reads.

Work is placed where it is cheapest. Structural extraction runs on-platform through utilities every shop already licenses and trusts — catalog unloads for Db2, archived DBD source for IMS, FDT and Predict reports for Adabas, dictionary punches for IDMS and Datacom. A busy-host guardrail defers job submission when the LPAR is loaded. The compact unload output moves off-host, where a C++ engine performs the CPU-intensive parsing, scanning, and graph-building that would waste MSU on-platform.

MAINFRAME native unload utilities · busy guard C++ ENGINE parse · scan · resolve provenance ledger CATALOG indexed · idempotent byte-stable re-runs THIN UI reads only <1 s at any scale CDGC facade export (CSV)
Fig. 1 — Placement of work. Compact unload output crosses the boundary; parsing never runs on-host, extraction never runs in the UI.

The separation is enforced, not encouraged: the interface process contains no parser and no resolver. Every view it renders is a single indexed read against the pre-built catalog, which is why the tree, detail cards, and lineage panels answer in under a second regardless of estate size.

04 · THE PROVENANCE LEDGER

Hallucination made structurally impossible.

Inside the engine, there is no code path that creates a catalog object or a lineage edge without a provenance tuple — the record type that carries a fact requires the record of its evidence:

FACT CUSTUPDT —EXEC SQL UPDATE→ BANKAPP.CUSTOMER confidence: VERIFIED ITS PROOF (REQUIRED) member: src/cobol/CUSTUPDT.cbl line: 17 stmt: UPDATE BANKAPP.CUSTOMER
Fig. 2 — The ledger entry. In the interface, hovering any fact reveals exactly this tuple; the export carries it to CDGC.

Resolution — connecting a scanned reference to a cataloged object — is permitted only two ways: exact-name match against parsed metadata, or a provable cross-artifact rule. The second is where the design shows its character. When an IDMS program says FIND OWNER WITHIN DEPT-EMPLOYEE, the scanner does not guess which record the owner is; it resolves through the schema's own SET … OWNER IS DEPARTMENT declaration — an inference in which every step has its own provenance line.

Everything else stays visible and flagged. A table referenced in COBOL with no DDL anywhere in the corpus; a DL/I call whose segment is chosen at runtime; a program executed by JCL but absent from source control; a Natural UPDATE whose loop scope a line scanner cannot prove — each is written with confidence: UNVERIFIED, rendered as a dashed edge, and labeled for manual confirmation.

Silent omission and confident fabrication are the same failure. The scanner commits neither: what it cannot prove, it shows as unproven.

The shipped sample corpus deliberately contains all four unprovable cases above, and the automated test suite asserts that each surfaces as UNVERIFIED — the honesty path is regression-tested, not merely claimed.

05 · IDEMPOTENCY

Re-runs that converge, never drift.

Reference identifiers are deterministic paths — orcl_table/DB2ZOS/BANKAPP/CUSTOMER — never sequence numbers; edge identifiers hash the full edge tuple. Each run writes the catalog in a single converge-by-key transaction, and a corpus content hash short-circuits unchanged inputs into a recorded no-op. The regression suite proves the property the strong way: scan twice, dump both catalogs, assert a zero-byte diff.

06 · PERFORMANCE

Measured on the production path, on deliberately weak hardware.

The specification requires 100 million lines of source metadata processed in five minutes. The shipped benchmark harness drives the exact scanning hot loop used in production over the real sample corpus replicated in memory, and reports the count of references it extracted — so the number cannot be flattered by a hollowed-out loop.

MetricMeasured
Lines processed100,000,000
Threads1
Elapsed18.48 s
Throughput5,411,042 lines/s
References extracted during the run30,303,024
Requirement333,334 lines/s
Margin16.2×, single core

Hardware stated in full: one Intel Xeon vCPU at 2.10 GHz, 3.9 GiB RAM, shared cloud container — deliberately the weakest plausible host. The member scan is embarrassingly parallel; on multi-core reference hardware the constraint moves to unload transfer and storage streaming, not compute. That extrapolation is labeled as such; the table above is measurement.

07 · SECURITY

Credentials with a single audited path.

Passwords are encrypted at rest with Fernet (AES-128-CBC with HMAC-SHA256 authentication) in a file separate from configuration, mode 0600, keyed from an environment variable or the operating-system keyring — the key never enters the repository. Non-secret settings live in plain INI; an assertion in the save path makes it impossible for a password to reach it. When the engine needs credentials, they arrive over an in-memory pipe at connection time — never a command-line argument, never a temp file, never a log line. Each invariant is exercised by the automated test suite on every build.

08 · VERIFICATION ITEMS

What we ask you to verify.

A provenance-first product owes its readers the same honesty about itself. Three items are flagged in the product and resolved during onboarding rather than assumed:

ItemWhy it's flagged
Connection field setsField names and ports follow the design specification and are verified against current vendor connection documentation for each site's stack during setup.
Report-layout grammarsAdabas FDT, IDMS punch, and Datacom dictionary layouts vary by shop and report options; parsers are validated against your actual utility output, and unrecognized lines are counted as warnings — never guessed at.
CDGC ingestion templateExport columns carry the full facade model and provenance; header labels are reconciled to your CDGC bulk-import template during implementation.

See your own estate mapped.

Bring a copybook, a DBD, or a schema punch to a 30-minute briefing. We'll show you the catalog, the lineage, and what it refuses to claim.

Request a Demo
Pacific Data Integrators — a data integration and data quality consultancy and Informatica partner.
All performance figures are produced by the benchmark harness that ships with the product and are reproducible with a single command; hardware is stated in §06. No customer data appears in this document; all examples derive from the product's published sample corpus.

Product › Data Quality › Metadata Scanner – Mainframe · July 2026 · © 2026 Pacific Data Integrators