The PDI Data Quality for Mainframe: architecture and benchmarks
Pushdown-first profiling across six mainframe datastores, linear-time pattern scoring, and a DAMA scorecard with a full audit trail — with every figure stated against its benchmark basis.
Executive summary
Most data quality tooling cannot reach the mainframe, so the conventional approach is bulk extraction: move the data off-host, then profile it. For 100-million-row tables this is slow, expensive in MIPS and network cost, and it creates copies of the enterprise's most sensitive data.
The solution inverts the model. Aggregate profiling — row counts, nulls, distincts, length bounds — is pushed down and executed by the datastore engine itself; one result row crosses the wire. Only the columns requiring pattern validation are streamed, in bounded fetches, to a Linux-side linear-time scoring engine. The output is a DAMA-dimension scorecard (Completeness, Validity, Uniqueness, Consistency per column) that is identical across Adabas, Datacom, Db2 for i, Db2 for z/OS, IDMS, and IMS.
Architecture
1 — Connect
The solution connects through the customer's licensed vendor access layers: Db2 Connect / DRDA for Db2 for z/OS, IBM i Access ODBC for Db2 for i, Datacom Server SQL, IDMS Server SQL Option, Adabas SQL Gateway (CONNX), and ODBM / IMS Connect for IMS. Nothing is installed on the host.
2 — Push down
One aggregate statement per table computes counts, null rates, distinct cardinalities, and length bounds on the host engine. On Db2 for z/OS, this DRDA-routed work is zIIP-eligible under IBM's processing rules. Pushdown depth varies by datastore: full for Db2 (both platforms) and Datacom, aggregate-level for IDMS and Adabas, streamed for IMS.
3 — Stream
Only profiled columns move, in 10,000-row array fetches under read-only/UR isolation. No locks are taken on production data.
4 — Score
A linear-time linear-time engine pattern engine validates streamed values across all cores: emails against an RFC 5322 subset, phones with E.164 and NANP tiering, URLs against a live IANA TLD snapshot, structured IDs, and junk or placeholder detection.
5 — Report
The scorecard reports the four DAMA dimensions per column, in text and CSV, identically formatted across all six datastores. Every check carries a registry ID and disclosure notes, and the report includes the exact SQL shipped to the host — the scorecard is fully retraceable.
Configuration, not code
One INI file declares connections, tables, and per-column rule chains. New rules are a regex section and a re-run — no recompile, no vendor ticket, no release cycle. Credentials are supplied via environment variables and never stored in the file.
Performance benchmarks
Measured by PDI on an 8-core, 256 GB Azure instance with 2 TB SSD. The rule-scoring figure is reproducible by any evaluator via the included make smoke harness, which runs the full pipeline on generated data with no mainframe access.
| Measure | Result | Basis |
|---|---|---|
| Rule-scoring throughput | 800K rows/sec per core | Reproducible — make smoke harness |
| Row design budget per profiling window | 100M rows / 30 min | PDI internal benchmark |
| Network cost of aggregate profiling | 1 result row per statement | Architectural property — pushdown |
| Stream batch size | 10,000-row array fetches | Product configuration |
Deployment model
One customer-controlled Linux server; C++ engine; open-standard ODBC; no cluster, no appliance, no host install. See Security & deployment for the full posture.
About Pacific Data Integrators
PDI is an enterprise data management consultancy with 15+ years of delivery and 100+ implementations. The PDI Data Quality for Mainframe is a PDI product.
Run the benchmark yourself.
The make smoke demonstration proves the pipeline in five minutes, on a laptop, with no host access.