DataEngUtils for Data Scientists
Data Scientists explore datasets, convert between formats, check data quality and track experiments. You might receive a Parquet export that needs a quick look before modelling, a CSV that should be validated before analysis, or a set of runs that need unique identifiers. Opening a notebook for each of those is slow, and web tools mean uploading files to a server — impractical for large datasets and often against governance rules. DataEngUtils gives you fast, client-side inspection and conversion in the browser.
Tools for data scientists
- Parquet InspectorInspect Parquet metadata, row groups, column statistics and encodings without uploading the file to
- Parquet Data ProfilerProfile a Parquet file in your browser and get per-column statistics — null percentage, distinct cou
- Parquet Row Group PrunerParquet Row Group Pruner is built for data engineers who need to inspect, debug, and explain data ar
- JSONL InspectorInspect JSONL (NDJSON) files line by line without uploading them to a converter. DataEngUtils valida
- CSV → ParquetTurn CSV exports into columnar Parquet without shipping them to a random web converter. DataEngUtils
- Parquet → CSVConvert Parquet to CSV in your browser without uploading the file to a cloud converter. DataEngUtils
- Parquet → JSONParquet → JSON is built for data engineers who need to convert between file formats and schema repre
- Avro File InspectorAvro File Inspector is built for data engineers who need to inspect, debug, and explain data artifac
- Avro Schema ValidatorAvro Schema Validator is built for data engineers who need to inspect, debug, and explain data artif
- JSON Schema ValidatorJSON Schema Validator is built for data engineers who need to validate and format pipeline configs,
- UUID GeneratorUUID Generator is built for data engineers who need to generate identifiers, test rows, and mock pay
- Hash GeneratorHash Generator is built for data engineers who need to generate identifiers, test rows, and mock pay
Workflow: Explore a dataset without a notebook
Open a Parquet file to inspect schema, column statistics and a preview before you spin up a kernel.
1. Open the dataset
Drop the Parquet file into Parquet Inspector for schema, row groups and column metadata.
2. Profile the columns
Use Parquet Data Profiler for null counts, distinct values, ranges and distributions.
3. Check pruning
Use the Row Group Pruner to see which row groups a filter can skip from the file's own statistics.
Workflow: Convert between formats
Move datasets between Parquet, CSV and JSON depending on what your analysis tools expect.
1. Convert CSV to Parquet
Use CSV → Parquet for columnar files that load faster in pandas or Polars.
2. Export Parquet to CSV or JSON
Use Parquet → CSV or Parquet → JSON to share data with tools that need row-based formats.
3. Verify the output
Re-open the result in Parquet Inspector to confirm schema and row counts.
Workflow: Validate schemas and version data
Check Avro and JSON Schema definitions and hash datasets for reproducible versioning.
1. Validate a schema
Use Avro Schema Validator or JSON Schema Validator before registering a schema in a feature store.
2. Inspect Avro records
Open an Avro file to view the schema, metadata and a record preview.
3. Version the data
Use Hash Generator for SHA-256 checksums of datasets and UUID Generator for run IDs.