DataEngUtils for Data Scientists
Data Scientists spend their days exploring datasets, converting between file formats, validating data quality, and tracking experiments. You might receive a Parquet export from a data engineer that needs inspection before modeling, a CSV file that needs validation before analysis, or a set of experiment runs that need unique identifiers for tracking. Online tools for these tasks require uploading files to a server, which is impractical for large datasets and often violates data governance policies. DataEngUtils gives you the same capabilities directly on your Mac, with no uploads and no network requests. This page highlights the tools most useful for Data Scientists, organized by the workflows you encounter daily.
Tools for data scientists
- Parquet InspectorInspect Parquet metadata, row groups, column statistics, and encodings without uploading files to a
- CSV to JSONCSV to JSON is built for data engineers who need to convert between file formats and schema represen
- JSON to CSVJSON to CSV is built for data engineers who need to convert between file formats and schema represen
- CSV to ParquetTurn CSV exports into columnar Parquet without shipping them to a random web converter. DataEngUtils
- Parquet to CSVConvert Parquet files to CSV on your Mac without uploading them to a cloud converter. DataEngUtils s
- CSV ValidatorCSV Validator is built for data engineers who need to validate and format pipeline configs, warehous
- JSON Schema ValidatorJSON Schema Validator is built for data engineers who need to validate and format pipeline configs,
- Avro Schema ValidatorAvro Schema Validator is built for data engineers who need to validate and format pipeline configs,
- UUID GeneratorUUID Generator is built for data engineers who need to generate identifiers, test rows, and mock pay
- ULID GeneratorULID Generator is built for data engineers who need to generate identifiers, test rows, and mock pay
- Snowflake ID GeneratorSnowflake ID Generator is built for data engineers who need to generate identifiers, test rows, and
- Hash GeneratorHash Generator is built for data engineers who need to generate identifiers, test rows, and mock pay
Workflow: Inspect dataset structure and types
Open Parquet, CSV, or JSON files to inspect schema, column types, and data distributions before starting analysis or modeling work.
1. Open a dataset
Drag a Parquet or CSV file into DataEngUtils. The Parquet Inspector immediately loads schema, row count, and column metadata.
2. Browse column statistics
Click any column to view data type, null count, min and max values, and distinct value estimates for quick quality assessment.
3. Export a preview
Use CSV to JSON to export sample rows as JSON for sharing with teammates or importing into a notebook for exploration.
Workflow: Convert data between formats
Convert datasets between Parquet, CSV, and JSON formats depending on the requirements of your analysis tools and modeling frameworks.
1. Convert CSV to Parquet
Use CSV to Parquet to convert a CSV export into columnar Parquet format for faster pandas or Polars loading.
2. Convert Parquet to CSV
Use Parquet to CSV when you need to share a Parquet dataset with a teammate who prefers CSV, or for tools that don't support Parquet.
3. Convert JSON to CSV
Use JSON to CSV to flatten nested JSON records from API exports into row-based CSV for analysis in Excel, R, or Python.
Workflow: Validate data quality and schema
Check CSV files for formatting issues and validate that data conforms to expected JSON Schema or Avro Schema specifications.
1. Validate a CSV file
Use CSV Validator to scan for inconsistent row lengths, encoding issues, and missing values before loading data into your analysis.
2. Check JSON Schema conformance
Use JSON Schema Validator to ensure your data conforms to expected schemas when working with structured data pipelines.
3. Validate Avro schemas
When working with feature stores or streaming data, use Avro Schema Validator to check schema definitions before registering them.
Workflow: Generate unique IDs for experiment tracking
Create UUIDs, ULIDs, Snowflake IDs, or hashes for tracking experiment runs, model versions, and dataset snapshots.
1. Generate experiment IDs
Use UUID Generator or ULID Generator to create unique identifiers for each experiment run. ULIDs are time-sortable, making them ideal for chronological tracking.
2. Generate Snowflake IDs
Use Snowflake ID Generator for distributed experiment tracking where chronological ordering across multiple workers is needed.
3. Hash datasets for versioning
Use Hash Generator to create SHA-256 hashes of dataset files for content-addressable versioning and reproducibility checks.