DataEngUtils for ML Engineers
ML Engineers work at the intersection of data pipelines and model training, handling dataset inspection, schema validation, and configuration management across multiple frameworks. You might need to inspect a Parquet training dataset, validate an Avro schema for a feature store, or convert YAML configs to JSON for a model serving endpoint. Online tools introduce latency, privacy risk, and a context switch away from your terminal. DataEngUtils provides a unified set of offline tools that handle these tasks directly on your Mac, alongside 40+ other data engineering utilities in the same sidebar. This page covers the tools ML Engineers reach for most often, organized by common workflows.
Tools for ml engineers
- Parquet InspectorInspect Parquet metadata, row groups, column statistics, and encodings without uploading files to a
- CSV to ParquetTurn CSV exports into columnar Parquet without shipping them to a random web converter. DataEngUtils
- Parquet to CSVConvert Parquet files to CSV on your Mac without uploading them to a cloud converter. DataEngUtils s
- JSON Schema ValidatorJSON Schema Validator is built for data engineers who need to validate and format pipeline configs,
- Avro Schema ValidatorAvro Schema Validator is built for data engineers who need to validate and format pipeline configs,
- YAML FormatterYAML Formatter is built for data engineers who need to validate and format pipeline configs, warehou
- TOML FormatterTOML Formatter is built for data engineers who need to validate and format pipeline configs, warehou
- JSON to YAMLJSON to YAML is built for data engineers who need to convert between file formats and schema represe
- YAML to JSONYAML to JSON is built for data engineers who need to convert between file formats and schema represe
- JSON to AvroJSON to Avro is built for data engineers who need to convert between file formats and schema represe
- Avro to JSONAvro to JSON is built for data engineers who need to convert between file formats and schema represe
- UUID GeneratorUUID Generator is built for data engineers who need to generate identifiers, test rows, and mock pay
Workflow: Inspect and convert training datasets
Open Parquet training datasets to inspect schema, feature distributions, and row counts, then convert between formats for different stages of the ML pipeline.
1. Open a training dataset
Drag a Parquet file into the Parquet Inspector to view schema, column types, row groups, and null counts without writing Python.
2. Check data quality
Inspect column statistics — min, max, distinct values, and null percentages — to spot issues before training.
3. Convert for downstream tools
Convert Parquet to CSV for pandas workflows, or CSV to Parquet for efficient distributed training pipelines.
Workflow: Validate ML pipeline schemas
Ensure your Avro, JSON Schema, and configuration files are valid before feeding them into feature stores, model registries, and serving infrastructure.
1. Validate a schema definition
Paste your Avro or JSON Schema into the corresponding validator. The tool checks for structural correctness and type consistency.
2. Format pipeline configs
Use YAML Formatter or TOML Formatter to standardize indentation and formatting across your ML pipeline configuration files.
3. Fix violations inline
Edit the schema or config directly in the tool and re-validate until clean, then copy back to your project.
Workflow: Convert config formats for model serving
Translate configuration files between YAML, JSON, and Avro formats when moving between experiment tracking, model registry, and serving environments.
1. Convert YAML config to JSON
Use YAML to JSON to convert your MLflow or Kubeflow pipeline configs for use with JSON-based serving APIs.
2. Translate between JSON and Avro
Use JSON to Avro or Avro to JSON to move schema definitions between your feature store and model registry.
3. Copy to the target system
Once converted, copy the output directly into your serving config or schema registry.
Workflow: Generate unique identifiers for experiments
Create UUIDs, ULIDs, Snowflake IDs, or hashes for experiment tracking, run IDs, and artifact versioning without leaving your keyboard.
1. Choose an ID format
Open UUID Generator for random IDs, ULID Generator for time-sortable IDs, or Snowflake ID Generator for distributed-system-compatible IDs.
2. Configure batch size
Generate a single ID or a batch of IDs at once. Copy them all to your clipboard in one click.
3. Identify existing hashes
Use Hash Identifier to detect the hash algorithm used in existing experiment IDs or artifact checksums.