LocalProc: local DataProc cluster images

Spin up a cloud-consistent DataProc cluster locally or in CI

Problem

Google provides no official Docker image for developers to run DataProc locally, so every dev or CI test spins up a real cloud cluster, which is slow and costly. Anyone verifying Spark or Hadoop logic lacks a local environment with matching component versions.

Solution

Ship DataProc Docker images aligned with cloud versions so a single command starts a local cluster for development and CI testing, keeping Hadoop, Spark, and related components consistent. Supports fast resets and scripting, and can be invoked by AI agents.

Users

Data engineers on Google Cloud DataProc, teams testing Spark jobs in CI, developers setting up environments for AI agents

MVP

Publish DataProc Docker images aligned with cloud versions; one-command start with self-test; CI integration scripts; fast environment reset; docs mapping to cloud versions

Revenue

Open-source images for acquisition, enterprise hosted registry, support subscription, and CI templates

Why now

DataProc users have long lacked an official local option, while local-first development and CI testing are becoming default, giving an indie developer room in niche infra tooling