LocalLLM GPU Scheduler

Lightweight scheduler to share one GPU among multiple local LLM services

Problem

A Show HN introduced GridCore to let several local LLMs share one GPU. Many developers run multiple models (coding assistant, transcription, chat) on a home server or single-GPU machine, but lack simple tools to queue, allocate VRAM, and avoid OOM.

Solution

A lightweight scheduler where users declare VRAM needs and priorities. It dynamically loads/unloads models, routes requests to available instances, and exposes a unified OpenAI-compatible API. Supports monitoring and oversubscription.

Users

Developers with a single consumer GPU, homelab users, small teams running internal AI services.

MVP

Listen on multiple model ports; queue requests based on VRAM and priority; auto start/stop model processes; simple web UI for GPU usage; OpenAI-compatible proxy.

Revenue

Open-source core + paid Pro (multi-GPU, advanced scheduling, team management).

Why now

Local LLMs are popular, but single-GPU users often can't run multiple models simultaneously due to VRAM limits; simple scheduling solutions are missing.