senkani ml-eval

Measure per-tier Gemma 4 output quality. Drives the 20-task harness (10 rationale + 10 vision) against every installed Gemma tier this machine can host, writes ~/.senkani/ml-tier-eval.json, and senkani doctor then surfaces per-tier ratings (excellent / acceptable / degraded) so 8 GB Mac users learn the smaller tier's quality cost up-front.

Syntax

Behavior

Iterates each Gemma 4 tier in ModelManager.visionModelIds (gemma4-26b-apex, gemma4-e4b, gemma4-e2b). For each tier:

• Insufficient RAM (tier requires more than this machine has) — recorded as notEvaluated with a named reason. Doctor surfaces the placeholder.
• Not installed (status not verified or downloaded) — recorded as notEvaluated with the actual ModelStatus in the reason; install via the Models pane and re-run.
• Eligible — loaded via VLMModelFactory.shared.loadContainer, the 20 tasks run through MLXInferenceLock.shared, and the tier is unloaded before the next one to avoid OOM.

The CLI itself is a thin shim. Real inference lives in MCPServer.MLTierEvalOrchestrator (which pulls in MLX); the everyday senkani binary stays MLX-free by shelling out to senkani-mcp eval.

Example

$ senkani ml-eval
[ml-eval] skip gemma4-26b-apex — insufficient RAM (8 GB; tier requires 16 GB)
[ml-eval] load gemma4-e4b (unsloth/gemma-4-E4B-it-UD-MLX-4bit)
[ml-eval]   gemma4-e4b: 16/20 (80% pass, excellent)
[ml-eval] load gemma4-e2b (unsloth/gemma-4-E2B-it-GGUF)
[ml-eval]   gemma4-e2b: 11/20 (55% pass, degraded)
[ml-eval] wrote /Users/you/.senkani/ml-tier-eval.json

$ senkani doctor | grep ml.tier
✓ ml.tier.gemma4-e4b: excellent (16/20, 80% pass, median 612ms, 1840 output tok)
✗ ml.tier.gemma4-e2b: degraded (11/20, 55% pass, median 388ms, 1100 output tok) — consider upgrading to a larger tier if RAM allows
- ml.tier.gemma4-26b-apex: notEvaluated — insufficient RAM (8 GB; tier requires 16 GB)

Flags