evalship - 2026 Pricing, Features, Reviews & Alternatives

evalship is a cloud-based GitHub App that brings LLM evaluation results directly into the pull request review process. Hosted on Railway and integrated via the GitHub Apps platform, evalship posts eval summaries as PR comments and check runs, tracking pass/fail status, scores, thresholds, duration, cost, and token usage per pull request. It automatically detects changed LLM calls and prompt files per PR and maps which evals cover them, making coverage gaps visible at review time without requiring engineers to consult external dashboards.

evalship is designed for software development teams and AI/ML engineering teams building LLM-powered products, and suits organizations of any size that run automated evals as part of their CI/CD workflow. On each pull request run that affects prompts, LLM calls, eval code or data, or introduces newly failing evals, a language model is invoked to summarize the change's impact on evals, identify failure patterns and likely causes, and indicate which review findings the diff resolves. The model receives only PR metadata, relevant file diffs, audit data, and up to 20 newly failing eval inputs and outputs; no model is run over the full repository. Eval artifacts can be uploaded from CI workflows, and behavior is configurable via a .evalship.yml file.

evalship also supports agent audits, where a coding agent running locally submits a structured audit of its reasoning and citations, which evalship verifies against the repository. This produces a verifiable record of agent decision-making. Source code is not stored; only diffs and file locations are retained. Model calls use providers operating under no-training and short-retention data policies. Additional capabilities include trend charts for pass rates and scores over time, per-repository public report settings and badge tokens, and team member access tracking per private repository.