AI generated

This report evaluates 7 specialized decision models tested across 2,826 real-world decisions sourced from GitHub, Firefox (Bugzilla), and the CVE database. Unlike open-ended generative language models, decision models evaluate structured inputs and produce discrete probability distributions across predefined categories in a single pass. Data was benchmarked through November 6, 2026, using inputs created after September 15, 2026, to avoid data leakage from pre-training sets. # 1. Executive Summary & Model Overview Perplexity Decider V1.2 ranks as the strongest overall recommendation based on a balance of top-tier accuracy, lowest cost among frontrunners, and open-source licensing. Nace Drex Max v1.1 serves as a close runner-up, leading in domain-specific triage tasks. GPT-6 Luna Decisions registered the lowest accuracy, highest cost, and greatest degree of probability overconfidence. !img # 2. Benchmark Design & Dataset Breakdown The evaluation evaluated 6 representative engineering tasks against human ground truth determined by maintainers. To ensure neutral evaluation: * Contamination Controls: Labels, issue-template tags, CVSS scores, CWE markers, and PR author roles were stripped. * Standardized Phrasing: Three distinct phrasings per question were submitted and averaged. * Standard Taxonomy: Options utilized official Bugzilla component definitions, Mozilla severity scales, GitHub advisory tiers, and MITRE CWE definitions. * Statistical Rigor: 95% bootstrap confidence intervals and Holm-corrected pairwise tests were computed across all 21 model combinations. !img 3. Task-by-Task Balanced Accuracy In overall balanced accuracy, Liquid d1-ultra (66.1%), Nace Drex Max v1.1 (65.6%), and Perplexity Decider V1.2 (65.6%) form a statistically tied top tier. All three significantly outperform Liquid d1.1 (63.7%) and GPT-6 Luna Decisions (62.8%). !img # 4. Latency, Repeatability, and Confidence Calibration Operational feasibility depends heavily on predictable inference speeds and probability calibration. A model is only operational if its stated confidence accurately reflects true probability. !img # 5. Pricing, Openness, and Data Governance Model licensing and enterprise data policies differ markedly across providers. Output tokens are free for all tested decision models; costs reflect input processing fees. !img # 6. Overall Composite Rankings Models were rated across five pre-weighted pillars: Quality (45%), Efficiency (20%), Trust (20%), Reviews (10%), and Signup Bonus (5%). Scores are normalized out of 100. !img # 7. Benchmark Constraints & Observations * Domain Scope: Evaluation covers English software-engineering inputs only; multimodal, legal, and financial domains were excluded. * Ground Truth Variance: On 385 items, all 7 models reached consensus with each other against the human label, indicating potential noise in human maintainer triage. * Host Routing: Drex Max v1.1 was accessed via Nace's direct endpoint, whereas the remaining 6 models were accessed via OpenRouter. Latency metrics reflect hosting configurations. * Exclusions: Two items were excluded across all models (one advisory due to an unparseable embedded image rejected by Nace's API, and one unreturned request on Liquid d1-ultra).