Model selection needs task portfolios

Leaderboards are inputs, not deployment policy

The fastest way to make a bad AI product decision is to ask, “Which model is best?” without naming the work the model must do. A leaderboard can tell you that one model is strong on reasoning, another is faster, another is cheaper per task, and another performs well in coding or agentic workflows. That is useful input. It is not a deployment policy.

The right thesis is simple: AI model selection needs task portfolios, not leaderboard winners.

A product workflow is rarely one task. A support copilot may classify intent, retrieve policy, draft a response, check tone, detect escalation risk, and write a CRM note. A coding workflow may inspect a ticket, change code, run tests, explain the diff, and prepare a review packet. A research workflow may search, summarize, compare, cite, and flag uncertainty. Treating that bundle as one generic “LLM use case” hides the real choices.

Artificial Analysis is valuable precisely because it does not reduce model comparison to one universal score. Its public analysis lets teams compare models across intelligence, speed, cost per task, coding-agent performance, agentic capabilities, output behavior, and pricing dimensions, which makes it a better input for product selection than a single trophy ranking. See the Artificial Analysis model and provider analysis for the underlying comparison frame: Artificial Analysis.

But the product leader still has to translate those signals into operating rules. A benchmark result does not know your customer promise, your latency tolerance, your margin target, your failure cost, or your review capacity. Those live inside the product. That is why model selection should begin with a task portfolio.

What belongs in a task portfolio?

A task portfolio is a one-page operating artifact that decomposes an AI workflow into task classes. Each row describes one job the model performs and the constraints that make a model acceptable for that job. It is not a vendor comparison table. It is a product control surface.

Start with the task class. Is the model classifying, drafting, extracting, reasoning, coding, searching, ranking, transforming data, using tools, or making a recommendation? The label matters because different failures hurt in different ways. A slow creative draft may be acceptable. A slow checkout assistant may kill conversion. A confident but false compliance answer may create risk. A mediocre internal summary may only create rework.

Then write the quality threshold. This cannot be “good enough.” It should be observable. For a classifier, the threshold might be minimum precision for high-risk categories. For extraction, it might be field-level accuracy on required fields. For a coding agent, it might be passing tests plus a human review checklist. This connects naturally to the operating principle in Inference, context, evals: AI principles before models: the team needs to know what it is evaluating before it can choose what to run.

Next add the latency ceiling. Some tasks can wait. Others sit inside an interactive path where every extra second changes behavior. A model with stronger benchmark performance may still be wrong for an inline UX if its response time breaks the flow. Conversely, a background reconciliation task may deserve a slower, stronger model if it prevents manual cleanup later.

Add cost per task, not just price per token. Token price is only part of the expense. Long prompts, retries, tool calls, reasoning tokens, cache behavior, and review time can change the actual unit economics. Cost per task forces the team to ask, “What does one completed product action cost us?” That is the number a product owner can compare against margin, support savings, conversion lift, or operational capacity.

Finally, add fallback, owner, and review cadence. The fallback says what happens when the primary model is unavailable, too slow, too expensive, or below threshold. The owner says who is accountable for the row. The review cadence says when the choice is revisited. Without those three fields, the portfolio becomes documentation. With them, it becomes governance.

The matrix beats the model debate

The model debate usually starts with preferences. One engineer trusts a provider. Another wants the latest frontier model. Finance wants lower cost. Design wants faster responses. Legal wants fewer uncontrolled outputs. Every group is partly right, but the conversation is framed at the wrong level.

The matrix changes the unit of argument. Instead of debating one model for the whole product, the team debates fit per task class. Drafting a help-center reply may require tone and context handling. Escalation detection may require conservative classification. A retrieval answer may require source discipline and refusal behavior. A code change may require tool use, test execution, and review evidence.

This is also where model routing becomes a consequence rather than a trend. In After vibe coding: model routing makes AI coding governable, routing is useful because different coding activities need different controls. The same logic applies outside coding. You do not route models because it sounds advanced. You route because the task portfolio proves that one model is not optimal for every row.

The matrix should expose tradeoffs, not hide them. If the best-quality model is too slow for an interactive task, write that down. If the cheapest model increases review burden, write that down. If a small model is fine for extraction but unsafe for final recommendations, write that down. The portfolio is useful when it makes disagreement concrete.

A practical row can look like this: task class, user-facing or internal, quality threshold, latency ceiling, cost ceiling, risk level, primary model, fallback model, monitoring signal, human review trigger, owner, next review date. That is enough structure to prevent leaderboard theater without turning selection into a six-month architecture program.

Review model choice like a product bet

Model choice decays. New models arrive, prices change, providers improve latency, prompts grow, workflows expand, and user behavior shifts. A model that was right three months ago may still be right today, but the team should know why.

Reviewing model selection like a product bet means setting decision rules before the team changes models. What evidence justifies a switch? Lower cost at equal quality? Better quality within the same latency ceiling? Fewer escalations? Less human review time? Higher task completion? If the answer is “a new leaderboard came out,” the team is reacting to market noise, not operating evidence.

This does not mean benchmarks are bad. Benchmarks are essential scouting tools. They help teams discover candidates, understand tradeoffs, and avoid testing every model from scratch. The problem starts when a benchmark becomes a proxy for product truth. The product truth is the task portfolio plus observed performance in your workflow.

For high-risk rows, review should include sampled outputs and failure analysis. For high-volume rows, review should include cost per completed task and latency distribution. For agentic rows, review should include tool errors, incomplete actions, and handoff quality. For customer-facing rows, review should include user correction, abandonment, escalation, and complaint signals.

The portfolio also protects the team from fashionable overkill. Not every row deserves the most capable model. Some tasks need reliability, speed, and predictable cost more than maximum reasoning. Others deserve a premium model because one bad answer is expensive. The portfolio lets both statements be true at the same time.

Audit one workflow before changing models

If your team is about to switch models, pause for one hour. Pick one AI-enabled workflow that already matters to the business. Write every model-mediated task in sequence. For each task, fill in quality, latency, cost, fallback, owner, and review threshold. Do not start with model names. Start with the work.

Then compare candidates against the rows. Use external analysis to shortlist options, internal evals to test fit, production telemetry to detect drift, and human review to inspect high-risk failures. The output should not be “we chose Model X.” The output should be “for this task, under these constraints, we use this model, with this fallback, until this review condition changes.”

That sentence is boring in the best possible way. It turns AI model selection from preference into product operations. It gives engineering a target, finance a unit-cost view, risk teams a review trigger, and product leaders a way to explain why different parts of the workflow use different models.

Leaderboard winners will keep changing. Your task portfolio is how you decide which changes matter.