If an AI output is hard to evaluate, the problem may not be the eval.
It may be the product boundary.
I have seen teams blame evaluation when the real issue was that the system produced one large, polished artifact nobody could inspect safely.
That pattern will not scale with AI. If a tool generates a strategy, a code change, a customer reply, an analysis, or a recommendation, the user needs to know what part can be trusted, what evidence supports it, and where judgment is still required. "It's hard to eval" is a useful product smell, because it forces a sharper question: did we design the work in a way that can be checked? A good AI product does not only produce output — it lowers the cost of trusting or rejecting that output. Decompose the work into pieces a human can accept or reject one at a time. Attach the source. Make the boundary between what the system decided and what it inferred visible. The output that cannot be verified cheaply will either be trusted blindly or ignored — and both are failures.
The easier it is to verify, the safer it is to delegate.
AI product quality is not only answer quality. It is verification quality.