AI demos are designed to impress. Useful evaluation starts when the demo ends.
Watch for these signals: the happy path is the only path shown; pricing is vague until you ask; switching costs are never mentioned; evaluation data lives only in the vendor's sandbox; and every competitor is framed as obsolete without a shared task.
A cleaner approach is a small reproducible harness: same inputs, same scoring, written failure notes. We keep a short comparison framework for that here: https://www.toolbit.ai/compare
If a demo cannot survive that process, it is not ready for a team commit.