A Model Card, Questioned

In practice

The card is professional and detailed, and the team reads it in the order the critical-reading procedure sets. First, intended use. The card lists summarizing commercial contracts in English as the primary use, and lists legal advice and non-English documents as out of scope. The team's contracts are in English, so they continue, but they note that a quarter of their supplier contracts have sections in Spanish. Second, caveats. The card warns that performance falls on documents longer than about fifty pages and on scanned copies. Many of the team's contracts are long scans. Third, factors and metrics. The card reports a high accuracy figure, measured on a public collection of short commercial agreements. It says nothing about long documents or about clauses like the team's most important ones, on liability and data handling. Fourth, what is missing. There is no breakdown by document length or by clause type, and no description of how accuracy was judged. The team writes down the gaps, then tests the tool on twenty of its own contracts, chosen to include long scans and mixed-language sections. The result matches the caveats: summaries of short, clean contracts are good, while the long scans miss clauses, including two liability caps. The decision is to approve the tool for short contracts only, require the full contract to be checked whenever liability or data handling is involved, and ask the vendor for clause-level results.