The evaluation of explainable AI (XAI) methods currently faces significant challenges primarily due to a lack of standardization. Existing metrics are often inconsistently defined, incompletely reported, and rarely validated against common baselines, hindering progress and reliability in XAI research.
To address this critical issue, researchers have identified insufficient transparency in evaluation reporting as a central, under-addressed problem. Their proposed solution is the "XAI Evaluation Card." This documentation template is analogous to "model cards" in AI, designed to accompany any study that introduces a new XAI evaluation metric.
The XAI Evaluation Card mandates the explicit declaration of several key aspects: target properties (what the metric aims to measure), grounding levels (the underlying concepts the metric relies on), metric assumptions, validation evidence (how the metric's effectiveness is demonstrated), potential "gaming" risks (ways the metric might be exploited or mislead), and known failure cases. The researchers argue that adopting this template as a community norm would significantly reduce evaluation fragmentation, support meta-analysis, and improve accountability and credibility in XAI research.
