Establishing Industry-Wide Standards and Benchmarks for Legal GenAI

Would you purchase a legal generative artificial intelligence product with no clear performance metrics, requiring your department or firm to conduct extensive testing to develop its own benchmarks? Unfortunately, according to Artificial Lawyer’s founder Richard Foman, there are currently no performance metrics or shared standards for AI tools. The remedy would be establishing industry-wide benchmarks for legal genAI. 

These benchmarks would provide clear performance metrics based on shared standards, making it easier for legal departments and law firms to evaluate products without extensive internal testing. Vendors could transparently share their product’s performance against these benchmarks, fostering trust.

Here are four scenarios that could be adopted if various parties agree on standards and benchmarks:

Scenario 1: Constructive conversation. Industry participants share ideas and agree to be transparent about their testing, leading to a general improvement in knowledge and practices without formal agreements.

Scenario 2: Formal agreement. Parties agree on specific benchmarks and standards, sharing information and committing to transparency. This collaboration might be sustained by a network rather than a regulatory or standards body.

Scenario 3: Formal body. An organization takes responsibility for maintaining standards, promoting transparency, and potentially acting as a testing center, supported by market participants.

Scenario 4: Regulatory body. A regulatory body enforces the standards with penalties for non-compliance, ensuring uniform adherence to the benchmarks.

LITIG, a UK-based group of legal innovation and IT experts, announced a project to address these challenges, supported by Artificial Lawyer and Legal IT Insider. This initiative aims to establish shared standards and benchmarks for legal GenAI tools. The project’s success remains to be seen, but it is a crucial step towards greater transparency and efficiency in adopting generative AI technologies in legal departments and law firms.