Tips for Vetting Vendors About GenAI Solutions for Legal Ops

Although generative AI (GenAI) offers significant benefits to corporate legal departments, there are critical concerns about its accuracy, reliability, data privacy, and security. That’s why it is important that Legal Ops carefully approaches vetting vendors about GenAI solutions to ensure trustworthy outputs and secure data handling. An article by Wolters Kluwer highlights three core questions to ask vendors.

How do you get your data? When evaluating GenAI solutions, Legal Ops should inquire about vendors’ data sources. GenAI relies on large language models (LLMs) trained on vast datasets. Ideally, vendors should use their in-house content and data or your company’s internal data to train their models, minimizing exposure to external risks. In addition, LLMs should incorporate diverse, external data from trusted, known sources that have been vetted to enhance accuracy and reduce bias.

How do you maintain your models? Maintaining the efficacy of AI models requires continuous monitoring, testing, and updates. To ensure current and relevant data feeds the models, vendors must have robust frameworks for data cleansing, refinement, and structuring. They should also have feedback loops in place to vet and approve model outputs and make recommendations for improvement. Regular audits by third parties can further ensure model accuracy and impartiality.

How will you guarantee that my data won’t be compromised? Vendors using their internal data sources or anonymized external data significantly reduce the risk of compromising client data. It is critical to select an established AI vendor with a proven track record in data governance, rather than new entrants to the field that lack the experience and frameworks allowing them to build reliable, trusted, secure and proven AI solutions.

By asking these targeted questions about data sources, model maintenance, and data security, Legal Ops can mitigate risks associated with GenAI and choose solutions that are both secure and effective.