Efficient AI Optimization with the mgm AI Evaluation Framework

The quality and performance of AI assistants play a critical role in their success in real-world applications. But how can companies accurately assess whether a large language model (LLM) meets their specific business needs? mgm technology partners has developed the AI Evaluation Framework – a systematic approach to objectively evaluate and optimize AI results.

Why an AI Evaluation Framework?

Organizations are deploying AI assistants to streamline business processes, but different use cases require customized model adjustments. mgm’s AI Evaluation Framework enables organizations to test AI models, evaluate their performance, and continually refine them. This data-driven approach helps determine whether a more powerful but more expensive model is a worthwhile investment.

The Four-Step Evaluation Process

1. Define the use case: Determine what information the AI will extract from unstructured data. Example: In the D&O insurance quoting process, AI analyzes emails to extract key details for premium calculations (coverage amount, deductible, etc.).
2. Collect sample data: Companies provide test data, such as 50 sample emails with target values. Data anonymization is optional.
3. Customize quality metrics: Error assessment and completeness are individually weighted. A revenue miscalculation may be more critical than an incorrect street name.
4. Perform evaluation: Automated tests compare different AI models and ask for variations. A detailed report highlights areas for improvement.

Weight quality metrics via drag & drop with a focus on premium calculation

Automated Tests & Structured Results

The test environment operates fully automatically:

• Data Input: Companies define requirements in an Excel file, specifying models, prompts, and expected outcomes.
• Upload: Test data is uploaded via API.
• Processing: The framework analyzes different model variations and evaluates their results.
• Output: A detailed Excel report with quality metrics visualizes strengths and weaknesses, including charts for easy comparison (see below: “AI Evaluation with Grading System”).

Integration with the mgm AI Assistant

The AI Evaluation Framework seamlessly integrates with the mgm AI Assistant, available as an Outlook add-in or via API. This assistant processes incoming emails, extracts relevant information, and presents it in a structured format. An interactive chat function enables users to work directly with extracted data, helping automate quoting processes and improve business workflows.

Transparency Through a Grading System

A key feature of the AI Evaluation Framework is its grading methodology. Results are not only assessed for syntactic or semantic accuracy but also weighted based on their relevance to business processes. For example, an insurance coverage amount holds more significance than a misidentified salutation. This creates a transparent foundation for optimizing AI models.

AI Evaluation with Grading System: Business Relevance Matters More Than Pure Accuracy.

Looking Ahead: More Automation

mgm is working on further automating the process. Customers provide test data, and mgm handles the analysis, delivering precise evaluations with minimal effort. In the future, businesses could determine within days whether an AI solution delivers real value to their operations.

Conclusion

The mgm AI Evaluation Framework offers a structured approach for assessing and refining AI applications. Automated testing, grading-based evaluation, and seamless integration make the AI Assistant a powerful tool for data-driven business processes. Companies benefit from more accurate data processing, streamlined workflows, and clear decision-making support for AI investments.