AI-Powered Multilingual Flashcards: A Case Study

A dynamic educational technology company approached us to enhance their AI-driven flashcard application. Our partnership focused on enhancing their language model (LLM) processes using the latest advancements in AI technology to improve the user experience and reduce operational costs.

Project Scope and Technological Implementation

The project’s scope was to optimize the generation of educational flashcards using a large language model (LLM) powered by ChatGPT and using a professional workflow. The application allows users to upload any type of content, including scanned documents, PDFs, or other forms of documents, which are processed using OCR and other technologies. Then, utilizing LLM, the learning flashcards are generated automatically according to the user’s specific needs.

There are three main operational modes:

• User provides questions and the LLM extracts the associated answers based on the documents
• User provides answers and an LLM defines a relevant question
• Both the questions and answers are extracted from the documents.

Challenges Addressed

The primary challenges were:

• Cost Reduction: Because the application employs a fixed monthly subscription structure, it is essential to keep the cost per user at an appropriate level, thus, it’s crucial to solve the LLM task with the most efficient utilization of the LLM capabilities.

• Flashcard Format Compliance: Ensuring that the AI-generated content adhered to specific flashcard format requirements such as length and question-answer style.

• Fact-Based Response Generation: Maintaining accuracy and relevance in AI responses is also crucial for such an application, avoiding the often biased hallucination effect of the LLM.

• Multilingual Support: Ensuring the system could operate in 30 languages while managing text recognition errors from OCR-processed documents.

Implementation and Results

We deployed a sophisticated prompt engineering and optimization pipeline, which included the following tasks:

Quantifiable Optimization Workflow

• Establishing an objective and comprehensive evaluation that could enable a KPI based co-optimization of multiple objectives.
  • LLM workflows are often hard to evaluate, but without hard metrics evaluation both optimization and testing can be a mess. Building trust in your AI-based system is crucial for any production application.
  • As there is no single good solution for comparing and scoring language generations as they are inherently subjective and context-dependent
  • We implemented a robust benchmarking system built up from complementary metrics(objective and subjective), and comprehensive test-corpus.
    • objective metrics we employed, which can be automatized and factually evaluate the test-corpus
      • logic-based evaluation 
      • syntactic validation of the formal requirements
    • subjective metrics
      • Model-based-Evaluation, harnessing LLM as a judge to semantically compare against an expected output
      • human-in-the-loop evaluation for the complex generations, and annotating hard-to-define anomalous behaviors
    • comprehensive test-corpuses
      • based on usage statistics we clustered test-pairs to the well defined test-sets of for both the diverse normal usage patterns and hard/anomalous patterns (like ambiguous uses, typos, adversarial prompting).
  • We made the evaluation process an integral part of the optimization and delivery workflow.

Prompt Engineering

With the language models, although at first glance they look like natural speech, special formulas often significantly increase the quality and consistency of the outputs.

• Few-shot examples prompting: some content requirements are too complex to define with strict rules. In these cases it’s beneficial to show some examples of those with the associated expected answer
• Chain-of-thought prompting: this enables complex reasoning capabilities through intermediate steps, guiding the LLM to break down solutions into smaller tasks and solve them step-by-step.

Adversarial Prompting

In any production LLM application, we should expect a malicious use. We employed multiple lines of defenses against user inputs that attempt to divert the AI from its defined tasks as:

• Jailbreaking: Hijacking the LLM to solve unintended tasks and using preventive measures to detect known attack vectors.
• Prompt Leaking: Prompts in production applications are akin to intellectual property like source code. Therefore, it was crucial to build defenses into the prompt to protect the system against adversarial attempts.

Cost Efficiency

By employing prompting techniques instead of custom fine-tuned models, we could make a 30% reduction in costs.  

The outcomes were highly satisfactory. The optimization led to considerable improvements in the quality of the responses, which consequently minimized the necessity to regenerate suboptimal responses or utilize costlier LLMs.

Client Feedback and Continued Collaboration

The client was thoroughly pleased with the project’s outcome, praising the effective collaboration and the tangible improvements in their service. The successful partnership has kept the doors open for future collaborations, with ongoing advisory roles in AI and language model applications.

The graph depicts the benchmarks of both the initial (blue) and improved (orange) prompt versions across our diverse test corpora.