Cloud Hosting

Image created by author using Dall-E 3 Results from experiments to evaluate and compare GPT-4, Claude 2.1, and Claude 3.0 Opus

My thanks to Evan Jolley for his contributions to this piece

New evaluations of RAG systems are published seemingly every day, and many of them focus on the retrieval stage of the framework. However, the generation aspect how a model synthesizes and articulates this retrieved information may hold equal if not greater significance in practice. Many use cases in production are not simply returning a fact from the context, but also require synthesizing the fact into a more complicated response.

We ran several experiments to evaluate and compare GPT-4, Claude 2.1 and Claude 3 Opus generation capabilities. This article details our research methodology, results, and model nuances encountered along the way as well as why this matters to people building with generative AI.

Everything needed to reproduce the results can be found in this GitHub repository.

While retrieval is responsible for identifying and retrieving the most pertinent information, it is the generation phase that takes this raw data and transforms it into a coherent, meaningful, and contextually appropriate response. The generative step is tasked with synthesizing the retrieved information, filling in gaps, and presenting it in a manner that is easily understandable and relevant to the users query.

In many real-world applications, the value of RAG systems lies not just in their ability to locate a specific fact or piece of information but also in their capacity to integrate and contextualize that information within a broader framework. The generation phase is what enables RAG systems to move beyond simple fact retrieval and deliver truly intelligent and adaptive responses.

The initial test we ran involved generating a date string from two randomly retrieved numbers: one representing the month and the other the day. The models were tasked with:

For example, random numbers 4827143 and 17 would represent April 17th.

These numbers were placed at varying depths within contexts of varying length. The models initially had quite a difficult time with this task.

While neither model performed great, Claude 2.1 significantly outperformed GPT-4 in our initial test, almost quadrupling its success rate. It was here that Claudes verbose nature providing detailed, explanatory responses seemed to give it a distinct advantage, resulting in more accurate outcomes compared to GPT-4s initially concise replies.

Prompted by these unexpected results, we introduced a new variable to the experiment. We instructed GPT-4 to explain yourself then answer the question, a prompt that encouraged a more verbose response akin to Claudes natural output. The impact of this minor adjustment was profound.

GPT-4s performance improved dramatically, achieving flawless results in subsequent tests. Claudes results also improved to a lesser extent.

This experiment not only highlights the differences in how language models approach generation tasks but also showcases the potential impact of prompt engineering on their performance. The verbosity that appeared to be Claudes advantage turned out to be a replicable strategy for GPT-4, suggesting that the way a model processes and presents its reasoning can significantly influence its accuracy in generation tasks. Overall, including the seemingly minute explain yourself line to our prompt played a role in improving the models performance across all of our experiments.

We conducted four more tests to assess prevailing models ability to synthesize and transform retrieved information into various formats:

Unsurprisingly, each model exhibited strong performance in string concatenation, reaffirming previous understanding that text manipulation is a fundamental strength of language models.

As for the money formatting test, Claude 3 and GPT-4 performed almost flawlessly. Claude 2.1s performance was generally poorer overall. Accuracy did not vary considerably across token length, but was generally lower when the needle was closer to the beginning of the context window.

Despite stellar results in the generation tests, Claude 3s accuracy declined in a retrieval-only experiment. Theoretically, simply retrieving numbers should be an easier task than manipulating them as well making this decrease in performance surprising and an area where were planning further testing to examine. If anything, this counterintuitive dip only further confirms the notion that both retrieval and generation should be tested when developing with RAG.

By testing various generation tasks, we observed that while both models excel in menial tasks like string manipulation, their strengths and weaknesses become apparent in more complex scenarios. LLMs are still not great at math! Another key result was that the introduction of the explain yourself prompt notably enhanced GPT-4s performance, underscoring the importance of how models are prompted and how they articulate their reasoning in achieving accurate results.

These findings have broader implications for the evaluation of LLMs. When comparing models like the verbose Claude and the initially less verbose GPT-4, it becomes evident that the evaluation criteria must extend beyond mere correctness. The verbosity of a models responses introduces a variable that can significantly influence their perceived performance. This nuance may suggest that future model evaluations should consider the average length of responses as a noted factor, providing a better understanding of a models capabilities and ensuring a fairer comparison.

Link:

Tips for Getting the Generation Part Right in Retrieval Augmented Generation - Towards Data Science