In the ever-evolving landscape of Artificial Intelligence (AI), generative AI has emerged as a key player, promising to revolutionize industries and business processes. However, it's not without its share of complexities and confusion. To shed light on this innovative technology, I committed to explore the latest breakthroughs and their applicability to Industrial AI with a set of ARC blogs, podcasts, Insights and Reports. However with the general confusion being created by generative AI myths and misconceptions, I've worked with the team at ARC to lay some AI foundations, including a glossary of terms, in our initial report on The Industrial AI (R)Evolution.

Industrial AI, a subset of the broader field of artificial intelligence (AI), refers to the application of AI technologies (including Generative AI) in industrial settings to augment the workforce in pursuit of growth, profitability, more sustainable products and production processes, enhanced customer service, and business outcomes. Industrial AI leverages machine learning, deep learning, neural networks, and other approaches. Some of these techniques have been used for decades to build AI systems using data from various sources within an industrial environment, such as sensors, machinery, industrial engineers, and frontline workers.

Among these AI techniques and technologies, generative AI has caught the attention of many, particularly within various industries such as Aerospace & Defense, Automotive, Electric Power & Smart Grid, Industrial Equipment, Oil & Gas, Semi-Conductors, and more.

Generative AI, powered by machine learning and neural networks used for decades in various Industrial AI use cases, but with new genuine breakthroughs in natural language processing (using GANs, transformers and LLMs), are revolutionizing how we interact with everything around us, whether those technologies are inherently smart, industrialized, or not.

However, this revolutionary technology often leads to confusion due to general media hype about AI, extravagant marketing claims from software suppliers struggling to get due credit for having invested in AI technologies long before the current wave of generative AI breakthroughs, its technical complexity and the rapid pace at which AI continues to evolve.

The current state of industrial AI presents a complex picture. On one hand, we have a myriad of AI solutions promising to revolutionize processes and boost efficiency. On the other hand, there is a lack of clarity regarding which technologies truly deliver on these promises.

One of the main challenges faced by organizations is discerning valuable AI breakthroughs from the hype. While many AI technologies have proven their worth, others are still emerging, and their long-term value remains uncertain.

Moreover, there are numerous myths and misconceptions surrounding which of these AI techniques and technologies relevant to industrial AI use cases. These include the belief that AI implementation requires massive upfront investment or that it will lead to widespread job displacement. Such misconceptions often deter organizations from exploring the potential benefits of AI.

To address this confusion, the ARC Advisory Group has embarked on a mission to simplify the complex, identify relevant breakthroughs, and cut through the hype surrounding industrial AI. Central to this mission is The Industrial AI (R)Evolution that cuts through the Generative AI hype, dispels myths and summarizes the latest developments and trends in the field.

The report covers a wide range of topics, including data governance, cybersecurity risks, high-value industrial AI use cases, and the societal impact of AI. Additionally, it dives into the intricacies of various AI techniques, including unsupervised, semi-supervised, supervised, and reinforcement learning, as well as Large Language Models (LLMs) and Foundation Models (FMs).

The Industrial AI Report also dispels myths and misconceptions. One common myth is that AI implementation requires substantial upfront investment. While initial costs can be high, the long-term benefits often outweigh these costs. Another prevalent misconception is that AI will eliminate jobs. However, while AI may automate certain tasks, it also creates new roles and opportunities.

It discusses the shift in priorities from Industrial Metaverse to Industrial AI, emphasizing the potential of AI to drive efficiency and innovation in industries. For more on this particular topic read my blog on how Industrial AI is paving the way for Industrial Metaverse(s).

Industrial organizations can leverage ARC's Industrial AI Impact Assessment Model used by ARC's own team of Analysts, to guide their own AI evaluation and implementation process. This model offers a structured approach to assess the potential impact of AI on various aspects of the organization, including operations, strategy, and workforce.

As we continue to explore the potential of generative AI and other breakthroughs in industrial AI, collaboration and knowledge sharing become increasingly important. We invite you to join us in this journey, sharing your questions, experiences, learnings, and solutions.

The future of Industrial AI is promising, with its potential to transform industries and societal structures. By deepening our understanding and effectively applying AI technologies, we can unlock their true potential in the industrial realm.

For more information or to contribute to Industrial AI research, please contact Colin Masson at cmasson@arcweb.com.

Excerpt from:

Industrial AI Challenges and the Path Forward - ARC Advisory Group

Advances in artificial intelligence (AI) are accelerating at breakneck speed. Systems like ChatGPT are approaching, and by some measures exceeding, human-level performance in many domains.

But what does the growth of these systems mean for schools?

We see tremendous potential for these technologies to enhance and augment human capabilities, making us smarter, more efficient and able to solve problems that currently seem impossible to manage.

However, we also see significant downsides. Without thoughtful intervention, AI could diminish human agency, stifle creativity and potentially stunt our collective progress.

Nowhere are the stakes higher than in education. Schools and universities have helped generations climb the ladder of knowledge and skills. But if machines can soon out-think us, whats the point of learning? Why invest time and effort acquiring expertise that could soon be obsolete?

To explore these questions, we recently co-authored a paper analysing the staggering pace of progress in AI and the potential implications for education.

Systems like GPT-4 are already scoring higher than well over 90 per cent of humans on academic tests of literacy and quantitative skills. Many experts predict AI will reach human-level reasoning across all domains in the next decade or two.

Once achieved, these artificial general intelligence systems could quickly exceed the combined brainpower of every person who has ever lived.

Faced with these exponential advances, how might society respond? We foresee four possible scenarios, all of which would have different implications for schools:

One option is that governments recognise the risks and halt further AI development, through regulation or restricting hardware supply. This might slow things down and buy some time.

Bans are hard to enforce, often porous and would mean that we would forfeit many potential benefits that carefully governed AI systems could bring. However, if AI advances get curtailed at, say, GPT-4.5 there is a greater chance that humanity stays in the driving seat and we still benefit from education.

In fact, with suitable guardrails, many of the recent AI advances might greatly accelerate our thinking skills, for example by providing high-quality supplementary AI tuition to all students and by acting as a digital personal assistant to teachers.

A second pathway is that AI takes over most jobs, but legislation forces companies to keep employing humans alongside the machines, in largely ceremonial roles. The risk here is that this fake work risks infantilising people.

As AI thinking accelerates, our stunted contributions could create bottlenecks, leaving us disempowered spectators rather than active participants.

This pathway also requires only a basic level of education - we would simply need to turn up and read out the script displayed in our AI glasses. After all, our own thinking and words would never exceed the abilities of the machines.

Wanting to remain competitive, some might opt to biologically or digitally upgrade their brains through gene editing or neural implants. This might sound like science fiction, but is not beyond the realm of possibility - and such a scenario would have profound implications for education.

We might be able to literally download new knowledge, skills and abilities in milliseconds. No more need for schooling.

But in making ourselves more machine-like, would we risk losing our humanity?

A final scenario is that we accept economic irrelevance and embrace universal basic income - paid for by taxing the fruits of AI labour. Freed from work, people would focus on sports, hobbies, rituals and human connections.

But devoid of productive purpose, might we lose our vital force and struggle to get out of bed in the morning?

All these paths are, in different ways, problematic. So, before we sleepwalk into one, we need urgent debate on the destination we want.

Our paper offers 13 pragmatic proposals to regulate and slow down AI, to buy time for this discussion by, for example: requiring frontier AI models to be government licensed before their release; making it illegal for systems to impersonate humans; implementing guardrails to stop AI systems from giving students the answers; and making system developers accountable for untruths, harms and bad advice generated by their systems.

At the same time, we must also re-examine educations role in society. If humans can add only marginal value working alongside AI, schools may need to pivot from preparation for employment to nurturing distinctly human traits: ethics, empathy, creativity, playfulness and curiosity.

As AI excels at information retrieval and analysis, we must double down on contextual reasoning, wisdom, judgement and morality. However, even here, we must be realistic that (eventually) AI is likely to be able to emulate all these human traits as well.

Some skills like literacy might also become less essential - for example, if we can learn through verbal discourse with AI or by porting into realistic simulations.

Yet foundational knowledge will likely remain crucial, enabling us to meaningfully prompt and critique AI. And direct instruction, whether by teacher or AI, will still help students to grasp concepts more quickly than trial-and-error discovery. We must, therefore, identify the irreducible core of timeless human competencies to pass on.

None of this is preordained. With vigilance, foresight and governance, AI can uplift humanity in the same way that prior innovations have. But we must act decisively. Timelines are highly uncertain. AI capabilities could exceed our own in a decade or two. Either way, the hinge point of history is now.

We hope these proposals stimulate urgent debate on the society and education system we want to build - before the choice is made for us.

Dylan Wiliam is emeritus professor of educational assessment at the UCL Institute of Education. John Hattie is emeritus laureate professor of education at the University of Melbourne. Arran Hamilton is group director, education, at Cognition Learning Group. His contributions had editorial support from Claude AI

Go here to read the rest:

What will the rise of AI really mean for schools? - TES

Large Language Models (LLMs) are smart enough to understand context. They can answer questions, leveraging their vast training data to provide coherent and contextually relevant responses, no matter whether the topic is astronomy, history or even physics. However, LLMs tend to hallucinate (deliver compelling yet false facts) when asked to answer questions outside the scope of their training data, or when they cant remember the details in the training data.

A new technique, Retrieval Augmented Generation (RAG), fills the knowledge gaps, reducing hallucinations by augmenting prompts with external data. Combined with a vector database (like MyScale), it substantially increases the performance gain in extractive question answering.

To this end, this article focuses on determining the performance gain with RAG on the widely-used MMLU dataset. We find that both the performance of commercial and open source LLMs can be significantly improved when knowledge can be retrieved from Wikipedia using a vector database. More interestingly, this result is achieved even when Wikipedia is already in the training set of these models.

You can find the code for the benchmark framework and this example here.

But first, lets describe Retrieval Augmented Generation (RAG).

Research projects aim to enhance LLMs like gpt-3.5 by coupling them with external knowledge bases (like Wikipedia), databases, or the internet to create more knowledgeable and contextually aware systems. For example, lets assume a user asks an LLM what Newtons most important result is. To help the LLM retrieve the correct information, we can search for Newtons wiki and provide the wiki page as context to the LLM.

This method is called Retrieval Augmented Generation (RAG). Lewis et al. in Retrieval Augmented Generation for Knowledge-Intensive NLP Tasks define Retrieval Augmented Generation as:

A type of language generation model that combines pre-trained parametric and non-parametric memory for language generation.

Moreover, the authors of this academic paper go on to state that they:

Endow pre-trained, parametric-memory generation models with a non-parametric memory through a general-purpose fine-tuning approach.

Note: Parametric-memory LLMs are massive self-reliant knowledge repositories like ChatGPT and Googles PaLM. Non-parametric memory LLMs leverage external resources that add additional context to parametric-memory LLMs.

Combining external resources with LLMs seems feasible as LLMs are good learners, and referring to specific external knowledge domains can improve truthfulness. But how much of an improvement will this combination be?

Two major factors affect a RAG system:

Both of these factors are hard to evaluate. The knowledge gained by the LLM from the context is implicit, so the most practical way to assess these factors is to examine the LLMs answer. However, the accuracy of the retrieved context is also tricky to evaluate.

Measuring the relevance between paragraphs, especially in question answering or information retrieval, can be a complex task. The relevance assessment is crucial to determine whether a given section contains information directly related to a specific question. This is especially important in tasks that involve extracting information from large datasets or documents, like the WikiHop dataset.

Sometimes, datasets employ multiple annotators to assess the relevance between paragraphs and questions. Using multiple annotators to vote on relevance helps mitigate subjectivity and potential biases that can arise from individual annotators. This method also adds a layer of consistency and ensures that the relevance judgment is more reliable.

As a consequence of all these uncertainties, we developed an open-sourced end-to-end evaluation of the RAG system. This evaluation considers different model settings, retrieval pipelines, knowledge base choices, and search algorithms.

We aim to provide valuable baselines for RAG system designs and hope that more developers and researchers join us in building a comprehensive and systematic benchmark. More results will help us disentangle these two factors and create a dataset closer to real-world RAG systems.

Note: Share your evaluation results at GitHub. PRs are very welcome!

In this article, we focus on a simple baseline evaluated on an MMLU (Massive Multitask Language Understanding Dataset), a widely used benchmark for LLMs, containing multiple-choice single-answer questions on many subjects like history, astronomy and economy.

We set out to find out if an LLM can learn from extra contexts by letting it answer multiple-choice questions.

To achieve our aim, we chose Wikipedia as our source of truth because it covers many subjects and knowledge domains. And we used the version cleaned by Cohere.aion Hugging Face, which includes 34,879,571 paragraphs belonging to 5,745,033 titles. An exhaustive search of these paragraphs will take quite a long time, so we need to use the appropriate ANNS (Approximate Nearest Neighbor Search) algorithms to retrieve relevant documents. Additionally, we use the MyScale database with the MSTG vector index to retrieve the relevant documents.

Semantic search is a well-researched topic with many models with detailed benchmarks available. When incorporated with vector embeddings, semantic search gains the ability to recognize paraphrased expressions, synonyms, and contextual understanding.

Moreover, embeddings provide dense and continuous vector representations that enable the calculation of meaningful metrics of relevance. These dense metrics capture semantic relationships and context, making them valuable for assessing relevance in LLM information retrieval tasks.

Taking into account the factors mentioned above, we have decided to use the paraphrase-multilingual-mpnet-base-v2 model from Hugging Face to extract features for retrieval tasks. This model is part of the MPNet family, designed to generate high-quality embeddings suitable for various NLP tasks, including semantic similarity and retrieval.

For our LLMs, we chose OpenAIs gpt-3.5-turbo and llama2-13b-chat with quantization in six bits. These models are the most popular in commercial and open-source trends. The LLaMA2 model is quantized by llama.cpp. We chose this 6-bit quantization setup because it is affordable without sacrificing performance.

Note: You can also try other models to test their RAG performance.

The following image describes how to formulate a simple RAG system:

Figure 1: Simple Benchmarking RAG

Note: Transform can be anything as long as it can be fed into the LLM, returning the correct answer. In our use case, Transform injects context into the question.

Our final LLM prompt is as follows:

```pythontemplate = ("The following are multiple choice questions (with answers) with context:""nn{context}Question: {question}n{choices}Answer: ")```

```python

template =

("The following are multiple choice questions (with answers) with context:"

"nn{context}Question: {question}n{choices}Answer: ")

```

Now lets move on to the result.

Our benchmark test results are collated in Table 1 below.

But first, our summarized findings are:

In these benchmarking tests, we compared performance with and without context. The test without context represents how internal knowledge can solve questions. Secondly, the test with context shows how an LLM can learn from context.

Note: Both llama2-13b-chat and gpt-3.5-turbo are enhanced by around 3-5% overall, even with only one extra context.

The table reports that some numbers are negative, for example, when we insert context into clinical-knowledge to gpt-3.5-turbo.

This might be related to the knowledge base, saying that Wikipedia does not have much information on clinical knowledge or because OpenAIs terms of use and guidelines are clear that using their AI models for medical advice is strongly discouraged and may even be prohibited. Despite this, the increase is quite evident for both models.

Notably, the gpt-3.5-turbo results claim that the RAG system might be powerful enough to compete with other language models. Some of the reported numbers, such as those on prehistory and astronomy are pushing towards the performance of gpt4 with extra tokens, suggesting RAG could be another solution to specialized Artificial General Intelligence (AGI) when compared to fine-tuning.

Note: RAG is more practical than fine-tuning models as it is a plug-in solution and works with both self-hosted and remote models.

Figure 2: Performance Gain vs. the Number of Contexts

The benchmark above suggests that you need as much context as possible. In most cases, LLMs will learn from all the supplied contexts. Theoretically, the model provides better answers as the number of retrieved documents is increased. However, our benchmarking shows that some numbers dropped the greater the contexts retrieved.

By way of validating our benchmarking results, a paper by Stanford University titled: Lost in the Middle: How Language Models Use Long Contexts suggests the LLM only looks at the contexts head and tail. Therefore, choose fewer but more accurate contexts from the retrieval system to augment your LLM.

The larger the LLM, the more knowledge it stores. Larger LLMs tend to have a greater capacity to store and understand information, which often translates to a broader knowledge base of generally understood facts. Our benchmarking tests tell the same story: the smaller LLMs lack knowledge and are hungrier for more knowledge.

Our results report that llama2-13b-chat shows a more significant increase in knowledge than gpt-3.5-turbo, suggesting context injects more knowledge into an LLM for information retrieval. Additionally, these results imply gpt-3.5-turbo was given information it already knows while llama2-13b-chat is still learning from the context.

Almost every LLM uses the Wikipedia corpus as a training dataset, meaning both gpt-3.5-turbo and llama2-13b-chat should be familiar with the contexts added to the prompt. Therefore, the questions that beg are:

We currently dont have any answers to these questions. As a result, research is still needed.

Contribute to research to help others.

We can only cover a limited set of evaluations in this blog. But we know more is needed. The results of every benchmark test matter, regardless of whether they are replications of existing tests or some new findings based on novel RAGs.

With the aim of helping everyone create benchmark tests to test their RAG systems, we have open sourced our end-to-end benchmark framework. To fork our repository, check out our GitHub page.

This framework includes the following tools:

Its up to you to create your own benchmark. We believe RAG can be a possible solution to AGI. Therefore, we built this framework for the community to make everything trackable and reproducible.

PRs are welcome.

We have evaluated a small subset of MMLU with a simple RAG system built with different LLMs and vector search algorithms and described our process and results in this article. We also donated the evaluation framework to the community and called for more RAG benchmarks. We will continue to run benchmarking tests and update the latest results to GitHub and the MyScale blog, so follow us on Twitter or join us on Discord to stay updated.

Here is the original post:

Discover the Performance Gain with Retrieval Augmented Generation - The New Stack

Photo by Milad Fakurian on Unsplash

Hey Futurists and AI Aficionados! Buckle up because were about to jump into a time machine and peek into the next decade of AI and Data Science. If you think what weve seen so far is mind-blowing, you aint seen nothin yet! So, lets put on our prediction hats and delve into the trends that will shape the AI landscape in the upcoming years.

First off, a reality check. AI has been the buzzword for years, and while weve made some amazing strides (Hello, self-driving cars and personalized medicine!), were not living in a sci-fi movie yet. No, robots arent taking over the world, but they are about to make our lives much more interesting!

Currently, most AI algorithms need robust servers and data centers to function. But what if that could change? Were looking at a future where AI models will run efficiently on your devices yes, your smartphone could soon be your AI assistant in a much more sophisticated way than Siri or Alexa could ever be!

Heres a big one: using AI to combat climate change. Algorithms are already getting better at predicting weather patterns, analyzing soil health, and tracking endangered species. But in the next decade, AI will play a crucial role in resource optimization and perhaps even in engineering solutions to reverse environmental damage. The planets heroes may be lines of code!

Currently, most AI falls under narrow or specialized intelligence good at one thing but pretty useless otherwise. However, were inching closer to Artificial General Intelligence (AGI), where machines can understand, learn, and apply knowledge across different domains. Imagine an AI that can compose music, diagnose diseases, and manage city traffic while teaching itself quantum physics!

Read the original:

What's Next in AI? Predicting the Trends for the Upcoming Decade - DataDrivenInvestor