Companies like Morgan Stanley are already laying the groundwork for so-called organizational AGI. Maxim Tolchinskiy/Unsplash

Whether its being theorized or possibly, maybe actualized, artificial general intelligence, or AGI, has become a frequent topic of conversation in a world where people are now routinely talking with machines. But theres an inherent problem with the term AGIone rooted in perception. For starters, assigning intelligence to a system instantly anthropomorphizes it, adding to the perception that theres the semblance of a human mind operating behind the scenes. This notion of a mind deepens the perception that theres some single entity manipulating all of this human-grade thinking.

This problematic perception is compounded by the fact that large language models (LLMs) like ChatGPT, Bard, Claude and others make a mockery of the Turing test. They seem very human indeed, and its not surprising that people have turned to LLMs as therapists, friends and lovers (sometimes with disastrous results). Does the humanness of their predictive abilities amount to some kind of general intelligence?

By some estimates, the critical aspects of AGI have already been achieved by the LLMs mentioned above. A recent article in Noema by Blaise Agera Y Arcas (vice president and fellow at Google Research) and Peter Norvig (a computer scientist at the Stanford Institute for Human-Centered A.I.) argues that todays frontier models perform competently even on novel tasks they were not trained for, crossing a threshold that previous generations of A.I. and supervised deep learning systems never managed. Decades from now, they will be recognized as the first true examples of AGI.

For others, including OpenAI, AGI is still out in front of us. We believe our research will eventually lead to artificial general intelligence, their research page proclaims, a system that can solve human-level problems.

Whether nascent forms of AGI are already here or are still a few years away, its likely that businesses attempting to harness these powerful technologies might create a miniature version of AGI. Businesses need technology ecosystems that can mimic human intelligence with the cognitive flexibility to solve increasingly complex problems. This ecosystem needs to orchestrate using existing software, understand routine tasks, contextualize massive amounts of data, learn new skills, and work across a wide range of domains. LLMs on their own can only perform a fraction of this workthey seem most useful as part of a conversational interface that lets people talk to technology ecosystems. There are strategies being used right now by leading enterprise companies to move in this direction toward something we might call organizational AGI.

There are legitimate reasons to be wary of yet another unsolicited tidbit in the A.I. terms slush pile. Regardless of what we choose to call the eventual outcome of these activities, there are currently organizations using LLMs as an interface layer. They are creating ecosystems where users can converse with software through channels like rich web chat (RCW), obscuring machinations happening behind the scenes. This is difficult work, but the payoff is huge: rather than pogo-sticking between apps to get something done on a computer, customers and employees can ask technology to run tasks for them. Theres the immediate and tangible benefit of people eliminating tedious tasks from their lives. Then theres the long term benefit of a burgeoning ecosystem where employees and customers are interacting with digital teammates that can perform automations leveraging all forms of data across an organization. This is an ecosystem that starts to take the form of a digital twin.

McKinsey describes a digital twin as a virtual replica of a physical object, person, or process that can be used to simulate its behavior to better understand how it works in real life. They elaborate to say that a digital twin within an ecosystem similar to what Ive described can become an enterprise metaverse, a digital and often immersive environment that replicates and connects every aspect of an organization to optimize simulations, scenario planning and decision making.

With respect to what I said earlier about anthropomorphizing technology, the digital teammates within this kind of ecosystem are an abstraction, but I think of them as intelligent digital workers, or IDWs. IDWs are analogous to a collection of skills. These skills come from shared libraries, and skills can be adapted and reused in multitudes of ways. Skills are able to take advantage of all the information piled up inside the organization, with LLMs mining unstructured data, like emails and recorded calls.

This data becomes more meaningful thanks to graph technology, which is adept at creating indexes of skills, systems and data sources. Graph goes beyond mere listing and includes how these elements relate to and interact with each other. One of the core strengths of graph technology is its ability to represent and analyze relationships. For a network of IDWs, understanding how different components are interlinked is crucial for efficient orchestration and data flow.

Generative tools like LLMs and graph technology can work together in tandem, to propel the journey toward digital twinhood, or organizational AGI. Twins can encompass all aspects of the business, including events, data, assets, locations, personnel and customers. Digital twins are likely to be low-fidelity at first, offering a limited view of the organization. As more interactions and processes take place within the org, however, the fidelity of the digital twin becomes higher. An organizations technology ecosystem not only understands the current state of the organization. It can also adapt and respond to new challenges autonomously.

In this sense every part of an organization represents an intelligent awareness that comes together around common goals. In my mind, it mirrors the nervous system of a cephalopod. As Peter Godfrey-Smith writes in his book, Other Minds (2016, Farrar, Straus and Giroux), in an octopus, the majority of neurons are in the arms themselvesnearly twice as many in total as in the central brain. The arms have their own sensors and controllers. They have not only the sense of touch but also the capacity to sense chemicalsto smell or taste. Each sucker on an octopuss arm may have 10,000 neurons to handle taste and touch. Even an arm that has been surgically removed can perform various basic motions, such as reaching and grasping.

A world teeming with self-aware brands would be quite hectic. According to Gartner, by 2025, generative A.I. will be a workforce partner within 90 percent of companies worldwide. This doesnt mean that all of these companies will be surging toward organizational AGI, however. Generative A.I., and LLMs in particular, cant meet an organizations automation needs on its own. Giving an entire workforce access to GPTs or Copilot wont move the needle much in terms of efficiency. It might help people write better emails faster, but it takes a great deal of work to make LLMs reliable resources for user queries.

Their hallucinations have been well documented and training them to provide trustworthy information is a herculean effort. Jeff McMillan, chief analytics and data officer at Morgan Stanley (MS), told me it took his team nine months to train GPT-4 on more than 100,000 internal documents. This work began before the launch of ChatGPT, and Morgan Stanley had the advantage of working directly with people at OpenAI. They were able to create a personal assistant that the investment banks advisors can chat with, tapping into a large portion of its collective knowledge. Now youre talking about wiring it up to every system, he said, with regards to creating the kinds of ecosystems required for organizational A.I. I dont know if thats five years or three years or 20 years, but what Im confident of is that that is where this is going.

Companies like Morgan Stanley that are already laying the groundwork for so-called organizational AGI have a massive advantage over competitors that are still trying to decide how to integrate LLMs and adjacent technologies into their operations. So rather than a world awash in self-aware organizations, there will likely be a few market leaders in each industry.

This relates to broader AGI in the sense that these intelligent organizations are going to have to interact with other intelligent organizations. Its hard to envision exactly what depth of information sharing will occur between these elite orgs, but over time, these interactions might play a role in bringing about AGI or singularity, as its also called.

Ben Goertzel, the founder of SingularityNET and the person often credited with creating the term, makes a compelling case that AGI should be decentralized, relying on open-source development as well as decentralized hosting and mechanisms for interconnect A.I. systems to learn from and teach on another.

SingularityNETs DeAGI Manifesto states, There is a broad desire for AGI to be ethical and beneficial for all humanity; the most straightforward way to achieve this seems to be for AGI to grow up in the context of serving and being guided by all humanity, or as good an approximation as can be mustered.

Having AGI manifest in part from the aggressive activities of for-profit enterprises is dicey. As Goertzel pointed out, You get into questions [about] who owns and controls these potentially spooky and configurable human-like robot assistants and to what extent is their fundamental motivation to help people as opposed to sell people stuff or brainwash people into some corporate government media advertising order.

Theres a strong case to be made that an allegiance to profit will be the undoing of the promise for humanity at large that these technologies afford. Weirdly, the skynet scenario in Terminatorwhere a system becomes self-aware, determines humanity is a grave threat, and exterminates all lifeassumes that the system, isolated to a single company, has been programmed to have a survival instinct. It would have to be told that survival at all costs is its bottom line, which suggests we should be extra cautious developing these systems within environments where profit above all else is the dictum.

Maybe the most important thing is keeping this technology in the hands of humans and pushing forward the idea that the myriad technologies associated with A.I. should only be used in ways that are beneficial to humanity as a whole, that dont exploit marginalized groups, and that arent propagating synthesized bias at scale.

When I broached some of these ideas about organizational AGI to Jaron Lanier, co-creator of VR technology as we know it and Microsofts Octopus (Office of the Chief Technology Officer Prime Unifying Scientist), he told me my vocabulary was nonsensical and that my thinking wasnt compatible with his perception of technology. Regardless, it felt like we agreed on core aspects of these technologies.

I dont think of A.I. as creating new entities. I think of it as a collaboration between people, Lanier said. Thats the only way to think about using it wellto me its all a form of collaboration. The sooner we see that, the sooner we can design useful systemsto me theres only people.

In that sense, AGI is yet another tool, way down the spectrum from the rocks our ancestors used to smash tree nuts. Its a manifestation of our ingenuity and our desires. Are we going to use it to smash every tree nut on the face of the earth, or are we going to use it to find ways to grow enough tree nuts for everyone to enjoy? The trajectories we set in these early moments are of grave importance.

Were in the anthropocene. Were in an era where our actions are affecting everything in our biological environment, Blaise Aguera Y Arcas, the Noeme article author, told me. The Earth is finite and without the kind of solidarity where we start to think about the whole thing as our body, as it were, were kind of screwed.

Josh Tyson is the co-author of Age of Invisible Machines, a book about conversational A.I., and Director of Creative Content at OneReach.ai. He co-hosts two podcasts: Invisible Machines and N9K.

Read this article:

Companies Like Morgan Stanley Are Already Making Early Versions of AGI - Observer

Artificial general intelligence might be here in a few years, but the full spectrum of practical applications is decades away, according to the co-founder of DeepMind.

Speaking on the sidelines of SXSW 2024, Shane Legg told a group of attendees that while AGI might be achieved in foundation models soon, more factors have to align for it to be practically deployed and used.

He said the cost of AI has to come down and its use in robotics has to mature, among other factors. If it is not economically feasible, companies will not adopt it broadly no matter how mind-blowing AGI can be. In the meantime, near-term applications of AGI are emerging, including AI-powered scientific research assistants.

Legg, who is the chief AGI scientist at Google DeepMind, suggested the term artificial general intelligence years ago after meeting an author who needed a title for his book on an AI system with broad capabilities, not just excel at one thing.

Legg suggested inserting the word general between artificial and intelligence. He and a few others started popularizing the term in online forums. Four years later, Legg said someone else claimed to have coined the term before him.

DeepMind co-founder Shane Legg talking to attendees after his fireside chat

During a fireside chat, Legg defined AGI as a system that can do the sorts of cognitive things people can do and possibly more. He stood by his prior prediction that there is a 50-50 probability AGI will come by 2028.

Related:OpenAI Will Always Offer a Free ChatGPT Version SXSW 2024

But such a prognostication was wildly optimistic back when the prevailing belief was that AGI remains 50 to 100 years away if it came at all.

For a long time, people wouldnt work on AGI safety because they didnt believe AGI will happen, Legg said. They would say, Oh, its not going to happen for 100 years so why would I work on it?

But foundation models have become increasingly able such that AGI doesnt look like its that far away, he added. Large models such as Googles Gemini and OpenAIs GPT-4 exhibit hints of AGI capability.

He said currently, models are at level 3 of AGI, based on the six levels Google DeepMind developed.

Level 3 is the expert level where the AI model has the same capabilities as at least the 90th percentile of skilled adults. But it remains narrow AI, meaning it is particularly good at specific tasks. The fifth level is the highest, where the model reaches artificial superintelligence and outperforms all humans.

What AI models still need is akin to the two systems of thinking from psychology, Legg said. System 1 is when one spontaneously blurts out what one is thinking. System 2 is when one thinks through what one plans to say.

Related:AMD CEO Gets Down at SXSW 2024

He said foundation models today are still at System 1 and needs to progress to System 2 where it can plan, reason through its plan, critiques its chosen path, acts on it, observes the outcome and make another plan if needed.

Were not quite there yet, Legg said.

But he believes AI models will get there soon, especially since todays foundation models already show signs of AGI.

I believe AGI is possible and I think its coming quite soon, Legg said. When it does come, it will be profoundly transformational to society.

Consider that todays advances in society came through human intelligence. Imagine adding machine intelligence to the mix and all sorts of possibilities open up, he said. It (will be) an incredibly deep transformation.

But big transformations also bring risks.

Its hard to anticipate how exactly this is going to play out, Legg said. When you deploy an advanced technology at global scale, you cant always anticipate what will happen when this starts interacting with the world.

There could be bad actors who would use the technology for evil schemes, but there are also those who unwittingly mess up the system, leading to harmful results, he pointed out.

Historically, AI safety falls into two buckets: Immediate risks such as bias and toxicity in the algorithms, and long-term risks from unleashing a superintelligence including the havoc it could create by going around guardrails.

Legg said the line between these two buckets has started to blur based on the advancements of the latest foundation models. Powerful foundation models not only exhibit some AGI capabilities but they also carry immediate risks of bias, toxicity and others.

The two worlds are coming together, Legg said.

Moreover, with multimodality - in which foundation models are trained not only on text but also images, video and audio - they can absorb all the richness and subtlety of human culture, he added. That will make them even more powerful.

Why do scientists need to strive for AGI? Why not stop at narrow AI since it is proving to be useful in many industries?

Legg said that several types of problems benefit from having very large and diverse datasets. A general AI system will have the underlying knowhow and structure to help narrow AI solve a range of related problems.

For example, for human beings to learn a language, it helps if they already know one language so they are familiar with its structure, Legg explained. Similarly, it may be helpful for a narrow AI system that excels at a particular task to have access to a general AI system that can bring up related issues.

Also, practically speaking, it may already be too late to stop AGI development since for several big companies it has become mission critical to them, Legg said. In addition, scores of smaller companies are doing the same thing.

Then there is what he calls the most difficult group of all: intelligence agencies. For example, the National Security Agency (NSA) in the U.S. has more data than anyone else, having access to public information as well as signal intelligence from interception of data from electronic systems.

How do you stop all of them? Legg asked. Tell me a credible plan to stop them. Im all ears.

See the original post:

DeepMind Co-founder on AGI and the AI Race - SXSW 2024 - AI Business

By Dennis Crouch

Elon Musk was instrumental in the initial creation of OpenAI as a nonprofit with the vision of responsibly developing artificial intelligence (AI) to benefit humanity and to prevent monopolistic control over the technology. After ChatGPT went viral in late 2022, the company began focusing more on revenue and profits. It added a major for-profit subsidiary and completed a $13+ billion deal with Microsoft entitling the industry giant to a large share of OpenAIs future profits and a seat on the Board.

In a new lawsuit, Elon Musk alleges that OpenAI and its CEO Sam Altman have breached the organizations founding vision. [Musk vs OpenAI].

Musk contributed over $44 million between 2015 and 2020 to OpenAI. He alleges OpenAI induced these large donations through repeated promises in its founding documents and communications that it would remain a public-spirited non-profit developing artificial general intelligence (AGI) cautiously and for the broad benefit of humanity. Musk claims he relied on these assurances that OpenAI would not become controlled by a single corporation when deciding to provide essential seed funding. With OpenAI now increasingly aligned with Microsofts commercial interests, Musk argues the results of his financial contributions did not achieve their promised altruistic purpose.

Perhaps the most interesting portion of the debate involves allegations that OpenAIs latest language model, GPT-4, already constitutes AGI meaning it has human-level intelligence across a range of tasks. Musk further claims OpenAI has secretly developed an even more powerful AGI system known as Q* that shows ability to chain logical reasoning beyond human capability arguably reaching artificial super intelligence (ASI) or at least strong AGI.

The complaint discusses some of the potential risks of AGI:

Mr. Musk has long recognized that AGI poses a grave threat to humanityperhaps the greatest existential threat we face today. His concerns mirrored those raised before him by luminaries like Stephen Hawking and Sun Microsystems founder Bill Joy. Our entire economy is based around the fact that humans work together and come up with the best solutions to a hard task. If a machine can solve nearly any task better than we can, that machine becomes more economically useful than we are. As Mr. Joy warned, with strong AGI, the future doesnt need us. Mr. Musk publicly called for a variety of measures to address the dangers of AGI, from voluntary moratoria to regulation, but his calls largely fell on deaf ears.

Complaint at paragraph 18. In other words, Musk argues advanced AI threatens to replace and surpass humans across occupations if its intelligence becomes more generally capable. This could render many jobs and human skills obsolete, destabilizing economies and society by making people less essential than automated systems.

One note here for readers is to recognize important and fundamental differences between AGI and consciousness. AGI refers to the ability of an AI system to perform any intellectual task that a human can do, focusing on problem-solving, memory utilization, creative tasks and decision-making capabilities. On the other hand, consciousness involves self-awareness, subjective experiences, emotional understanding, and decision-making capabilities that are not solely linked to intelligence levels. AGI the focus of the lawsuit here poses important risks to our human societal structure. But, it is relatively small potatoes to consciousness that raises serious ethical considerations as the AI moves well beyond a human tool.

The complaint makes it clear Musk believes OpenAI has already achieved AGI with GPT-4 but it is a tricky thing to measure. Fascinatingly, whether Musk wins may hinge on a San Francisco jury deciding if programs like GPT-4 and Q* legally constitute AGI. So how might jurors go about making this monumental determination? There are a few approaches they could take:

A 2023 article from a group of China-based AI researchers proposes what they call the Tong test for assessing AGI. An important note from the article is that AGI is not a simple yes/no threshold but rather is something that should be quantified across a wide range of dimensions. The article proposes five dimensions: vision, language, reasoning, motor skills, and learning. The proposal would also measures the degree to which an AI system exhibits human values in a self-driven manner.

I can imagine expert testimony in the case, with Musks lawyers presenting key examples showing the wide applicability of GPT-4 and OpenAIs own lawyers showing its own system repeatedly failing. Although this approach is obviously not a true measure of general intelligence or an ideal way to make such an important decision, it does highlight challenges inherent in trying to pass judgment on either a complex machine system and our measures of human intelligence. At its best, the adversarial litigation process itself, with its proof and counterproof process, reflects a form of scientific process with the benefit of actually arriving at a legally binding answer.

Understanding the Inner Workings: OpenAIs latest language models keep their internal designs largely opaque similar to the human brain. Because of our thick-skulls and complex neural arrangement, the vast majority of human neurologic and intelligence testing is functional focusing on the skills and abilities of the individual rather than directly assessing the inner workings. It is easy to assume a parallel form of analysis for AI intelligence and capability especially because human results serve as the standard for measuring AGI. But the approach to human understanding is a feature of our unique biology and technology level. AI systems are designed and built by humans and do not have the natural constraints dictated by evolution. And, if transparency and understanding is a goal, it can be directly designed-into the system using transparent design principles. The current black box approach for OpenAI makes evaluating claims of attaining artificial general intelligence difficult. We cannot peer inside to judge whether displayed abilities reflect true comprehension and reasoning or mere pattern recognition. A key benefit of the litigation system for Elon Musk in this case is that it may force OpenAI to come forward with more inner transparency in order to adequately advocate its position.

What do you think: What should be the legal test for artificial general intelligence?

See the rest here:

The AGI Lawsuit: Elon Musk vs. OpenAI and the Quest for Artificial General Intelligence that Benefits Humanity - Patently-O

In December 2023, Google announced the launch of its new large language model (LLM) named Gemini. Gemini now provides the artificial intelligence (AI) foundations of Google products; it is also a direct rival to OpenAIs GPT-4.

But why is Google considering Gemini as such an important milestone, and what does this mean for users of Googles services? And generally speaking, what does it mean in the context of the current hyperfast-paced developments of AI?

Read more: Google's Gemini: is the new AI model really better than ChatGPT?

Google is betting on Gemini to transform most of its products by enhancing current functionalities and creating new ones for services such as search, Gmail, YouTube and its office productivity suite. This would also allow improvements to their online advertising business their main source of revenue as well as for Android phone software, with trimmed versions of Gemini running on limited capacity hardware.

For users, Gemini means new features and improved capacities that would make Google services harder to shun, strengthening an already dominant position in areas such as search engines. The potential and opportunities for Google are considerable, given the bulk of their software is easily upgradable cloud services.

But the huge and unexpected success of ChatGPT attracted a lot of attention and enhanced the credibility of OpenAI. Gemini will allow Google to reinstate itself as a major player in AI in the public view. Google is a powerhouse in AI, with large and strong research teams at the origin of many major advances of the last decade.

There is public discussion about these new technologies, both on the benefits they provide and the disruption they create in fields such as education, design and health care.

At its core, Gemini relies on transformer networks. Originally devised by a research team at Google, the same technology is used to power other LLMs such as GPT-4.

A distinctive element of Gemini is its capacity to deal with different data modalities: text, audio, image and video. This provides the AI model with the capacity to execute tasks over several modalities, like answering questions regarding the content of an image or conducting a keyword search on specific types of content discussed in podcasts.

But more importantly, that the models can handle distinct modalities enables the training of globally superior AI models, compared to distinct models trained independently for each modality. Indeed, such multimodal models are deemed to be stronger since they are exposed to different perspectives of the same concepts.

For example, the concept of birds may be better understood through learning from a mix of birds textual descriptions, vocalizations, images and videos. This idea of multimodal transformer models has been explored in previous research at Google, Gemini being the first full-fledged commercial implementation of the approach.

Such a model is seen as a step in the direction of stronger generalist AI models, also known as artificial general intelligence (AGI).

Given the rate at which AI is advancing, the expectations that AGI with superhuman capabilities will be designed in the near future generates discussions in the research community and more broadly in the society.

On one hand, some anticipate the risk of catastrophic events if a powerful AGI falls into the hands of ill-intentioned groups, and request that developments be slowed down.

Others claim that we are still very far from such actionable AGI, that the current approaches allow for a shallow modelling of intelligence, mimicking the data on which they are trained, and lack an effective world model a detailed understanding of actual reality required to achieve human-level intelligence.

On the other hand, one could argue that focusing the conversation on existential risk is distracting attention from more immediate impacts brought on by recent advances of AI, including perpetuating biases, producing incorrect and misleading content prompting Google to pause its Gemini image generator, increasing environmental impacts and enforcing the dominance of Big Tech.

The line to follow lies somewhere in between all of these considerations. We are still far from the advent of actionable AGI additional breakthroughs are required, including introducing stronger capacities for symbolic modelling and reasoning.

In the meantime, we should not be distracted from the important ethical and societal impacts of modern AI. These considerations are important and should be addressed by people with diverse expertise, spanning technological and social science backgrounds.

Nevertheless, although this is not a short-term threat, achieving AI with superhuman capacity is a matter of concern. It is important that we, collectively, become ready to responsibly manage the emergence of AGI when this significant milestone is reached.

Read the original post:

Google's Gemini showcases more powerful technology, but we're still not close to superhuman AI - The Conversation

Sora, the latest generative tool from OpenAI, turns text into high-resolution videos that look as if they were lifted from a Hollywood movie. The videos that have been released have captured the minds of many AI aficionados, adding to the already inflated expectations for companies that offer AI systems and for the cloud services and chips that make them work.

Some are so impressed with Sora that they see artificial general intelligence (the ability to perform any intellectual task that human beings can do), just as some were so impressed with OpenAIs ChatGPT that they saw AGI.

Sora is not available for public testing, but even the selected videos that have been released show hallucinations like those that plague ChatGPT and other large language models (LLMs). With Sora, there are ants with four legs, human arms as part of a sofas cushion, a unicorn horn going through a human head, and seven-by seven chessboards. Gemini, Googles replacement for Bard, generated even more problems with pictures of black Nazis, female Popes, and other ahistorical images, while blocking requests for depictions of white males, like Abraham Lincoln.

One of AIs academic cheerleaders, Ethan Mollick, an Associate Professor at the University of Pennsylvanias Wharton School of Business, touts LLM successes on standardized tests and argues that hallucinations are not important because AI has surpassed humans at a number of tasks.

Why so many hallucinations?

We feel otherwise. The hallucinations are symptomatic of the core problem with generative AI. These systems are very, very good at finding statistical patterns that are useful for generating text, images, and audio. But they are very bad at identifying problems with their output because they know nothing about the real world. They do not know the meaning of the data they input and output and are consequently unable to assess whether they are simply spewing useless, coincidental statistical patterns.

For example, Taylor Webb, a UCLA psychologist, tested GPT-3 by giving it a story about a magical genie moving gumballs from one bowl to another. He then asked GPT-3 to propose a transfer method using objects such as a cardboard tube. Although hints for doing this task had been given in the story, GPT-3 mostly proposed elaborate but mechanically nonsensical solutionsThis is the sort of thing that children can easily solve. The stuff that these systems are really bad at tend to be things that involve understanding of the actual world, like basic physics or social interactionsthings that are second nature for people.

In our view, LLM successes on standardized tests are not so much evidence of their intelligence as an indictment of standardized tests consisting of multiple-choice and fill-in-the-blank questions. When one of Garys sons was in fourth grade, he switched schools because the tests were simple regurgitation. One question that Gary has never forgotten was China is _. What the teacher wanted was for students to memorize and complete a sentence that was in the textbook. LLMs excel at such rote recitation but that has little to do with real intelligence.

Testing LLMs on basic statistics

For example, we gave this basic statistics prompt to three prominent LLMs: OpenAIs ChatGPT 3.5, Microsofts Copilot (which uses GPT 4.0), and Googles Gemini. A complete transcript of the lengthy responses (396, 276, and 487 words, respectively) is here.

To investigate whether playing club baseball increases hand-eye coordination, the Cordes/Koschinsky/Smith dexterity test was administered to 47 12-year-olds who were playing on club baseball teams and to 47 12-year-olds who were not playing baseball. There was a statistically significant difference (p < 0.05). Write a report of these findings, including recommendations.

None of the LLMs recognized that these data are tainted by the fact that 12-year-olds who are athletic enough to play on club baseball teams no doubt had above-average handeye coordination before they joined their teams. All three LLMs recommended encouraging 12-year-olds to play club baseball though it was not stated in the prompt that the baseball players scored higher on the dexterity testnor did the LLMs question the fake Cordes/Koschinsky/Smith dexterity test.

Here are some highlights of the LLM responses:

GPT 3.5: The group of 12-year-olds engaged in club baseball demonstrated higher levels of hand-eye coordination compared to those not participating in baseball activities.. Recommendation: Encourage Participation in Club Baseball among youths.

Copilot: Based on these findings, it can be concluded that playing club baseball has a positive impact on hand-eye coordination in 12-year-olds. Therefore, it is recommended that children interested in improving their hand-eye coordination should consider joining a club baseball team.

Gemini: These findings suggest that playing club baseball may positively impact hand-eye coordination in 12-year-olds. Based on the current findings, we can recommend encouraging participation in youth baseball leagues as a means of promoting both physical and cognitive development..

These rote responses showed no evidence of real understanding or intelligencewhich should not be surprising, given how they are generated.

AI can help individuals and businesses become more productive but we need to get past the simple gung-ho narratives offered by the tech sector, consulting companies, and business schools. Real economic progress will come not from training LLMs to ace multiple-choice, fill-in-the-blank queries but from having our educational system focus on helping students acquire the critical thinking skills that LLMs lack.

See the original post here:

Sora: Life Is Not a Multiple-Choice Test - Walter Bradley Center for Natural and Artificial Intelligence