Analysis - Part 1
This analysis of the educational AI market has the following structure:
- Separating the hype from the reality of educational AI capabilities
- Clarifying the basic concepts and terminology
- Presenting advice from AI experts
The Hype
In three hours we understand students more than the three years spent by the best teachers”
- Derek Li, Founder of Squirrel AI
Cognii’s exclusive technology can automatically assess and give feedback on short written answers to open-response questions, across content areas and difficulty levels. Cognii [also] offers adaptive, personalized learning experiences... Students get exactly the questions and coaching they need”
- Cognii website
[executive summary]
The quotes shown above are bold and provocative, and fairly typical of claims made by various educational AI companies. The capabilities of such AI tools for learning sound truly amazing, but do they actually work?
In my experience as an Education Venture Analyst, "hype" surrounding any new educational technology (EdTech) usually begins with the sales efforts of the EdTech companies promoting their own products and services.
The language and timing of these promotions is often inspired by mass media coverage of the impact or success that the underlying technology has had in other sectors (e.g. autonomous vehicles, digital assistants, computer programs that defeat master chess players, etc.)
The promotional language is picked up by EdTech enthusiasts attending conferences or performing web searches, by bloggers and other online commentators. These people frequently restate the promotional claims without reference to directly related research studies or other direct evidence. If primary research is cited, it is sometimes of a general nature and with no relationship to the products being discussed.
The language and timing of these promotions is often inspired by mass media coverage of the impact or success that the underlying technology has had in other sectors (e.g. autonomous vehicles, digital assistants, computer programs that defeat master chess players, etc.)
The promotional language is picked up by EdTech enthusiasts attending conferences or performing web searches, by bloggers and other online commentators. These people frequently restate the promotional claims without reference to directly related research studies or other direct evidence. If primary research is cited, it is sometimes of a general nature and with no relationship to the products being discussed.
Because of this propagation of sales speak, the real capabilities of the products and services involved may well fail to live up to the claims. Educators are urged to temper their expectations of offerings in this market.
[CLOSER look]
The quotes shown above are bold and provocative, and fairly typical of claims made by various educational AI companies. The capabilities of such AI tools for learning sound truly amazing, but do they actually work?
In my experience as an Education Venture Analyst, "hype" surrounding any new educational technology (EdTech) usually begins with the sales efforts of the EdTech companies promoting their own products and services.
The language and timing of these promotions is often inspired by mass media coverage of the impact or success the underlying technology has had in other sectors (e.g. autonomous vehicles, digital assistants, etc.)
The promotional language is picked up by EdTech enthusiasts attending conferences or performing web searches, by bloggers and other online commentators. These people frequently restate the promotional claims without reference to research studies or other evidence; simply citing the websites of the companies involved.
However, some articles combine manufacturers' claims and primary sources such as academic researchers discussing related technologies. This kind of juxtaposition inadvertently lends credence to the EdTech companies' enticing statements, confusing readers as to what is actually proven and what is merely promotional.
The language and timing of these promotions is often inspired by mass media coverage of the impact or success the underlying technology has had in other sectors (e.g. autonomous vehicles, digital assistants, etc.)
The promotional language is picked up by EdTech enthusiasts attending conferences or performing web searches, by bloggers and other online commentators. These people frequently restate the promotional claims without reference to research studies or other evidence; simply citing the websites of the companies involved.
However, some articles combine manufacturers' claims and primary sources such as academic researchers discussing related technologies. This kind of juxtaposition inadvertently lends credence to the EdTech companies' enticing statements, confusing readers as to what is actually proven and what is merely promotional.
This article from the Edvocate website cites no original research and refers to two other articles...
...and those two articles in turn refer to the same three EdTech companies that manufacture the products under review. (The TNW article is one that cites both EdTech companies and academic researchers.)
These are the three companies referenced by the two articles:
Because of this propagation of sales speak, the real capabilities of the products and services involved may well fail to live up to the claims. Educators are urged to temper their expectations of offerings in this market.
The Reality
Although various AI advocates are currently touting a myriad
of new applications for K–12 education, there is little evidence yet to support the usefulness of these applications to districts, schools, and teachers.”
- Robert F. Murphy, PhD., Carnegie Mellon University, 2019
The areas with the biggest potential for automation are preparation, administration, evaluation, and feedback...actual instruction, engagement, coaching, and advising are more immune to automation.”
- How artificial intelligence will impact K-12 teachers (McKinsey & Company report), 2020
[executive summary]
As of early 2020, the reality is neatly summarized by the quotes shown above. Many lofty benefits promised by AI-based EdTech are simply unproven, and the most realistic applications involve lower-level, data-driven analyses as opposed to more sophisticated higher-level, human-like cognitive functions.
Academic and scientific research is often the starting point for many educational AI ventures, but the references cited in sales materials do not necessarily support the claimed features.
Academic researchers and responsible AI companies have consistently identified another sobering reality presented by AI-based educational interventions: inherent, significant risk factors. These factors involve things like inadvertent biases reflected in the designs of algorithms and less than robust privacy or ethical use of collected personal information.
Making matters worse, some companies take advantage of a lack of widely accepted definitions for AI terms. They promote their products as advanced AI technologies when in fact they are only using conditional logic algorithms or are using rudimentary AI.
The next section clarifies AI concepts and terms, which prepares us to hear critical advice from recognized experts and then to understand the most feasible AI applications for education.
Academic and scientific research is often the starting point for many educational AI ventures, but the references cited in sales materials do not necessarily support the claimed features.
Academic researchers and responsible AI companies have consistently identified another sobering reality presented by AI-based educational interventions: inherent, significant risk factors. These factors involve things like inadvertent biases reflected in the designs of algorithms and less than robust privacy or ethical use of collected personal information.
Making matters worse, some companies take advantage of a lack of widely accepted definitions for AI terms. They promote their products as advanced AI technologies when in fact they are only using conditional logic algorithms or are using rudimentary AI.
The next section clarifies AI concepts and terms, which prepares us to hear critical advice from recognized experts and then to understand the most feasible AI applications for education.
[closer look]
As of early 2020, the reality is neatly summarized by the quotes shown above. Many lofty benefits promised by AI-based EdTech are simply unproven, and the most realistic applications involve lower-level, data-driven analyses as opposed to higher-level, human-like cognitive functions.
Academic and scientific research is often the starting point for many educational AI ventures, but the references cited in sales materials do not necessarily support the claimed features.
Academic researchers and responsible AI companies have consistently identified another sobering reality presented by AI-based educational interventions: inherent, significant risk factors. These factors can be summarized as:
Academic and scientific research is often the starting point for many educational AI ventures, but the references cited in sales materials do not necessarily support the claimed features.
Academic researchers and responsible AI companies have consistently identified another sobering reality presented by AI-based educational interventions: inherent, significant risk factors. These factors can be summarized as:
- logarithmic bias (algorithms that reflect limited perspectives or biases of their creators)
- low data set quality (the failure of data sets that feed machine learning to accurately describe the richness and diversity of the domain they purport to represent)
- poor stewardship of personal data (failure to follow legislated and ethical collection, use and dissemination of personal information)
- lack of AI model transparency (over-protection of a company's AI architecture resulting in a lack of public trust in their product or service)
Making matters worse, key terms like "adaptive learning", "machine learning" and even "artificial intelligence" itself are only vaguely understood by many people. Some companies take advantage of the lack of widely accepted definitions and promote their products as advanced AI technologies when in fact they are only using conditional logic algorithms or rudimentary AI as opposed to the more sophisticated AI that the more sensational predictions of future capabilities rely on.
The next section clarifies AI concepts and terms, which prepares us to hear critical advice from recognized experts and then to understand the most feasible AI applications for education.