Healthcare is by nature conservative; despite a huge level of interest for artificial intelligence (AI) and machine learning (ML) techniques, significant penetration in standard workflows of care will depend on demonstrations of safety and efficacy. As a result, it is today’s pipeline of clinical studies using AI/ML that will determine mainstream usage tomorrow. In order to better understand this pipeline, we have mined clinicaltrials.gov for characteristics and trends of registered studies making use of AI/ML.
Unsurprisingly we see strong growth, but starting from a very small baseline. Although most of the studies are still related to the processing of diagnostic data for care delivery, we observe significant uptake in the registration of “discovery studies”, i.e. studies which, rather than substituting AI for current standardized human activity/judgment, attempt to surface new patterns of correlation as yet undetermined; for instance: a system trying to discover relationships between patterns of speech and neurological disease.
***
In previous work, we have proposed a segmentation of AI healthcare applications based on primary users and functions:
For example AI can make a diagnosis from a radiographic image (“measure”), provide decision support e.g. to admit to the hospital (“recommend”), or even perform an intervention such as sending motivational messages to a person in recovery (“execute”). Most of the traditional CAD (computer aided diagnosis) would fall under “measure” with the clinical delivery domain. We have added one more function, “discover” which means the use of AI methods to surface previously unknown relationships that can be built on to make predictions, or simply building a new AI from scratch prior to testing its performance in the real world.
We mined clinicaltrials.gov by searching for studies with US site(s) with the following key words: “artificial intelligence”, “machine learning”, and “deep learning” and excluded studies with the status “withdrawn”. We reviewed each entry to ensure that AI use was germane to the study, and then classified them by domain and function (see Table 1). The first thing that is apparent is a greater than 5-fold increase in study starts in 5 years. This growth is seen across all 4 domains of application.
In absolute numbers, 71 registered studies using AI/ML starting in 2020 is still extremely small relative to thousands of drug and device trials, and an undercount. Many studies using AI/ML are conducted with a low degree of formality and not registered. However, it’s nearly a sine qua non in health that in order to be used broadly (and reimbursed), evidence needs to be published, and in order to be published in reputable medical journals, studies need to be registered.
We have also analyzed trends over the last 4 years at the level of individual segments (i.e. domain and function) and while the numbers are necessarily small a few patterns emerge.
The lion’s share of the registered studies pertain to the validation of diagnostic algorithms, commonly medical images, but also other type of datasets.
Recently, more studies have been registered looking to gather large datasets and mine them for relationships with clinical outcomes without a clear a priori hypothesis beyond the fact that a relationship might exist. In many cases, these are observational studies looking to build a de novo AI engine that will then move into formal testing; an activity that more and more health systems sitting on data troves are starting to undertake, sometime with help of a specialized third party such as Rhino Health to help ensure that there is appropriately diverse and current data to train robust AIs.
Finally we take a look at the fraction of studies where there is participation from industry through funding. Overall, this is robust at ~40% with the highest level of emphasis on care delivery use-cases which is where the value proposition / ROI is clearest. A surprise is that the level of industry funding in patient engagement solutions is lowest; however this is an area where the lines between consumerism/wellness and clinical applications are blurred and some efforts that are nevertheless health related may not be captured under the typical scope of clinical studies.
***
These are early days for AI/ML in healthcare, numbers are still too small to evaluate trends, and 2020 was heavily disrupted by Covid-19. Nevertheless it is worth looking at because what is in the development pipeline today will be in the clinic tomorrow. We will come back in year to look at 2021, but let’s risk a few predictions at the 80% confidence level:
- There will more than 100 starts for AI/ML related registered studies in 2021
- There will be continuing strong growth in the “Discovery” area, i.e. building new AI models from scratch, although that’s the tip of the iceberg considering it is not required for these studies to register as they typically use retrospective data
- Although growing and still the largest segment, care delivery will represent a smaller relative share
And one speculation: at some point soon, studies testing the “recommend” function will take off, if not 2021, then in 2022.
At the same time, we expect an increasingly formalized process (cf. FDA’s push for GMLP – good machine learning practices) to emerge with growth in structured AI/ML ops services. So far, a big part of AI/ML has been a cottage industry; the infrastructure that we are familiar with in drug and device R&D is only nascent in AI/ML. And as this happens, there will not only be more studies, but it will become more likely that they are registered in a study database like clinicaltrials.gov.