"The pace of innovation and development of AI tools is outrunning the FDA and other regulators’ ability to stay on top of AI innovations"

Q: Can you talk about the report you recently published highlighting the possibility of racial bias in some of the FDA approved AI enabled products and devices?

Casey: I built a database of all the FDA cleared AI algorithms to date. As a reporter, I’m always getting press releases from companies talking about the clearances that they’ve gained from the FDA. But there is no real systematic way to look at those products. There is no database that identifies them to look in totality about what has been approved. So, I took a step further after identifying the products and looking at the level of validation that was done on them, like what was the size of the validation sets? What were the methods used? What is in those data sets? How diverse are they by race, by gender? Where were the data sets gained to get a sense of what level of information was disclosed? What is publicly available? And what I found was that it’s really all over the map in terms of the sample sizes that are used to validate these algorithms. And there’s also really very little information about the demographics of the data sets in a way that raises questions about the ability of these products to generalize across populations. And I found that variation happening even within products that are designed to do the same thing, like assess patients for intracranial hemorrhage or stroke or even things like breast cancer.

Q: What kind of products we talking about here? Are these medical devices, software products, and how many of them did you really scrutinize?

Casey: The category is sort of software as a medical device. These are used as decision support tools that supply data to physicians on patients, that helps them make decisions and helps them diagnose and treat those patients. There were 161 products that I identified within specific product codes. You can search the FDA’s databases to try find these and figure out what validation was done on them. I have read medical studies that suggest there is up to 220 of these products and these are all deep learning AI products. So, it is machine learning technology which have all been approved. We see a vast amount of innovation going on in that area over the past six years.

Q: On your reports you focus on the breast cancer related products. Can you talk about that?

Casey: Yeah, that was an area where I’m especially interested in looking at because diversity really matters, and breast cancer varies so widely among patients. And it’s particularly important to have diversity in those data sets so that any AI system that might be advising a doctor or a physician on how to care for these patients sees enough patients and can give good advice so that its conclusions can be generalized to broader populations of patients. What we’ve seen over time with a lot of medical products and algorithms that have made their way into the market is that they’re not tested on diverse groups of people. And instead, their recommendations, their reliability mainly exists within European Caucasian populations, which shouldn’t be acceptable to patients or medical providers.

Q: So there is reason to be concerned about the lack of a standardized validation process and a lack of disclosure specifically around the data that is being used to develop these algorithms and there is a real potential for racial discrimination. Is it correct?

Casey: I think that’s right. It’s the lack of standards there and in particular, disclosure of the contents of the data sets that is troubling from that point of view.

Q: Based on all your reporting, do you think the challenge lies in the quality of the data or maybe even the sufficiency of the data? Or is it more to do with the deficiencies in the algorithms or is it both?

Casey: I think the biggest issue is the quality of the data and the access to the data such that you can have really, truly representative data across populations and have enough of it to be able to train an algorithm to adequately perform the task you’re asking it to perform. There have been some studies done that suggest that the vast majority of data supplied for AI research comes from institutions in three states in California, New York, and Massachusetts. That’s missing a huge part of the places that we sit in now. So many people in so many communities end up getting excluded from that. This is the major hole right now that this ecosystem needs to figure out how to remedy.

Q: You make a very provocative statement at the beginning of one of your reports – ‘AI is now a lawless frontier in medicine.’ Some people might say maybe it’s just a little bit harsh, perhaps because it has had some success in other areas in healthcare, like administrative functions, revenue cycle operations, claim management, fraud and abuse, or even in chronic disease management. What would you say to those who feel that?

Casey: I’m making a comparison to sort of frontier development, like the development of the American West. I’m sort of making that comparison because I’m trying to crystallize the notion that the sheriff isn’t in town yet, that the pace of innovation, the pace of development, the pace of that building process is outrunning the ability of the FDA and other regulators to stay on top of the questions that innovation is raising. That is a big concern right now. I think the FDA is trying very hard, but I think it’s under-resourced and it can’t keep up with the very important questions that this is raising. The other part of that metaphor that is worth diving into is, does that mean that there are a bunch of bandits out there that are a bunch of evildoers who are trying to gather data and do bad things with it? By and large, from the companies and the people that I’ve talked to, I would say no. I would say that most of them are very well-meaning and altruistic. But there is still the issue of unintended consequences that may arise from the use of products that are not fully and carefully vetted. I think once that process begins to fully mature and catch up with the innovation, everyone will be better for it.

Q: You made a comment a little bit earlier about not been enough data available to do a rigorous training of the algorithms. There is a vast amount of data available in the form of images, more so than other forms of healthcare data. What can we do with the large amount of data available, especially the data sitting in our systems, for instance, in hospitals?

Casey: It’s very hard for researchers to come by to aggregate that data to do anything meaningful with it. EHR data is notoriously siloed and kept in environments where it’s just very difficult to access the data and make use of it for meaningful research and purposes that could really benefit people. I think it’s very difficult to harness that data, even though there is so much of it. And about the imaging data, I think a big question for the industry and a big problem right now, is the issue of transparency. Where are those data sets from? What is in them? We need to know the ingredients of these algorithms. We need to know who these people are, where they come from. We don’t need to know their identities. I don’t mean to suggest that, but we need to know how these algorithms are being built on what data so that there can be some confidence in these products, that they can generalize and do what the developers intend.

Q: What are you hearing from policymakers and industry executives, especially tech firms, on how they’re wrestling with the ethical use of data and how they’re moving forward with this?

Casey: Over the past six months a lot of companies are realizing that this is an issue and they’re bringing it out into the light and wanting to talk about it at industry conferences and on virtual gatherings and so forth, to be able to set forth, OK, well, you know what? This is an issue for us in terms of optics. We want to be inclusive companies. We want to emphasize that. And you’re seeing a lot of those companies’ fund research and hold events to talk about it. But there isn’t yet sort of a consensus that emerged on the best way to accomplish this. What are the set of practices that ought to be used to ensure that these products are inclusive and don’t unintentionally discriminate against certain groups? So, I think there’s kind of a recognition that these issues need to be addressed. But how to do that really has not been agreed upon, there really aren’t any clear best practice standards that have been identified. There is just a process that’s beginning to confront those issues.

Q: Is this a question for the FDA or is this more for the industry to self-regulate and self-governance and come up with the best practices and hope that the outcomes are good? What is your thought?

Casey: That is really the big question right now. Whose responsibility is that? Where should that vetting process take place? Should it take place at the FDA before these products get onto the market? That is not happening right now. Some of the people I have talked to, executives of companies say, the FDA clearance, the 510 K clearance that’s granted to most of these products has never really filled that role for any kind of product. So, usually what happens is there are follow up studies done at conferences and by clients of these products to bear out their efficacy. And there is a process that takes place normally in the private market to verify that these products are the best things for patients. The responsibility lies on the health systems to adopt products that are really going to benefit the people. Data is the main thing that these products use in order to deliver services, to help inform physicians to provide care to patients. You wouldn’t say to somebody – ‘you should just take this drug. Don’t worry about it. We don’t need to talk about the ingredients or where it came from or what’s in it. Just take it, OK? It’s fine.’ You would tell them the ingredients. It would be studied rigorously. You would know who is in those validation data sets, you would be able to analyse it in all the different cohorts and how it affects different racial subgroups. That’s done now in public at the FDA for drugs. Now, drugs have a different risk profile. Hence, the data analysis should be rigorously done and must have transparency.

Q: We’ve recently seen some initiatives, especially the one where several health systems come together and formed Truveta, that is going to pool patient data from several leading health systems and use it to analyze it for insights and help improve healthcare outcomes. There are also some other initiatives like the synthetic data challenge that the ONC has come up with. All are looking to address the same problem that there isn’t enough data for us to really analyze or train the algorithms and come up with some kind of heuristics or benchmarks for us to drive the outcomes. Would you care to comment on these initiatives? And is that an alternative? Is this a viable alternative that is taking shape?

Casey: It’s a timely question. I’ve been talking to the executives and stakeholders that founded Truveta over the past week or so to talk to them about that initiative. I think it is interesting in something that the industry, by and large, has just failed to do to date, and that is aggregate a large amount of data that comes from health systems all over the country and not just health systems that are on the coast. Those 14 health systems that are gathered in Truveta represent patients who are spread throughout 40 states all over the country. So, I think that’s really exciting and potentially provides a really great resource that researchers can tap to be able to gain access to large amounts of representative patient data. There still are a lot of questions though with that because we all know about controversies that have arisen from, say, given the hospital system, working with a tech company and sharing their data with that tech company because of all the privacy questions and questions of economic exploitation that might arise from that. It is like you’re using data from the patients that got care at your institution. Then you are selling that data to another entity to do research on it to build a product that that entity will profit from and not necessarily the patient. So, there are issues of consent that get raised in that. There are questions that should be raised and talked about so that there can be a consensus or at least an open public discussion about how to get access to that data, who does it benefit, how to do this in a way that respects the patients and all of the stakeholders?

Q: What are the top two or three items of the unfinished agenda in harnessing data for us to really make a difference in healthcare outcomes? One is interoperability. Can you share your thoughts on this?

Casey: I think interoperability is a key issue and that issue is part of developing data sets at scale, large enough data sets that can be used by researchers and companies to be able to build meaningful and generalizable AI products that will benefit everybody. I think the biggest issues in my mind about that are really transparency, disclosure and some of those regulatory questions. I think it’s really important to think about the nature of these products, which are machine learning. It’s a computer that is able to comb the contours of a data set to form conclusions on its own without being explicitly sort of programed. I think when you have a system like that where it might be somewhat of a black box about how it is reaching the conclusions that it’s especially important for people to know what is going into those training sets. How is it being tested on what data is it being validated? Are these things at the end of the day going to improve care or are they just going to layer on top of care an additional level of cost without providing the benefit that they advertise? And I think that process just must unfold in a meaningful way so that, before we start paying for these things, before they get into the market and start providing care for people, we know that they are fair. We need to know that they are safe. We need to know that they stand some chance of improving care to people. So, I think those are the things that sort of need to be front and center questions that are addressed over the next few years.

Q: To sum it up in one word, would that be transparency?

Casey: I would say that would be the word I would choose as the one word that the industry needs to sort of focus on in the next couple of years.