AI Improves Endpoints and Evidence in Clinical Trials

Rohit: Hi Greg, welcome to the Big Unlock podcast. It’s great to have you on the show today. Thank you for having me, Greg. Absolutely. I’m Rohit Mahajan. I’m the CEO of Damo Consulting and BigRio, a Boston-based consulting company. Damo does strategy consulting for healthcare providers and payers and works with several pharma companies on AI, data, and analytics initiatives. I think we are in for a treat with our listeners.

Greg: Thanks, Rohit, and thank you for having me. I understand that most of the time your focus is on healthcare, and biopharma is healthcare-adjacent. It has some crossover interest, so I appreciate the invitation to talk to your audience. My name is Greg Goldmacher. I am Associate VP of Clinical Research at Merck Research Laboratories and head of the Clinical Imaging and Pathology function. My team oversees the use of scans and tissue in assessing trial outcomes and clinical trial endpoints.

Rohit: As we all know, Greg, there are so many clinical trials at this point in time. I think the number is very large, possibly approaching a million. So what kind of clinical trials do you focus on with your group?

Greg: My group has a broad base, and we support trials across every therapeutic area. We go where the endpoints are. What is an endpoint? It is a measurement of whether a clinical trial has met its objective. You make a measurement in one group of trial participants, compare it with another group, and see whether you have shown safety and efficacy.

The majority of trials where clinical imaging is used for endpoints are in oncology. Intuitively, you can think about what you measure to see whether a cancer drug is working. Are tumors growing or shrinking over time? You assess that by doing a series of scans.

There are several hundred clinical trials my team supports. The large majority are in oncology, but we also support central nervous system, immunology, cardio-respiratory, metabolic disease, and other therapeutic areas.

Rohit: That is fantastic, Greg. From what I know about clinical trials, it is a long and expensive process because patients are involved. It is usually a big hurdle for biopharma companies to complete clinical trials in a way that allows the drug to go to market. Could you tell us a little more, for listeners who may not be aware of this long and expensive process, about how this works at a high level?

Greg: Yeah, sure. I have to preface everything by saying I am speaking on behalf of myself. I am not speaking on behalf of Merck, and I will not discuss any specific Merck products, programs, or trials. I will speak broadly about biopharma, the processes, and then we can get into the use of technology and AI.

So let’s start with the big picture. There is the preclinical process where molecules are designed, evaluated for molecular properties, first in vitro and then in preclinical in vivo models. A lot of it is looking at pharmacodynamics, pharmacokinetics, the distribution of the drug in the organism, whether it is getting to its target, and preclinical safety. You also get an initial idea of dose.

Then comes the clinical part. Once you move into clinical trials, phase one trials focus on safety and establishing dose. Then there is a transition into phase two, where you look at finalizing the dose and ideally gaining some signal of efficacy. Safety information is collected throughout.

Then in phase three trials, you are really determining efficacy. These are often referred to as pivotal trials. Phase twos may be pivotal as well. Data from phase twos, if appropriately designed, and phase threes can be submitted for regulatory approval for marketing the drug.

Occasionally there are phase four trials, which typically happen post-approval and collect additional safety information or information on efficacy in the real-world setting.

Rohit: That is a great high-level intro, Greg. Thank you. Before we segue, and it is not too long in any podcast before AI comes up, we are going to talk about the use of AI and new technologies in the clinical trial process so it can be more efficient, cost effective, and faster. What are some of your thoughts on that?

And before that, if you could give some insights into how you got to this position and how you started your role. I understand you trained as a physician, so I would love to hear that story before we dive into the AI side.

Greg: I started off thinking I was going to be a basic science researcher working in a lab. In college, I was working in a lab during a summer research position, and someone said I should get an MD-PhD rather than a straight PhD because it would be easier to get research grants.

So I did an MD-PhD at UT Southwestern. My PhD was in neuroscience. I briefly thought about being a clinical neurologist, but in the mid-nineties the bread and butter of neurology was stroke and neurodegenerative disorders. In both cases, you made the diagnosis and then said good luck, go see occupational therapy because there was nothing disease-modifying.

So instead I went into diagnostic radiology. I returned to the Boston area, did my residency in diagnostic radiology, and then was a fellow at Mass General in neuroimaging. I was doing stroke imaging research and got involved in a clinical trial. I was helping run an NIH-sponsored clinical trial, and my mentor got me involved in a small industry-sponsored trial using a thrombolytic drug. They were looking at reopening of the artery on CT angiography to get an initial sense of efficacy.

I effectively became the core lab as a fellow for this trial.

A couple of years later, I was doing more stroke research at Jefferson in Philly and realized the lab was not really my jam. The chief medical officer from Icon came and gave a talk about things you can do as a physician-scientist in industry and mentioned that they had a radiology core lab. This was an ecosystem I had no idea about. You never hear about these things in medical training.

I went to Icon for five years and retrained as a cancer imager because that is where a lot of the endpoints are. Then in 2015, I came to Merck. I supported part of the cancer imaging portfolio, and then as organizational changes happened, I took on supporting clinical imaging across the entire portfolio, all therapeutic areas. Later we added clinical pathology as well, because assessing outcomes using pathology conceptually functions the same way. It is newer to have approvable endpoints based on pathology. Imaging processes had been worked out over a long time, so we were transposing those processes into that space.

Now going to the technology side of things, I like to say in radiology we were doing AI research before ChatGPT made it cool. AI is pattern recognition, and visual pattern recognition is something humans do all the time. You show a picture of a dog or a cat and you know immediately, even if you cannot describe explicit rules.

In radiology, models can see patterns that human eyes cannot, no matter how good a radiologist you are. There is a lot of interest in that. Shortly after I joined Merck, I started pulling together people from data science, statistics, clinicians, biomarker groups, regulatory, digital, and others across the organization to do research in this space.

Rohit: So given your deep experience in clinical trials and AI, especially now with new models like ChatGPT, what are some of the things you are seeing that are applicable in the clinical trials world?

Greg: So there are a couple of broad categories. Let’s start with one that is the easiest and most obvious business case for the use of AI in the clinical trials world. Then we can segue into things that are less obvious but, I think, scientifically and from a business strategy point of view, very interesting.

The easiest place to make the business case is that clinical trials are expensive and a lot of it is human resource cost. So there are many applications of AI for efficiency. In the preclinical space, AI is used for genome searches, target identification, drug design, assessing protein folding, and things of that nature.

Once you get into the clinical space, there is a lot of use of AI to support clinical operations. That includes creating documents, protocols, clinical study reports, informed consent forms, and reports of various kinds. There is also AI analysis. A huge amount of data gets collected, and manual review of all that data is extremely labor intensive. That is an area where AI can help.

One example of manual effort is the many kinds of reconciliation in clinical trials where data is gathered in one place and also in another place, and you have to make sure things match. That can be extremely labor intensive manually. That is where general AI use and specifically agents can help. You can say to an agent, retrieve data from this place, do the comparison, take an action based on the comparison, such as issuing a query to a site to ask for clarification. That kind of use is great for efficiency and is the most straightforward business case. That is being adopted across many industries, and in clinical trials where reports and reconciliation and analysis are important, it is a valuable tool.

Rohit: That’s amazing. I think that’s great to know. And then in drug development decisions, if we talk about AI applications in drug development decisions and automating tasks and making strategic decisions for the organization, what do you think about that aspect, Greg?

Greg: Efficiency is the obvious use case. The slightly less obvious but at least as impactful use case, and of course I’m speaking from my perspective as somebody who focuses on imaging assessments, is thinking about endpoints. Endpoints are measurements. Why do you make measurements? To make decisions. AI allows us to make better measurements for better decisions.

As an example, let’s use cancer. The traditional way of assessing whether a cancer drug is working is you do a scan before treatment starts. You find the tumors, pick a few to measure, add up their measurements, then do multiple scans during treatment. If they shrink, that’s a response. If they grow, that’s progression. At each assessment time point, you apply simple rules to classify as complete response, partial response, stable disease, or progressive disease. Then you extract an endpoint like objective response rate or progression free survival using rules like RECIST, a standard tool in clinical trials.

What we really care about is not tumors growing or shrinking. What we care about is patient survival. Objective response rate is a good predictor of survival at large N. When you have a large population, improvements in response rate indicate improvements in survival. The problem is that as N gets smaller, that correlation gets worse. There are many examples of drugs that looked good in early phase trials with dozens of subjects and then failed in phase three. That is a terrible outcome for patients and a huge waste of resources. A phase three trial can cost $150 million. Those pipeline decisions are made early in phase one B or early phase two.

Another application is picking the right dose of a drug. Traditional dose finding cranks up the dose until the patient has adverse events and then backs off. There are reasons why that may not be optimal. If you could compare survival in groups of 30 or 40 patients, you could get to a confident dose for late phase trials.

Here is where AI can come into its own. Scans contain information not visible to the naked eye. Models can see pixel patterns. When you look at a tumor, there is more information than size. Pixel patterns may correlate with necrosis, vascularity, or inflammation, reflecting aspects of the tumor microenvironment. If you have a training set where you can associate these patterns with a gold standard like biopsy, then you can do non-invasive assessment of the tumor microenvironment.

You could use that as a pharmacodynamic biomarker. For instance, drugs in immuno-oncology work better when tumors are inflamed. If you could measure that non-invasively across the entire patient, you could assess whether a combination partner is good. If all tumors light up with inflammation, that is a potentially good drug partner.

There is also the potential to look at early changes in tumors and directly predict survival rather than predicting shrinkage first. A downside of systems like RECIST is that they use fixed percentage changes and do not take into account kinetics. A common academic approach is to take total tumor burden and assume it consists of a treatment-sensitive fraction that decays and a treatment-resistant fraction that grows. If you fit a curve with a decay constant and a growth constant, the growth constant has been shown to be better associated with survival. That makes sense because tumors kill by growing.

The challenge is that systems like RECIST depend on picking a few tumors and measuring them. Anytime you sample, you assume the average drives the outcome, but that is not true. What kills the patient is the worst tumor, the most resistant and most aggressive. If you did not pick that tumor for measurement, you are blind to it. You want to measure everything.

You also want a full 3D outline of the entire tumor. Radiologist time is expensive, and drawing every tumor in 3D is not scalable. AI can do it. Tools do not yet automatically find every tumor, but once you point to a tumor, AI can draw it in 3D, propagate it across scans, make all the measurements, and feed that into modeling.
With a combination of direct AI assessments and assessments assisted by AI, you could get better measurements of efficacy and make better decisions about a drug pipeline, combinations, and dose selection.

Rohit: That’s a great explanation, thank you for that, Greg. So as you mentioned, a lot of this is exploratory and academic in nature. When do you see it moving into industry grade regulatory practices, because pharma, biotech, biopharma is a highly regulated industry segment? What are some of your thoughts around that space and any FDA insights you can provide?

Greg: What I would say is, first of all, with regard to the kinds of measurements I’ve talked about, you notice that I talked about using them in early phase decision making. This is purely internal decision making.

Now, as you move forward, pharma decision makers are conservative and risk averse. You have to do a lot of validation first. You validate internally, and the initial uses are internal uses. As evidence starts to accumulate, you can start building these more advanced assessments into phase two trials as exploratory endpoints.

In order for the FDA to accept a measurement of any kind, whether AI based or not, as an outcome in a trial, there needs to be strong evidence that those measurements are strongly associated with clinical benefit. Essentially, the process is that you start with internal decision making, then build it in as exploratory endpoints, then accumulate data and submit that to regulators as part of the overall package. As experience builds up, you can start thinking about using it to supplement regulatory decisions like supporting breakthrough designation or accelerated approval, and then someday potentially as primary endpoints.

The FDA put out guidance in January of this year on the use of AI in regulatory decision making. They laid out a framework for building what they call a credibility assessment framework. You have to define the context of use clearly, then assess the model risk. If you are making decisions about treatment or endpoints based on the output of an AI model, you have to consider whether it is the only factor in the decision or just a contributing factor, and what the potential risk is if it is wrong.

If you are making a treatment decision based on AI output, that is high risk. If the decision is not as directly about care, the risk is lower. You have to assess model risk and generate evidence commensurate with that level of risk. The evidence has to be generated in intelligent ways, avoiding pitfalls like data commingling or circularity. If you train a model on data and then test it on overlapping data, that can give inaccurate results that do not generalize. You have to think through all the pitfalls to build evidence the FDA can accept.

Right now, everybody is in exploratory mode and figuring out how to do this. The guidance from FDA in January was draft guidance. These draft guidances can remain in draft for years and reflect the latest thinking. There have been responses and published analyses, and pharma and bio have generated organized responses. That is where I would suggest people go to read about the FDA’s thinking.

The bottom line is whatever you are thinking of doing with AI in biopharma, if you want to bring it to regulators eventually, you need to engage early. Sit down and say, here is our plan, here is the technology, here is the data we trained it on, here is how we validated it, here is the context of use, here is the risk, and use that to build the credibility assessment framework.

Rohit: That’s amazing. And then Greg, you are AI in a very large biopharma organization. What are some of the challenges you run into in such environments and how do you overcome them? What are some strategies that you have followed successfully, if you would like to share?

Greg: Sure. So it’s a great question. One thing that big pharma has in massive amounts is data. If you are in a pharma that has run a large clinical development program, you are drowning in data. Some challenges are around data standardization. You will have legacy systems, and a big pharma may have dozens of legacy systems that collect data and cannot necessarily talk to each other. When you want a model to look at data, it helps to have a data standard.

That is something that helped radiology, which adopted AI early because radiology has the DICOM data standard. No matter the scanner manufacturer, they all generate DICOM data. That was possible because there were only a few major equipment manufacturers whose data engineers could agree on a standard. Other kinds of data are not standardized that way. That is an important challenge and a market opportunity for companies that want to create tools to interconvert, harmonize, and unify data. AI would be useful for this.

Another aspect is sensitivity around data access and patient consent in clinical trials. Clinical trial data is sensitive, and you have to put risk mitigation strategies in place to ensure you are not generating outputs that confuse or undermine what is known about a medication. You also have to think about whether subjects in a trial consented to that use of their data. The mitigation is working with people writing informed consent language to clarify that these additional uses are permissible.

Finally, aside from data, there is the people side. Data scientists and AI engineers are brilliant, but they may not understand the problems that need to be solved. They may come up with solutions in search of problems or solutions that are difficult to apply. Any organization building an AI team needs to bring together people who understand the challenges, like clinical developers and strategists, with people who understand the tools, like data scientists and statisticians. Statisticians are critical because of pitfalls like overlapping data and overtraining. What you want in an AI team is a multidimensional view of the problems and tools, and collaboration to solve the problems.

Rohit: That’s a great insight. So as we come to the end of the podcast, Greg, what are your thoughts for the future? When you look into the crystal glass, what small, medium, or big changes do you see coming our way?

Greg: People are enthusiastic that it will do everything, walk your dog, run your clinical trial, and butter your bread. AI is a set of tools, potentially very powerful, but you have to use it with caution because how you train and validate it matters.

In the future, what I envision is the early phase of drug development getting faster as AI is applied to molecule design, preclinical aspects, and extracting more information in clinical development from each subject. In later phases, AI will make trials more efficient in workload, resourcing, and insights around recruitment, like where to find the right patients.

So I see an acceleration of drug development through each stage that AI can enable as long as it is done with appropriate ethical safeguards and scientific rigor.

Rohit: Absolutely. And it will bring drugs to market faster if we are able to envision that. Thank you, Greg. I really appreciate it. These were very insightful thoughts you provided in this podcast. Anything you would like to say in closing?

Greg: Thank you very much for inviting me, and I look forward to seeing some other guests on your show now that I am aware of it.

Rohit: Thank you, Greg. Really appreciate it.

————

Subscribe to our podcast series at www.thebigunlock.com and write us at [email protected]    

Disclaimer: This Q&A has been derived from the podcast transcript and has been edited for readability and clarity.