Evaluation of Clinical Prediction Models Using Net Benefit versus ROC Curves
Performance evaluation of clinical prediction models involves comparing predicted outcomes with ground truth data using ROC curves. The area under the ROC curve (AUC) is commonly used to assess model performance. A novel method for comparing different models is proposed. Decision-making on treatments, such as drug administration or surgery, can be guided by predictive models that estimate the probability of a patient having a disease. The net benefit versus loss is calculated to determine the threshold probability for treatment decision. The net benefit is weighed against the expected loss for treating a patient without the disease.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Curve decisionali per valutazione di modelli di previsione clinici OVVERO NET BENEFIT VS. ROC CURVE
Per valutare le performance di un modello di previsone solitamente si applica a un dataset noto e si compara in risultato ottenuto (prediction) con la reale situazione del paziente (Ground truth). I risultati sono espressi come area sotto la curva ROC Receiver operating characteristic . Per confrontare due modelli si possono confrontare le rispettive AUC Gli autori propongono un NOVEL* method
NOVEL* NOVEL* method Conferenza di Berlino, 1884 Scramble for Africa
Background La decisione se applicare o meno un trattamento : Somministrazione di un farmaco Applicazione di un protocollo Intervento chirurgico si pu basare sul risultato di un modello di previsione che in genere restituisce una certa probabilit p che il paziente abbia una malattia. Se p abbastanza grande si applica il trattamento. Bisogna pesare il profitto P ottenuto nel curare un paziente con la malattia rispetto alla loss L causata dal trattare un paziente sano. La probabilit di soglia ??sar diversa da situazione a situazione, da medico a medico, da denuncia a denuncia
Background ?? ?? ? > ??with the desease ?? ?? ? > ??without the desease ?? ?? ? < ??with the desease ?? ?? ? < ??without the desease Normalized as TP + FP + FN + TN = 1
Net benefit for the treated Profit P : It is convenient to treat if : Expected benefit for treating a correctly classified patient ? ? > (1 ?) ? e.g. Giving antibiotics for pneumonia Focusing on who recieve the treatment the benefit is : Loss L : ????????= ?? ?? ? ?? ?? ? Expected loss for treating a patient without the desease e.g. Giving antibiotics for the flu Correctly classified Uncorrectly classified
Net benefit for the treated Profit P : Threshold probability: Expected benefit for treating a correctly classified patient ?? ? ? ? (1 ??)= ??= ? + ? e.g. Giving antibiotics for pneumonia The net benefit : Loss L : ?? ?? ? Expected loss for treating a patient without the desease ????????= ?? ?? ?= ?? = ?? ?? ?? ?? e.g. Giving antibiotics for the flu (1 ??)
Extreme Strategy / Perfect Model Prevalence : ? = ?? + ?? =? ??????? ? ????? If I choose to treat regardless of the outcome p: ?????? ???= ? 1 ? ? A perfect model (TP = ? , FN = 0) has the maximun performance ???????????= ? ?= ?? = ? 1 ? (1 ??)
Threshold probability (in this case is high!): CR : Complete responder. Patients who has done CRT and have no tumor left so they DO NOT need surgery ? ??= Loss : ? + ? Someone with a tumor doesn t get surgery I get sued In this case to treat means deciding to avoid surgery Profit : I avoid surgery when not necessary Benefit from a watch and wait strategy
Threshold probability (in this case is high!): CR : Complete responder. Patients who has done CRT and have no tumor left so they DO NOT need surgery ? ??= ? + ? In this case to treat means deciding to avoid surgery At this ??Surgery is the best choise if a prediction is not avaible Currently the protocol is to operate all patients
Net benefit for the Untreated Profit lost from the wrong classification of these patiens. If I have had treated these patiens I would have get this LOSS (so I benefit from my decision) Patient that should have recieved the treatment. The benefit ??????????= ?? ?? ? ?? ?? ? The net benefit : ?? ? ? (1 ??)= ?? ?? (1 ??) ??????????= ?? ?? ??
Extreme Strategy / Perfect Model 1 - Prevalence : 1 ? = ?? + ?? =? ????? ? ????? If I choose not to treat regardless of the outcome p: ?????? ????= (1 ?) ? ? A perfect model (TN = 1 ? , FP = 0) has the maximun performance ?= ???????????= 1 ? = (1 ?) ? (1 ??) ??
Threshold probability (in this case is high!): CR : Complete responder. Patients who has done CRT and have no tumor left so they DO NOT need surgery ? ??= Loss : (that I avoid) Someone with a tumor doesn t get surgery I get sued ? + ? In this case to treat means deciding to avoid surgery Profit : (that I don t make) I avoid surgery when not necessary Benefit from a watch and wait strategy Currently the protocol is to operate all patients
Overall Net benefit Benefit lost from the wrong classification of patiens. Benefit from correct classification: Treated patients with desease and untreated healty patients Patients that should have recieved the treatment or that have recived a wrong terapy The oveall net benefit : ????????= ( ?? ?? + ?? ?? ) ?? ?? (1 ??) ?? ?? ?? ?? (1 ??)
Prevalence : ? = ?? + ?? =? ??????? Relationship with ROC ? ????? The net benefit : Sensitivity ?? ????????= ?? ?? ?? ?? ??+??=?? ?? ?? (1 ??) ??? = ? 1 - specificity ??+??=?? ?? ?? 1 ??? = 1 1 ? ?? ????????= ? ??? 1 ? (1 ???) (1 ??)
ROC curve shows all possible couples of specificity and sensitivity that can be achieve by a prediction model when varying the threshold probability. ?? The optimal point (Youden Index) is the point with the largest vertical distance from the diagonal. Utility of a prediction is the performance near a p that is chosen independently from the model
Conclusions For our model : Clinicians can benefit from automatic classification with texture analysis, but the high LOSS make difficult the application at least in the CR case. For NR patients (a few slide after the conclusion) the situation is worse. Still, texture analysis highlight information that currently is ignored. In general for the net benefit Useful to compare different models when utility is the main concern instead of accuracy. Net benefit is increasingly used in practical applications, including high profile publications.
NR : Non responder. Patients who are still doing CRT but they will not respond to the therapy they SHOULD STOP CRT In this case to treat means deciding to stop CRT ? ??= ? + ?
Loss : Someone with a CRT responsive tumor doesn t get CRT As usual, I get sued ? ??= ? + ? Profit : I avoid CRT (dangerous) I can try another terapy (less dangerous)
