i4Health - Publications

scACORN: Context-engineered Agent Orchestration of Specialized Small Language Models for single-cell Transcriptomic Interpretation

Single-cell atlases now exceed 66 million cells, but turning a ranked expression profile and a free-form biological question into a reliable, evidence-grounded answer remains unsolved. Scaling a single model does not resolve this, because single-cell interpretation is a heterogeneous family of tasks whose correct answer depends on tissue, cohort, perturbation and annotation resolution. Here we present scACORN, an agentic alternative to monolithic single-cell language models that combines specialized small language models with context-engineered agent orchestration for their selection and composition at inference time. Each expert is built in two stages: domain-aligned contrastive adaptation fits a pretrained cell-to-text backbone to the transcriptomic geometry of a target dataset, and geometry-preserving specialization learns question-conditioned biological completions without eroding that geometry. A fixed orchestrating language model agent then selects and combines experts under a natural-language playbook that is itself optimized from textual feedback, with no gradient updates to the orchestrator. Across 10 Tabula Sapiens tissues, domain alignment raised transfer macro-F1 from 0.36 to 0.64 and Recall@5 from 0.87 to 0.97; specialized experts reached 0.89 mean exact-match annotation accuracy; and playbook optimization reduced unsupported gene citations from 14.5% to 3.5%. Our findings support specialization and orchestration as complementary responses to the heterogeneity and evidentiary demands of single-cell analysis.

[arxiv]

OPENMEDREASON: Scientific Reasoning Supervision for Medical Vision–Language Models

High-stakes clinical use of large vision–language models (LVLMs) requires reasoning that is grounded in visual evidence and clinical knowledge, not just correct final answers. We introduce OPENMEDREASON, a large-scale, open multimodal medical reasoning corpus comprising approximately 450K image–question–answer instances whose reasoning traces are primarily derived from curated biomedical, human-authored scientific articles. OPENMEDREASON provides high-fidelity supervision beyond synthetic chains of thought, covering diverse medical domain vision modalities such as radiological scans, microscopic images, visible light photographs, charts, and others. We complement it with OPENMEDREASON-Bench, a held-out benchmark that allows fine-grained evaluation of LVLMs along three complementary axes of capability, including perception, medical knowledge, and rationale, enabling diagnostic evaluation beyond final-answer accuracy. OPENMEDREASON is a rich training resource that exhibits its effectiveness in both supervised fine-tuning (SFT) and reinforcement-based alignment. Training with OPENMEDREASON yields a 20% average improvement in VQA accuracy over the base model and achieves performance within 4.2% of the strongest comparablescale medical LVLMs. Fine-grained performance analysis confirms that the gains are not concentrated in any single axis: OPENMEDREASON improves perception, medical knowledge, and rationale jointly, and its reasoning traces are preferred over those of the base model in 86.1% of pairwise comparisons. We release the code and dataset at huggingface.co/datasets/neginb/OpenMedReason

[EACL]

When Can We Trust LLMs in Mental Health? Large-Scale Benchmarks for Reliable LLM Evaluation

Evaluating Large Language Models (LLMs) for mental health support is challenging due to the emotionally and cognitively complex nature of therapeutic dialogue. Existing benchmarks are limited in scale, reliability, often relying on synthetic or social media data, and lack frameworks to assess when automated judges can be trusted. To address the need for largescale dialogue datasets and judge-reliability assessment, we introduce two benchmarks that provide a framework for generation and evaluation. MentalBench-100k consolidates 10,000 one-turn conversations from three real scenarios datasets, each paired with nine LLM-generated responses, yielding 100,000 response pairs. MentalAlign-70k reframes evaluation by comparing four high-performing LLM judges with human experts across 70,000 ratings on seven attributes, grouped into Cognitive Support Score (CSS) and Affective Resonance Score (ARS). We then employ the Affective–Cognitive Agreement Framework, a statistical methodology using intraclass correlation coefficients (ICC) with confidence intervals to quantify agreement, consistency, and bias between LLM judges and human experts. Our analysis reveals systematic inflation by LLM judges, strong reliability for cognitive attributes such as guidance and informativeness, reduced precision for empathy, and some unreliability in safety and relevance. Our contributions establish new methodological and empirical foundations for reliable, largescale evaluation of LLMs in mental health. We release the benchmarks and codes at: https://github.com/abeerbadawi/MentalBench/

[MICCAI]

Advancing Medical Representation Learning Through High-Quality Data

Despite the growing scale of medical Vision-Language datasets, the impact of dataset quality on model performance remains underexplored. We introduce Open-PMC, a high-quality medical dataset from PubMed Central, containing 2.2 million image-text pairs, enriched with image modality annotations, subfigures, and summarized in-text references. Notably, the in-text references provide richer medical context, extending beyond the abstract information typically found in captions. Through extensive experiments, we benchmark Open-PMC against larger datasets across retrieval and zero-shot classification tasks. Our results show that dataset quality-not just size-drives significant performance gains. We complement our benchmark with an in-depth analysis of feature representation. Our findings highlight the crucial role of data curation quality in advancing multimodal medical AI. We release Open-PMC , along with the trained models and our codebase.

Collaboration work with Vector Institute

[JMIR]

Advancing Psychiatric Safety With the Predictive Risk Identification for Mental Health Events Tool: Retrospective Cohort Study

We introduce the Predictive Risk Identification for Mental Health Events (PRIME) tool, a deep learning–based EWS trained on longitudinal psychiatric electronic medical record (EMR) data to anticipate adverse events in 24-hour windows. A retrospective cohort study using routinely collected EMR data to train and validate machine learning (ML) models for short-term risk prediction was conducted. A total of 4651 patients and 403,098 encounters from January 2020 to August 2024 were included. For model evaluation, the 2024 test set included 900 patients and 48,313 encounters.

Collaboration work with Waypoint Center

[ICML]

Position: Beyond Assistance – Reimagining LLMs as Ethical and Adaptive Co-Creators in Mental Health Care

This position paper argues for a fundamental shift in how Large Language Models (LLMs) are integrated into the mental health care domain. We advocate for their role as co-creators rather than mere assistive tools. While LLMs have the potential to enhance accessibility, personalization, and crisis intervention, their adoption remains limited due to concerns about bias, evaluation, over-reliance, dehumanization, and regulatory uncertainties. To address these challenges, we propose two structured pathways: SAFE-i (Supportive, Adaptive, Fair, and Ethical Implementation) Guidelines for ethical and responsible deployment, and HAAS-e (Human-AI Alignment and Safety Evaluation) Framework for multidimensional, human-centred assessment. SAFE-i provides a blueprint for data governance, adaptive model engineering, and real-world integration, ensuring LLMs align with clinical and ethical standards. HAAS-e introduces evaluation metrics that go beyond technical accuracy to measure trustworthiness, empathy, cultural sensitivity, and actionability. We call for the adoption of these structured approaches to establish a responsible and scalable model for LLM-driven mental health support, ensuring that AI complements, rather than replaces, human expertise.

[JAMANetworkOpen]

Diagnosing and remediating harmful data shifts for the responsible deployment of clinical AI models

Harmful data shifts occur when the distribution of data used to train a clinical AI system differs significantly from the distribution of data encountered during deployment, leading to erroneous predictions and potential harm to patients. We evaluated the impact of data shifts on an early warning system for in-hospital mortality that uses electronic health record data from patients admitted to a general internal medicine service, across 7 large hospitals in Toronto, Canada. To explore the robustness of the model, we evaluated potentially harmful data shifts across demographics, hospital types, seasons, time of hospital admission, and whether the patient was admitted from an acute care institution or nursing home, without relying on model performance. Interestingly, many of these harmful data shifts were unidirectional. Overall, our study is a crucial step towards the deployment of clinical AI models, by providing strategies and workflows to ensure the safety and efficacy of these models in real-world settings.

Collaboration work with Vector Institute and GEMINI

[npj Digital Medicine]

The FAIIR conversational AI agent assistant for youth mental health service provision

Frontline crisis support plays a critical role in youth mental health services, where Crisis Responders (CRs) engage in conversations and assign issue tags to guide interventions. To enhance this process, we introduce FAIIR (Frontline Assistant: Issue Identification and Recommendation), an ensemble of domain-adapted transformer models trained on 780,000 conversations. FAIIR aims to reduce CR’s cognitive burden, enhance issue identification accuracy, and streamline post-conversation administrative tasks. Evaluated on retrospective data, FAIIR achieves an average AUC ROC of 94%, an average F1-score of 64%, and an average recall score of 81%. During the silent testing phase, its performance remained robust, with less than a 2% drop in all metrics. CRs exhibited 90.9% agreement with its predictions, and expert agreement with FAIIR exceeded their agreement with original labels. These findings highlight FAIIR’s potential to assist CRs in prioritizing urgent cases and ensuring appropriate resource allocation in crisis interventions.

Collaboration work with Vector Institute and Kids Help Phone

[PlusOne]

Academic case reports lack diversity

Understanding disparities Understanding disparities in the prevalence of Post COVID-19 Condition (PCC) amongst vulnerable populations is crucial to improving care and addressing intersecting inequities. This study aims to develop a comprehensive framework for integrating social determinants of health (SDOH) into PCC research by leveraging natural language processing (NLP) techniques to analyze disparities and variations in SDOH representation within PCC case reports. Following construction of a PCC Case Report Corpus, comprising over 7,000 case reports from the LitCOVID repository, a subset of 709 reports were annotated with 26 core SDOH-related entity types using pre-trained named entity recognition (NER) models, human review, and data augmentation to improve quality, diversity and representation of entity types. An NLP pipeline integrating NER, natural language inference (NLI), trigram and frequency analyses was developed to extract and analyze these entities. Both encoder-only transformer models and RNN-based models were assessed for the NER objective.

Fine-tuned encoder-only BERT models outperformed traditional RNN-based models in generalizability to distinct sentence structures and greater class sparsity, achieving a macro F1-score of 0.72 and macro AUC of 0.99 on a held-out generalization set. Exploratory analysis revealed variability in entity richness, with prevalent entities like condition, age, and access to care, and under-representation of sensitive categories like race and housing status. Trigram analysis highlighted frequent co-occurrences among entities, including age, gender, and condition. The NLI objective (entailment and contradiction analysis) showed attributes like “Experienced violence or abuse” and “Has medical insurance” had high entailment rates (82.4%–80.3%), while attributes such as “Is female-identifying,” “Is married,” and “Has a terminal condition” exhibited high contradiction rates (70.8%–98.5%). Our results highlight the effectiveness of transformer-based NER in extracting SDOH information from case reports. However, the findings also expose critical gaps in the representation of marginalized groups within PCC-related academic case reports, e.g., across gender, insurance status, and age. This work underscores the need for standardized SDOH documentation and inclusive reporting practices to enable more equitable research and inform future health policy and AI model development.

Collaboration work with Vector Institute, Public Health Ontario

[ICML 2025 Workshop on Multi-modal Foundation Models and Large Language Models for Life Sciences (FM4LS)]

Benchmarking Vision-Language Contrastive Methods for Medical Representation Learning

We perform a comprehensive benchmarking of contrastive frameworks for learning multimodal representations in the medical domain. Through this study, we aim to answer the following research questions: (i) How transferable are general domain representations to the medical domain? (ii) Is multimodal contrastive training sufficient, or does it benefit from unimodal training as well? (iii) What is the impact of feature granularity on the effectiveness of multimodal medical representation learning? To answer these questions, we investigate eight contrastive learning approaches under identical training setups, and train them on 2.8 million image-text pairs from four datasets, and evaluate them on 25 downstream tasks, including classification (zero-shot and linear probing), image-to-text and text-to image retrieval, and visual question-answering. Our findings suggest a positive answer to the first question, a negative answer to the second question, and the benefit of learning fine-grained features. Finally, we make our code publicly available.

Collaboration work with Vector Institute

[IEEE Access]

Evaluating knowledge transfer in the neural network for medical images

In this study, we implement different experiments for standard transfer learning approaches as our baseline and introduce the adoption of a novel knowledge transfer approach, called teacher-student learning framework, to improve the performance of diagnostic predictive models in medical imaging. Specifically, we investigate various configurations in the teacher-student learning framework inspired by the activation attention transfer in computer vision models to help address some of the challenges faced in medical imaging, such as the limited availability of annotated data and limited computing resources. We show that the teacher-student learning approach has the potential to significantly improve the performance of diagnostic predictive models. Our findings could have a positive impact on healthcare accessibility and affordability, as they may enable the development of more cost effective and widely available medical imaging technologies under a limited data environment.

[NATURE Scientific Report]

Transitioning sleeping position detection in late pregnancy using computer vision from controlled to real-world settings: an observational study

Sleeping on the back after 28 weeks of pregnancy has recently been associated with giving birth to a small-for-gestational-age infant and late stillbirth, but whether a causal relationship exists is currently unknown and difficult to study prospectively. This study was conducted to build a computer vision model that can automatically detect sleeping position in pregnancy under real-world conditions. Real-world overnight video recordings were collected from an ongoing, Canada-wide, prospective, four-night, home sleep apnea study and controlled-setting video recordings were used from a previous study. Images were extracted from the videos and body positions were annotated. Five-fold cross validation was used to train, validate, and test a model using state-of-the-art deep convolutional neural networks. The dataset contained 39 pregnant participants, 13 bed partners, 12,930 images, and 47,001 annotations.

[YouTube],

Collaboration work with UHN

[PLoS Digital Health]

Sleep in Late Pregnancy: Artificial Intelligence for the Detection and Diagnosis of Disturbances and Disorders (SLeeP AID4)

Recognizing that approximately one-third of the pregnancy is spent asleep, there has been a surge of clinical and research interest over the last two decades in the potential roles that sleep (and poor sleep) during pregnancy might play in adverse pregnancy outcomes. In this project, we are conducting multi-night, multi-participant, in-home sleep apnea studies in late pregnancy and developing video-based sleep apnea diagnosis technology. We believe the old adage, "diagnosis is treatment." This technology will eventually be used to streamline the diagnosis of sleep apnea in pregnancy so that it can be urgently triaged for appropriate management. This technology will also be able to diagnose sleep apnea in the bed partner simultaneously.

[YouTube]

Collaboration work with UHN

[JMIR]

Using Social Media to Help Understand Patient-Reported Health Outcomes of Post–COVID-19 Condition: Natural Language Processing Approach

In response to the emergence of long COVID, we developed an NLP pipeline to facilitate extracting information from user-reported experiences on social media platforms. In this study, we examined the validity and effectiveness of our NLP pipeline to provide insights into patient-reported long COVID-related health outcomes across two popular social media platforms, Twitter and Reddit. In doing so, we extracted symptoms and conditions (SyCo) and estimated their occurrence frequency. We compared the outputs with human annotations and highly utilized clinical outcomes grounded in the medical literature. Lastly, we tracked occurrences of SyCo terms over time and geographies to explore the pipeline's potential to be used as a surveillance tool reflecting users’ opinions and experiences. The outcome of our social media-derived pipeline is comparable with the results of

peer-reviewed articles relevant to long COVID symptoms. Overall, this study provides unique

insights into patient-reported health outcomes from long COVID and valuable information about

the patient’s journey that can help healthcare providers anticipate future needs.

Collaboration work with Vector Institute, Roche, Deloitte, TELUS, and UHN

[JMIR]

Natural Language Processing for Clinical Laboratory Data Repository Systems: Implementation and Evaluation for Respiratory Viruses

This study explores the feasibility of using a deep learning-based natural language processing (NLP) model for information extraction from unstructured laboratory reports. The NLP model, trained on a large corpus of annotated laboratory reports, demonstrated strong performance in extracting clinically meaningful medical concepts. The model's stability and generalizability were evaluated across different test sets, showing consistently high accuracy. The study highlights the potential of deep learning-based NLP models to automate the parsing of laboratory data, enabling scalable and efficient access to valuable information for decision support and analysis.

Collaboration work with Vector Institute and ICES

[EMNLP - ACL Anthology]

Bringing the State-of-the-Art to Customers: A Neural Agent Assistant Framework for Customer Service Support

Building Agent Assistants that can help improve customer service support requires inputs from industry users and their customers, as well as knowledge about state-of-the-art Natural Language Processing (NLP) technology. We combine expertise from academia and industry to bridge the gap and build task/domain-specific Neural Agent Assistants (NAA) with three high-level components for: (1) Intent Identification, (2) Context Retrieval, and (3) Response Generation. In this paper, we outline the pipeline of the NAA's core system and also present three case studies in which three industry partners successfully adapt the framework to find solutions to their unique challenges. Our findings suggest that a collaborative process is instrumental in spurring the development of emerging NLP models for Conversational AI tasks in industry.

Collaboration work with Vector Institute, KPMG, PwC, and CIBC

[IEEE] Journal of Translational Engineering in Health and Medicine

A Computer Vision Approach to Identifying Ticks Related to Lyme Disease

In this work, we build an automated detection tool that can differentiate blacklegged ticks from other tick species using advanced computer vision approaches in real-time. Specially, we use convolution neural network models, trained end-to-end, to classify tick species. Also, advanced knowledge transfer techniques are adopted to improve the performance of convolution neural network models. Our best convolution neural network model achieves 92% accuracy on unseen tick species. Our proposed vision-based approach simplifies tick identification and contributes to the emerging work on public health surveillance of ticks and tick-borne diseases. In addition, it can be integrated with the geography of exposure and potentially be leveraged to inform the risk of Lyme disease infection. This is the first report of using deep learning technologies to classify ticks, providing the basis for automation of tick surveillance, and advancing tick-borne disease ecology and risk management.

Collaboration work with Vector Institute and Public Health Ontario

Page updated

Google Sites

Report abuse