Transform your business data workflows with Matasoft’s AI-driven spreadsheet processing services and software. (Un)Perplexed Spready, powered by Perplexity AI, automates data extraction, categorizatio...

The ODC-TBI Editorial Board has just voted to encourage the use of a set of standard Common Data Elements (CDEs) for datasets submitted to ODC-TBI.

Transform your business data workflows with Matasoft’s AI-driven spreadsheet processing services and software. (Un)Perplexed Spready, powered by Perplexity AI, automates data extraction, categorizatio...

Background: Integrating data is essential for advancing clinical and epidemiological research. However, because datasets often describe variables (e.g., demographic, health conditions, etc.) in diverse ways, the process of integrating and harmonizing variables from research studies remains a major bottleneck. Objective: The objective was to assess a natural language processing (NLP)-based method to automate variable harmonization to achieve a scalable approach to integration of multiple datasets. Methods: We developed a fully connected neural network (FCN) method, enhanced with contrastive learning, using domain-specific embeddings from the BioBERT language representation model, using three cardiovascular datasets: the Atherosclerosis Risk in Communities (ARIC) study, the Framingham Heart Study (FHS) and the Multi-Ethnic Study of Atherosclerosis (MESA). We used metadata variable descriptions and curated harmonized concepts as ground truth. We framed the problem as a paired sentence classification task. The accuracy of this method was compared to a logistic regression baseline method. To assess the generalizability of the trained models, we also evaluated their performance by separating the three datasets when preparing the training and validation sets. Results: The newly developed fully connected neural network (FCN) achieved a top-5 accuracy of 98.95% (95% CI: 98.31%-99.47%) and an AUC of 0.990 (95% CI: 0.988-0.991), outperforming the standard logistic regression model, which exhibited a top-5 accuracy of 22.23% (95% CI: 19.91% - 24.87%) and an AUC of 0.824 (95% CI: 0.815 – 0.834). The contrastive learning enhancement also outperformed the logistic regression model, although slightly below the base FCN model, exhibiting a top-5 accuracy of 89.88% (95% CI: 87.88% - 91.68%) and an AUC of 0.977 (95% CI: 0.975 – 0.979). Conclusions: This novel approach provides a scalable solution for harmonizing metadata across large-scale cohort studies. The proposed method significantly enhances the performance over the baseline method by utilizing learned representations to categorize harmonized concepts more accurately for cohorts in cardiovascular disease and stroke.

Data harmonization is essential to produce multi-cohort research. Learn more about best practices for harmonizing large-scale cohort data.