When papers are published, even in major medical journals, they are almost never published with the code used to retrieve or analyze the data and you can rarely even get the data. I know from reviewing student work, that it can take months to vet an ad hoc pipeline.

I am struggling to assemble a systematic pipeline for observational studies using routinely collected clinical data. The data we start with is a mess and I can only find general suggestions from data science to begin to vet and clean the data. Cleaning decisions affect later analysis and results.

Yikes. That's bad.

www.youtube.com/watch?v=gk9-...

My parents are on social security and get food from a local food bank. And yes, they voted for this. They have going further and further down the Fox News rabbit hole for the past 20 years.

Evidence that 4 year olds are better than Fox News viewers ;-)

We have to validate all of the output, but that is easier than filling in the tables ourselves, since most of the extraction was correct.

We gave it a blank table with column and row names and a paper or link to the registry and had it fill the table in. In one case, the paper was a simulation, not an RCT/TTE so Claude changed the table to fit a simulation study and extracted related elements.

A PhD student and I are currently finishing up an analysis in which we had Claude 3.7 extract elements of RCTs and matching TTEs from papers and the clinical trial registry. Claude was nearly perfect except for some edge conditions and confusion over dates. It was zero shot learning.

Until Claude 3.5 (12/24 for me) I saw similar results. LLMs were largely useless. In many cases Claude is still better than GPT, even the more expensive models. Copilot was very stripped down. I would not have expected much from it.

I found the same thing a year ago. That changed with Claude 3.5. 4.5 is even better. What kinds of questions are you asking? It might be domain-specific.

I have been learning about causal inference and doing informatics work in it for several years. The chat above came after all of that and then hours reading primary sources. Note the use of critical thinking (on my part) throughout the chat.

The point of showing these chats and LLM output is to demonstrate how LLMs can help humans better understand complex topics and also encourage critical thinking. There are reasons to be concerned about AI, but they don't have destroy education or critical thinking.

It is going to take quite some time and reading of other primary sources to turn this into a full blown teaching tutorial. But the LLM really helped me understand issues beyond what I read from authoritative sources.