I would love to see if findings replicate once you remove keywords that obviously indicate that the researchers had some statistically significant and theoretically important findings.

Might there be a chance that the researchers show are publishing in high-impact journals (which is the indicator of "quality" here) after Chat-GPT release are actually finding more "significant" results for unrelated reasons? (e.g., luck, better research ideas)

The authors could have learned a thing or two from sociology! www.journals.uchicago.edu/doi/10.1086/...

There may be a trade-off between the cognitive resources we need to process large amounts of data and carefully examining data. By the same logic, feedback from 1-2 careful readers may be more useful than from several readers who lack the skills or willingness to appropriately engage with your work.

But can we reduce noise in the data we do have? I think yes, and much less advice exists out there about how to do that. I think we can reduce noise through thoughtful, unemotional reflection about the data that we already have. It means, not necessarily reading more, but reading carefully.

How to get more data? Read more. Write & submit more. And get tons of feedback from others before submitting. We've all heard this advice.