Eva Vivalt
@evavivalt.bsky.social
Assistant prof in economics at the University of Toronto, research on cash transfers and evidence-based decision-making, J-PAL affiliate. https://evavivalt.com/
That's a great question. We hope that soon there will be a few "sets" of related projects to forecast, via collaborations, which could let us look at this, but to date we have only had studies on similar topics be posted on the platform by chance, so I'd say there's not enough info quite yet.
November 25, 2025 at 3:16 AM
That's a great question. We hope that soon there will be a few "sets" of related projects to forecast, via collaborations, which could let us look at this, but to date we have only had studies on similar topics be posted on the platform by chance, so I'd say there's not enough info quite yet.
Several excellent RAs worked on this project, most recently Kevin Didi, Malek Hassouneh, Rohan Jha, and Francis Priestland.
Kevin is a pre-doc of mine currently applying to PhD programs! He also assisted with the recent guaranteed income papers. He's great - watch out for him!
Kevin is a pre-doc of mine currently applying to PhD programs! He also assisted with the recent guaranteed income papers. He's great - watch out for him!
November 24, 2025 at 6:39 PM
Several excellent RAs worked on this project, most recently Kevin Didi, Malek Hassouneh, Rohan Jha, and Francis Priestland.
Kevin is a pre-doc of mine currently applying to PhD programs! He also assisted with the recent guaranteed income papers. He's great - watch out for him!
Kevin is a pre-doc of mine currently applying to PhD programs! He also assisted with the recent guaranteed income papers. He's great - watch out for him!
Together, these results show that forecast data contain real, exploitable information that can help decision-makers prioritize projects and treatment arms, benchmark expected effect sizes, and design better studies.
For more results, check out the paper! www.nber.org/papers/w34493
For more results, check out the paper! www.nber.org/papers/w34493
Forecasting Social Science: Evidence from 100 Projects
Founded in 1920, the NBER is a private, non-profit, non-partisan organization dedicated to conducting economic research and to disseminating research findings among academics, public policy makers, an...
www.nber.org
November 24, 2025 at 3:43 PM
Together, these results show that forecast data contain real, exploitable information that can help decision-makers prioritize projects and treatment arms, benchmark expected effect sizes, and design better studies.
For more results, check out the paper! www.nber.org/papers/w34493
For more results, check out the paper! www.nber.org/papers/w34493
Again, one of the advantages of this paper is that we can track individuals. There is a lot of work on, e.g., the wisdom of crowds, perhaps in part because you can estimate effects of crowds without a panel. But getting the right people to forecast matters a lot, too. (Both are important!)
November 24, 2025 at 3:43 PM
Again, one of the advantages of this paper is that we can track individuals. There is a lot of work on, e.g., the wisdom of crowds, perhaps in part because you can estimate effects of crowds without a panel. But getting the right people to forecast matters a lot, too. (Both are important!)
Our results also highlight that who is placing the forecasts really matters. We see a clear gap between academics and non-academics, but field and subfield expertise don't improve accuracy in a meaningful way.
November 24, 2025 at 3:43 PM
Our results also highlight that who is placing the forecasts really matters. We see a clear gap between academics and non-academics, but field and subfield expertise don't improve accuracy in a meaningful way.
Our panelists are great!
We have a paid forecaster panel that takes the majority of the surveys posted on the platform. And they do very well in comparison to other users.
We have a paid forecaster panel that takes the majority of the surveys posted on the platform. And they do very well in comparison to other users.
November 24, 2025 at 3:43 PM
Our panelists are great!
We have a paid forecaster panel that takes the majority of the surveys posted on the platform. And they do very well in comparison to other users.
We have a paid forecaster panel that takes the majority of the surveys posted on the platform. And they do very well in comparison to other users.
In other words, controlling for forecaster fixed effects, we regain the result that confidence has a (weak) positive correlation with accuracy. But some people are much better forecasters than others, and those who place more accurate forecasts also tend to be more uncertain.
November 24, 2025 at 3:43 PM
In other words, controlling for forecaster fixed effects, we regain the result that confidence has a (weak) positive correlation with accuracy. But some people are much better forecasters than others, and those who place more accurate forecasts also tend to be more uncertain.
Interestingly, high self-reported confidence is associated with lower accuracy. This is dissimilar to most of the literature.
In our setting, we can track individual forecasters over time. And thus we can observe: this result is driven by overconfident forecasters.
In our setting, we can track individual forecasters over time. And thus we can observe: this result is driven by overconfident forecasters.
November 24, 2025 at 3:43 PM
Interestingly, high self-reported confidence is associated with lower accuracy. This is dissimilar to most of the literature.
In our setting, we can track individual forecasters over time. And thus we can observe: this result is driven by overconfident forecasters.
In our setting, we can track individual forecasters over time. And thus we can observe: this result is driven by overconfident forecasters.
If you are a funding body, policymaker, or researcher, you could benefit from collecting forecasts, but shrink their effect sizes by about 1/2.
November 24, 2025 at 3:43 PM
If you are a funding body, policymaker, or researcher, you could benefit from collecting forecasts, but shrink their effect sizes by about 1/2.
First result: forecasters tend to overestimate treatment effects - but there is a lot of signal in the forecasts made.
This means that forecasts can be informative in power calculations or determining which interventions to trial.
This means that forecasts can be informative in power calculations or determining which interventions to trial.
November 24, 2025 at 3:43 PM
First result: forecasters tend to overestimate treatment effects - but there is a lot of signal in the forecasts made.
This means that forecasts can be informative in power calculations or determining which interventions to trial.
This means that forecasts can be informative in power calculations or determining which interventions to trial.
Many of the forecasts are thus of *causal* phenomena, like "what effect will X have on Y?"
Super relevant for decision-makers trying to understand the potential impact of their actions.
Super relevant for decision-makers trying to understand the potential impact of their actions.
November 24, 2025 at 3:43 PM
Many of the forecasts are thus of *causal* phenomena, like "what effect will X have on Y?"
Super relevant for decision-makers trying to understand the potential impact of their actions.
Super relevant for decision-makers trying to understand the potential impact of their actions.
For context, this paper is based on the most comprehensive set of forecasts of research results, to our knowledge. It uses data from the Social Science Prediction Platform, a platform researchers use to collect forecasts of what their studies will find.
November 24, 2025 at 3:43 PM
For context, this paper is based on the most comprehensive set of forecasts of research results, to our knowledge. It uses data from the Social Science Prediction Platform, a platform researchers use to collect forecasts of what their studies will find.
Those are my three favorite new plots in the revised paper. Follow for more updates as we continue to put out results about this exciting program.
And here is the full paper: evavivalt.com/wp-content/u... 17/17
And here is the full paper: evavivalt.com/wp-content/u... 17/17
evavivalt.com
September 2, 2025 at 8:38 PM
Those are my three favorite new plots in the revised paper. Follow for more updates as we continue to put out results about this exciting program.
And here is the full paper: evavivalt.com/wp-content/u... 17/17
And here is the full paper: evavivalt.com/wp-content/u... 17/17
But you can check out results from different years in the paper. 16/
September 2, 2025 at 8:38 PM
But you can check out results from different years in the paper. 16/
Most people pointing to the pandemic seem to think effects would be better afterwards.
If anything, during the pandemic the effects on labor supply were more muted. 15/
If anything, during the pandemic the effects on labor supply were more muted. 15/
September 2, 2025 at 8:38 PM
Most people pointing to the pandemic seem to think effects would be better afterwards.
If anything, during the pandemic the effects on labor supply were more muted. 15/
If anything, during the pandemic the effects on labor supply were more muted. 15/
Also, some people have wondered about whether results were driven by the COVID-19 pandemic. We can't make conclusive statements here, but it's important to note the majority of the negative labor supply effects only materialized late. 14/
September 2, 2025 at 8:38 PM
Also, some people have wondered about whether results were driven by the COVID-19 pandemic. We can't make conclusive statements here, but it's important to note the majority of the negative labor supply effects only materialized late. 14/
Importantly, if the transfer were of a shorter duration or if we had followed participants for a shorter period of time, we might have come to very different conclusions! 13/
September 2, 2025 at 8:38 PM
Importantly, if the transfer were of a shorter duration or if we had followed participants for a shorter period of time, we might have come to very different conclusions! 13/
As you can see, the results show a clear time trend, with impacts on employment growing over time until near the end of the program, when the gap starts to close. 12/
September 2, 2025 at 8:38 PM
As you can see, the results show a clear time trend, with impacts on employment growing over time until near the end of the program, when the gap starts to close. 12/
3) We previously included quarterly regression results, but we obtained some updated administrative data and made some nicer plots. For example, here is an event study plot looking at employment status. 11/
September 2, 2025 at 8:38 PM
3) We previously included quarterly regression results, but we obtained some updated administrative data and made some nicer plots. For example, here is an event study plot looking at employment status. 11/
Yes, there are some negative impacts on labor supply and income excluding the transfers. People also do stuff with that money.
This second new figure helps illustrate the overall effects - and what doesn't move. 10/
This second new figure helps illustrate the overall effects - and what doesn't move. 10/
September 2, 2025 at 8:38 PM
Yes, there are some negative impacts on labor supply and income excluding the transfers. People also do stuff with that money.
This second new figure helps illustrate the overall effects - and what doesn't move. 10/
This second new figure helps illustrate the overall effects - and what doesn't move. 10/
2) Zooming out a bit, what can we say about the broader effects of cash transfers?
Some people have focused on the negative effects of cash transfers on labor supply. Others have focused on the consumption the transfers have enabled. 9/
Some people have focused on the negative effects of cash transfers on labor supply. Others have focused on the consumption the transfers have enabled. 9/
September 2, 2025 at 8:38 PM
2) Zooming out a bit, what can we say about the broader effects of cash transfers?
Some people have focused on the negative effects of cash transfers on labor supply. Others have focused on the consumption the transfers have enabled. 9/
Some people have focused on the negative effects of cash transfers on labor supply. Others have focused on the consumption the transfers have enabled. 9/
In any case, improving people's subjective well-being over the long term is hard. This isn't necessarily a fault of the intervention - it could be something about human nature or subjective well-being measures. 8/
September 2, 2025 at 8:38 PM
In any case, improving people's subjective well-being over the long term is hard. This isn't necessarily a fault of the intervention - it could be something about human nature or subjective well-being measures. 8/