arxiv.org/abs/2009.14050

leave parrots alone!!

Cool that those experiments changed your mind :) The referenced appendix was important to convince myself of what we eventually concluded (that the correlations indicate procedural knowledge). And thank you for the praise!! What kind of ideas did you get?

If that's how you define retrieval, then they are doing retrieval under your definition. The heavy lifting is of course done by the word "synthesize", how do they do that? That's what we're characterising in the paper

It should be much less computationally expensive to do for fine tuning data

To do this you can use Juhan's recent work on Source! arxiv.org/abs/2405.12186

Just want to add to Stella's responses that the reason we went with procedural knowledge very much came from the correlation results; documents influence each query with the same underlying task similarly, even though the task is applied to different numbers for different queries.

Definitely! Next time will be Christmas so I presume that's not ideal, but I can reach out when I know the next time I will be there?

What did you think

Yeah. I do think as you become more senior you become better at determining from the intro whether a paper is likely to be good or bad. The point is just that we should still actively keep an open mind when reading the rest of the paper

I learn so much from reviewing, it’s the papers I review that I keep coming back to for my own ideas and citations. They broaden and deepen my view on the field. Let’s give it the time it deserves.

It’s actually pretty cool if you as a reviewer get to make papers better by suggesting improvements. This cycle, I’ve given an 8 where all other reviewers gave a rejecting rating. Now, the scores are 8, 5, 8, 6, 8. Pretty exciting, if you ask me.

You don’t have to add these to the review (unless it’s TMLR). But hold yourself accountable when you are rejecting it. What could the authors do to lift your scores? If the answer is nothing, be sure to have a good reason for this. If there is something, tell the authors.

There’s an easy way to hold yourself accountable (thanks TMLR guidelines ✌️): "make a list of proposed adjustments to the submission, specifying for each whether they are critical to securing your recommendation for acceptance or would simply strengthen the work in your view."

The art of rebuttal is to learn how to stick firmly to the points you believe are important, while at the same time allowing yourself to be wrong. Admitting when you might be misunderstanding (after all, the authors probably spent about ~1000x more time thinking about it).

The hardest part is to keep an open mind all the way down 🐢. The rebuttal phase is the kicker. If you don’t spend enough time in this phase, just don’t sign up to be a reviewer, because it’s incredibly demoralising to people who work months to years on a submission.

I’ve heard people say they know whether they will accept or reject a paper after reading the abstract/intro. That’s great, but what is even greater is when you realise that is *just presentation*, and the soundness and contribution are *not* going to be determined by that part.

Reviewing requires constant questioning of the motive behind your responses, every step of the way. Which btw, according to chatty, will help you become a better scientist yourself.