If you want to learn more about how to apply influence functions to diffusion models, and the key take-aways for their use in this setting, check-out the paper! arxiv.org/abs/2410.13850

For example: for even moderately sized datasets, the trained diffusion models' marginal probability distribution stays the same irrespective of which examples were removed from the training data, potentially making the influence functions task vacuous.

We also point out several empirical challenges to the use of influence functions in diffusion models.

In our paper, we empirically show that the choice of GGN and K-FAC approximation is crucial for the performance of influence functions, and that following our recommended design principles leads to the better performing approximations.

Influence functions require the training loss Hessian matrix. Typically, a K-FAC approximation to a Generalised Gauss-Newton (GGN) matrix is used instead of the Hessian. However, it's not immediately obvious which GGN and K-FAC approximations to use in the diffusion

Influence functions are already being used in deep learning, from classification and regression through to autoregressive LLMs. What's the challenge in adapting them to the diffusion setting?

• Identifying and removing data responsible for undesirable behaviours (e.g. generating explicit content)
• Data valuation (how much did each training datapoint contribute towards generating the samples my users pay me for?)

Answering how a model's behaviour changes upon removing training datapoints could help with:
• Quantifying impact of copyrighted data on a given sample (how much less likely is it that the model would generate this image if not for the works of a given artist?)

Influence functions attempt to answer: how would the model's behaviour (e.g. probability of generating an image) change if the model was trained from scratch with some training datapoints removed.

They give an approximate answer, but without actually retraining the model.

It’s an awesome piece of work, done on a surprisingly small budget compared to the performance