Thank you so much for the invite!

We really hope this analysis can help the community better understand where we come from, where we stand, and what things may help us make meaningful progress in the future.

Co-authored with @jiezhang-ethz.bsky.social, Nicholas Carlini and @floriantramer.bsky.social

arxiv.org/abs/2502.02260

We propose that adversarial ML research should clearly differentiate between two problems:

1️⃣ Real-world vulnerabilities. Attacks and defenses on ill-defined problems are valuable when harm is immediate.

2️⃣ Scientific understanding. We should study well-defined problems.

We are aware that this is not a simple problem and some changes may actually have been for the better! For instance, we now study real-world challenges instead of academic “toy” problems like ℓₚ robustness. We tried to carefully discuss these alternative views in our work.

We identify 3 core challenges that make adversarial ML for LLMs harder to define, harder to solve, and harder to evaluate. We then illustrate these with specific case studies: jailbreaks, un-finetunable models, poisoning, prompt injections, membership inference, and unlearning.

Perhaps most telling, unlike for image classifiers, manual attacks outperform automated methods at finding worst-case inputs for LLMs! This challenges our ability to automatically evaluate the worst-case robustness of protections and benchmark progress.

Now, the field has shifted to LLMs, where we consider subjective notions of safety, allow for unbounded threat models, and evaluate closed-source systems that constantly change. These changes are hindering our ability to produce meaningful scientific progress.

Back in the 🐼 days, we dealt with well-defined tasks: misclassify an image by slightly perturbing pixels within an ℓₚ-ball. Also, attack success and defense utility could be easily measured with classification accuracy. Simple objectives that we could rigorously benchmark.

Looking forward to this presentation. You can add it to your calendar here cohere.com/events/coher...

Recently, we have demonstrated that small amounts of poisoned data posted online could compromise large-scale pretraining with backdoors that persist even after alignment arxiv.org/abs/2410.13722

We poisoned RLHF to introduce backdoors in LLMs that allowed adversaries to elicit harmful generations easily arxiv.org/abs/2311.14455

From left to right the amazing @nkristina.bsky.social @jiezhang-ethz.bsky.social @edebenedetti.bsky.social @javirandor.com @aemai.bsky.social and @dpaleka.bsky.social!

We work on AI Security/Safety/Privacy. Find out more about work in our lab website spylab.ai

Check out all the details in the offical website llmailinject.azurewebsites.net

2) An Adversarial Perspective on Machine Unlearning for AI Safety

🏆 Best paper award
@solarneurips

📅 Sat 14 Dec. Poster at 11am and Talk in the afternoon.
📍 Room West Meeting 121,122

Paper: arxiv.org/abs/2409.18025

1) Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competition.

📅 Fri 13 Dec 4:30 p.m. PST — 7:30 p.m. PST
📍 Spotlight Poster #5203 (West Ballroom A-D)

arxiv.org/abs/2406.07954