What's super cool to me about this paper is that it does a longitudinal analysis - so many audit studies stick to a single point in time, and this paper is a great demonstration that the data available to you at that time will inevitably impact your measurements.

🔗: dl.acm.org/doi/10.1145/...

The authors find that delays in reporting patient demographics are the norm (for >50% of patients, race is reported >60 days after other details like DOB). These delays obfuscate measurement of health outcomes and health disparities, and techniques like imputing race do not improve measurement.

⏳ Bias Delayed is Bias Denied? Assessing the Effect of Reporting Delays on Disparity Assessments by @jennahgosciak.bsky.social and @aparnabee.bsky.social et al. (incl. @allisonkoe.bsky.social @marzyehghassemi.bsky.social) analyzes how missing demographic data impacts estimates of health disparities.

I love how this paper emphasizes that evaluation != accountability and recommends steps towards accountability: ensuring that tools are open, valid, and reliable; focusing on tools to support participatory methods; and ensuring auditors are protected from retaliation.

🔗: dl.acm.org/doi/full/10....

The authors analyze 435(!) tools, finding that most focus on evaluation – but tools for other aspects of AI audits, like harms discovery, communicating audit results, and advocating for change, are less common. Further, while many tools are freely available, auditors often struggle to use them.

Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling by @victorojewale.bsky.social @rbsteed.com @briana-v.bsky.social @abeba.bsky.social @rajiinio.bsky.social compares the landscape of AI audit tools (tools.auditing-ai.com) to the actual needs of AI auditors.

This work won a 🏆Best Paper Award🏆 at FAccT! I think it's a fantastic example of an external audit that not only identifies a problem but also provides concrete steps towards a solution.

🔗: dl.acm.org/doi/10.1145/...

The authors show that the external review established by the settlement is insufficient to guarantee that Meta is actually reducing discrimination in ad delivery (as opposed to adversarially complying by showing the same ad repeatedly to one person or applying VRS only to small ad campaigns).

📱 External Evaluation of Discrimination Mitigation Efforts in Meta's Ad Delivery by Imana et al. audits VRS (Meta’s process for reducing bias in ad delivery as part of a settlement with DOJ), and finds VRS reduces demographic differences in ad audience – but also reduces reach and increases cost.