Crowdsourcing Human Oversight on Image Tagging Algorithms: An initial study of image diversity

On November 15th and 17th of 2021 at 20:30 EET, I presented our work-in-progress that has been published at the Ninth AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2021) as a Work-in-Progress Paper. The conference has been held online on GatherTown. For the current session, the pre-recorded presentations were presented to the conference participants where they were able to visit the virtual poster booth of the corresponding authors to discuss with them further. This work is part of my doctoral work and the first published paper under the topic of Human Agency and Oversight in Algorithmic Processes.

The paper

Abstract

Various stakeholders have called for human oversight of algorithmic processes, as a means to mitigate the possibility for automated discrimination and other social harms. This is even more crucial in light of the democratization of AI, where data and algorithms, such as Cognitive Services, are deployed into various applications and socio-cultural contexts. Inspired by previous work proposing human-in-the-loop governance mechanisms, we run a feasibility study involving image tagging services. Specifically, we ask whether micro-task crowdsourcing can be an effective means for collecting a diverse pool of data for evaluating fairness in a hypothetical scenario of analyzing professional profile photos in a later phase. In this work-in-progress paper, we present our proposed oversight approach and framework for analyzing the diversity of the images provided. Given the subjectivity of fairness judgments, we first aimed to recruit a diverse crowd from three distinct regions. This study lays the groundwork for expanding the approach, to offer developers a means to evaluate Cognitive Services before and/or during deployment.

Read the paper: https://www.humancomputation.com/assets/wips_demos/HCOMP_2021_paper_104.pdf