A part of our work-in-progress on the overall area of human agency and oversight in algorithmic processes has been recently accepted to the 9th AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2021) that will be held November 14–18th as a virtual conference. This work is a preliminary and early first step towards trustworthy AI and algorithmic systems as described in my Ph.D. proposal. This is the first published work that goes under my research activity as a doctoral student.
A few words about the conference
HCOMP is the premier venue for disseminating the latest research findings on human computation and crowdsourcing. While artificial intelligence (AI) and human-computer interaction (HCI) represent traditional mainstays of the conference, HCOMP believes strongly in fostering and promoting broad, interdisciplinary research. Our field is particularly unique in the diversity of disciplines it draws upon and contributes to, including human-centered qualitative studies and HCI design, social computing, artificial intelligence, economics, computational social science, digital humanities, policy, and ethics. We promote the exchange of advances in human computation and crowdsourcing not only among researchers, but also engineers and practitioners, to encourage dialogue across disciplines and communities of practice.
HCOMP 2021 builds on a successful history of past meetings: eight HCOMP conferences (2013–2020) and four earlier workshops, held at the AAAI Conference on Artificial Intelligence (2011–2012), and the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (2009–2010).
Conference Website: https://www.humancomputation.com/
Abstract
Various stakeholders have called for human oversight of algorithmic processes, as a means to mitigate the possibility for automated discrimination and other social harms. This is even more crucial in light of the democratization of AI, where data and algorithms, such as Cognitive Services, are deployed into various applications and socio-cultural contexts. Inspired by previous work proposing human-in-the-loop governance mechanisms, we run a feasibility study involving image tagging services. Specifically, we ask whether micro-task crowdsourcing can be an effective means for collecting a diverse pool of data for evaluating fairness in a hypothetical scenario of analyzing professional profile photos in a later phase. In this work-in-progress paper, we present our proposed oversight approach and framework for analyzing the diversity of the images provided. Given the subjectivity of fairness judgments, we first aimed to recruit a diverse crowd from three distinct regions. This study lays the groundwork for expanding the approach, to offer developers a means to evaluate Cognitive Services before and/or during deployment.
