Preview abstract
As generative AI technologies are increasingly being launched across the globe, assessing their competence to operate in different cultural contexts is increasingly becoming a priority. Recent years have seen numerous and much-needed efforts on cultural benchmarking, these efforts tend to focus on specific aspects of culture. While these efforts contribute to our understanding of cultural competence, a unified and systematic evaluation approach is needed for us as a field to comprehensively assess diverse cultural dimensions at scale. In this paper, we propose a principled approach towards developing a unified framework for evaluating cultural intelligence of models. We start by developing a working definition of culture that includes identifying core domains of culture. We then introduce a broad-purpose, systematic, and extensible framework for assessing cultural intelligence of AI systems. Drawing on theoretical framing from psychometric measurement validity theory, we decouple the background concept (i.e., cultural intelligence) from its operationalization via measurement. We conceptualize cultural intelligence in terms of a set of core capabilities that span different cultural domains, before operationalizing them in terms of a set of indicators designed to reliably measure these capabilities collectively. Finally, we identify the considerations, challenges, and research pathways to meaningfully measure these indicators, specifically focusing on data collection, probing strategies, and evaluation metrics.View details
Preview abstract
This course explores how researchers and practitioners can engage ethically with Indigenous communities when
developing AI- and data-intensive applications. Some key issues such as fair engagement, legal constraints, reciprocity, and informed consent are discussed based on the examples drawn from the instructors’ experience. The course also examines good practices in terms of co-designing and co-development processes, data governance and sovereignty issues and systems, decolonial software licensing, and processes of technology transfer and appropriation. In its practical part, the course critically discusses examples and cases gathered from the audience to explore the diversity of issues and solutions when working with Indigenous communities.View details
Preview abstract
This course explores how researchers and practitioners can engage ethically with Indigenous communities when
developing AI- and data-intensive applications. Some key issues such as fair engagement, legal constraints, reciprocity, and informed consent are discussed based on the examples drawn from the instructors’ experience. The course also examines good practices in terms of co-designing and co-development processes, data governance and sovereignty issues and systems, decolonial software licensing, and processes of technology transfer and appropriation. In its practical part, the course critically discusses examples and cases gathered from the audience to explore the diversity of issues and solutions when working with Indigenous communities.View details
Preview abstract
Large Language Models (LLMs) have rapidly increased in size and apparent capabilities in the last three years but their training
data is largely English text. There is growing interest in language inclusivity in LLMs, and various efforts are striving for models
to accommodate language communities outside of the Global North1
, which include many languages that have been historically
underrepresented digitally. These languages have been coined as “low resource languages” or “long tail languages”, and LLMs
performance on these languages is generally poor. While expanding the use of LLMs to more languages may bring many potential
benefits, such as assisting cross-community communication and language preservation, great care must be taken to ensure that
data collection on these languages is not extractive and that it does not reproduce exploitative practices of the past. Collecting
data from languages spoken by previously colonized people, indigenous people, and non-Western languages raises many complex
sociopolitical and ethical questions, e.g., around consent, cultural safety, and data sovereignty. Furthermore, linguistic complexity and
cultural nuances are often lost in LLMs. This position paper builds on recent scholarship, and our own work, and outlines several
relevant social, cultural, and ethical considerations and potential ways to mitigate them through qualitative research, community
partnerships and participatory design approaches. We provide twelve recommendations for consideration when collecting language
data on underrepresented language communities outside of the Global North.View details