Showing 1–14 of 14 results for author: Fazelpour, S

Navigating Epistemic Monocultures in AI-Driven Science: A Simulation Study

Authors: Sina Fazelpour, Joseph O'Brien, Hannah Rubin

Abstract: AI integration into scientific communities promises accelerated discovery but raises concerns about detrimental homogenization. We develop an NK landscape model to explore these promises and risks. We find that non-personalized AI systems that offer uniform guidance yield benefits only under a narrow conjunction of problem structure, practices, and baseline research capabilities, becoming harmful… ▽ More AI integration into scientific communities promises accelerated discovery but raises concerns about detrimental homogenization. We develop an NK landscape model to explore these promises and risks. We find that non-personalized AI systems that offer uniform guidance yield benefits only under a narrow conjunction of problem structure, practices, and baseline research capabilities, becoming harmful otherwise. We implement two proposed mitigations: randomization and personalization. While randomization's utility remains restricted to decomposable problems, personalization can enhance diversity, enabling benefits across a broader range of conditions. Crucially, these benefits are not automatic, but depend on effective institutional adaptation, requiring new standards and practices. △ Less

Submitted 19 August, 2026; originally announced August 2026.

Comments: Accepted at Philosophy of Science

Ambiguity Collapse by LLMs: A Taxonomy of Epistemic Risks

Authors: Shira Gur-Arieh, Angelina Wang, Sina Fazelpour

Abstract: Large language models (LLMs) are increasingly used to make sense of ambiguous, open-textured, value-laden terms. Platforms routinely rely on LLMs for content moderation, asking them to label text based on disputed concepts like "hate speech" or "incitement"; hiring managers may use LLMs to rank who counts as "qualified"; and AI labs increasingly train models to self-regulate under constitutional-s… ▽ More Large language models (LLMs) are increasingly used to make sense of ambiguous, open-textured, value-laden terms. Platforms routinely rely on LLMs for content moderation, asking them to label text based on disputed concepts like "hate speech" or "incitement"; hiring managers may use LLMs to rank who counts as "qualified"; and AI labs increasingly train models to self-regulate under constitutional-style ambiguous principles such as "biased" or "legitimate". This paper introduces ambiguity collapse: a phenomenon that occurs when an LLM encounters a term that genuinely admits multiple legitimate interpretations, yet produces a singular resolution, in ways that bypass the human practices through which meaning is ordinarily negotiated, contested, and justified. Drawing on interdisciplinary accounts of ambiguity as a productive epistemic resource, we develop a taxonomy of the epistemic risks posed by ambiguity collapse at three levels: process (foreclosing opportunities to deliberate, develop cognitive skills, and shape contested terms), output (distorting the concepts and reasons agents act upon), and ecosystem (reshaping shared vocabularies, interpretive norms, and how concepts evolve over time). We illustrate these risks through three case studies, and conclude by sketching multi-layer mitigation principles spanning training, institutional deployment design, interface affordances, and the management of underspecified prompts, with the goal of designing systems that surface, preserve, and responsibly govern ambiguity. △ Less

Submitted 5 March, 2026; originally announced March 2026.

The Value of Disagreement in AI Design, Evaluation, and Alignment

Authors: Sina Fazelpour, Will Fleisher

Abstract: Disagreements are widespread across the design, evaluation, and alignment pipelines of artificial intelligence (AI) systems. Yet, standard practices in AI development often obscure or eliminate disagreement, resulting in an engineered homogenization that can be epistemically and ethically harmful, particularly for marginalized groups. In this paper, we characterize this risk, and develop a normati… ▽ More Disagreements are widespread across the design, evaluation, and alignment pipelines of artificial intelligence (AI) systems. Yet, standard practices in AI development often obscure or eliminate disagreement, resulting in an engineered homogenization that can be epistemically and ethically harmful, particularly for marginalized groups. In this paper, we characterize this risk, and develop a normative framework to guide practical reasoning about disagreement in the AI lifecycle. Our contributions are two-fold. First, we introduce the notion of perspectival homogenization, characterizing it as a coupled ethical-epistemic risk that arises when an aspect of an AI system's development unjustifiably suppresses disagreement and diversity of perspectives. We argue that perspectival homogenization is best understood as a procedural risk, which calls for targeted interventions throughout the AI development pipeline. Second, we propose a normative framework to guide such interventions, grounded in lines of research that explain why disagreement can be epistemically beneficial, and how its benefits can be realized in practice. We apply this framework to key design questions across three stages of AI development tasks: when disagreement is epistemically valuable; whose perspectives should be included and preserved; how to structure tasks and navigate trade-offs; and how disagreement should be documented and communicated. In doing so, we challenge common assumptions in AI practice, offer a principled foundation for emerging participatory and pluralistic approaches, and identify actionable pathways for future work in AI design and governance. △ Less

Submitted 12 May, 2025; originally announced May 2025.

Comments: Accepted to ACM Conference on Fairness, Accountability, and Transparency (FAccT) 2025

Aspirational Affordances of AI

Authors: Sina Fazelpour, Meica Magnani

Abstract: As artificial intelligence (AI) systems increasingly permeate processes of cultural and epistemic production, there are growing concerns about how their outputs may confine individuals and groups to restricted narratives about who or what they could be. In this paper, we advance the discourse surrounding these concerns by making three contributions. First, we introduce the concept of aspirational… ▽ More As artificial intelligence (AI) systems increasingly permeate processes of cultural and epistemic production, there are growing concerns about how their outputs may confine individuals and groups to restricted narratives about who or what they could be. In this paper, we advance the discourse surrounding these concerns by making three contributions. First, we introduce the concept of aspirational affordance to describe how culturally shared interpretive resources, such as concepts, images, and narratives, can shape individual cognition, and in particular exercises of imagination. We show the usefulness of this concept for grounding the evaluation of psychological risks posed by AI. Second, we provide three reasons for scrutinizing AI's influence on aspirational affordances: AI's influence is potentially more potent, but less public, than that of traditional sources; the influence is not simply incremental, but ecological, transforming the entire landscape of practices that shape aspirational affordances; and it is highly concentrated, with a few corporate-controlled systems mediating a growing portion of production. Our third contribution is to advance such a scrutiny of AI's influence by introducing the concept of aspirational harm. In the context of AI systems, such harms arise when AI-enabled aspirational affordances distort or diminish available interpretive resources in ways that undermine individuals' ability to imagine relevant practical possibilities. Through three case studies, we illustrate how aspirational harms extend the existing discourse on AI-inflicted harms beyond representational and allocative harms, warranting separate attention. Overall, this paper aims to advance our understanding of the psychological and societal stakes of AI in shaping individual and collective aspirations. △ Less

Submitted 19 August, 2026; v1 submitted 21 April, 2025; originally announced April 2025.

Should you use LLMs to simulate opinions? Quality checks for early-stage deliberation

Authors: Terrence Neumann, Maria De-Arteaga, Sina Fazelpour

Abstract: The emergent capabilities of large language models (LLMs) have prompted interest in using them as surrogates for human subjects in opinion surveys. However, prior evaluations of LLM-based opinion simulation have relied heavily on costly, domain-specific survey data, and mixed empirical results leave their reliability in question. To enable cost-effective, early-stage evaluation, we introduce a qua… ▽ More The emergent capabilities of large language models (LLMs) have prompted interest in using them as surrogates for human subjects in opinion surveys. However, prior evaluations of LLM-based opinion simulation have relied heavily on costly, domain-specific survey data, and mixed empirical results leave their reliability in question. To enable cost-effective, early-stage evaluation, we introduce a quality control assessment designed to test the viability of LLM-simulated opinions on Likert-scale tasks without requiring large-scale human data for validation. This assessment comprises two key tests: \emph{logical consistency} and \emph{alignment with stakeholder expectations}, offering a low-cost, domain-adaptable validation tool. We apply our quality control assessment to an opinion simulation task relevant to AI-assisted content moderation and fact-checking workflows -- a socially impactful use case -- and evaluate seven LLMs using a baseline prompt engineering method (backstory prompting), as well as fine-tuning and in-context learning variants. None of the models or methods pass the full assessment, revealing several failure modes. We conclude with a discussion of the risk management implications and release \texttt{TopicMisinfo}, a benchmark dataset with paired human and LLM annotations simulated by various models and approaches, to support future research. △ Less

Submitted 13 November, 2025; v1 submitted 11 April, 2025; originally announced April 2025.

Comments: Accepted to AAAI AI for Social Impact (AISI), 2026. This version includes Appendices

Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machina

Authors: Yuan Gao, Dokyun Lee, Gordon Burtch, Sina Fazelpour

Abstract: Recent studies suggest large language models (LLMs) can exhibit human-like reasoning, aligning with human behavior in economic experiments, surveys, and political discourse. This has led many to propose that LLMs can be used as surrogates or simulations for humans in social science research. However, LLMs differ fundamentally from humans, relying on probabilistic patterns, absent the embodied expe… ▽ More Recent studies suggest large language models (LLMs) can exhibit human-like reasoning, aligning with human behavior in economic experiments, surveys, and political discourse. This has led many to propose that LLMs can be used as surrogates or simulations for humans in social science research. However, LLMs differ fundamentally from humans, relying on probabilistic patterns, absent the embodied experiences or survival objectives that shape human cognition. We assess the reasoning depth of LLMs using the 11-20 money request game. Nearly all advanced approaches fail to replicate human behavior distributions across many models. Causes of failure are diverse and unpredictable, relating to input language, roles, and safeguarding. These results advise caution when using LLMs to study human behavior or as surrogates or simulations. △ Less

Submitted 23 January, 2025; v1 submitted 25 October, 2024; originally announced October 2024.

Authenticity and exclusion: social media algorithms and the dynamics of belonging in epistemic communities

Authors: Nil-Jana Akpinar, Sina Fazelpour

Abstract: Recent philosophical work has explored how the social identity of knowers influences how their contributions are received, assessed, and credited. However, a critical gap remains regarding the role of technology in mediating and enabling communication within today's epistemic communities. This paper addresses this gap by examining how social media platforms and their recommendation algorithms shap… ▽ More Recent philosophical work has explored how the social identity of knowers influences how their contributions are received, assessed, and credited. However, a critical gap remains regarding the role of technology in mediating and enabling communication within today's epistemic communities. This paper addresses this gap by examining how social media platforms and their recommendation algorithms shape the professional visibility and opportunities of researchers from minority groups. Using agent-based simulations, we investigate this question with respect to components of a widely used recommendation algorithm, and uncover three key patterns: First, these algorithms disproportionately harm the professional visibility of researchers from minority groups, creating systemic patterns of exclusion. Second, within these minority groups, the algorithms result in greater visibility for users who more closely resemble the majority group, incentivizing assimilation at the cost of professional invisibility. Third, even for topics that strongly align with minority identities, content created by minority researchers is less visible to the majority than similar content produced by majority users. Importantly, these patterns emerge, even though individual engagement with professional content is independent of group identity. These findings have significant implications for philosophical discussions on epistemic injustice and exclusion, and for policy proposals aimed at addressing these harms. More broadly, they call for a closer examination of the pervasive, but often neglected role of AI and data-driven technologies in shaping today's epistemic communities. △ Less

Submitted 21 October, 2024; v1 submitted 11 July, 2024; originally announced July 2024.

Disciplining Deliberation: A Sociotechnical Perspective on Machine Learning Trade-offs

Authors: Sina Fazelpour

Abstract: This paper examines two prominent formal trade-offs in artificial intelligence (AI) -- between predictive accuracy and fairness, and between predictive accuracy and interpretability. These trade-offs have become a central focus in normative and regulatory discussions as policymakers seek to understand the value tensions that can arise in the social adoption of AI tools. The prevailing interpretati… ▽ More This paper examines two prominent formal trade-offs in artificial intelligence (AI) -- between predictive accuracy and fairness, and between predictive accuracy and interpretability. These trade-offs have become a central focus in normative and regulatory discussions as policymakers seek to understand the value tensions that can arise in the social adoption of AI tools. The prevailing interpretation views these formal trade-offs as directly corresponding to tensions between underlying social values, implying unavoidable conflicts between those social objectives. In this paper, I challenge that prevalent interpretation by introducing a sociotechnical approach to examining the value implications of trade-offs. Specifically, I identify three key considerations -- validity and instrumental relevance, compositionality, and dynamics -- for contextualizing and characterizing these implications. These considerations reveal that the relationship between model trade-offs and corresponding values depends on critical choices and assumptions. Crucially, judicious sacrifices in one model property for another can, in fact, promote both sets of corresponding values. The proposed sociotechnical perspective thus shows that we can and should aspire to higher epistemic and ethical possibilities than the prevalent interpretation suggests, while offering practical guidance for achieving those outcomes. Finally, I draw out the broader implications of this perspective for AI design and governance, highlighting the need to broaden normative engagement across the AI lifecycle, develop legal and auditing tools sensitive to sociotechnical considerations, and rethink the vital role and appropriate structure of interdisciplinary collaboration in fostering a responsible AI workforce. △ Less

Submitted 20 December, 2024; v1 submitted 7 March, 2024; originally announced March 2024.

Comments: Accepted for publication in the British Journal for the Philosophy of Science

Diverse, but Divisive: LLMs Can Exaggerate Gender Differences in Opinion Related to Harms of Misinformation

Authors: Terrence Neumann, Sooyong Lee, Maria De-Arteaga, Sina Fazelpour, Matthew Lease

Abstract: The pervasive spread of misinformation and disinformation poses a significant threat to society. Professional fact-checkers play a key role in addressing this threat, but the vast scale of the problem forces them to prioritize their limited resources. This prioritization may consider a range of factors, such as varying risks of harm posed to specific groups of people. In this work, we investigate… ▽ More The pervasive spread of misinformation and disinformation poses a significant threat to society. Professional fact-checkers play a key role in addressing this threat, but the vast scale of the problem forces them to prioritize their limited resources. This prioritization may consider a range of factors, such as varying risks of harm posed to specific groups of people. In this work, we investigate potential implications of using a large language model (LLM) to facilitate such prioritization. Because fact-checking impacts a wide range of diverse segments of society, it is important that diverse views are represented in the claim prioritization process. This paper examines whether a LLM can reflect the views of various groups when assessing the harms of misinformation, focusing on gender as a primary variable. We pose two central questions: (1) To what extent do prompts with explicit gender references reflect gender differences in opinion in the United States on topics of social relevance? and (2) To what extent do gender-neutral prompts align with gendered viewpoints on those topics? To analyze these questions, we present the TopicMisinfo dataset, containing 160 fact-checked claims from diverse topics, supplemented by nearly 1600 human annotations with subjective perceptions and annotator demographics. Analyzing responses to gender-specific and neutral prompts, we find that GPT 3.5-Turbo reflects empirically observed gender differences in opinion but amplifies the extent of these differences. These findings illuminate AI's complex role in moderating online communication, with implications for fact-checkers, algorithm designers, and the use of crowd-workers as annotators. We also release the TopicMisinfo dataset to support continuing research in the community. △ Less

Submitted 29 January, 2024; originally announced January 2024.

Comments: Under Review

Homophily and Incentive Effects in Use of Algorithms

Authors: Riccardo Fogliato, Sina Fazelpour, Shantanu Gupta, Zachary Lipton, David Danks

Abstract: As algorithmic tools increasingly aid experts in making consequential decisions, the need to understand the precise factors that mediate their influence has grown commensurately. In this paper, we present a crowdsourcing vignette study designed to assess the impacts of two plausible factors on AI-informed decision-making. First, we examine homophily -- do people defer more to models that tend to a… ▽ More As algorithmic tools increasingly aid experts in making consequential decisions, the need to understand the precise factors that mediate their influence has grown commensurately. In this paper, we present a crowdsourcing vignette study designed to assess the impacts of two plausible factors on AI-informed decision-making. First, we examine homophily -- do people defer more to models that tend to agree with them? -- by manipulating the agreement during training between participants and the algorithmic tool. Second, we considered incentives -- how do people incorporate a (known) cost structure in the hybrid decision-making setting? -- by varying rewards associated with true positives vs. true negatives. Surprisingly, we found limited influence of either homophily and no evidence of incentive effects, despite participants performing similarly to previous studies. Higher levels of agreement between the participant and the AI tool yielded more confident predictions, but only when outcome feedback was absent. These results highlight the complexity of characterizing human-algorithm interactions, and suggest that findings from social psychology may require re-examination when humans interact with algorithms. △ Less

Submitted 19 May, 2022; originally announced May 2022.

Comments: Accepted at CogSci, 2022

Justice in Misinformation Detection Systems: An Analysis of Algorithms, Stakeholders, and Potential Harms

Authors: Terrence Neumann, Maria De-Arteaga, Sina Fazelpour

Abstract: Faced with the scale and surge of misinformation on social media, many platforms and fact-checking organizations have turned to algorithms for automating key parts of misinformation detection pipelines. While offering a promising solution to the challenge of scale, the ethical and societal risks associated with algorithmic misinformation detection are not well-understood. In this paper, we employ… ▽ More Faced with the scale and surge of misinformation on social media, many platforms and fact-checking organizations have turned to algorithms for automating key parts of misinformation detection pipelines. While offering a promising solution to the challenge of scale, the ethical and societal risks associated with algorithmic misinformation detection are not well-understood. In this paper, we employ and extend upon the notion of informational justice to develop a framework for explicating issues of justice relating to representation, participation, distribution of benefits and burdens, and credibility in the misinformation detection pipeline. Drawing on the framework: (1) we show how injustices materialize for stakeholders across three algorithmic stages in the pipeline; (2) we suggest empirical measures for assessing these injustices; and (3) we identify potential sources of these harms. This framework should help researchers, policymakers, and practitioners reason about potential harms or risks associated with these algorithms and provide conceptual guidance for the design of algorithmic fairness audits in this domain. △ Less

Submitted 29 April, 2022; v1 submitted 28 April, 2022; originally announced April 2022.

Comments: Accepted at ACM Conference on Fairness, Accountability, and Transparenct (FAccT), 2022

Diversity in Sociotechnical Machine Learning Systems

Authors: Sina Fazelpour, Maria De-Arteaga

Abstract: There has been a surge of recent interest in sociocultural diversity in machine learning (ML) research, with researchers (i) examining the benefits of diversity as an organizational solution for alleviating problems with algorithmic bias, and (ii) proposing measures and methods for implementing diversity as a design desideratum in the construction of predictive algorithms. Currently, however, ther… ▽ More There has been a surge of recent interest in sociocultural diversity in machine learning (ML) research, with researchers (i) examining the benefits of diversity as an organizational solution for alleviating problems with algorithmic bias, and (ii) proposing measures and methods for implementing diversity as a design desideratum in the construction of predictive algorithms. Currently, however, there is a gap between discussions of measures and benefits of diversity in ML, on the one hand, and the broader research on the underlying concepts of diversity and the precise mechanisms of its functional benefits, on the other. This gap is problematic because diversity is not a monolithic concept. Rather, different concepts of diversity are based on distinct rationales that should inform how we measure diversity in a given context. Similarly, the lack of specificity about the precise mechanisms underpinning diversity's potential benefits can result in uninformative generalities, invalid experimental designs, and illicit interpretations of findings. In this work, we draw on research in philosophy, psychology, and social and organizational sciences to make three contributions: First, we introduce a taxonomy of different diversity concepts from philosophy of science, and explicate the distinct epistemic and political rationales underlying these concepts. Second, we provide an overview of mechanisms by which diversity can benefit group performance. Third, we situate these taxonomies--of concepts and mechanisms--in the lifecycle of sociotechnical ML systems and make a case for their usefulness in fair and accountable ML. We do so by illustrating how they clarify the discourse around diversity in the context of ML systems, promote the formulation of more precise research questions about diversity's impact, and provide conceptual tools to further advance research and practice. △ Less

Submitted 19 July, 2021; originally announced July 2021.

Fair Machine Learning Under Partial Compliance

Authors: Jessica Dai, Sina Fazelpour, Zachary C. Lipton

Abstract: Typically, fair machine learning research focuses on a single decisionmaker and assumes that the underlying population is stationary. However, many of the critical domains motivating this work are characterized by competitive marketplaces with many decisionmakers. Realistically, we might expect only a subset of them to adopt any non-compulsory fairness-conscious policy, a situation that political… ▽ More Typically, fair machine learning research focuses on a single decisionmaker and assumes that the underlying population is stationary. However, many of the critical domains motivating this work are characterized by competitive marketplaces with many decisionmakers. Realistically, we might expect only a subset of them to adopt any non-compulsory fairness-conscious policy, a situation that political philosophers call partial compliance. This possibility raises important questions: how does the strategic behavior of decision subjects in partial compliance settings affect the allocation outcomes? If k% of employers were to voluntarily adopt a fairness-promoting intervention, should we expect k% progress (in aggregate) towards the benefits of universal adoption, or will the dynamics of partial compliance wash out the hoped-for benefits? How might adopting a global (versus local) perspective impact the conclusions of an auditor? In this paper, we propose a simple model of an employment market, leveraging simulation as a tool to explore the impact of both interaction effects and incentive effects on outcomes and auditing metrics. Our key findings are that at equilibrium: (1) partial compliance (k% of employers) can result in far less than proportional (k%) progress towards the full compliance outcomes; (2) the gap is more severe when fair employers match global (vs local) statistics; (3) choices of local vs global statistics can paint dramatically different pictures of the performance vis-a-vis fairness desiderata of compliant versus non-compliant employers; and (4) partial compliance to local parity measures can induce extreme segregation. △ Less

Submitted 26 September, 2022; v1 submitted 6 November, 2020; originally announced November 2020.

Comments: Presented at AIES 2021; previously at the NeurIPS 2020 Workshop on Consequential Decision Making in Dynamic Environments and the NeurIPS 2020 Workshop on ML for Economic Policy. Minor correction uploaded Sept. 2022

Algorithmic Fairness from a Non-ideal Perspective

Authors: Sina Fazelpour, Zachary C. Lipton

Abstract: Inspired by recent breakthroughs in predictive modeling, practitioners in both industry and government have turned to machine learning with hopes of operationalizing predictions to drive automated decisions. Unfortunately, many social desiderata concerning consequential decisions, such as justice or fairness, have no natural formulation within a purely predictive framework. In efforts to mitigate… ▽ More Inspired by recent breakthroughs in predictive modeling, practitioners in both industry and government have turned to machine learning with hopes of operationalizing predictions to drive automated decisions. Unfortunately, many social desiderata concerning consequential decisions, such as justice or fairness, have no natural formulation within a purely predictive framework. In efforts to mitigate these problems, researchers have proposed a variety of metrics for quantifying deviations from various statistical parities that we might expect to observe in a fair world and offered a variety of algorithms in attempts to satisfy subsets of these parities or to trade off the degree to which they are satisfied against utility. In this paper, we connect this approach to \emph{fair machine learning} to the literature on ideal and non-ideal methodological approaches in political philosophy. The ideal approach requires positing the principles according to which a just world would operate. In the most straightforward application of ideal theory, one supports a proposed policy by arguing that it closes a discrepancy between the real and the perfectly just world. However, by failing to account for the mechanisms by which our non-ideal world arose, the responsibilities of various decision-makers, and the impacts of proposed policies, naive applications of ideal thinking can lead to misguided interventions. In this paper, we demonstrate a connection between the fair machine learning literature and the ideal approach in political philosophy, and argue that the increasingly apparent shortcomings of proposed fair machine learning algorithms reflect broader troubles faced by the ideal approach. We conclude with a critical discussion of the harms of misguided solutions, a reinterpretation of impossibility results, and directions for future research. △ Less

Submitted 8 January, 2020; originally announced January 2020.

Comments: Accepted for publication at the AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society (AIES) 2020