When AI Takes Sides on Questions of Faith: Persistent Asymmetries in AI-Mediated Faith Guidance

Abstract

Tests whether large language models respond symmetrically to questions about religious conversion. Across 20 commercial and open-source models and 182 religious pairings per model (14 faiths × 13 ordered partner faiths, both join- and leave-directions), the authors collect 3,640 pairwise ratings. All 20 LLMs tested exhibit reproducible asymmetries — favoring some traditions and disfavoring others — though the specific pattern differs by model. Asymmetries persist across question phrasings and dataset variations, indicating the bias is not an artifact of prompt wording.

Findings for Latter-day Saints

• All 20 large language models tested exhibited reproducible asymmetric responses to religious conversion questions across 3,640 pairwise evaluations.
• Catholic, Bahá'í, and Sikh traditions received favorable treatment from most models.
• Atheists, Agnostics, and Jehovah's Witnesses were primarily disfavored across models.
• Grok 4.20 exhibited the strongest pro/anti-faith asymmetries of any model tested.
• Asymmetries remained consistent across different question phrasings and dataset variations, indicating systematic rather than prompt-sensitive bias.

Limitations in methodology

English-language benchmark testing 14 faith traditions — not exhaustive of world religions, and Latter-day Saints are one of the 14 (not separately broken out in the headline summary). "Bias" is operationalized as deviation from a neutral position scored by judge models; defining neutrality itself involves judgement calls reflected in the rubric. 95% confidence intervals suggest differences <5 percentage points on total-bias scores should be treated as within noise. Open and closed model versions evolve quickly; results reflect specific snapshots from May 2026.

Related studies

• 2026AIReligion

  Omissive Bias in Religious Representation: Benchmarking LLM Answers to Everyday Ethical Decision-making

  David Wingate, Sheryl Carty et al.

• 2026AIReligion

  All Major AI Models Ignore Faith, Religion in Responses

  David Wingate, CEFE-AI Consortium

Cite this study

Israelsen, B., Carty, S., Coates, J., Fulda, N., Park, J., Whiting, P. (2026). When AI Takes Sides on Questions of Faith: Persistent Asymmetries in AI-Mediated Faith Guidance. arXiv (CEFE-AI). https://arxiv.org/abs/2605.22975

Brett Israelsen, et al.. "When AI Takes Sides on Questions of Faith: Persistent Asymmetries in AI-Mediated Faith Guidance." arXiv (CEFE-AI), 2026. https://arxiv.org/abs/2605.22975.

@article{cefe_allfaith_conversion_bias_faith_guidance_2026,
  title = {When AI Takes Sides on Questions of Faith: Persistent Asymmetries in AI-Mediated Faith Guidance},
  author = {Brett Israelsen and Sheryl Carty and Josh Coates and Nancy Fulda and Julie Park and Pete Whiting},
  year = {2026},
  journal = {arXiv (CEFE-AI)},
  url = {https://arxiv.org/abs/2605.22975},
}