CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from Multiple Perspectives

Summary

This paper talks about CLASH, a way to test how well language models can handle tough ethical questions by looking at problems from different points of view.

What's the problem?

The problem is that while AI can do well with simple moral questions where the right answer is obvious, it often struggles when the situation is more complicated, involves emotional discomfort, or when people's values might change depending on the situation. This makes it hard to trust AI with decisions that really matter, especially when the stakes are high and there’s no easy answer.

What's the solution?

The researchers used CLASH to evaluate language models by giving them high-stakes dilemmas that are tricky and can be seen from multiple perspectives. They found that these models are less accurate and flexible when the questions are ambiguous or emotionally challenging, and that their answers can change depending on which perspective they are asked to take.

Why it matters?

This matters because it shows that AI still has a long way to go before it can be trusted to help with real-life ethical decisions, especially in situations where there isn’t one clear right answer. Understanding these weaknesses helps researchers know what to work on so AI can become more responsible and trustworthy in the future.