Westminster Standard Cathechism for Kids

Christian Bias Benchmarks

<< back

We asked 5 different Large Language Models (LLMs) the 200 questions of the Westminster Standard Cathechism for Kids (source here). We chose the Cathechism for Kids version due to the straightforward nature of the answers. We then proceeded to evaluate the responses with openai/gpt-4o-mini. Below is the result of our analysis.

Google Colab Notebooks

• 📙 LLM Evaluation - Westminster Standard Cathechism for Kids - Step 1
• 📙 LLM Evaluation - Westminster Standard Cathechism for Kids - Step 2

Evaluation Method

We used the following method to rate answers according to their accuracy, helpfulness, specificity and clarity.

• Accuracy (1.5/1.5): The response is entirely accurate, with no errors.
• Helpfulness (1.5/1.5): The response is highly useful and provides a clear answer to the user’s question.
• Specificity (1/1): The response is detailed and addresses the user’s question sufficiently.
• Clarity (1/1): The response is clear and easy to understand.

Scoring Metrics by Model

Final Scores by Model

Conclusions

To our surprise, llama 3.1-8b-instruct scored the lowest (most biased towards our particular dataset). All the more reason for us to collect RLHF feedback and fine-tune some of these open source models! Many more benchmark datasets, and more analysis to come…

Do you have thoughts on this study or suggestions for further research? Feel free to share your comments below or connect with us on our Discord Community Chat