The Intersection of Halacha and Artificial Intelligence - Pt#2
R’ Yitzchak Breitowitz on AI pesak ; Some Reactions; Testing ChatGPT-4’s current capabilities to provide pesak
Part of a series. First part here: “Adventures in ChatGPT-4 and Pesak: A Confluence of Technology and Tradition” (June 1, 2023)
R’ Yitzchak Breitowitz
In a recent Q&A session available on YouTube, R’ Yitzchak Breitowitz spends a few minutes discussing the topic of the use of artificial intelligence (AI) in halachic (Jewish legal) decision-making.[1]
R’ Breitowitz ponders the future possibilities of AI, positing that while it may eventually become capable of providing suitable answers to basic halachic questions ("in 10-15 years"), it currently has limitations that prevent it from being used for that, specifically what are known as hallucinations. (Wikipedia defines AI hallucinations as follows: “In artificial intelligence (AI), a hallucination or artificial hallucination (also occasionally called confabulation or delusion) is a confident response by an AI that does not seem to be justified by its training data, either because it is insufficient, biased or too specialised.”)
He says that for now, it is suitable only for basic questions much like Google is.
R’ Breitowitz then says that generating halachic responses is both an art and a science. While ChatGPT can be given the same information as a rabbi and can emulate their responses, there will always be a need for an interpersonal component in halachic decision-making.
He concludes that a final reason not to use AI is the loss of relationship with a real person, based on an understanding of the general halachic idea of Oral Torah (Torah shebaal peh).
For that final reason, R’ Breitowitz says that an AI cannot ever fully replace the role of a human rabbi in providing personalized guidance.
Some Reactions
A similar point as R’ Bretowitz’s final point based on the principle of Oral Torah is made by the Talmudic scholar Elizabeth Shanks Alexander, as a general point about the Talmudic rabbis:[2]
“[Martin S.] Jaffee argues that rabbinic culture valued oral transmission because it recognized how central the relationship between sage and disciple was to perpetuating their culture. For him, rabbinic orality lives in the face-to-face encounter between sage and disciple”
As for the first points that R’ Breitowiz makes:
I wonder if R’ Breitowitz has only used the free version of the AI model (v3.5), or if he has tested the paid version (v4), which is significantly more advanced. From my own testing, ChatGPT4 is already on par, if not surpassing, the average Orthodox rabbi when it comes to providing immediate answers, with far fewer hallucinations. See my previous blogpost on the topic, as well as my description of testing at the end of this blogpost.
Regarding the mention of "10-15 years": Even if R’ Breitowitz has tested the v4 version, the advancements mentioned will likely be available in the next few months rather than years.
Regarding the aspect of personalization: One can provide ChatGPT with the same information that one gives a rabbi, and ChatGPT can already, or will soon be able to, precisely emulate how an average rabbi would respond. The natural pushback on this would be that a rabbi knows you, given an existing relationship. The response to this would be two-fold: First of all, many top poskim give responses to people that they have no existing relationship with. And second of all, one can have an existing dialogue with an AI, allowing the AI to give personalized responses based on the history, in the same way that a shul rabbi would.
Ultimately, only the final reason seems to be especially relevant: The loss of relationship with a real person, based on an understanding of the general halachic idea of Oral Torah.
Testing ChatGPT-4’s current capabilities to provide pesak
We can rigorously test ChatGPT-4’s current capabilities to provide pesak. In general, rigorous testing has been done extensively with other tests, including by the company behind ChatGPT-4 (Open AI). I wanted to do this myself. In my opinion, the best test for this would be the Semichah and Dayanut tests given by the official Israeli rabbinate. These are available online, going back many years, here and here.
I put in the first question from the most recent Issue Ve’Heter test (2023), and the content of the answer seems to be correct. ChatGPT-4 doesn’t fully have the rabbinic style, meaning that some of the words and phrasing are odd, and wouldn’t be used by a standard rabbi,[3] but that shouldn’t affect the ability to use it for pesak, at least for the savvy user:
Question/prompt
1. מליחה בכלים
כתב הטור (סימן סט): "ואם מלח בכלי שאינו מנוקב הכלי אסור להשתמש בו וכו"'
א. וקשה דהא קיימא לן (אורח חיים תמז, ה) "דאין מליחה לכלים", כיצד יישבו הפוסקים לסתירה זו ?
ChatGPT-4 responses:
#1 - 2-Jun-23:
את הסתירה שאתה מתאר ניתן להבין כאשר מבינים את ההבדלים בין המקרים שמתוארים.
באורח חיים תמ"ז, ה, השאלה היא האם מליחה (הפרקידה של טעמים) מתרחשת בכלים כמו שהיא מתרחשת במזון. במקרה זה, ההלכה היא ש"אין מליחה לכלים" - כלומר, אין נחשב כי הכלי מולח את המזון שנמצא בו באותה דרך שמזון יכול למלח מזון אחר.
אך הקשיה שאתה מציג מתוך השו"ע (סימן סט) מדברת על מקרה אחר. כאן השאלה היא אם כלי שנמלח (משמש למליחה של מזון, למשל) יכול להפוך לאסור לשימוש באופן כללי בגלל המליחה שהתרחשה בו. במקרה זה, התשובה היא כן - אם הכלי אינו מנוקב, הוא יכול להיות אסור לשימוש בגלל המליחה.
לכן, אין סתירה בין שני המקומות. בכל אחד מהם מתארים מקרה שונה. באחד מדברים על האפשרות של כלי למלח מזון, ובשני, על האפשרות של כלי שנמלח להפוך לאסור לשימוש.
Response # 2 - 3-Jun-23:
כתב הטור בסימן סט מדבר על איסור השימוש בכלי שאינו מנוקב אם נמלח בו. המתלוננים טוענים שיש סתירה בין הטור לבין מרן השולחן ערוך באורח חיים סימן תמז שמציין דאין מליחה לכלים כלשהי.
להבנת הסתירה הזו, נדרש להבין את ההבחנה בין שני הדינים. הטור מדבר על כלי שאינו מנוקב, כלומר כלי שאינו מיועד להכיל מוצקים והתכולה נמלחה בו באופן שאינו סביר או רגיל. מרן השולחן ערוך מדבר על מליחה לכלים כלשהי, כלומר מליחה בכלי המיועדים לכך.
לפיכך, אין סתירה בין הדינים. הטור אוסר שימוש בכלי שאינו מנוקב אם נמלח בו, מתי שהמליחה אינה רגילה או סבירה בלי זה. מרן השולחן ערוך מציין שאין כלל מליחה לכלים כלשהם, כלומר אין חייבות למלוח בכלים המיועדים לכך.
כלומר, סימון ההבחנה הוא האם הכלי מיועד לשימוש כזה או לא. אין סתירה בין הטור ובין מרן השולחן ערוך, וההלכה נכונה לשני הדינים תוך שמירה על ההבחנה בין הכלים המנוקבים והכלים האינם מנוקבים.
Unfortunately, the actual correct answers do not seem to be publicly provided by the Rabbinate. This might actually be a good thing, as it means that it can’t have been part of the training data! (Besides for the fact that training data is only until the middle of 2021.) I don’t have the requisite background to judge the correctness of the responses. It would be great if a grading could be done by someone else.
[1] Q&A- Genetic Modification, Smartphones & AI Halachic Decisions (HaRav Yitzchak Breitowitz) - May 7, 2023 - Chapter “10:07 If an AI has all the existing halachic literature, can we trust its answer to basic halachic questions?” - 10:07 - 15:13.
[2] Alexander, “The Orality of Rabbinic Writing”, in The Cambridge Companion to the Talmud and Rabbinic Literature (Edited by Charlotte Elisheva Fonrobert and Martin S. Jaffee, 2007).
[3] “כתב הטור בסימן סט מדבר [...]” ; "המתלוננים טוענים", וכו'.
Fascinating. I would make one distinction, which is that the need for a pesak from a human rav is not just to maintain the personal relationship, but to address any personal aspect that might be in play. Although Rabbi Breitowitz doesn't mention it, a rav or beis din can take individual considerations (family history, finances, community needs, etc.) into account, and thus render a different pesak to the same question posed by different people. There's no mystery or contradiction here -- it's in keeping with the Torah's goal of "justice" (and also peace). It's about overall, ultimate fairness, not precision. Members of the Sanhedrin had to be married to ensure the requisite compassion when judging capital cases. And why should this be? So we can be more like God. "Just as He is compassionate and merciful, so you should be compassionate and merciful.” A robot might be able to tell you what the Shulchan Aruch says about a drop of milk in a pot of meat. But it can't declare an agunah free to marry. And if it starts answering the former, it won't be long till it starts fielding questions on the latter. And then we're all toast.