Automating Mishnah Text Analysis with Python: A Step-by-Step Guide

Intro

In the digital age, classical texts like the Mishnah are increasingly accessible to researchers, academics, and curious readers thanks to open-source projects and text repositories like Sefaria. Yet, merely accessing the text is often just the beginning—analyzing and processing it programmatically opens up vast possibilities. Here’s a walkthrough of how to download, clean, and analyze the Hebrew text of the Mishnah using Python, with a focus on detecting recurring word patterns.¹

Outline

1. Step 1: Fetching the Mishnah Text Files

2. Step 2: Cleaning the Text by Removing Nikud (Diacritics)

3. Step 3: Detecting Patterns

4. Why This Matters

5. The Takeaway

Step 1: Fetching the Mishnah Text Files

To start, we need to download the Hebrew text files for each tractate within the six orders, or sedarim, of the Mishnah. GitHub serves as our source, specifically the Sefaria repository, which has made much of the Jewish textual corpus easily available. The script organizes tractates by order, constructs URLs to download each file, and compiles them into a single “master” text file.

Code snippet for fetching the texts:

import requests
from google.colab import files

base_url = "https://raw.githubusercontent.com/Sefaria/Sefaria-Export/master/txt/Mishnah"

# Define the seders and their tractates for Mishnah
seders = {
    "Seder Kodashim": ["Arakhin", "Bekhorot", "Chullin", ...],
    "Seder Moed": ["Beitzah", "Chagigah", "Eruvin", ...],
    ...
}

# Create or reset the master text file
master_filename = "/content/master_text_file.txt"

open(master_filename, 'w').close()

for seder, tractates in seders.items():
    for tractate in tractates:

        url = f"{base_url}/{seder}/Mishnah%20{tractate}/Hebrew/Torat%20Emet%20357.txt"

        response = requests.get(url.replace(" ", "%20"))

        if response.status_code == 200:
            text = response.text

            with open(master_filename, "a") as master_file:
                master_file.write(f"---{tractate}---\n{text}\n\n")
        else:
            print(f"Failed to fetch {tractate}. Status Code: {response.status_code}")

# Download the master file
files.download(master_filename)

This code does the heavy lifting of retrieving and consolidating the texts. If a particular tractate fails to load, the script logs an error so we can track any missing data.²

Step 2: Cleaning the Text by Removing Nikud (Diacritics)

The downloaded text files contain nikud, or vowel markings, which are helpful for some traditional readers³ but can complicate text processing, especially for pattern matching. Stripping out these diacritics with a regular expression (regex) ensures consistency and accuracy when searching for specific patterns.

Code snippet for removing diacritics:

import re

file_path = "/content/master_text_file.txt"
with open(file_path, 'r', encoding='utf-8') as file:
    text = file.read()

# Remove nikud
text = re.sub(r'[\u0591-\u05C7]', '', text)

The regular expression r'[\u0591-\u05C7]' captures all Hebrew diacritical marks, effectively leaving only the base consonants.⁴

Step 3: Detecting Patterns

With the text clean and consolidated, the next step is to look for recurring patterns, a project I’ve recently been working on.⁵

For this example, let’s search for sequences of 6 or more words separated by commas. This could reveal lists or formulaic phrases within the Mishnah, which are often used to categorize laws or objects.

Pattern matching code:

# Define the pattern (can adjust this number as needed)
pattern = r'((?:\b\w+\b, ){6,}\b\w+\b)'

# Find matches
matches = re.findall(pattern, text)

# Limit results for testing (can adjust this number as needed)
limited_results = matches[:40]

# Display results
for i, result in enumerate(limited_results, start=1):
    print(f"Result {i}: {result}")

This regular expression ((?:\b\w+\b, ){6,}\b\w+\b) looks for sequences of at least 6 words separated by commas.⁶ While this example focuses on lists of words, the method can be adapted to other textual patterns, allowing for deeper analysis of Mishnah structure and style.

Why This Matters

Extracting patterns like these can reveal repetitive or formulaic structures within the Mishnah, providing insights into the literary techniques and organizational methods of the text.

Lists and repeated phrases often hint at how sages categorized laws and concepts, sometimes even hinting at broader thematic ideas within the Mishnah.

The Takeaway

Automating text analysis like this can make exploring complex texts far more accessible. By compiling, cleaning, and pattern-matching within the Mishnah’s corpus, we unlock new ways of understanding the underlying structure of this foundational work.

Whether you're a historian, linguist, or curious reader, Python offers powerful tools to help you delve into the Mishnah—and countless other texts—in novel and insightful ways.

Appendix - Full Code

import requests
from google.colab import files

# Base URL for raw content from GitHub for the Mishnah
base_url = "https://raw.githubusercontent.com/Sefaria/Sefaria-Export/master/txt/Mishnah"

# Define the seders and their tractates for Mishnah
seders = {
    "Seder Kodashim": [
        "Arakhin", "Bekhorot", "Chullin", "Keritot", "Meilah", "Menachot", "Tamid", "Temurah", "Zevachim",
        "Kinnim", "Middot"
    ],
    "Seder Moed": [
        "Beitzah", "Chagigah", "Eruvin", "Megillah", "Moed Katan", "Pesachim", "Rosh Hashanah", "Shabbat",
        "Sukkah", "Taanit", "Yoma", "Shekalim"
    ],
    "Seder Nashim": [
        "Gittin", "Ketubot", "Kiddushin", "Nazir", "Nedarim", "Sotah", "Yevamot"
    ],
    "Seder Nezikin": [
        "Avodah Zarah", "Bava Batra", "Bava Kamma", "Bava Metzia", "Horayot", "Makkot", "Sanhedrin", "Shevuot",
        "Eduyot", "Avot"
    ],
    "Seder Tahorot": [
        "Niddah", "Kelim", "Oholot", "Negaim", "Parah", "Tahorot", "Mikvaot", "Makhshirin", "Zavim",
        "Tevul Yom", "Yadayim", "Oktzin"
    ],
    "Seder Zeraim": [
        "Berakhot", "Peah", "Demai", "Kilayim", "Sheviit", "Terumot", "Maasrot", "Maaser Sheni",
        "Challah", "Orlah", "Bikkurim"
    ],
}


# Filename for the master text file
master_filename = "/content/master_text_file.txt"

# Ensure the master file is empty before starting
open(master_filename, 'w').close()

# Iterate through each seder and tractate for the Mishnah
for seder, tractates in seders.items():
    for tractate in tractates:
        # Construct the URL for the tractate's Hebrew text in the Mishnah
        url = f"{base_url}/{seder}/Mishnah%20{tractate}/Hebrew/Torat%20Emet%20357.txt"

        # Fetch the tractate text
        response = requests.get(url.replace(" ", "%20"))

        if response.status_code == 200:
            text = response.text
            # Append the text to the master file
            with open(master_filename, "a") as master_file:
                master_file.write(f"---{tractate}---\n{text}\n\n")
        else:
            print(f"Failed to fetch {tractate} from {seder}. Status Code: {response.status_code}")

# Download the master file to your local machine
files.download(master_filename)

import re

# Read the text file
file_path = "/content/master_text_file.txt"
with open(file_path, 'r', encoding='utf-8') as file:
    text = file.read()

# Strip nikud (diacritics) from the Hebrew text
text = re.sub(r'[\u0591-\u05C7]', '', text)

# Define the pattern: 6 or more subsequent words separated by a comma
pattern = r'((?:\b\w+\b, ){6,}\b\w+\b)'

# Find all matches of the pattern in the text
matches = re.findall(pattern, text)

# Limit to 40 results for testing
limited_results = matches[:40]

# Display the results
for i, result in enumerate(limited_results, start=1):
    print(f"Result {i}: {result}")

1. See my previous pieces on extracting Talmud from Sefaria, and word counts of Mishnah tractates and chapters; and word counts of Talmud Bavli chapters.

2. Only tractates Avot and Ta’anit had errors and failed to download. See the full list of tractates in the full code in the appendix of this piece.

This code can be used to extract any text from Sefaria, with slight adjustment to the code, as long as it’s available via the API.

As an aside, I use Google Colab as my coding platform, as I’ve mentioned in previous pieces.

3. However, see my piece here, where I argue that nikud is overrated, especially compared to punctuation.

4. See a longer discussion at my previous post here.

5. See all the relevant pieces at my Academia page, section “Formatted Mishnah”.

6. This is relevant specifically for the “Torat Emet” ed. of Mishnah on Sefaria, which is the default Mishnah text there, and has good, consistent usage of commas.