#!/usr/bin/env python
# coding: utf-8
# # A Review of Incident Reports Related to Women
# by LB, KL, BP Group Red
#
# ## Data Overview, Telling Our Story
# The goal of this research was to develop a better understanding of the role that female prisoners played within the camp from the perspective of the incident reports, which are a small sliver of the daily life that prisoners endured during their time at the prison camps. We gained a better sense of women’s riot activities at Tule Lake -- whether participants or informants. As another way of lending context to the data we had refined, we also compared the trends we found within the camp to national crime rates at the time. Note we are making a good faith effort to utilize the terminology outlined by the Denshō project: “prisoner”, “concentration camp”, “prison camp”, and more (Denshō, 2019). Some such terms are at odds with those often used to downplay these events within American history textbooks. (KL)
#
# Throughout the process, our group needed to remember and acknowledge that those data weren’t numbers, but people and lives. Moore states that researchers have a “responsibility to think carefully about how they reconstruct lives in the present and for the future” (2010, p. 266). Further examination of the incident cards revealed individual stories and struggles. There were stories of death and loss. Moore goes on to say “there has been little sustained reflection about the dilemmas researchers face when research subjects are dead and when the research is of a sensitive nature” (2010, p. 266). Sensitive issues were discussed throughout the analysis, and decisions were made on how best to properly include information.
# ## Dataset Exploration related to Scale and Levels within Collection
# Determining the computational story to tell began with establishing familiarity with the dataset. The project began with individual exploration of the data and examination of both the cards and Excel file. That period of individual reflection was to ensure each group member had an understanding of the structure of the data and the information contained within it. From those sessions, fairly early on our group expressed interest in understanding the treatment of the female prisoners within the Tule Lake prison camp. Our research and manipulation concerned the incident reports related specifically to women (KL & LB). Data were manipulated further to see what types of criminal incidents were reported against female prisoners.
#
# ## Data Cleaning and Preparation
# Starting with an Excel file of all index card data, as well as a separate Excel file of prisoner demographic data, we were able to manipulate the incident reports in order to identify those that involved female prisoners. We then worked to categorize each incident based on whether the woman involved was an offender, victim, witness, non-crime, or other. Based on that categorized and refined data, we were able to start noticing unique finds among the data, such as particularly prolific criminals, or an alarmingly high rate of dog bites.
# Given the sheer number of individual incident cards in the dataset, BP contacted Geoff Froh, Deputy Director at Denshō, to see if we could acquire a list of individuals with their (assigned) genders. Director Froh directed BP to the Form WRA-26 masterlist file that Denshō has on their website: https://ddr.densho.org/names/. It contains the intake information for individuals who were placed in the concentration camp system, transcribed into an Excel format. BP downloaded the WRA-26 file, and used OpenRefine software to filter the list down to individuals at Tule Lake, and then filtered further to display only (assigned gender) females. BP sent the resulting spreadsheet to LB and KL. (BP)
#
# Using the vlookup feature in Excel, KL combined the WRA-26 masterlist and the original Excel file of the incident reports, using the prisoner name field as the common field between both sets of data. We recognized that this approach would certainly have some degree of inconsistency and errors, due to the non-standard formatting of names within the incident card Excel file, but considered it a worthwhile first step in having a computer scan and combine the data, versus having to manually compare every single record. The vlookup resulted in a list of only forty-odd female individuals with incident reports, out of approximately 15,000 total incidents. Although we understood that the occurrence rates of female-related incident reports would likely be below 50%, this was an unexpectedly low number. (KL)
#
# In an effort to double check the results in case of an error, BP went back through the main Excel spreadsheet of incidents from Tule Lake, and quickly noticed there were female names that Larenitis’s search had not flagged. BP went back to the original, unfiltered Form WRA-26 masterlist file, and discovered that some of the names in the incident card Excel spreadsheet belonged to individuals on the Form WRA-26 masterlist not listed as being at Tule Lake. It is unclear whether the concentration camp listed for each individual is a reflection of the first camp they were processed at, or the last one they were at before release. However, with that in mind, BP refiltered the Form WRA-26 masterlist file using Open Refine to display all the recorded females, not just those at Tule Lake. BP then used the “text facet” feature on the first names of the individuals, in order to get a list of first names by frequency of occurrence.
#
# BP checked this list of first names against the original Excel incident spreadsheet by a simple process of searching each individual first name. The incidents found were copied into a larger shared Google Docs file so that the entire group could see. While this second search yielded hundreds of previously unflagged examples of incident reports involving females, in hindsight it was an incredibly tedious and time-consuming process that would likely have been sped up if the data had been handed off to KL to re-run her search algorithm. Additionally, BP chose to include incidents where the main named offender was a male, if a female was a victim or had in some way been central to the male’s conviction. (BP)
#
# Once female cards were isolated, they were analyzed to determine the nature of the incident. Cards were assigned to one of five broad categories: Non-crime, Offender, Other, Victim, and Witness/Informant.
#
# Cards categorized as Offender were then moved to a new spreadsheet. Those incident cards were examined further and assigned into one of ten subcategories based on the nature of the incident. Subcategories were: Assault, Disturbing the Peace, Contraband, Infraction of Project Regulations, Riot, Political/Social Activism, Speeding, Theft, Threat to Kill, and Other. Using Google Sheets, charts were created to display the results. (LB)
#
# ## Modeling: Computation and Transformation
# With the gender data on one sheet and the incident reports on another, we switched the data to Excel and utilizing its powerful vlookup feature (KL wrote search algorithms), allowed the two sheets to merge into one functional document. Incident reports were now associated with a gender. Once this merged sheet was created, the data was then uploaded and converted into Google Sheet format. Google Sheets allowed our remote team to have consistent, real time access to the same data. This software choice was important as further filters were applied and the data was more finely refined. Google Sheets also allowed our team to share screens and work synchronously on manipulations, finding the computational story we were looking for. (KL)
#
# ## Modeling: Visualization
# The final categorized incident cards were reviewed to examine two different trends. First, by looking more closely at the ‘riot’ category, we were able to glimpse the role of females in the riot at Tule Lake. Secondly, cards in the offender category were compared to crime statistics at a national level. (LB)
#
# The largest single type of incident in our dataset was Riot, as shown in **Figure 1**. A further breakdown of those incident cards, shown in **Figure 2**, reveals the most prevalent category was that of “Offender,” in which the Japanese female prisoners were charged with participation in the November 4th, 1943 riot at Tule Lake. It is unclear whether or not formal charges and punishments were brought against these prisoners, or if a note was simply made in their files. The second largest dataset was somewhat surprisingly “Witness/Informant.” Some of the females involved in this category were of Japanese descent, but based on the names and listed positions of the majority of the women, we can assume that most of the informants for the incidents were non-Japanese females who helped run the camp. In particular the head nurse AKF, who is described in the records as the “Chief Caucasian Nurse,” was responsible for 12% of the “Witness/Informant” charges under Riot. Two cases were categorized as “Victim,” with the incident charge being for Riot. For that charge, they are included in this dataset, but this may be up for debate. In some cases, we were unable to reach a firm agreement on how to label certain incidents under Riot, and those make up the 5.2% of incidents categorized as “Other.” (BP)
#
# In the second portion of the analysis the riot category was removed. Our goal was to see how female crimes reported in the concentration camp matched crimes committed by females at the national level. Data for comparison was provided by 1943 historical statistics compiled by the US Census Bureau. The data are consolidated in **Figure 3**. The Census data were recategorized to match the categories in our computational model. For example ‘possession of drugs’ and ‘violation of liquor laws’ were reclassified as Contraband. Incidents of infraction of project regulations (being caught out of bounds or visiting someone in the stockades) were specific to incidents that occurred at the camps; the category did not exist at a national level. There were also no arrests for women at a national level that fit the category of political/social activism. Men were arrested for violating National Defense Laws, or draft dodging, but such did not apply to women. Now it can be seen that in both cases Other is the majority of incidents, Theft is second, and Disturbing the peace is the third most common crime committed by females. (LB)
#
# 
#
# 
#
# 
# ## Ethics and Values Considerations
# We are entirely aware and respectful of the fact that in reality, the gender data we have may not accurately reflect the true gender preferences of the individual people that these records represent. Our research and manipulations reflect only the assigned gender of the unique individuals that comprised our data set. Gender concepts have evolved quite a bit from the 1940s, and while we recognize today that individuals who were identified as female may not have identified that way themselves, no such consideration is present in the Tule Lake incident report cards. In addition, some individuals’ (assigned) gender could not be determined from their names, as the names given were unisex. It is entirely possible that they have been missed out from our dataset.
#
# The lack of accurate camp listings on the Form WRA-26 masterlist, as well as the fact that some individuals in the Tule Lake incident cards do not appear on the Form WRA-26 masterlist at all, contributes to our belief that we have likely missed some individuals in our analysis. That realization was a good lesson for us in not necessarily assuming that files, such as the Form WRA-26 masterlist, are accurate and comprehensive. Finally some of the incident cards provided too little information that to determine which category an incident might belong to. Regarding the roles of non-Japanese staff at Tule Lake, we had to assume that they were likely functioning as either Victims or Witness/informants, though we cannot be completely sure.
#
# ## Summary and Suggestions
# During and upon conclusion of our research, we saw several further avenues for exploration. The woman nicknamed “Japanese Mata Hari” within the otherwise formal incident report cards, warrants additional exploration across camp records, to see if a more complete picture can be painted of who she was, and where she went after being released from the camps (BP’s web search did not turn up much additional information. Her nicknamesake was an exotic dancer convicted as a German spy during WWI). We also found some female but no male members of the Aredas family, listed as Missing People. More generally many of the incident reports were for seemingly innocuous “crimes”, such as playing the trumpet. Such observations lead us to ask do these infractions match up with the posted rules and regulations of the camps? Were prisoners even aware that they weren’t allowed to play the trumpet that they were allowed to bring into the camp with them? Additional research on these “harmless” crimes may result in noteworthy findings pertaining to the fairness of the incident report cards, and the transparency of the rules that the prisoners had to follow. The thousands of cards that the authors filtered through represent thousands of unique individuals who all have their own stories to tell.
#
# ## References
# * Denshō. (2019, April 17). Terminology - Densho: Japanese American Incarceration and Japanese Internment. https://densho.org/terminology/.
# * Moore, F.P.L. (2010). Tales from the archive: Methodological and ethical issues in historical geography research. Area, 42(3), 262-270. http://doi.org/10.1111/j.1475-4762.2009.00923.x
# * U.S. Census Bureau. 1944. Crime and criminals. Statistical Abstract of the United States: 1943. (Vol 65). https://www2.census.gov/library/publications/1944/compendia/1943statab/1943-03.pdf
#