When a text uses natural language, i.e. normal human speech or prose, then data is embedded in the particular syntax and lexicon of that language. Natural Language Processing employs sophisticated models, trained on millions of documents, to parse natural language for meaningful information.
In the case of the Charlotte directory, the text resembles a table with rows and columns, more so than prose. So there are no sentence structures for NLP to use as hints. Instead it reverts to using the English lexicon alone to establish parts of speech and relationships. The result is not much data, in the form of named entity recognition.
In fact, the lines in our text file have non-English rules that we can establish and use to extract data directly. For instance, each line starts with a description of the household "race" category. Then within households there are often members of another race and these are also labeled. After each mention of race there is usually a surname and then a given name, but not always. Our job is to describe and encode these rules in formal patterns.
import re
import os
cwd = os.getcwd()
data_loc = cwd + "/data"
output_loc = cwd + "/output"
allfiles = []
# find all the text files
for root, dirs, files in os.walk(data_loc):
for file in files:
if file.endswith(".txt"):
allfiles.append(os.path.join(root, file))
print('%d files' % len(allfiles))
alltext = []
for file in allfiles:
with open(file, "r") as a_file:
for line in a_file: alltext.append(line)
print('%d lines of text' % len(alltext))
4 files 173 lines of text
def parse(pattern):
data = []
for line in alltext:
match = pattern.search(line)
if match is None:
data.append(None)
print(line)
else:
data.append(match.groupdict())
return data
race_pattern = re.compile("^(Black|White).*$") # Match any line starting with Black or White
data = parse(race_pattern)
print(data[0:3])
[{}, {}, {}]
race_pattern = re.compile("^(?P<race>Black|White)\t(?P<surname>\w*).*$") # Match surname after a tab character
data = parse(race_pattern)
print(data[0:3])
[{'race': 'White', 'surname': 'Adams'}, {'race': 'White', 'surname': 'Adams'}, {'race': 'Black', 'surname': 'Adams'}]
pattern = re.compile("^(?P<race>Black|White)\t(?P<surname>\w*)\t+(?P<title>Miss|Dr|Rev|Mrs).*$") # Match specific titles
data = parse(pattern)
print(data[0:3])
White Adams Henry L Fannie S route agent Southern Railway Co House 327 n Tryon
White Adams James Gertrude manager House 419 Elizabeth av
Black Adams Jane teacher House 1021 s Church
Black Adams John laborer House 1031 s Church
White Adams John J president Adams G & P Co and Char Pepsi-Cola Co House 309 e 6th
White Adams John W Cora conductor S A L Railway House 21st nr Caldwell
Black Adams Joseph Violet cooper House 1011 s Church
Black Adams Kate laundress House Groveton
White Adams Lafayette N clerk Southern Railway House 327 n Tryon
White Adams Lawrence A salesman B S Moore & Co Rooms 405 s Tryon
White Adams Laurie A Margaret N mill head House ElizabetMills
Black Adams Leland porter J P Stowe & Co House 403 s Myers
Black Adams Lizzie House 309 1/2 w Morehead
White Adams wid (Geo O) Grace M House 601 n College
White Adams Luther M Mamie House Groveton
Black Adams Major waiter Buford Hotel
Black Adams Mattie House 719 n Graham ext
Black Adams Reuben Belle laborer Y & B Co House 419 w 2d
Black Adams Rosa House 714 s Caldwell
Black Adams Rufus laborer House Greenville
Black Adams Rufus driver Stand I & F Co House Ross Town
Black Adams Violet servant 508 w Trade
White Adams Wheeler F Mamie moulder House 303 s Cedar
White Adams William E The Chronicle Rooms 300 1/2 s Church
White Adcock John F mill worker House 916 Calvine av
White Adcock wid (Jas M) Millie M House 916 Calvine av
White Adelsheimer Henry S Lizzie mill worker House 1216 Louise av
Black Adkins King laborer 600 s Myers
White Adkins Walter D (Leona E) Walter D Leona E lineman House (r) 305 e 13th
Black Agers Nancy cook House 206 Wilson
Black Agers Sallie laundress House 420 Jackson
White Ahaus Herman Frances E tailor 203 w 4th House 204 s Church
White Aikel Joseph confectioner 317 e Trade Rooms 225 w Trade
White Aiken George W M Barbara superintendent Queen City M & G Wks House 1120 s Caldwell
White Aiken Henry Rooms 9 e 3d
Black Aiken Walter Ella laborer House 600 e 2d
Black Aiken William Ella driver W I Henderson Gro Co House 528 e 8th
White Akers Joseph J chief clerk superintendent Southern Railway House 908 s Tryon
White Albea John C pressman News Ptg Co House 310 s McDowell
White Albea wid (J F) Emma S House 310 s McDowell
White Albrecht wid (Mathias) Gesha House 402 Elizabeth av
White Albright Fay L clerk House 15 s Caldwell
White Albright Hal C clerk Post Office House 15 s Caldwell
White Albright Judson D Belle C officer US Internal Revenue House 15 s Caldwell
White Aldred Curtis M Victoria machinist House Seversville
White Aldred James A Mattie clerk House Seversville
White Aldridge wid (Wm A) Millie S House Harrill st Belmont Park
White Alexain Nicholas employment Met Café Rooms 26 n Tryon
Black Alexander Adelaide House 910 (909) e 1st
Black Alexander Adeline House Lutheran st
White Alexander Albert W Alice foreman Charlotte Cordage Co House 621 e 5t
Black Alexander Alfred Willie driver House 1016 e 3d
Black Alexander Allen Lizzie laborer House 6 10th st al
Black Alexander Alvey laborer House Lutheran st
Black Alexander Andrew Bessie grocer 15 1/2 e Boundary House 514 e Stonewall
Black Alexander Andrew W Rooms 415 n Myers
White Alexander Arnold E Alice laundryman House 1424 e 5th
Black Alexander Bernard porter Young's Steam Baking Co
Black Alexander Bessie House 1006 e 1st
White Alexander Blanche madam House 17 Spring
Black Alexander Burdette porter Colonial Club
White Alexander C P House Providence rd
Black Alexander Calvin Emeline laborer House 213 1/2 w 1st
White Alexander Carl L clerk House 200 e Oak
Black Alexander Carrie House Gaither's al
Black Alexander Carson chauffeur J E S Davidson House 209 s Middle
Black Alexander Charles laborer House 410 n A
White Alexander Charles F Mary clerk Post Office House 308 e Worthington av
White Alexander Charles L Sue dentist 913-918 Realty Bldg House 900 s Tryon
White Alexander Charles Y teacher King's Bus College Rooms 305 s Church
White Alexander Clarence W employment J M Porter House 1116 s Tryon
White Alexander Clyde C painter House 810 n Brevard
Black Alexander Daniel Etta laborer House 527 e 8th
Black Alexander Decatur C Alta porter Colonial Club House 515 e 1st
Black Alexander Edward Louvenia laborer House 1012 e Stonewall
Black Alexander Eli Jennie grocer Oil Town House 900 s Tryon
Black Alexander Elizabeth laundress House 414 s College
Black Alexander Ellen House 606 s McDowell
White Alexander wid (Wm S) Emma V House 610 n Graham ext
Black Alexander Ephraim Alice driver Coles & Smith House Stonewall bet A & B
Black Aaron Amelia domestic 506 s Tryon
White Abbey Simeon A Mary A superintendent construction General Fire Ext Co House 104 Central av
White Abee Junius A telegraph operator Southern Railway Boards 507 n Graham
White Abel Abram Fannie traveling salesman House 506 w 10t
Black Abel Belle House 14 Boundary al
Black Abel George laborer House 12 Boundary al
White Abernathy C L House Lawyer's rd
White Abernathy Clement D Cora employment Charlotte Lea Belt Co House 1117 s Tryon
White Abernathy David M Enola House 1310 e 4text
Black Abernathy Hannah House 4 Bellinger
White Abernathy John W Nannie carpenter House Sunnyside
Black Abernathy Joseph laborer House 422 w Hill
Black Abernathy Lewis Annie carpenter House 422 w Hill
White Abernethy Glenn E salesman Burwell & Dunn Co House 409 w 11th
White Abernethy Lee J Ida painter h 810 n Brevard House Seversville
White Abernethy James F Alice blacksmith Trade ext House Seversville
White Abernethy wid (Jas C) Margaret K House 3 e 1st
White Abernethy Thomas J Lucy employment City House 414 Templeton av
White Abernethy Leslie W machinist W S Abernethy House 409 w 11th
White Abernethy William S Mamie automobile representative 29 w 4th House 409 w 11th
White Abraham William Lula House 516 s College
Black Adanis Anna teacher House 1021 s Church
Black Adams Belle maid Realty Bldg House 419 w 2d
Black Adams Berry A Edith porter W S Cramer House Seversville
Black Adams Bessie cook 247 e Trade
White Adams Compton Cora mill head House Elizabeth Mills
Black Adams Edward Mittie driver Rhyne Bros House 305 w Palmer
White Adams George tailor Henry Miller Jr Boards 15 & Church
Black Adams Henry Dicie laborer House 308 Middle
Black Aaron Amelia domestic 506 s Tryon
White Abbey Simeon A Mary A superintendent construction General Fire Ext Co House 104 Central av
White Abee Junius A telegraph operator Southern Railway Boards 507 n Graham
White Abel Abram Fannie traveling salesman House 506 w 10t
Black Abel Belle House 14 Boundary al
Black Abel George laborer House 12 Boundary al
White Abernathy C L House Lawyer's rd
White Abernathy Clement D Cora employment Charlotte Lea Belt Co House 1117 s Tryon
White Abernathy David M Enola House 1310 e 4text
Black Abernathy Hannah House 4 Bellinger
White Abernathy John W Nannie carpenter House Sunnyside
Black Abernathy Joseph laborer House 422 w Hill
Black Abernathy Lewis Annie carpenter House 422 w Hill
White Abernethy Glenn E salesman Burwell & Dunn Co House 409 w 11th
White Abernethy Lee J Ida painter h 810 n Brevard House Seversville
White Abernethy James F Alice blacksmith Trade ext House Seversville
White Abernethy wid (Jas C) Margaret K House 3 e 1st
White Abernethy Thomas J Lucy employment City House 414 Templeton av
White Abernethy Leslie W machinist W S Abernethy House 409 w 11th
White Abernethy William S Mamie automobile representative 29 w 4th House 409 w 11th
White Abraham William Lula House 516 s College
Black Adanis Anna teacher House 1021 s Church
Black Adams Belle maid Realty Bldg House 419 w 2d
Black Adams Berry A Edith porter W S Cramer House Seversville
Black Adams Bessie cook 247 e Trade
White Adams Compton Cora mill head House Elizabeth Mills
Black Adams Edward Mittie driver Rhyne Bros House 305 w Palmer
White Adams George tailor Henry Miller Jr Boards 15 & Church
Black Adams Henry Dicie laborer House 308 Middle
[{'race': 'White', 'surname': 'Adams', 'title': 'Rev'}, {'race': 'White', 'surname': 'Adams', 'title': 'Miss'}, {'race': 'White', 'surname': 'Adams', 'title': 'Miss'}]
Now that we have captured all of the lines that have titles, we need to deal with the fact that many lines do not include a title at all. So we need to make the title pattern group optional, using a leading question mark.
pattern = re.compile("^(?P<race>Black|White)\t(?P<surname>\w*)\t+(?P<title>Miss|Dr|Rev|Mrs)?.*$") # Match specific titles
data = parse(pattern)
print(data[0:3])
White Adkins Walter D (Leona E) Walter D Leona E lineman House (r) 305 e 13th
[{'race': 'White', 'surname': 'Adams', 'title': None}, {'race': 'White', 'surname': 'Adams', 'title': None}, {'race': 'Black', 'surname': 'Adams', 'title': None}]
From the output above we learned that our new pattern with optional title matches every line except one. The reason is the '\t' or tab character we are expecting between the surname and the title. In the one exceptional line, printed above by our parse() function, the surname and title are separated by a space. Instead of using the tab character, let's use a generic "whitespace" detector, which is '\w'.
pattern = re.compile("^(?P<race>Black|White)\t(?P<surname>\w*)\w+(?P<title>Miss|Dr|Rev|Mrs)?.*$") # Match specific titles
data = parse(pattern)
print(data[0:3])
[{'race': 'White', 'surname': 'Adam', 'title': None}, {'race': 'White', 'surname': 'Adam', 'title': None}, {'race': 'Black', 'surname': 'Adam', 'title': None}]
Now all the text lines are matching again and we have the titles, when they are present.
The next group is the head of household's given name.
pattern = re.compile("^(?P<race>Black|White)\t(?P<surname>\w*)\W+(?P<title>Miss|Dr|Rev|Mrs)?(?P<hohgiven>\w+).*$") # Match specific titles
data = parse(pattern)
print(data[0:3])
[{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'Henry'}, {'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'James'}, {'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Jane'}]
Sometimes there is a middle initial for the head of the household.. It appears that the initial is only ever one whitespace character away from the head of household given name. We are also going to change our display output so that we can look more closely at the results..
pattern = re.compile("^(?P<race>Black|White)\t(?P<surname>\w*)\W+(?P<title>Miss|Dr|Rev|Mrs)?(?P<hohgiven>\w+)(\W(?P<hohmi>\w))?.*$") # Match specific titles
data = parse(pattern)
for i in range(0, len(alltext)):
print(alltext[i])
print(data[i])
White Adams Henry L Fannie S route agent Southern Railway Co House 327 n Tryon
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'Henry', 'hohmi': 'L'}
White Adams James Gertrude manager House 419 Elizabeth av
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'James', 'hohmi': None}
Black Adams Jane teacher House 1021 s Church
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Jane', 'hohmi': None}
Black Adams John laborer House 1031 s Church
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'John', 'hohmi': None}
White Adams John J president Adams G & P Co and Char Pepsi-Cola Co House 309 e 6th
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'John', 'hohmi': 'J'}
White Adams John W Cora conductor S A L Railway House 21st nr Caldwell
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'John', 'hohmi': 'W'}
Black Adams Joseph Violet cooper House 1011 s Church
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Joseph', 'hohmi': None}
White Adams Rev Joseph Q Leslie House 1509 s Boulevard
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'Rev', 'hohmi': None}
White Adams Miss Julia M House 707 n Church
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'J'}
Black Adams Kate laundress House Groveton
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Kate', 'hohmi': None}
White Adams Lafayette N clerk Southern Railway House 327 n Tryon
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'Lafayette', 'hohmi': 'N'}
White Adams Lawrence A salesman B S Moore & Co Rooms 405 s Tryon
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'Lawrence', 'hohmi': 'A'}
White Adams Laurie A Margaret N mill head House ElizabetMills
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'Laurie', 'hohmi': 'A'}
Black Adams Leland porter J P Stowe & Co House 403 s Myers
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Leland', 'hohmi': None}
Black Adams Lizzie House 309 1/2 w Morehead
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Lizzie', 'hohmi': None}
White Adams Miss Lula clerk Belk Bros Rooms Y W C A
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'L'}
White Adams wid (Geo O) Grace M House 601 n College
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'wid', 'hohmi': None}
White Adams Luther M Mamie House Groveton
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'Luther', 'hohmi': 'M'}
Black Adams Major waiter Buford Hotel
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Major', 'hohmi': None}
Black Adams Mattie House 719 n Graham ext
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Mattie', 'hohmi': None}
White Adams Miss Pattie V stenographer Boards 708 n Caldwell
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'P'}
Black Adams Reuben Belle laborer Y & B Co House 419 w 2d
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Reuben', 'hohmi': None}
Black Adams Rosa House 714 s Caldwell
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Rosa', 'hohmi': None}
Black Adams Rufus laborer House Greenville
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Rufus', 'hohmi': None}
Black Adams Rufus driver Stand I & F Co House Ross Town
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Rufus', 'hohmi': None}
White Adams Miss Salie H assistant Carnegie Library House 707 n Church
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'S'}
Black Adams Violet servant 508 w Trade
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Violet', 'hohmi': None}
White Adams Wheeler F Mamie moulder House 303 s Cedar
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'Wheeler', 'hohmi': 'F'}
White Adams William E The Chronicle Rooms 300 1/2 s Church
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'William', 'hohmi': 'E'}
White Adcock John F mill worker House 916 Calvine av
{'race': 'White', 'surname': 'Adcock', 'title': None, 'hohgiven': 'John', 'hohmi': 'F'}
White Adcock wid (Jas M) Millie M House 916 Calvine av
{'race': 'White', 'surname': 'Adcock', 'title': None, 'hohgiven': 'wid', 'hohmi': None}
White Adelsheimer Henry S Lizzie mill worker House 1216 Louise av
{'race': 'White', 'surname': 'Adelsheimer', 'title': None, 'hohgiven': 'Henry', 'hohmi': 'S'}
Black Adkins King laborer 600 s Myers
{'race': 'Black', 'surname': 'Adkins', 'title': None, 'hohgiven': 'King', 'hohmi': None}
White Adkins Walter D (Leona E) Walter D Leona E lineman House (r) 305 e 13th
{'race': 'White', 'surname': 'Adkins', 'title': None, 'hohgiven': 'Walter', 'hohmi': 'D'}
Black Agers Nancy cook House 206 Wilson
{'race': 'Black', 'surname': 'Agers', 'title': None, 'hohgiven': 'Nancy', 'hohmi': None}
Black Agers Sallie laundress House 420 Jackson
{'race': 'Black', 'surname': 'Agers', 'title': None, 'hohgiven': 'Sallie', 'hohmi': None}
White Ahaus Herman Frances E tailor 203 w 4th House 204 s Church
{'race': 'White', 'surname': 'Ahaus', 'title': None, 'hohgiven': 'Herman', 'hohmi': None}
White Aikel Joseph confectioner 317 e Trade Rooms 225 w Trade
{'race': 'White', 'surname': 'Aikel', 'title': None, 'hohgiven': 'Joseph', 'hohmi': None}
White Aiken George W M Barbara superintendent Queen City M & G Wks House 1120 s Caldwell
{'race': 'White', 'surname': 'Aiken', 'title': None, 'hohgiven': 'George', 'hohmi': 'W'}
White Aiken Henry Rooms 9 e 3d
{'race': 'White', 'surname': 'Aiken', 'title': None, 'hohgiven': 'Henry', 'hohmi': None}
Black Aiken Walter Ella laborer House 600 e 2d
{'race': 'Black', 'surname': 'Aiken', 'title': None, 'hohgiven': 'Walter', 'hohmi': None}
Black Aiken William Ella driver W I Henderson Gro Co House 528 e 8th
{'race': 'Black', 'surname': 'Aiken', 'title': None, 'hohgiven': 'William', 'hohmi': None}
White Akers Joseph J chief clerk superintendent Southern Railway House 908 s Tryon
{'race': 'White', 'surname': 'Akers', 'title': None, 'hohgiven': 'Joseph', 'hohmi': 'J'}
White Albea John C pressman News Ptg Co House 310 s McDowell
{'race': 'White', 'surname': 'Albea', 'title': None, 'hohgiven': 'John', 'hohmi': 'C'}
White Albea Miss Lottie clerk Floyd L Liles Co House 10 w 11th
{'race': 'White', 'surname': 'Albea', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'L'}
White Albea wid (J F) Emma S House 310 s McDowell
{'race': 'White', 'surname': 'Albea', 'title': None, 'hohgiven': 'wid', 'hohmi': None}
White Albrecht wid (Mathias) Gesha House 402 Elizabeth av
{'race': 'White', 'surname': 'Albrecht', 'title': None, 'hohgiven': 'wid', 'hohmi': None}
White Albright Fay L clerk House 15 s Caldwell
{'race': 'White', 'surname': 'Albright', 'title': None, 'hohgiven': 'Fay', 'hohmi': 'L'}
White Albright Hal C clerk Post Office House 15 s Caldwell
{'race': 'White', 'surname': 'Albright', 'title': None, 'hohgiven': 'Hal', 'hohmi': 'C'}
White Albright Judson D Belle C officer US Internal Revenue House 15 s Caldwell
{'race': 'White', 'surname': 'Albright', 'title': None, 'hohgiven': 'Judson', 'hohmi': 'D'}
White Alderman Miss Elizabeth nurse Charlotte Sanatorium Rooms 12 w 7th
{'race': 'White', 'surname': 'Alderman', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'E'}
White Aldred Curtis M Victoria machinist House Seversville
{'race': 'White', 'surname': 'Aldred', 'title': None, 'hohgiven': 'Curtis', 'hohmi': 'M'}
White Aldred James A Mattie clerk House Seversville
{'race': 'White', 'surname': 'Aldred', 'title': None, 'hohgiven': 'James', 'hohmi': 'A'}
White Aldred Rev William L Marian evangelist House Seversville
{'race': 'White', 'surname': 'Aldred', 'title': None, 'hohgiven': 'Rev', 'hohmi': None}
White Aldridge wid (Wm A) Millie S House Harrill st Belmont Park
{'race': 'White', 'surname': 'Aldridge', 'title': None, 'hohgiven': 'wid', 'hohmi': None}
White Alexain Nicholas employment Met Café Rooms 26 n Tryon
{'race': 'White', 'surname': 'Alexain', 'title': None, 'hohgiven': 'Nicholas', 'hohmi': None}
Black Alexander Adelaide House 910 (909) e 1st
{'race': 'Black', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Adelaide', 'hohmi': None}
Black Alexander Adeline House Lutheran st
{'race': 'Black', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Adeline', 'hohmi': None}
White Alexander Albert W Alice foreman Charlotte Cordage Co House 621 e 5t
{'race': 'White', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Albert', 'hohmi': 'W'}
Black Alexander Alfred Willie driver House 1016 e 3d
{'race': 'Black', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Alfred', 'hohmi': None}
Black Alexander Allen Lizzie laborer House 6 10th st al
{'race': 'Black', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Allen', 'hohmi': None}
Black Alexander Alvey laborer House Lutheran st
{'race': 'Black', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Alvey', 'hohmi': None}
Black Alexander Andrew Bessie grocer 15 1/2 e Boundary House 514 e Stonewall
{'race': 'Black', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Andrew', 'hohmi': None}
Black Alexander Andrew W Rooms 415 n Myers
{'race': 'Black', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Andrew', 'hohmi': 'W'}
White Alexander Miss Annie stenographer C M Stiefif House 610 n Graham ext
{'race': 'White', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'A'}
White Alexander Miss Annie L physician 410 n Tryon House 610 n Graham ext
{'race': 'White', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'A'}
White Alexander Arnold E Alice laundryman House 1424 e 5th
{'race': 'White', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Arnold', 'hohmi': 'E'}
Black Alexander Bernard porter Young's Steam Baking Co
{'race': 'Black', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Bernard', 'hohmi': None}
Black Alexander Bessie House 1006 e 1st
{'race': 'Black', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Bessie', 'hohmi': None}
White Alexander Blanche madam House 17 Spring
{'race': 'White', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Blanche', 'hohmi': None}
White Alexander Miss Blandina millliner J B Ivey Co House 610 n Graham ext
{'race': 'White', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'B'}
Black Alexander Burdette porter Colonial Club
{'race': 'Black', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Burdette', 'hohmi': None}
White Alexander C P House Providence rd
{'race': 'White', 'surname': 'Alexander', 'title': None, 'hohgiven': 'C', 'hohmi': 'P'}
Black Alexander Calvin Emeline laborer House 213 1/2 w 1st
{'race': 'Black', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Calvin', 'hohmi': None}
White Alexander Carl L clerk House 200 e Oak
{'race': 'White', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Carl', 'hohmi': 'L'}
Black Alexander Carrie House Gaither's al
{'race': 'Black', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Carrie', 'hohmi': None}
Black Alexander Carson chauffeur J E S Davidson House 209 s Middle
{'race': 'Black', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Carson', 'hohmi': None}
Black Alexander Charles laborer House 410 n A
{'race': 'Black', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Charles', 'hohmi': None}
White Alexander Charles F Mary clerk Post Office House 308 e Worthington av
{'race': 'White', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Charles', 'hohmi': 'F'}
White Alexander Charles L Sue dentist 913-918 Realty Bldg House 900 s Tryon
{'race': 'White', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Charles', 'hohmi': 'L'}
White Alexander Charles Y teacher King's Bus College Rooms 305 s Church
{'race': 'White', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Charles', 'hohmi': 'Y'}
White Alexander Clarence W employment J M Porter House 1116 s Tryon
{'race': 'White', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Clarence', 'hohmi': 'W'}
White Alexander Clyde C painter House 810 n Brevard
{'race': 'White', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Clyde', 'hohmi': 'C'}
Black Alexander Daniel Etta laborer House 527 e 8th
{'race': 'Black', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Daniel', 'hohmi': None}
Black Alexander Decatur C Alta porter Colonial Club House 515 e 1st
{'race': 'Black', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Decatur', 'hohmi': 'C'}
White Alexander Miss Edna nurse St Peters Hospital Rooms 515 e 1st
{'race': 'White', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'E'}
Black Alexander Edward Louvenia laborer House 1012 e Stonewall
{'race': 'Black', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Edward', 'hohmi': None}
White Alexander Miss Eleanor House 900 s Tryon
{'race': 'White', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'E'}
Black Alexander Eli Jennie grocer Oil Town House 900 s Tryon
{'race': 'Black', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Eli', 'hohmi': None}
Black Alexander Elizabeth laundress House 414 s College
{'race': 'Black', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Elizabeth', 'hohmi': None}
Black Alexander Ellen House 606 s McDowell
{'race': 'Black', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Ellen', 'hohmi': None}
White Alexander wid (Wm S) Emma V House 610 n Graham ext
{'race': 'White', 'surname': 'Alexander', 'title': None, 'hohgiven': 'wid', 'hohmi': None}
Black Alexander Ephraim Alice driver Coles & Smith House Stonewall bet A & B
{'race': 'Black', 'surname': 'Alexander', 'title': None, 'hohgiven': 'Ephraim', 'hohmi': None}
Black Aaron Amelia domestic 506 s Tryon
{'race': 'Black', 'surname': 'Aaron', 'title': None, 'hohgiven': 'Amelia', 'hohmi': None}
White Abbey Simeon A Mary A superintendent construction General Fire Ext Co House 104 Central av
{'race': 'White', 'surname': 'Abbey', 'title': None, 'hohgiven': 'Simeon', 'hohmi': 'A'}
White Abbott Miss Margaret House 1804 s Boulevard
{'race': 'White', 'surname': 'Abbott', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'M'}
White Abee Junius A telegraph operator Southern Railway Boards 507 n Graham
{'race': 'White', 'surname': 'Abee', 'title': None, 'hohgiven': 'Junius', 'hohmi': 'A'}
White Abel Abram Fannie traveling salesman House 506 w 10t
{'race': 'White', 'surname': 'Abel', 'title': None, 'hohgiven': 'Abram', 'hohmi': None}
Black Abel Belle House 14 Boundary al
{'race': 'Black', 'surname': 'Abel', 'title': None, 'hohgiven': 'Belle', 'hohmi': None}
Black Abel George laborer House 12 Boundary al
{'race': 'Black', 'surname': 'Abel', 'title': None, 'hohgiven': 'George', 'hohmi': None}
White Abernathy C L House Lawyer's rd
{'race': 'White', 'surname': 'Abernathy', 'title': None, 'hohgiven': 'C', 'hohmi': 'L'}
White Abernathy Clement D Cora employment Charlotte Lea Belt Co House 1117 s Tryon
{'race': 'White', 'surname': 'Abernathy', 'title': None, 'hohgiven': 'Clement', 'hohmi': 'D'}
White Abernathy Mrs Cora boarding House 1117 s Tryon
{'race': 'White', 'surname': 'Abernathy', 'title': None, 'hohgiven': 'Mrs', 'hohmi': 'C'}
White Abernathy David M Enola House 1310 e 4text
{'race': 'White', 'surname': 'Abernathy', 'title': None, 'hohgiven': 'David', 'hohmi': 'M'}
White Abernathy Miss Elizabeth House 430 Mint
{'race': 'White', 'surname': 'Abernathy', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'E'}
Black Abernathy Hannah House 4 Bellinger
{'race': 'Black', 'surname': 'Abernathy', 'title': None, 'hohgiven': 'Hannah', 'hohmi': None}
White Abernathy John W Nannie carpenter House Sunnyside
{'race': 'White', 'surname': 'Abernathy', 'title': None, 'hohgiven': 'John', 'hohmi': 'W'}
Black Abernathy Joseph laborer House 422 w Hill
{'race': 'Black', 'surname': 'Abernathy', 'title': None, 'hohgiven': 'Joseph', 'hohmi': None}
Black Abernathy Lewis Annie carpenter House 422 w Hill
{'race': 'Black', 'surname': 'Abernathy', 'title': None, 'hohgiven': 'Lewis', 'hohmi': None}
White Abernethy Miss Connie stenographer Burwell & Dunn Co House 414 Templeton av
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'C'}
White Abernethy Glenn E salesman Burwell & Dunn Co House 409 w 11th
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'Glenn', 'hohmi': 'E'}
White Abernethy Miss Gertrude fitter Little-Long Co House 306 w 7th
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'G'}
White Abernethy Lee J Ida painter h 810 n Brevard House Seversville
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'Lee', 'hohmi': 'J'}
White Abernethy Dr J S House Beatty's Ford rd
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'Dr', 'hohmi': None}
White Abernethy James F Alice blacksmith Trade ext House Seversville
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'James', 'hohmi': 'F'}
White Abernethy Miss Lillian saleslady Efird's Dept Store House Seversville
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'L'}
White Abernethy wid (Jas C) Margaret K House 3 e 1st
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'wid', 'hohmi': None}
White Abernethy Miss Mildred Rooms 603 n Davidson
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'M'}
White Abernethy Miss Nettie I bookkeeper Pound & Moore Co House 311 n College
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'N'}
White Abernethy Thomas J Lucy employment City House 414 Templeton av
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'Thomas', 'hohmi': 'J'}
White Abernethy Leslie W machinist W S Abernethy House 409 w 11th
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'Leslie', 'hohmi': 'W'}
White Abernethy William S Mamie automobile representative 29 w 4th House 409 w 11th
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'William', 'hohmi': 'S'}
White Abraham William Lula House 516 s College
{'race': 'White', 'surname': 'Abraham', 'title': None, 'hohgiven': 'William', 'hohmi': None}
Black Adanis Anna teacher House 1021 s Church
{'race': 'Black', 'surname': 'Adanis', 'title': None, 'hohgiven': 'Anna', 'hohmi': None}
Black Adams Belle maid Realty Bldg House 419 w 2d
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Belle', 'hohmi': None}
Black Adams Berry A Edith porter W S Cramer House Seversville
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Berry', 'hohmi': 'A'}
Black Adams Bessie cook 247 e Trade
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Bessie', 'hohmi': None}
White Adams Miss Beulah House 419 Elizabeth av
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'B'}
White Adams Compton Cora mill head House Elizabeth Mills
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'Compton', 'hohmi': None}
White Adams Miss Dorothy stenographer Oconee Mills Co Rooms YWCA
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'D'}
Black Adams Edward Mittie driver Rhyne Bros House 305 w Palmer
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Edward', 'hohmi': None}
White Adams George tailor Henry Miller Jr Boards 15 & Church
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'George', 'hohmi': None}
Black Adams Henry Dicie laborer House 308 Middle
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Henry', 'hohmi': None}
Black Aaron Amelia domestic 506 s Tryon
{'race': 'Black', 'surname': 'Aaron', 'title': None, 'hohgiven': 'Amelia', 'hohmi': None}
White Abbey Simeon A Mary A superintendent construction General Fire Ext Co House 104 Central av
{'race': 'White', 'surname': 'Abbey', 'title': None, 'hohgiven': 'Simeon', 'hohmi': 'A'}
White Abbott Miss Margaret House 1804 s Boulevard
{'race': 'White', 'surname': 'Abbott', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'M'}
White Abee Junius A telegraph operator Southern Railway Boards 507 n Graham
{'race': 'White', 'surname': 'Abee', 'title': None, 'hohgiven': 'Junius', 'hohmi': 'A'}
White Abel Abram Fannie traveling salesman House 506 w 10t
{'race': 'White', 'surname': 'Abel', 'title': None, 'hohgiven': 'Abram', 'hohmi': None}
Black Abel Belle House 14 Boundary al
{'race': 'Black', 'surname': 'Abel', 'title': None, 'hohgiven': 'Belle', 'hohmi': None}
Black Abel George laborer House 12 Boundary al
{'race': 'Black', 'surname': 'Abel', 'title': None, 'hohgiven': 'George', 'hohmi': None}
White Abernathy C L House Lawyer's rd
{'race': 'White', 'surname': 'Abernathy', 'title': None, 'hohgiven': 'C', 'hohmi': 'L'}
White Abernathy Clement D Cora employment Charlotte Lea Belt Co House 1117 s Tryon
{'race': 'White', 'surname': 'Abernathy', 'title': None, 'hohgiven': 'Clement', 'hohmi': 'D'}
White Abernathy Mrs Cora boarding House 1117 s Tryon
{'race': 'White', 'surname': 'Abernathy', 'title': None, 'hohgiven': 'Mrs', 'hohmi': 'C'}
White Abernathy David M Enola House 1310 e 4text
{'race': 'White', 'surname': 'Abernathy', 'title': None, 'hohgiven': 'David', 'hohmi': 'M'}
White Abernathy Miss Elizabeth House 430 Mint
{'race': 'White', 'surname': 'Abernathy', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'E'}
Black Abernathy Hannah House 4 Bellinger
{'race': 'Black', 'surname': 'Abernathy', 'title': None, 'hohgiven': 'Hannah', 'hohmi': None}
White Abernathy John W Nannie carpenter House Sunnyside
{'race': 'White', 'surname': 'Abernathy', 'title': None, 'hohgiven': 'John', 'hohmi': 'W'}
Black Abernathy Joseph laborer House 422 w Hill
{'race': 'Black', 'surname': 'Abernathy', 'title': None, 'hohgiven': 'Joseph', 'hohmi': None}
Black Abernathy Lewis Annie carpenter House 422 w Hill
{'race': 'Black', 'surname': 'Abernathy', 'title': None, 'hohgiven': 'Lewis', 'hohmi': None}
White Abernethy Miss Connie stenographer Burwell & Dunn Co House 414 Templeton av
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'C'}
White Abernethy Glenn E salesman Burwell & Dunn Co House 409 w 11th
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'Glenn', 'hohmi': 'E'}
White Abernethy Miss Gertrude fitter Little-Long Co House 306 w 7th
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'G'}
White Abernethy Lee J Ida painter h 810 n Brevard House Seversville
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'Lee', 'hohmi': 'J'}
White Abernethy Dr J S House Beatty's Ford rd
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'Dr', 'hohmi': None}
White Abernethy James F Alice blacksmith Trade ext House Seversville
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'James', 'hohmi': 'F'}
White Abernethy Miss Lillian saleslady Efird's Dept Store House Seversville
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'L'}
White Abernethy wid (Jas C) Margaret K House 3 e 1st
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'wid', 'hohmi': None}
White Abernethy Miss Mildred Rooms 603 n Davidson
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'M'}
White Abernethy Miss Nettie I bookkeeper Pound & Moore Co House 311 n College
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'N'}
White Abernethy Thomas J Lucy employment City House 414 Templeton av
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'Thomas', 'hohmi': 'J'}
White Abernethy Leslie W machinist W S Abernethy House 409 w 11th
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'Leslie', 'hohmi': 'W'}
White Abernethy William S Mamie automobile representative 29 w 4th House 409 w 11th
{'race': 'White', 'surname': 'Abernethy', 'title': None, 'hohgiven': 'William', 'hohmi': 'S'}
White Abraham William Lula House 516 s College
{'race': 'White', 'surname': 'Abraham', 'title': None, 'hohgiven': 'William', 'hohmi': None}
Black Adanis Anna teacher House 1021 s Church
{'race': 'Black', 'surname': 'Adanis', 'title': None, 'hohgiven': 'Anna', 'hohmi': None}
Black Adams Belle maid Realty Bldg House 419 w 2d
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Belle', 'hohmi': None}
Black Adams Berry A Edith porter W S Cramer House Seversville
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Berry', 'hohmi': 'A'}
Black Adams Bessie cook 247 e Trade
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Bessie', 'hohmi': None}
White Adams Miss Beulah House 419 Elizabeth av
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'B'}
White Adams Compton Cora mill head House Elizabeth Mills
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'Compton', 'hohmi': None}
White Adams Miss Dorothy stenographer Oconee Mills Co Rooms YWCA
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'Miss', 'hohmi': 'D'}
Black Adams Edward Mittie driver Rhyne Bros House 305 w Palmer
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Edward', 'hohmi': None}
White Adams George tailor Henry Miller Jr Boards 15 & Church
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'George', 'hohmi': None}
Black Adams Henry Dicie laborer House 308 Middle
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Henry', 'hohmi': None}
Next is often the name of a second person in the household, but not always. Sometimes the next word is the start of the head of household occupation. These are distinct in that names start with an upper case character. So our second given name pattern will ask for an upper case first character.
pattern = re.compile("^(?P<race>Black|White)\t(?P<surname>\w*)\W+(?P<title>Miss|Dr|Rev|Mrs)?(?P<hohgiven>\w+)(\W(?P<hohmi>\w))?\W+(?P<given2>[A-Z]{1}\w*)?.*$") # Match specific titles
data = parse(pattern)
for i in range(0, 5):
print(alltext[i])
print(data[i])
White Adams Henry L Fannie S route agent Southern Railway Co House 327 n Tryon
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'Henry', 'hohmi': 'L', 'given2': 'Fannie'}
White Adams James Gertrude manager House 419 Elizabeth av
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'James', 'hohmi': None, 'given2': 'Gertrude'}
Black Adams Jane teacher House 1021 s Church
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Jane', 'hohmi': None, 'given2': None}
Black Adams John laborer House 1031 s Church
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'John', 'hohmi': None, 'given2': None}
White Adams John J president Adams G & P Co and Char Pepsi-Cola Co House 309 e 6th
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'John', 'hohmi': 'J', 'given2': None}
The second person may also optionally have a middle initial.
pattern = re.compile("^(?P<race>Black|White)\t(?P<surname>\w*)\W+(?P<title>Miss|Dr|Rev|Mrs)?(?P<hohgiven>\w+)(\W(?P<hohmi>\w))?\W+(?P<given2>[A-Z]{1}\w*)?(\W(?P<mi2>\w))?.*$") # Match specific titles
data = parse(pattern)
for i in range(0, 5):
print(alltext[i])
print(data[i])
White Adams Henry L Fannie S route agent Southern Railway Co House 327 n Tryon
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'Henry', 'hohmi': 'L', 'given2': 'Fannie', 'mi2': 'S'}
White Adams James Gertrude manager House 419 Elizabeth av
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'James', 'hohmi': None, 'given2': 'Gertrude', 'mi2': None}
Black Adams Jane teacher House 1021 s Church
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Jane', 'hohmi': None, 'given2': None, 'mi2': None}
Black Adams John laborer House 1031 s Church
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'John', 'hohmi': None, 'given2': None, 'mi2': None}
White Adams John J president Adams G & P Co and Char Pepsi-Cola Co House 309 e 6th
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'John', 'hohmi': 'J', 'given2': None, 'mi2': None}
Now we come to the occupation, which is a series of lower case words, separated by single spaces.
pattern = re.compile("^(?P<race>Black|White)\t(?P<surname>\w*)\W+(?P<title>Miss|Dr|Rev|Mrs)?(?P<hohgiven>\w+)(\W(?P<hohmi>\w))?\W+(?P<given2>[A-Z]{1}\w*)?(\W(?P<mi2>\w))?\W+(?P<occupation>[a-z ]+)?.*$") # Match specific titles
data = parse(pattern)
for i in range(0, 5):
print(alltext[i])
print(data[i])
White Adams Henry L Fannie S route agent Southern Railway Co House 327 n Tryon
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'Henry', 'hohmi': 'L', 'given2': 'Fannie', 'mi2': 'S', 'occupation': 'route agent'}
White Adams James Gertrude manager House 419 Elizabeth av
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'James', 'hohmi': None, 'given2': 'Gertrude', 'mi2': None, 'occupation': 'manager'}
Black Adams Jane teacher House 1021 s Church
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Jane', 'hohmi': None, 'given2': None, 'mi2': None, 'occupation': 'teacher'}
Black Adams John laborer House 1031 s Church
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'John', 'hohmi': None, 'given2': None, 'mi2': None, 'occupation': 'laborer'}
White Adams John J president Adams G & P Co and Char Pepsi-Cola Co House 309 e 6th
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'John', 'hohmi': 'J', 'given2': None, 'mi2': None, 'occupation': 'president'}
The next bit seems to be either "House" or "Boards" or the capitalized, often multi-word, name of a business. Let's add a required "House" or "Boards" later in the pattern as required for a match. Then the workplace pattern will be before that and optional.
pattern = re.compile("^(?P<race>Black|White)\t(?P<surname>\w+)(\W+(?P<title>Miss|Dr|Rev|Mrs))?(\W+(?P<hohgiven>\w+))(\W+(?P<hohmi>[A-Z]{1}))?(\W+(?P<given2>[A-Z]{1}\w*))?(\W+(?P<mi2>[A-Z]{1}))?(\W+(?P<occupation>[a-z ]+))?(\W+(?P<workplace>[A-Z0-9]{1}[A-Za-z0-9 /&-]+))?(\W+(?P<la>House|Boards|Rooms))?(\W+(?P<address>[A-Za-z0-9 /]+))?$")
data = parse(pattern)
for i in range(0, 5):
print(alltext[i])
print(data[i])
White Adams wid (Geo O) Grace M House 601 n College
Black Adams Major waiter Buford Hotel
White Adcock wid (Jas M) Millie M House 916 Calvine av
White Adkins Walter D (Leona E) Walter D Leona E lineman House (r) 305 e 13th
White Aiken George W M Barbara superintendent Queen City M & G Wks House 1120 s Caldwell
White Albea wid (J F) Emma S House 310 s McDowell
White Aldridge wid (Wm A) Millie S House Harrill st Belmont Park
White Alexain Nicholas employment Met Café Rooms 26 n Tryon
Black Alexander Adelaide House 910 (909) e 1st
Black Alexander Bernard porter Young's Steam Baking Co
Black Alexander Burdette porter Colonial Club
White Alexander Charles Y teacher King's Bus College Rooms 305 s Church
White Alexander wid (Wm S) Emma V House 610 n Graham ext
Black Alexander Ephraim Alice driver Coles & Smith House Stonewall bet A & B
White Abernethy Lee J Ida painter h 810 n Brevard House Seversville
White Abernethy Miss Lillian saleslady Efird's Dept Store House Seversville
White Abernethy wid (Jas C) Margaret K House 3 e 1st
White Adams George tailor Henry Miller Jr Boards 15 & Church
White Abernethy Lee J Ida painter h 810 n Brevard House Seversville
White Abernethy Miss Lillian saleslady Efird's Dept Store House Seversville
White Abernethy wid (Jas C) Margaret K House 3 e 1st
White Adams George tailor Henry Miller Jr Boards 15 & Church
White Adams Henry L Fannie S route agent Southern Railway Co House 327 n Tryon
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'Henry', 'hohmi': 'L', 'given2': 'Fannie', 'mi2': 'S', 'occupation': 'route agent', 'workplace': 'Southern Railway Co', 'la': 'House', 'address': '327 n Tryon '}
White Adams James Gertrude manager House 419 Elizabeth av
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'James', 'hohmi': None, 'given2': 'Gertrude', 'mi2': None, 'occupation': 'manager', 'workplace': 'House', 'la': None, 'address': '419 Elizabeth av '}
Black Adams Jane teacher House 1021 s Church
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'Jane', 'hohmi': None, 'given2': None, 'mi2': None, 'occupation': 'teacher', 'workplace': 'House', 'la': None, 'address': '1021 s Church'}
Black Adams John laborer House 1031 s Church
{'race': 'Black', 'surname': 'Adams', 'title': None, 'hohgiven': 'John', 'hohmi': None, 'given2': None, 'mi2': None, 'occupation': 'laborer', 'workplace': 'House', 'la': None, 'address': '1031 s Church'}
White Adams John J president Adams G & P Co and Char Pepsi-Cola Co House 309 e 6th
{'race': 'White', 'surname': 'Adams', 'title': None, 'hohgiven': 'John', 'hohmi': 'J', 'given2': None, 'mi2': None, 'occupation': 'president', 'workplace': 'Adams G & P Co and Char Pepsi-Cola Co', 'la': 'House', 'address': '309 e 6th'}
Now looking at the unmatched lines, we see a number of problems remaining, each a special case that we need to add to the overall pattern..
White Adams wid (Geo O)
White Adkins Walter D (Leona E)
The parentheses above are unexpected in our current pattern.
pattern = re.compile(r'^(?P<race>Black|White)\t(?P<surname>\w+)(\W+wid \((?P<deadhusband>[\w ])\))?(\W+(?P<title>Miss|Dr|Rev|Mrs))?(\W+(?P<hohgiven>\w+))(\W+(?P<hohmi>[A-Z]{1})\W)?(\W*(?P<given2>(?!House|Boards|Rooms)[A-Z]{1}\w*))?(\W+(?P<mi2>[A-Z]{1})\W)?(\W*(?P<occupation>[a-z ]+))?(\W+(?P<workplace>(?!House|Boards|Rooms)[A-Z0-9]{1}[A-Za-z0-9- /&\']+))?(\W+(?P<la>House|Boards|Rooms))?(\W+(?P<address>[A-Za-z0-9 ]+))?.*$')
data = parse(pattern)
for i in range(0, 5):
print(alltext[i])
print(data[i])
White Adams Henry L Fannie S route agent Southern Railway Co House 327 n Tryon
{'race': 'White', 'surname': 'Adams', 'deadhusband': None, 'title': None, 'hohgiven': 'Henry', 'hohmi': 'L', 'given2': 'Fannie', 'mi2': 'S', 'occupation': 'route agent', 'workplace': 'Southern Railway Co', 'la': 'House', 'address': '327 n Tryon '}
White Adams James Gertrude manager House 419 Elizabeth av
{'race': 'White', 'surname': 'Adams', 'deadhusband': None, 'title': None, 'hohgiven': 'James', 'hohmi': None, 'given2': 'Gertrude', 'mi2': None, 'occupation': 'manager', 'workplace': None, 'la': 'House', 'address': '419 Elizabeth av '}
Black Adams Jane teacher House 1021 s Church
{'race': 'Black', 'surname': 'Adams', 'deadhusband': None, 'title': None, 'hohgiven': 'Jane', 'hohmi': None, 'given2': None, 'mi2': None, 'occupation': 'teacher', 'workplace': None, 'la': 'House', 'address': '1021 s Church'}
Black Adams John laborer House 1031 s Church
{'race': 'Black', 'surname': 'Adams', 'deadhusband': None, 'title': None, 'hohgiven': 'John', 'hohmi': None, 'given2': None, 'mi2': None, 'occupation': 'laborer', 'workplace': None, 'la': 'House', 'address': '1031 s Church'}
White Adams John J president Adams G & P Co and Char Pepsi-Cola Co House 309 e 6th
{'race': 'White', 'surname': 'Adams', 'deadhusband': None, 'title': None, 'hohgiven': 'John', 'hohmi': 'J', 'given2': None, 'mi2': None, 'occupation': 'president', 'workplace': 'Adams G & P Co and Char Pepsi-Cola Co', 'la': 'House', 'address': '309 e 6th'}
import json
with open(os.path.join(output_loc,'data.json'), 'w') as outfile:
json.dump(data, outfile)