Sources of Surname Data

We are primarily interested in sources of surname data that contain both surnames and countries of birth for surnames.

Internet Movie Database (IMDB)

A list of all actors and thier birth countries was extracted from the IMDB biographies file

Olympic Athletes

The Olympic Athletes data was taken from the relevant pages on wikipedia. Local copies of all of the individual country pages from the 2004 Summer Games were retrieved by a script (Olympics-RetrievePageURLs.pl) that uses an offline version (Olympics-OfflineSource.html) of the wikipedia Nations at the 2004 Summer Olympics page. This script also constructs a list of participating countries (Olympics-ParticipatingCountries.txt).

The offline pages were then parsed by another script (Olympics-ExtractOlypiads.pl) and the resulting output (Olympics-RawOutput.txt) was checked by hand. This output is the basic names set with countries for the 2004 Olympic Athletes used here. Because some individuals competed in multiple events, identical full name strings were collapsed to produce a single record with a count. It seems unlikely that many John Joe Smiths entered, making such a reduction erroneous. Users of these scripts should the wikipedia source files have likely changed and should check results carefully.

The country names were then corrected to the UN GeoRegions and coded using SQL scripts, and country with idiosyncratic name reversals were marked to produce a normalization input file (Olympics-RawOutputWithUNReversal.txt). The NormalizeSurnames.pl script was with following options (and defaults):

perl -i=Olympics-RawOutputWithUNReversal.txt -ncol=1 -rcol=3

The resultant output (Olympics-RawOutputWithUNReversalShort-Normalized.txt) can be used to create the n-gram variables.