Difference between revisions of "Sources of Surname Data"
imported>Ed |
imported>Ed |
||
| Line 1: | Line 1: | ||
We are primarily interested in sources of surname data that contain both surnames and countries of birth for surnames. | We are primarily interested in sources of surname data that contain both surnames and countries of birth for surnames. | ||
| − | |||
| − | |||
| − | |||
| − | |||
==Olympic Athletes== | ==Olympic Athletes== | ||
| Line 15: | Line 11: | ||
<tt>perl -i=Olympics-RawOutputWithUNReversal.txt -ncol=1 -rcol=3</tt> | <tt>perl -i=Olympics-RawOutputWithUNReversal.txt -ncol=1 -rcol=3</tt> | ||
| − | The resultant output ([http://www.edegan.com/repository/Olympics- | + | The resultant output ([http://www.edegan.com/repository/Olympics-RawOutputWithUNReversal-Normalized.txt Olympics-RawOutputWithUNReversal-Normalized.txt]) was used to create the n-gram variables. |
| + | |||
| + | ==Internet Movie Database (IMDB)== | ||
| + | |||
| + | A list of all actors and their birth countries was extracted from the [http://www.imdb.com/interfaces#plain IMDB biographies file] using a simple one-off script ([http://www.edegan.com/repository/IMDB-ExtractNamesAndCountries.pl IMDB-ExtractNamesAndCountries.pl]). This produced the input file http://www.edegan.com/repository/IMDB-BioData.txt IMDB-BioData.txt]. Much as with the Olympics data the country names were then corrected to the UN standard name, with individuals who were born in non-recognized jurisdictions, such as on a cruise ship at sea, excluded (see [http://www.edegan.com/repository/IMDB-BiosUNCountryCodes.txt IMDB-BiosUNCountryCodes.txt]). | ||
| + | |||
| + | The NormalizeSurnames.pl script was with following options (and defaults): | ||
| + | |||
| + | <tt>perl -i=IMDB-BiosUNCountryCodes.txt -ncol=1 -comma=1</tt> | ||
| + | |||
| + | The resultant output ([http://www.edegan.com/repository/IMDB-BiosUNCountryCodes-Normalized.txt IMDB-BiosUNCountryCodes-Normalized.txt]) was used to create the n-gram variables. | ||
==World Leaders== | ==World Leaders== | ||
Revision as of 19:48, 8 July 2009
We are primarily interested in sources of surname data that contain both surnames and countries of birth for surnames.
Olympic Athletes
The Olympic Athletes data was taken from the relevant pages on wikipedia. Local copies of all of the individual country pages from the 2004 Summer Games were retrieved by a script (Olympics-RetrievePageURLs.pl) that uses an offline version (Olympics-OfflineSource.html) of the wikipedia Nations at the 2004 Summer Olympics page. This script also constructs a list of participating countries (Olympics-ParticipatingCountries.txt).
The offline pages were then parsed by another script (Olympics-ExtractOlypiads.pl) and the resulting output (Olympics-RawOutput.txt) was checked by hand. This output is the basic names set with countries for the 2004 Olympic Athletes used here. Because some individuals competed in multiple events, identical full name strings were collapsed to produce a single record with a count. It seems unlikely that many John Joe Smiths entered, making such a reduction erroneous. Users of these scripts should the wikipedia source files have likely changed and should check results carefully.
The country names were then corrected to the UN GeoRegions and coded using SQL scripts, and country with idiosyncratic name reversals were marked to produce a normalization input file (Olympics-RawOutputWithUNReversal.txt). The NormalizeSurnames.pl script was with following options (and defaults):
perl -i=Olympics-RawOutputWithUNReversal.txt -ncol=1 -rcol=3
The resultant output (Olympics-RawOutputWithUNReversal-Normalized.txt) was used to create the n-gram variables.
Internet Movie Database (IMDB)
A list of all actors and their birth countries was extracted from the IMDB biographies file using a simple one-off script (IMDB-ExtractNamesAndCountries.pl). This produced the input file http://www.edegan.com/repository/IMDB-BioData.txt IMDB-BioData.txt]. Much as with the Olympics data the country names were then corrected to the UN standard name, with individuals who were born in non-recognized jurisdictions, such as on a cruise ship at sea, excluded (see IMDB-BiosUNCountryCodes.txt).
The NormalizeSurnames.pl script was with following options (and defaults):
perl -i=IMDB-BiosUNCountryCodes.txt -ncol=1 -comma=1
The resultant output (IMDB-BiosUNCountryCodes-Normalized.txt) was used to create the n-gram variables.
