NHL
Old Material
Downloading Postgresql on Mac
Download package from:
http://www.enterprisedb.com/products-services-training/pgdownload#osx
Follow instructions given on the website. Macs already come with Perl, using the stackbuilder application which was also downloaded through the same link, download the PL/Perl package.
Additional Findings
The NHL as an organization cares about players birthdays as of June 30th of the season year, this determines players eligibility for contracts UFA and 35-plus. Due to this as of now age is calculated based on June 30th cut off.
RFA: Player has fewer than seven years of accrued seasons AND younger than 27 at the end of a league year using June 30th as cut off. UFA: More than 7 years of accrued seasons OR is 27 or older at the end of a league year. 35-plus: Automatically in effect when a player reaches 35 in the cut off of a certain season. Exceptions to the RFA, UFA rules occur when listed in source.
There are no loans in the NHL (trades only), so when a player has switched teams we take the last team in the string of teams as that is where he probably spent most of the season. Also contract ownership switches from team to team with a caphit somewhere between 50-100% taken on by the new team.
Is CPI necessary??
Use this syntax to dump a database:
pg_dump dbname > outfile
Restoring a database:
psql dbname < infile
Variables
List of necessary variables and where to find them in the dropbox.
For all skaters we need:
NHLIDDetails.txt (likely a file we generate) ID (int) Playername from NHL, Playername from CapGeek, Playername from GeneralFanager DOB (transform to ISO8601)
NHLHistoric_Player_summary.txt & NHLPlayer_summary.txt (historic data set includes NHL Player summary except for two games of 2013-2014 season) Playername Current Team (string) Position (F, D) season (YYYY) goals (int) TOI (float)
NHLPlayer_points.txt Playername DOB PPG (float)
NHLPlayer_bios.txt playername dob game type (overtime or no overtime) weights (int) height (int) age (int) - calculated from DOB
NHLPlayer_faceOffPercentageAll.txt playername face-off wins (int)
Capgeek_10_processed-notepad.txt playername dob salary (int) length (int) contract start date (MM/DD/YYYY) contract type (EL, RFA, UFA, TFP) caphit (int) In a separate Table: Year and CPI (2010 Base Year)
Next Tasks
Spec General Fanager!
General Fanager Webcrawler
The Perl Libraries I used to create this webcrawler are
use strict; use LWP::Simple; use HTML::Tree;
Using the LWP::Simple library makes it easy to rip the HTML off the website by simply doing,
$content = get(your url as a string here);
The URL used to access the General Fanager page containing data from all the players is http://www.generalfanager.com/players. Occasionally the function will pull a webpage without any actual content in it. I don't know why this happens and it appears to be very inconsistent.
Now the HTML::Tree library allows us to parse the HTML code into a more accessible tree structure.
$tree = HTML::Tree->new(); $tree->parse($content);
Now, with the HTML code parsed we can look down the tree to find what we are searching for.
$tree->look_down( '_tag', 'tag of what you are looking for here')
Will return an array with each element of the array containing the HTMl tree down from where the tag was found. I used the tag table because it was the most specific tag above the player stat, and put the resuls into the @tables variable. Now in order to access the data of each individual player you must look inside the @tables variable.
@{$tables[0]->{_content}[1]->{_content}}
is where I found an array containing an HTML tree for each player. However the content of the first element of this array is an empty array and the last 2 elements of this array have no content. the rest of the elements should be players. There are 2 different ways the HTML tree can be formed, one for a player without his own page, and one for a player with his own page. I created
my %playerdicit;
to store all the data from each player. For the players without their own page these are the locations I found their data
foreach my $player (@{$tables[0]->{_content}[1]->{_content}}){
$name = $player->{_content}[0]->{_content}[0];
$position = $player->{_content}[1]->{_content}[0];
$age = $player->{_content}[2]->{_content}[0];
for players with their own page, the position and age can be found at the same place but the name and link to their page are found elsewhere.
$name = $player->{_content}[0]->{_content}[0]->{_content}[0];
$link = $player->{_content}[0]->{_content}[0]->{href};
the link should be of the form /players/playerid# to which you can add http://www.generalfanager.com to get http://www.generalfanager.com/players/playerid# which is the link to that player's page. Using that link you can use the same method as described above to pull the HTML from that page and parse it into a tree structure. In order to do this I looped through all the players in playerdict, making sure to avoid any players without their own page.
foreach my $loopplayer (keys %playerdict){
if ( @{$playerdict{$loopplayer}}[2]) {
I constructed the url for the player using the following line, it should produce a structure similar to the one described above
my $playerurl = "http://www.generalfanager.com". @{$playerdict{$loopplayer}}[2];
Similarly I grabbed the data from that URL and parsed it into the variable $playertree. I found the player's team and birth date at the following locations
my $teamstring = $playertree->{_content}[1]->{_content}[4]->{_content}[1]->{_content}[0]->{_content}[1]->{_content}[0]->{_content}[0]->{href};
my $birthstring = $playertree->{_content}[1]->{_content}[4]->{_content}[1]->{_content}[0]->{_content}[1]->{_content}[1]->{_content}[0];
I then proceeded to clean up the strings using regexes. I removed unnecesarry information and spaces before and after the information like so
$teamstring =~ s/\/teams\///; $teamstring =~ s/-|^\s+|\s+$/ /g;
$birthstring=~ s/Birthdate:\s//; $birthstring=~s/^\s+|\s+$//g;
now by looking down the playertree for tables we should find each contract as a table. I placed them into the array @playertables. Due to the irregular structure of the webpage I also had to look down the tree for the "contract_source" like so
my @contract_sources = $playertree->look_down('class', 'contract_source');
I then matched up the source with the contract using an index and began to loop through the contracts
my $contidx = 0;
foreach my $contract (@playertables){
I then found the cap hit, aav, Total Value, Contract Length and Expiry Status and cleaned up the data using more regexes like so
my $caphit = $contract->{_content}[1]->{_content}[0]->{_content}[0];
$caphit=~s/Cap Hit:\s\$|,|^\s+|\s+$//g;
my $aav = $contract->{_content}[1]->{_content}[0]->{_content}[2];
$aav=~s/AAV:\s\$|,|^\s+|\s+$//g;
my $totalvalue = $contract->{_content}[1]->{_content}[0]->{_content}[4];
$totalvalue=~s/Total Value:\s\$|,|^\s+|\s+$//g;
my $contlength = $contract->{_content}[1]->{_content}[1]->{_content}[0];
$contlength=~s/Length:\s|\syears|^\s+|\s+$//g;
my $expirystatus = $contract->{_content}[1]->{_content}[1]->{_content}[4];
$expirystatus=~s/Expiry Status:\s|^\s+|\s+$//g;
Now in order to get the source, I used several conditional statements that look like below, I then cleaned up the Source using regexes
my $source;
if ((ref $contract_sources[$contidx]->{_content}[0] eq "HTML::Element") or ($contract_sources[$contidx]->{_content}[0] eq " ")) {
$contidx++;
}
if (not $contract_sources[$contidx]->{_content}[1]) {
$source = $contract_sources[$contidx]->{_content}[0];
unless (ref $source eq "") {
$source = $source->{_content}[0];
}
}
elsif ($contract_sources[$contidx]->{_content}[1]->{_content}) {
$source = $contract_sources[$contidx]->{_content}[1]->{_content}[0];
}
else {
$source = $contract_sources[$contidx]->{_content}[0];
}
$source =~ s/\s+Source:\s+|^\s+//;
$contidx++;
These conditionals ensured that I always got the correct source for each contract. Finally I got the year, Salary, and bonuses of the contract, avoided any table rows that were not useful information, and cleaned up the numbers using regexes
for (my $row = 3; $row<scalar(@{$contract->{_content}})-1; $row++){
unless ((ref $contract->{_content}[$row]->{_content}[0]->{_content}[0] eq "HTML::Element") or not (ref $contract->{_content}[$row]->{_content}[1] eq "HTML::Element")){
my $year = $contract->{_content}[$row]->{_content}[0]->{_content}[0];
$year =~ s/-\d+|^\s+|\s+$//g;
my $nhlsalary = $contract->{_content}[$row]->{_content}[1]->{_content}[0];
$nhlsalary =~ s/[^\d]//g;
my $perfbonus = $contract->{_content}[$row]->{_content}[3]->{_content}[0];
$perfbonus =~ s/[^\d]//g;
my $signbonus = $contract->{_content}[$row]->{_content}[4]->{_content}[0];
$signbonus =~ s/[^\d]//g;
Now with all that you should have all of the data that I looked for in its own variable, to do whatever you want with.
Deciding Contract Type
Simple decisions:
- We have contract type declared as Entry Level or Thirty-Five Plus -- Just use this
- If previous contract was TFP then TFP
- We have contract type declared as Standard or NULL then:
- Age based choices:
- If age >=27 & <35 and new contract then UFA
- If age > 35 and new contract then TFP
- If Age is <=21 and length is 3 then Entry Level
- If Age is >=22 <24 and length is 2 then Entry Level
- If Age is >=24 <25 and length is 1 then Entry Level
- If Age is <25 and he doesn't have the right length then RFA
- If age > 25 <= 27 then RFA
- If contract type is Standard and Exp Status is RFA then RFA
- If the player has a previous contract, use its expiry status, irrespective of whether type is Standard or NULL
