NHL

Old Material

• Winner's Curse

Downloading Postgresql on Mac

Download package from:

http://www.enterprisedb.com/products-services-training/pgdownload#osx

Follow instructions given on the website. Macs already come with Perl, using the stackbuilder application which was also downloaded through the same link, download the PL/Perl package.

Additional Findings

The NHL as an organization cares about players birthdays as of June 30th of the season year, this determines players eligibility for contracts UFA and 35-plus. Due to this as of now age is calculated based on June 30th cut off.

RFA: Player has fewer than seven years of accrued seasons AND younger than 27 at the end of a league year using June 30th as cut off. UFA: More than 7 years of accrued seasons OR is 27 or older at the end of a league year. 35-plus: Automatically in effect when a player reaches 35 in the cut off of a certain season. Exceptions to the RFA, UFA rules occur when listed in source.

There are no loans in the NHL (trades only), so when a player has switched teams we take the last team in the string of teams as that is where he probably spent most of the season. Also contract ownership switches from team to team with a caphit somewhere between 50-100% taken on by the new team.

Is CPI necessary??

Use this syntax to dump a database:

pg_dump dbname > outfile

Restoring a database:

psql dbname < infile

Sources: [1] [2]

Variables

List of necessary variables and where to find them in the dropbox.

For all skaters we need:

NHLIDDetails.txt (likely a file we generate)
 ID (int) 
 Playername from NHL, Playername from CapGeek, Playername from GeneralFanager
 DOB (transform to ISO8601)

NHLHistoric_Player_summary.txt & NHLPlayer_summary.txt (historic data set includes NHL Player summary except for two games of 2013-2014 season)
 Playername
 Current Team (string)
 Position (F, D) 
 season (YYYY) 
 goals (int) 
 TOI (float)

NHLPlayer_points.txt
 Playername
 DOB
 PPG (float)

NHLPlayer_bios.txt
 playername
 dob 
 game type (overtime or no overtime)
 weights (int)
 height (int)
 age (int) - calculated from DOB

NHLPlayer_faceOffPercentageAll.txt
 playername
 face-off wins (int) 

Capgeek_10_processed-notepad.txt
 playername
 dob
 salary (int)
 length (int)
 contract start date (MM/DD/YYYY)
 contract type (EL, RFA, UFA, TFP)
 caphit (int)
 
In a separate Table:
 Year and CPI (2010 Base Year)

Next Tasks

Spec General Fanager!

General Fanager Webcrawler

The Perl Libraries I used to create this webcrawler are

use strict;
use LWP::Simple;
use HTML::Tree;

Using the LWP::Simple library makes it easy to rip the HTML off the website by simply doing,

$content = get(your url as a string here);

The URL used to access the General Fanager page containing data from all the players is http://www.generalfanager.com/players. Occasionally the function will pull a webpage without any actual content in it. I don't know why this happens and it appears to be very inconsistent.

Now the HTML::Tree library allows us to parse the HTML code into a more accessible tree structure.

$tree = HTML::Tree->new();
$tree->parse($content);

Now, with the HTML code parsed we can look down the tree to find what we are searching for.

$tree->look_down( '_tag', 'tag of what you are looking for here')

Will return an array with each element of the array containing the HTMl tree down from where the tag was found. I used the tag table because it was the most specific tag above the player stat, and put the resuls into the @tables variable. Now in order to access the data of each individual player you must look inside the @tables variable.

@{$tables[0]->{_content}[1]->{_content}}

is where I found an array containing an HTML tree for each player. However the content of the first element of this array is an empty array and the last 2 elements of this array have no content. the rest of the elements should be players. There are 2 different ways the HTML tree can be formed, one for a player without his own page, and one for a player with his own page. I created

my %playerdicit;

to store all the data from each player. For the players without their own page these are the locations I found their data

foreach my $player (@{$tables[0]->{_content}[1]->{_content}}){
    $name = $player->{_content}[0]->{_content}[0];
    $position = $player->{_content}[1]->{_content}[0];
    $age = $player->{_content}[2]->{_content}[0];

for players with their own page, the position and age can be found at the same place but the name and link to their page are found elsewhere.

    $name = $player->{_content}[0]->{_content}[0]->{_content}[0];
    $link = $player->{_content}[0]->{_content}[0]->{href};

the link should be of the form /players/playerid# to which you can add http://www.generalfanager.com to get http://www.generalfanager.com/players/playerid# which is the link to that player's page. Using that link you can use the same method as described above to pull the HTML from that page and parse it into a tree structure. In order to do this I looped through all the players in playerdict, making sure to avoid any players without their own page.

foreach my $loopplayer (keys %playerdict){
    if ( @{$playerdict{$loopplayer}}[2]) {

I constructed the url for the player using the following line, it should produce a structure similar to the one described above

        my $playerurl = "http://www.generalfanager.com". @{$playerdict{$loopplayer}}[2];

Similarly I grabbed the data from that URL and parsed it into the variable $playertree. I found the player's team and birth date at the following locations

        my $teamstring = $playertree->{_content}[1]->{_content}[4]->{_content}[1]->{_content}[0]->{_content}[1]->{_content}[0]->{_content}[0]->{href};
        my $birthstring = $playertree->{_content}[1]->{_content}[4]->{_content}[1]->{_content}[0]->{_content}[1]->{_content}[1]->{_content}[0];

I then proceeded to clean up the strings using regexes. I removed unnecesarry information and spaces before and after the information like so

        $teamstring =~ s/\/teams\///; $teamstring =~ s/-|^\s+|\s+$/ /g;
        $birthstring=~ s/Birthdate:\s//; $birthstring=~s/^\s+|\s+$//g;

now by looking down the playertree for tables we should find each contract as a table. I placed them into the array @playertables. Due to the irregular structure of the webpage I also had to look down the tree for the "contract_source" like so

        my @contract_sources = $playertree->look_down('class', 'contract_source');

I then matched up the source with the contract using an index and began to loop through the contracts

        my $contidx = 0;
        foreach my $contract (@playertables){

I then found the cap hit, aav, Total Value, Contract Length and Expiry Status and cleaned up the data using more regexes like so

            my $caphit = $contract->{_content}[1]->{_content}[0]->{_content}[0];
            $caphit=~s/Cap Hit:\s\$|,|^\s+|\s+$//g;
            my $aav = $contract->{_content}[1]->{_content}[0]->{_content}[2];
            $aav=~s/AAV:\s\$|,|^\s+|\s+$//g;
            my $totalvalue = $contract->{_content}[1]->{_content}[0]->{_content}[4];
            $totalvalue=~s/Total Value:\s\$|,|^\s+|\s+$//g;
            my $contlength = $contract->{_content}[1]->{_content}[1]->{_content}[0];
            $contlength=~s/Length:\s|\syears|^\s+|\s+$//g;
            my $expirystatus = $contract->{_content}[1]->{_content}[1]->{_content}[4];
            $expirystatus=~s/Expiry Status:\s|^\s+|\s+$//g;

Now in order to get the source, I used several conditional statements that look like below, I then cleaned up the Source using regexes

            my $source;
            if ((ref $contract_sources[$contidx]->{_content}[0] eq "HTML::Element") or ($contract_sources[$contidx]->{_content}[0] eq " ")) {
                $contidx++;
            }
            if (not $contract_sources[$contidx]->{_content}[1]) {
                $source = $contract_sources[$contidx]->{_content}[0];
                unless (ref $source eq "") {
                    $source = $source->{_content}[0];
                } 
            }
            elsif ($contract_sources[$contidx]->{_content}[1]->{_content}) {
                $source = $contract_sources[$contidx]->{_content}[1]->{_content}[0];
            }
            else {
                $source = $contract_sources[$contidx]->{_content}[0];
            }
            $source =~ s/\s+Source:\s+|^\s+//;
            $contidx++;

These conditionals ensured that I always got the correct source for each contract. Finally I got the year, Salary, and bonuses of the contract, avoided any table rows that were not useful information, and cleaned up the numbers using regexes

            for (my $row = 3; $row<scalar(@{$contract->{_content}})-1; $row++){
                unless ((ref $contract->{_content}[$row]->{_content}[0]->{_content}[0] eq "HTML::Element") or not (ref $contract->{_content}[$row]->{_content}[1] eq "HTML::Element")){
                    my $year = $contract->{_content}[$row]->{_content}[0]->{_content}[0];
                    $year =~ s/-\d+|^\s+|\s+$//g;
                    my $nhlsalary = $contract->{_content}[$row]->{_content}[1]->{_content}[0];
                    $nhlsalary =~ s/[^\d]//g;
                    my $perfbonus = $contract->{_content}[$row]->{_content}[3]->{_content}[0];
                    $perfbonus =~ s/[^\d]//g;
                    my $signbonus = $contract->{_content}[$row]->{_content}[4]->{_content}[0];
                    $signbonus =~ s/[^\d]//g;

Now with all that you should have all of the data that I looked for in its own variable, to do whatever you want with.

Deciding Contract Type

Simple decisions:

• We have contract type declared as Entry Level or Thirty-Five Plus -- Just use this
• If previous contract was TFP then TFP
• We have contract type declared as Standard or NULL then:
  • Age based choices:
  • If age >=27 & <35 and new contract then UFA
  • If age > 35 and new contract then TFP
  • If Age is <=21 and length is 3 then Entry Level
  • If Age is >=22 <24 and length is 2 then Entry Level
  • If Age is >=24 <25 and length is 1 then Entry Level
  • If Age is <25 and he doesn't have the right length then RFA
  • If age > 25 <= 27 then RFA
• If contract type is Standard and Exp Status is RFA then RFA
• If the player has a previous contract, use its expiry status, irrespective of whether type is Standard or NULL