Changes
Jump to navigation
Jump to search
Finished up documentation on how webcrawler was made
Next the useragent object is created.
my $ua = new LWP::UserAgent;
Now the parameters used to search the api are decided. Currently we are searching for the 107 bills related to entrepreneurship during the 114th congress. my $queryName = "Entrepreneurship''insert query here''"; my $congressNo = "114''insert congress num here''"; my $limit = "107''insert maximum number of bills to search here''";
Using these parameters the url can be constructed.
my $genUrl = "https://www.govtrack.us/api/v2/bill?order_by=-current_status_date&congress=". $congressNo."&q=".$queryName."&limit=".$limit;
The useragent object can now retrieve and decode the text from the url into a JSON datastring.
my $genResponse = $ua->get($genUrl);
my $genContent=$genResponse->decoded_content;
Using the JSON module the JSON string can be converted into a data structure that can be searched through for relevant data. my $JSONcontent = JSON::XS::decode_json($genResponse->decoded_content);After getting the resulting JSON Data an array of all the bills matching the parameters bill id's can now be locatedat @{$JSONcontent->{objects}}Iterating over this array more specific bill information can be found by constructing a url using the bill's ID number as such, my $billurl = "https://www. Using govtrack.us/api/v2/bill/" . $bill->{id};After finding this page the ids another page data can once again be found which contains more specific information about each billparsed from a string into a data structure. my $billresponse = $ua->get($billurl); my $billcontent = JSON::XS::decode_json($billresponse->decoded_content);From this bill specific page the tags of each bill can be used to determine whether or not the bill is relevant and should be reviewed by Mcnair Center staff. An array of these tags can be found at @{$billcontent->{terms}}The data retrieved from this search is then placed into several tab delimited text files containing sets of useful information about bills deemed relevant.Currently tags that are considered relevant:
*Commerce: ID 5914
*Business Investment and Capital: ID 5918
*Intellectual Property: ID 5927
*Small Business: ID 5935
*Advanced Technology and Technological Innovations: ID 6294
*Computers and Information Tech: ID 6300
*Small Business Administration: ID 6769