83
edits
Changes
Jump to navigation
Jump to search
The crawler performs searches using google and collects the title and url for results. Searches are in the format of "incubator" + "city, state".==Implementation==
no edit summary
|Depends upon it=Ecosystem Organization Classifier, Incubator Seed Data
}}
==Background==
We wanted to create a google web crawler that could collect data from web searches specific to individual cities. The searches could be in the format of "incubator" + "city, state". It was modeled off of previous researcher's web crawler which collected information on accelerators. We could not simply modify their web crawler as it used an outdated python module.
The crawler opens the text file containing a list of locations in the format "city, state", with each entry separated by a newline. It appends the google search query domain "https://www.google.com/search?q=" to the front of the key term "incubator" and appropriately attaches the city and state name, using google escape characters for commas and spaces. Then, using beautifulsoup, the script opens each of the generated urls and parses the resulting page to collect the titles and urls of the results.
The titles and urls are stored in a csv file in the following format
