LinkedIn Crawler (Python)

McNair Project

LinkedIn Crawler (Python)
Project Information
Project Title	LinkedIn Crawler (Python)
Start Date	March 2, 2017
Deadline
Keywords	Selenium, LinkedIn, Crawler
Primary Billing
Notes
Has project status
Copyright © 2016 edegan.com. All Rights Reserved.

Overview

This page is dedicated to a new LinkedIn Crawler built using Selenium and Python. The goal of this project is to be able to crawl LinkedIn without being caught by LinkedIn's aggressive anti-scraping rules. To do this, we will use Selenium to behave like a human, and use time delays to hide bot-like tendencies.

The documentation for Selenium Web Driver can be found [here http://selenium-python.readthedocs.io/index.html].

Relevant scripts can be found in the following directory:

E:\McNair\Projects\LinkedIn Crawler

The main script for this crawler is:

crawl_linkedin.py

The old documentation said that the programs/scripts (see details below) are located on our Bonobo Git Server.

repository: Web_Crawler
branch: researcher/linkedin
directory: /linkedin

Accounts

Test Account:

email: testapplicat6@gmail.com

pass: McNair2017

Functionality

This section lists functions in the crawl_linkedin.py script that can be combined for higher functionality.

login(username, password)

This function opens the LinkedIn home page and logs in using the credentials given to the function. You will be taken to the home news feed for your account.

search(query)

This function assumes you are already logged into LinkedIn. It will type in the search bar the query that it is given, and begin the search for the given query.

To what extent are we able to reproduce the network structure in LinkedIn

Example 1: 1st degree contact- You are connected to his profile Albert Nabiullin (485 connections)

Example 2: 2nd degree contact- You are connected to someone who is connected to him Amir Kazempour Esmati (63 connections)

Example 3: 3rd degree contact- You are connected to someone who is connected to someone else who is connected to her. Linda Szabados(500+ connections)

Any profile with a distance greater than three is defined as out your network.

Summary: Individual specific network information are not accessible even for the first degree connections. Therefore, any such plans to construct a network structure based on the connection of every individuals is not feasible.

It seems that the only possible direction would be using the advanced search feature.