Selenium Documentation
Selenium Web Driver is a framework often used for automated web application testing. It uses an API to launch a web browser and browse sites from the client's perspective. Popular Selenium bindings exist for Python, Java, Javascript, and other languages. This documentation covers Selenium Web Driver using Python3.
| Selenium Documentation | |
|---|---|
| Project Information | |
| Has title | Selenium Documentation |
| Has owner | Peter Jalbert |
| Has start date | |
| Has deadline date | |
| Has project status | Active |
| Has sponsor | McNair Center |
| Copyright © 2019 edegan.com. All Rights Reserved. | |
Installation
A full list of installation documentation can be found here.
This documentation assumes you have Python 3.6 or later installed. If you do not, visit the Python Download page.
From the command line, enter
pip install selenium
The Basics
A folder with tutorial code can be found on the RDP in:
E:\McNair\Software\Selenium Tutorial
Launching a Driver
The first step is to launch a driver. This is an object that has information on the current page including its url and web elements, and is the object you interact with to do any sort of navigation. First, import the webdriver:
from selenium import webdriver
Then, create an instance of the web driver. The RDP has bindings for Google Chrome and Mozilla Firefox. The following will launch a web browser on Google Chrome.
driver = webdriver.Chrome()
The GET method is used to visit a website. The get() command in Selenium takes a string url.
driver.get("http://www.google.com")
From here, different methods can be used to interact with the page. Most interactions involve some type of exchange with a web element. Selenium comes with many different ways to locate specific elements. To see the attributes of the element you want to work with, it is often a good idea to visit that page on your own browser, right click on the element you want your program to interact with, and select INSPECT. This will bring up the developer console and display the HTML representation of that element. From there, you can use one of the following selectors that best matches what you need.
Selectors
These functions deal with web elements on the current page the driver is on. Any function that contains find_element_by returns a single web element, and any function that contains find_elements_by returns a list of web elements.
driver.find_element_by_class_name(class_name)
This function takes a string class name of the element you're looking for, and finds the first element on the page that has that class name. If there is a possibility that more than one web element on the page has the same class, you are probably better of using find_elements_by_class_name.
driver.find_element_by_name(name)
This function takes a string name of the element you're looking for, and finds the first element on the page that has a name attribute matching the string. Similar to the find_element_by_class_name function, this is not your best bet if there are multiple objects with the same name attribute.
driver.find_element_by_id(id)
This function takes a string id of the element you're looking for, and finds the element on the page that has an id attribute matching the string. Since ids are guaranteed to be unique, this will always find the element you're looking for. This function is not helpful if the element you want to select does not have an id attribute.
driver.find_element_by_xpath(xpath)
This function takes an XPATH, and returns the first web element that matches the path. This should not be your first choice if many elements can share the same XPATH. Contrary to all the above functions, XPATH can be used to find any web element, regardless of its attributes. However, XPATH takes some time to learn, and is more complex than all of the above. After investing some time, XPATH is the most secure way to find the elements you're looking for.
driver.find_elements_by_class_name(class_name)
This is similar to find_element_by_class_name, except it returns a list of all matches with the class name. This allows you to iterate over the results or index them accordingly. This is often useful for search results, or any sort of list based queries.
driver.find_elements_by_name(name)
Same as find_element_by_name, except it returns a list of all matches.
driver.find_elements_by_xpath(xpath)
Same as find_element_by_xpath, but returns a list of all matches.
A tutorial on XPATH can be found here.
Javascript
If you are familiar with Javascript, you can inject Javascript into the driver to envoke certain behaviors. Simply use:
driver.execute_script(someJavascriptCode)
For example, the following could be used to scroll to the bottom of the page:
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
Clicking
If an element is clickable (such as a link or a button), you can click on that element by doing:
element.click()
This will route you to the linked page, or execute the action of the button. The driver will now be on the new page, and commands will deal with elements on the new page.
Web Elements
This area contains methods that can be called on a web element.
element.get_attribute(attr)
This function gets an attribute of the web element by using the attribute identifier. For example, element.get_attribute("href") would get the URL of a button or link.
Open a New Window
OPTION 1: Cntrl + Click
This method will utilize Action Chains and Keys. Action Chains simply queue commands, and the string of commands are executed once the perform() method is called. First, import these dependencies:
from selenium.webdriver.common.action_chains import ActionChains from selenium.webdriver.common.keys import Keys
Next, select the element using one of the selectors listed above. We will assume this web element is stored in the variable called element. Then, add a key down command to Action Chains using the Shift Key to simulate the shift button being pressed down:
ActionChains(driver).key_down(Keys.SHIFT).perform()
The Shift button is now "pressed" down. A click will now be a Shift + Click.
element.click()
This will simulate a Shift + Click on a link. A new window should open with the new link. Execute a key up command with Action Chains on the Shift key so your future commands are not combined with shift:
ActionChains(driver).key_up(Keys.SHIFT).perform()
Now the browser is displaying a new window, but the driver thinks it is still on the previous window. To change this, access the window handlers:
handles = driver.window_handles
And switch the driver's "focus" to the new window (the last window in this list of windows is the newest one):
driver.switch_to_window(handles[-1])
Now you can execute commands on the new window! To exit this window, use:
driver.close()
When closing a window, you will also need to return the driver to the previous handler by repeating the code:
handles = driver.window_handles driver.switch_to_window(handles[-1])
OPTION 2: Open Blank Window + get()
This method finds URLs that you wish to traverse, and opens up a new window for each URL you would like to visit. It utlizies Javascript injection and the GET method.
PROS: This method allows you to check a URL string for correctness before visiting the page. This can save you trouble from TimeOut Errors if someone has linked a rotten link on their webpage.
This assumes you have a URL retrieved by using element.get_attribute("href") or some other method. We assume this is stored in the variable url.
First, open a new window by injecting javascript:
driver.execute_script("window.open();")
Now the browser is displaying a new window, but the driver thinks it is still on the previous window. To change this, access the window handlers:
handles = driver.window_handles
And switch the driver's "focus" to the new window (the last window in this list of windows is the newest one):
driver.switch_to_window(handles[-1])
Now, you can execute commands on the new window. To get to the webpage you want to visit:
driver.get(url)
Now you can interact with anything on the new page using the current driver. To exit this window, use:
driver.close()
When closing a window, you will also need to return the driver to the previous handler by repeating the code:
handles = driver.window_handles driver.switch_to_window(handles[-1])
Downloading Files
In general, I recommend using Selenium to do browsing, and use another method to download the file.
driver.current_url
This will get the current url. From there, standard libraries such as wget or urlretrieve can be used to download the file if the url ends in .pdf, or the html of the page if it is a regular webpage.
If you are trying to retrieve a body of text, find the element using selectors. Then:
element.text
will retrieve the text in that element as a string. This can then be written to a file in any way you see fit.
Helpful link for downloading files in python
Helpful Links
Advanced
The folder for the Web Driver Executables can be found:
C:\SeleniumDriver
chromedriver.exe is an executable to launch Google Chrome, and geckodriver.exe is an executable to launch Mozilla Firefox. Any new drivers for different web browsers should be placed in this folder.
