Selenium with Python: A Practical Web Automation Guide

A hands-on reference for automating browsers with Selenium 4 and Python, with runnable examples for every common task.

1. Setup

Install Selenium (Python 3.9+ recommended):

pip install selenium

Selenium 4.6+ includes Selenium Manager, which downloads the right browser driver automatically. You no longer need to install chromedriver by hand. You only need Chrome, Firefox or Edge installed.

from selenium import webdriver

driver = webdriver.Chrome()   # or webdriver.Firefox(), webdriver.Edge()
driver.get("https://example.com")
print(driver.title)
driver.quit()                 # always close the browser

Use driver.quit() (closes everything and ends the session) rather than driver.close() (closes only the current window).

2. Core Imports Cheat Sheet

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import WebDriverWait, Select
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import (
    NoSuchElementException, TimeoutException,
    StaleElementReferenceException, ElementClickInterceptedException,
)

3. Browser and Navigation Basics

driver = webdriver.Chrome()
driver.get("https://www.python.org")

print(driver.title)
print(driver.current_url)
print(len(driver.page_source))

driver.get("https://www.python.org/about/")
driver.back()
driver.forward()
driver.refresh()

driver.maximize_window()
driver.set_window_size(1280, 800)
driver.set_window_position(0, 0)
# driver.fullscreen_window()
# driver.minimize_window()

driver.quit()

4. Locating Elements

Selenium 4 uses find_element(By.X, value) and find_elements(By.X, value). The old find_element_by_* methods were removed.

Strategy	Example
By.ID	find_element(By.ID, "search")
By.NAME	find_element(By.NAME, "q")
By.CLASS_NAME	find_element(By.CLASS_NAME, "btn") (single class only)
By.TAG_NAME	find_element(By.TAG_NAME, "h1")
By.LINK_TEXT	find_element(By.LINK_TEXT, "Docs")
By.PARTIAL_LINK_TEXT	find_element(By.PARTIAL_LINK_TEXT, "Doc")
By.CSS_SELECTOR	find_element(By.CSS_SELECTOR, "form#login input[type='email']")
By.XPATH	find_element(By.XPATH, "//button[text()='Submit']")

driver.get("https://www.python.org")

search = driver.find_element(By.ID, "id-search-field")
links = driver.find_elements(By.CSS_SELECTOR, "#top ul.menu li a")   # returns a list (empty if none)
print([a.text for a in links])

# Search inside an element
menu = driver.find_element(By.CSS_SELECTOR, "ul.menu")
first_link = menu.find_element(By.TAG_NAME, "a")

Useful XPath patterns

"//input[@name='email']"                         # by attribute
"//a[contains(text(), 'Sign in')]"               # partial text
"//div[contains(@class, 'card')]"                # partial class
"//table//tr[3]/td[2]"                           # table cell
"//label[text()='Email']/following-sibling::input"  # relative to a label
"//li[@class='item'][last()]"                    # last match

Relative locators (Selenium 4)

from selenium.webdriver.support.relative_locator import locate_with

password = driver.find_element(By.ID, "password")
username = driver.find_element(locate_with(By.TAG_NAME, "input").above(password))
# also: .below() .to_left_of() .to_right_of() .near()

Tip: Prefer IDs, then stable CSS selectors. Avoid long absolute XPaths such as /html/body/div[2]/div[1]/... because they break easily.

5. Interacting with Elements

box = driver.find_element(By.NAME, "q")

box.clear()
box.send_keys("selenium python")
box.send_keys(Keys.ENTER)            # or box.submit() for forms

btn = driver.find_element(By.CSS_SELECTOR, "button.search")
btn.click()

# Reading element info
el = driver.find_element(By.TAG_NAME, "h1")
print(el.text)                        # visible text
print(el.get_attribute("class"))      # attribute value
print(el.get_attribute("href"))
print(el.value_of_css_property("color"))
print(el.tag_name, el.size, el.location)
print(el.is_displayed(), el.is_enabled(), el.is_selected())

Keyboard shortcuts:

box.send_keys(Keys.CONTROL, "a")     # select all (use Keys.COMMAND on macOS)
box.send_keys(Keys.BACKSPACE)
box.send_keys(Keys.TAB)
box.send_keys(Keys.ESCAPE)

6. Waits (the most important topic)

Pages load asynchronously. Never use time.sleep() as your main strategy.

Implicit wait

Global polling time for find_element. Set once.

driver.implicitly_wait(5)

Explicit wait (recommended)

wait = WebDriverWait(driver, 10)

element = wait.until(EC.presence_of_element_located((By.ID, "results")))
button  = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.submit")))
visible = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, "toast")))
wait.until(EC.invisibility_of_element_located((By.ID, "spinner")))
wait.until(EC.title_contains("Dashboard"))
wait.until(EC.url_contains("/home"))
wait.until(EC.text_to_be_present_in_element((By.ID, "status"), "Done"))
wait.until(EC.number_of_windows_to_be(2))
wait.until(EC.alert_is_present())

Custom condition

wait.until(lambda d: len(d.find_elements(By.CSS_SELECTOR, ".row")) >= 10)

Fluent wait

wait = WebDriverWait(
    driver, timeout=20, poll_frequency=0.5,
    ignored_exceptions=[NoSuchElementException, StaleElementReferenceException],
)

Do not mix implicit and explicit waits. It can cause unpredictable delays. Pick explicit waits.

7. Forms: Inputs, Checkboxes, Radios, Dropdowns

driver.get("https://www.selenium.dev/selenium/web/web-form.html")

driver.find_element(By.ID, "my-text-id").send_keys("Hello")
driver.find_element(By.NAME, "my-password").send_keys("secret")
driver.find_element(By.NAME, "my-textarea").send_keys("Multi\nline")

# Checkbox / radio
checkbox = driver.find_element(By.ID, "my-check-1")
if not checkbox.is_selected():
    checkbox.click()
driver.find_element(By.ID, "my-radio-2").click()

# Dropdown (<select>)
select = Select(driver.find_element(By.NAME, "my-select"))
select.select_by_visible_text("Two")
select.select_by_value("3")
select.select_by_index(1)
print(select.first_selected_option.text)
print([o.text for o in select.options])

# Multi-select extras
# select.deselect_all(); select.deselect_by_value("1")

# Submit
driver.find_element(By.CSS_SELECTOR, "button[type='submit']").click()

For custom (non-<select>) dropdowns, click the trigger, wait for the options, then click the one you want:

driver.find_element(By.CSS_SELECTOR, ".dropdown-toggle").click()
wait.until(EC.element_to_be_clickable((By.XPATH, "//ul[@class='menu']/li[text()='Option B']"))).click()

8. Alerts, Confirms and Prompts

driver.find_element(By.ID, "alert-btn").click()

alert = WebDriverWait(driver, 5).until(EC.alert_is_present())
print(alert.text)
alert.accept()          # OK
# alert.dismiss()       # Cancel
# alert.send_keys("text")  # for prompt dialogs, then accept()

9. Frames and iFrames

driver.switch_to.frame("frame-name-or-id")           # by name/id
driver.switch_to.frame(0)                            # by index
iframe = driver.find_element(By.CSS_SELECTOR, "iframe.editor")
driver.switch_to.frame(iframe)                       # by element

driver.find_element(By.ID, "inside-frame").click()

driver.switch_to.parent_frame()
driver.switch_to.default_content()                   # back to main page

Safer pattern: wait.until(EC.frame_to_be_available_and_switch_to_it((By.ID, "frame"))).

10. Multiple Windows and Tabs

original = driver.current_window_handle

driver.find_element(By.LINK_TEXT, "Open new tab").click()
WebDriverWait(driver, 10).until(EC.number_of_windows_to_be(2))

for handle in driver.window_handles:
    if handle != original:
        driver.switch_to.window(handle)
        break

print(driver.title)
driver.close()                          # closes the new tab
driver.switch_to.window(original)       # return

# Open a new tab/window programmatically (Selenium 4)
driver.switch_to.new_window("tab")      # or "window"
driver.get("https://example.com")

11. Mouse and Keyboard with ActionChains

actions = ActionChains(driver)

# Hover
menu = driver.find_element(By.ID, "menu")
actions.move_to_element(menu).perform()

# Double click, right click
actions.double_click(driver.find_element(By.ID, "dbl")).perform()
actions.context_click(driver.find_element(By.ID, "ctx")).perform()

# Drag and drop
src = driver.find_element(By.ID, "draggable")
dst = driver.find_element(By.ID, "droppable")
actions.drag_and_drop(src, dst).perform()

# Click and hold, then release
actions.click_and_hold(src).move_to_element(dst).release().perform()

# Key combos
actions.key_down(Keys.CONTROL).send_keys("c").key_up(Keys.CONTROL).perform()

# Type into a focused element, with pauses
actions.send_keys_to_element(box, "hello").pause(1).send_keys(Keys.ENTER).perform()

# Scroll to element (Selenium 4)
actions.scroll_to_element(driver.find_element(By.ID, "footer")).perform()

12. Running JavaScript and Scrolling

# Scroll
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
driver.execute_script("arguments[0].scrollIntoView({block:'center'});", element)

# Click when a normal click is intercepted
driver.execute_script("arguments[0].click();", element)

# Read values
height = driver.execute_script("return document.body.scrollHeight")

# Set a value directly
driver.execute_script("arguments[0].value = 'abc';", element)

# Infinite scroll loop
import time

last = driver.execute_script("return document.body.scrollHeight")
while True:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(1.5)   # or wait for a specific new element to appear
    new = driver.execute_script("return document.body.scrollHeight")
    if new == last:
        break
    last = new

13. Screenshots

driver.save_screenshot("page.png")                           # viewport
driver.find_element(By.ID, "chart").screenshot("chart.png") # single element
png_bytes = driver.get_screenshot_as_png()
b64 = driver.get_screenshot_as_base64()

# Print page to PDF (Chrome, headless)
import base64
pdf = driver.print_page()
with open("page.pdf", "wb") as f:
    f.write(base64.b64decode(pdf))

14. Cookies and Storage

driver.get("https://example.com")

driver.add_cookie({"name": "session", "value": "abc123", "path": "/"})
print(driver.get_cookie("session"))
print(driver.get_cookies())
driver.delete_cookie("session")
driver.delete_all_cookies()

# Save and restore cookies between runs
import json
json.dump(driver.get_cookies(), open("cookies.json", "w"))

driver.get("https://example.com")          # must be on the same domain first
for c in json.load(open("cookies.json")):
    driver.add_cookie(c)
driver.refresh()

# Local storage
driver.execute_script("localStorage.setItem('theme','dark');")
print(driver.execute_script("return localStorage.getItem('theme');"))

15. Browser Options: Headless, Proxy, User Agent, Profiles

opts = Options()
opts.add_argument("--headless=new")             # run without a window
opts.add_argument("--window-size=1920,1080")
opts.add_argument("--disable-gpu")
opts.add_argument("--no-sandbox")               # often needed in Docker/CI
opts.add_argument("--disable-dev-shm-usage")    # Docker memory fix
opts.add_argument("--user-agent=Mozilla/5.0 (X11; Linux x86_64) Chrome/124.0")
opts.add_argument("--proxy-server=http://host:8080")
opts.add_argument("--incognito")
opts.add_argument("--user-data-dir=/tmp/chrome-profile")   # persistent profile (stay logged in)
opts.add_argument("--lang=en-US")
opts.add_experimental_option("excludeSwitches", ["enable-automation"])
opts.page_load_strategy = "eager"               # 'normal' | 'eager' | 'none'

driver = webdriver.Chrome(options=opts)

Firefox:

from selenium.webdriver.firefox.options import Options as FFOptions
ff = FFOptions()
ff.add_argument("-headless")
driver = webdriver.Firefox(options=ff)

16. File Upload and Download

Upload: send the absolute path to the <input type="file">. Do not click it.

import os
file_input = driver.find_element(By.CSS_SELECTOR, "input[type='file']")
file_input.send_keys(os.path.abspath("resume.pdf"))

Download: set the download folder in Chrome preferences.

import os, time, glob

dl_dir = os.path.abspath("downloads")
os.makedirs(dl_dir, exist_ok=True)

opts = Options()
opts.add_experimental_option("prefs", {
    "download.default_directory": dl_dir,
    "download.prompt_for_download": False,
    "plugins.always_open_pdf_externally": True,
})
driver = webdriver.Chrome(options=opts)
driver.get("https://example.com/files")
driver.find_element(By.LINK_TEXT, "Download report").click()

# Wait until the download finishes (no .crdownload files left)
for _ in range(60):
    if glob.glob(os.path.join(dl_dir, "*.pdf")) and not glob.glob(os.path.join(dl_dir, "*.crdownload")):
        break
    time.sleep(1)

17. Handling Common Exceptions

try:
    el = WebDriverWait(driver, 10).until(
        EC.element_to_be_clickable((By.ID, "go"))
    )
    el.click()
except TimeoutException:
    print("Element never became clickable")
except ElementClickInterceptedException:
    # something (banner, modal) covers it: scroll or use JS
    driver.execute_script("arguments[0].click();", el)
except StaleElementReferenceException:
    # DOM re-rendered: locate the element again
    el = driver.find_element(By.ID, "go")
    el.click()
except NoSuchElementException:
    print("Locator is wrong or element is not on the page yet")

Exception	Typical cause and fix
NoSuchElementException	Wrong locator, element in an iframe, or not loaded yet. Check frames and use waits.
ElementNotInteractableException	Hidden or off-screen. Wait for visibility or scroll into view.
StaleElementReferenceException	Page re-rendered. Re-find the element.
ElementClickInterceptedException	Overlay or sticky header. Close the overlay, scroll, or use JS click.
TimeoutException	Condition never met. Verify the locator and the condition.

18. Example Project 1: Automated Login

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)

try:
    driver.get("https://the-internet.herokuapp.com/login")
    wait.until(EC.visibility_of_element_located((By.ID, "username"))).send_keys("tomsmith")
    driver.find_element(By.ID, "password").send_keys("SuperSecretPassword!")
    driver.find_element(By.CSS_SELECTOR, "button[type='submit']").click()

    flash = wait.until(EC.visibility_of_element_located((By.ID, "flash"))).text
    assert "You logged into a secure area" in flash
    print("Login OK")
finally:
    driver.quit()

Never hard-code real credentials. Read them from environment variables: os.environ["APP_PASSWORD"].

19. Example Project 2: Scraping a Table and Paginated List to CSV

import csv
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
rows_out = []

driver.get("https://books.toscrape.com/")

while True:
    wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "article.product_pod")))
    for book in driver.find_elements(By.CSS_SELECTOR, "article.product_pod"):
        title = book.find_element(By.CSS_SELECTOR, "h3 a").get_attribute("title")
        price = book.find_element(By.CSS_SELECTOR, ".price_color").text
        rows_out.append([title, price])

    try:
        next_btn = driver.find_element(By.CSS_SELECTOR, "li.next a")
    except Exception:
        break                      # no more pages
    next_btn.click()

with open("books.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.writer(f)
    writer.writerow(["title", "price"])
    writer.writerows(rows_out)

print(f"Saved {len(rows_out)} books")
driver.quit()

Before scraping any site, check its terms of service and robots.txt, and add polite delays.

20. Example Project 3: Form Filling from a Spreadsheet

import csv
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)

with open("contacts.csv", newline="", encoding="utf-8") as f:
    for row in csv.DictReader(f):          # columns: name,email,message
        driver.get("https://example.com/contact")
        wait.until(EC.visibility_of_element_located((By.NAME, "name"))).send_keys(row["name"])
        driver.find_element(By.NAME, "email").send_keys(row["email"])
        driver.find_element(By.NAME, "message").send_keys(row["message"])
        driver.find_element(By.CSS_SELECTOR, "button[type='submit']").click()
        wait.until(EC.visibility_of_element_located((By.CLASS_NAME, "success")))
        print("Submitted:", row["email"])

driver.quit()

21. Page Object Model (POM)

POM keeps locators and actions in classes so tests stay readable and easy to maintain.

# pages/base_page.py
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

class BasePage:
    def __init__(self, driver, timeout=10):
        self.driver = driver
        self.wait = WebDriverWait(driver, timeout)

    def find(self, locator):
        return self.wait.until(EC.visibility_of_element_located(locator))

    def click(self, locator):
        self.wait.until(EC.element_to_be_clickable(locator)).click()

    def type(self, locator, text):
        el = self.find(locator)
        el.clear()
        el.send_keys(text)

# pages/login_page.py
from selenium.webdriver.common.by import By
from .base_page import BasePage

class LoginPage(BasePage):
    URL = "https://the-internet.herokuapp.com/login"
    USERNAME = (By.ID, "username")
    PASSWORD = (By.ID, "password")
    SUBMIT = (By.CSS_SELECTOR, "button[type='submit']")
    FLASH = (By.ID, "flash")

    def open(self):
        self.driver.get(self.URL)
        return self

    def login(self, user, pwd):
        self.type(self.USERNAME, user)
        self.type(self.PASSWORD, pwd)
        self.click(self.SUBMIT)
        return self

    def message(self):
        return self.find(self.FLASH).text

22. Testing with pytest

pip install pytest

# conftest.py
import pytest
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

@pytest.fixture
def driver():
    opts = Options()
    opts.add_argument("--headless=new")
    opts.add_argument("--window-size=1920,1080")
    d = webdriver.Chrome(options=opts)
    yield d
    d.quit()

# test_login.py
import pytest
from pages.login_page import LoginPage

def test_valid_login(driver):
    page = LoginPage(driver).open().login("tomsmith", "SuperSecretPassword!")
    assert "You logged into a secure area" in page.message()

@pytest.mark.parametrize("user,pwd", [("bad", "bad"), ("tomsmith", "wrong")])
def test_invalid_login(driver, user, pwd):
    page = LoginPage(driver).open().login(user, pwd)
    assert "invalid" in page.message().lower()

Run with pytest -v. Add pytest-xdist and run pytest -n 4 for parallel tests.

23. Selenium 4 Extras

Chrome DevTools Protocol (Chrome and Edge)

# Emulate geolocation
driver.execute_cdp_cmd("Emulation.setGeolocationOverride",
                       {"latitude": 20.2961, "longitude": 85.8245, "accuracy": 100})

# Block images/URLs
driver.execute_cdp_cmd("Network.enable", {})
driver.execute_cdp_cmd("Network.setBlockedURLs", {"urls": ["*.png", "*.jpg", "*.gif"]})

# Emulate a mobile device
driver.execute_cdp_cmd("Emulation.setDeviceMetricsOverride",
                       {"width": 390, "height": 844, "deviceScaleFactor": 3, "mobile": True})

Capturing console logs

opts = Options()
opts.set_capability("goog:loggingPrefs", {"browser": "ALL"})
driver = webdriver.Chrome(options=opts)
driver.get("https://example.com")
for entry in driver.get_log("browser"):
    print(entry["level"], entry["message"])

Shadow DOM

host = driver.find_element(By.CSS_SELECTOR, "my-component")
root = host.shadow_root
inner = root.find_element(By.CSS_SELECTOR, "button.inner")
inner.click()

24. Remote Execution with Selenium Grid or Docker

docker run -d -p 4444:4444 --shm-size=2g selenium/standalone-chrome

opts = Options()
driver = webdriver.Remote(command_executor="http://localhost:4444/wd/hub", options=opts)
driver.get("https://example.com")
print(driver.title)
driver.quit()

25. Best Practices

1. Use explicit waits and avoid time.sleep().
2. Always clean up with try/finally or a context manager so quit() runs even on errors.
3. Prefer stable locators: ID, data-testid, or short CSS selectors.
4. Use Page Objects to keep locators in one place.
5. Keep tests independent: each test sets up its own state.
6. Run headless in CI and take a screenshot on failure.
7. Handle secrets safely with environment variables, never source code.
8. Respect websites: follow terms of service, rate-limit your requests, and prefer an official API when one exists.
9. Keep browsers up to date; Selenium Manager handles drivers, but old browsers can still break things.

A reusable context manager:

from contextlib import contextmanager

@contextmanager
def browser(headless=True):
    opts = Options()
    if headless:
        opts.add_argument("--headless=new")
    d = webdriver.Chrome(options=opts)
    try:
        yield d
    except Exception:
        d.save_screenshot("error.png")
        raise
    finally:
        d.quit()

with browser() as d:
    d.get("https://example.com")
    print(d.title)

26. Troubleshooting

Problem	Fix
SessionNotCreatedException (version mismatch)	Upgrade Selenium (pip install -U selenium) and update the browser.
Chrome crashes in Docker or CI	Add --no-sandbox, --disable-dev-shm-usage, and use --headless=new.
Element found but click does nothing	Wait for element_to_be_clickable, scroll into view, or check for overlays.
Works headed but fails headless	Set an explicit window size (--window-size=1920,1080).
Site blocks automation	Respect the site's rules. Check for an official API instead of working around protections.
Slow page loads	Use page_load_strategy = "eager" and wait only for the elements you need.

27. Where to Go Next

• Official docs: selenium.dev/documentation
• Practice sites: the-internet.herokuapp.com, books.toscrape.com, saucedemo.com
• Related tools worth comparing: Playwright for Python (built-in auto-waiting) and requests with BeautifulSoup for static pages that need no browser.