Question
How do I fix this for those of us who use Firefox browser instead of Chrome browser?
from selenium.webdriver.firefox.service import Service as FirefoxServicefrom selenium.webdriver.firefox.options import Options as FirefoxOptionsfrom selenium.webdriver.common.by import By
Note that Firefox has some different preferences compared to Chrome. For example, setting the user agent or using specific preferences like general.useragent.override is handled differently.# Set Firefox optionsfirefox_options = FirefoxOptions() firefox_options.add_argument("--headless") # Running Firefox in headless modefirefox_options.add_argument(f"--lang={language_code}") # Set languagefirefox_options.set_preference("general.useragent.override", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Firefox/91.0 Safari/537.36") # User agent
# Path to geckodrivergeckodriver_path = os.path.join(current_dir, "geckodriver.exe")# Ensure geckodriver existsif not os.path.exists(geckodriver_path): logging.error(f"Geckodriver not found at path: {geckodriver_path}")sys.exit(f"Geckodriver not found at path: {geckodriver_path}")# Initialize FirefoxDriverservice = FirefoxService(executable_path=geckodriver_path)driver = webdriver.Firefox(service=service, options=firefox_options)logging.info(f"Geckodriver started from: {geckodriver_path}")
# Import the necessary components for FirefoxSummary of Changes:
from selenium.webdriver.firefox.service import Service as FirefoxService
from selenium.webdriver.firefox.options import Options as FirefoxOptions
from selenium.webdriver.common.by import By
import os
import logging
import sys
# Set Firefox options
firefox_options = FirefoxOptions()firefox_options.add_argument("--headless") # Running Firefox in headless mode
firefox_options.add_argument(f"--lang={language_code}") # Set language
firefox_options.set_preference("general.useragent.override", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Firefox/91.0 Safari/537.36") # User agent
# Path to geckodriver
geckodriver_path = os.path.join(current_dir, "geckodriver.exe")
# Ensure geckodriver exists
if not os.path.exists(geckodriver_path):
logging.error(f"Geckodriver not found at path: {geckodriver_path}")
sys.exit(f"Geckodriver not found at path: {geckodriver_path}")
# Initialize FirefoxDriver
service = FirefoxService(executable_path=geckodriver_path)
driver = webdriver.Firefox(service=service, options=firefox_options)
logging.info(f"Geckodriver started from: {geckodriver_path}")
Thanks for the comprehensive explanation for Firefox browsers.
Thank you very much.
I never tried it, so I am not sure at all how Firefox would download pages (clicking "See more" pages, "Storyline" sections and other), and if final html code would be the same as downloaded with Chrome, so it might be frustrating to realize that there are differences actually in scraped hmtls with either.
I am sorry to hear that. If you could be a bit more specific maybe I'd get an idea what it might be.
1. Did you rename any files, in this case did you rename Selenium_Chrome_Base_page_v4.py? If so, rename everywhere in the .psf too.
2. What says in the correspondent base.log file in \Tmp folder?
2025-03-15 10:08:08,399 - DEBUG - Starting the Python script.
2025-03-15 10:08:08,403 - DEBUG - Starting new HTTP connection (1): ipinfo.io:80
2025-03-15 10:08:08,641 - DEBUG - http://ipinfo.io:80 "GET /country HTTP/1.1" 200 27
2025-03-15 10:08:08,642 - DEBUG - Country code: SI, Language code: sl
Ok, that's better. Let's move debugging to the other topic, you may call it Firefox Selenium?
When the driver is called in python script, is it called as chrome, or gecko? It would be good to share you script so we could visually see how you adjusted it.
No, I didn't rename Selenium_Chrome_Base_page_v4.py or the other python files, because it would be too time-consuming to do it everywhere in the .psf files. Everything I did in all the python files, I only changed the settings on firefox and geckodriver.
That's all that was written in python_script_base_page.txt.Quote2025-03-15 10:08:08,399 - DEBUG - Starting the Python script.
2025-03-15 10:08:08,403 - DEBUG - Starting new HTTP connection (1): ipinfo.io:80
2025-03-15 10:08:08,641 - DEBUG - http://ipinfo.io:80 (http://ipinfo.io:80) "GET /country HTTP/1.1" 200 27
2025-03-15 10:08:08,642 - DEBUG - Country code: SI, Language code: sl
pyhton FullPathToTheScript titleIMDb "10 Things...." (with the double qoutes, or single quotes, it depends on your setting, try them both)
pyhton FullPathToTheScript "MovieURL" "FullPathToThe\downpage-UTF8_NO_BOM.htm" (with the double qoutes, or single quotes, it depends on your setting, try them both)
Interesting, I also did a test with Chromedriver (without using the Chrome browser), and it works great there.
Here are the Scripts adapted for firefox so that you can see my adaptation and maybe errors or I did something wrong or did not change anything.
firefox_options_options.add_argument("--headless") # Running Firefox in headless mode
firefox_options_options.add_argument(f"--lang={language_code}")
firefox_options_options.add_argument("--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36")
firefox_options_options.add_argument("--enable-unsafe-swiftshader") # Add this flag to use unsafe SwiftShader
firefox_options.add_argument("--headless") # Running Firefox in headless mode
firefox_options.add_argument(f"--lang={language_code}")
firefox_options.add_argument("--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36")
firefox_options.add_argument("--enable-unsafe-swiftshader") # Add this flag to use unsafe SwiftShader
I must have made a mistake, I just didn't notice it (I still have a lot of things to sort out for my mother's passing, so some details are missing and I don't notice them). I'll fix that and see if it works.