Author Topic: Firefox Selenium Script Discussion  (Read 308 times)

0 Members and 1 Guest are viewing this topic.

Offline Ivek23

  • Global Moderator
  • *****
  • Posts: 2830
    • View Profile
Firefox Selenium Script Discussion
« on: March 14, 2025, 08:44:35 am »
Question

How do I fix this for those of us who use Firefox browser instead of Chrome browser?
« Last Edit: March 16, 2025, 08:31:02 pm by afrocuban »
Ivek23
Win 10 64bit (32bit)   PVD v0.9.9.21, PVD v1.0.2.7, PVD v1.0.2.7 + MOD


Offline afrocuban

  • Moderator
  • *****
  • Posts: 623
    • View Profile
Re: Firefox Selenimu Script Discussion
« Reply #1 on: March 14, 2025, 11:34:39 am »
Question

How do I fix this for those of us who use Firefox browser instead of Chrome browser?



To adapt your existing Chrome Selenium script to use Firefox with geckodriver, you'll need to make a few key changes. Here's a detailed explanation of the necessary modifications:
 1. Change WebDriver to Firefox Selenium has a Firefox WebDriver (webdriver.Firefox()), similar to how you're using the Chrome WebDriver (webdriver.Chrome()).
 2. Install and Use Geckodriver You need to install geckodriver (the WebDriver for Firefox) and make sure it is available in your system’s PATH or specify its path explicitly.
 3. Modify Chrome-Specific Options to Firefox-Specific Options Some Chrome-specific options (like chrome_options) need to be replaced with their Firefox counterparts. The FirefoxOptions object is used to set browser-specific configurations.
 4. Remove Chrome-specific arguments and replace them with Firefox-specific ones For Firefox, you would use FirefoxOptions and its methods instead of ChromeOptions.
 

Step-by-Step Adaptation
  • Install Firefox and Geckodriver
     
    • Firefox: If you don’t have Firefox installed already, you can install it from the official website.
    • Geckodriver: You can download it from the Geckodriver GitHub releases page. Make sure to download the version that matches your operating system and place it in a directory that's included in your PATH or specify the path in the script.
  • Modify Imports You need to import the Firefox-specific classes instead of Chrome.
    Quote
    from selenium.webdriver.firefox.service import Service as FirefoxService
    from selenium.webdriver.firefox.options import Options as FirefoxOptions
    from selenium.webdriver.common.by import By 
  • Set Firefox Options Replace the chrome_options with firefox_options. Also, you will replace Chrome-specific arguments with their Firefox equivalents.
    Quote
    # Set Firefox options
    firefox_options = FirefoxOptions() firefox_options.add_argument("--headless")  # Running Firefox in headless mode
    firefox_options.add_argument(f"--lang={language_code}")  # Set language
    firefox_options.set_preference("general.useragent.override", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Firefox/91.0 Safari/537.36")  # User agent 
    Note that Firefox has some different preferences compared to Chrome. For example, setting the user agent or using specific preferences like general.useragent.override is handled differently.
  • Replace Chrome WebDriver with Firefox WebDriver When initializing the WebDriver, use webdriver.Firefox instead of webdriver.Chrome.
    Quote
    # Path to geckodriver
    geckodriver_path = os.path.join(current_dir, "geckodriver.exe")
    # Ensure geckodriver exists
    if not os.path.exists(geckodriver_path): logging.error(f"Geckodriver not found at path: {geckodriver_path}")
    sys.exit(f"Geckodriver not found at path: {geckodriver_path}")
    # Initialize FirefoxDriver
    service = FirefoxService(executable_path=geckodriver_path)
    driver = webdriver.Firefox(service=service, options=firefox_options)
    logging.info(f"Geckodriver started from: {geckodriver_path}") 
  • Make Other Necessary Changes for Firefox You don't need to make major changes for general functionality, but if you're handling Firefox-specific settings (like cookies or scrolling behavior), you may need to adjust those based on Firefox’s behavior.


Full Example of Key Changes:Here’s an example of how you can modify the initialization and WebDriver setup for Firefox:
Quote
# Import the necessary components for Firefox
from selenium.webdriver.firefox.service import Service as FirefoxService
from selenium.webdriver.firefox.options import Options as FirefoxOptions
from selenium.webdriver.common.by import By
import os
import logging
import sys



# Set Firefox options
firefox_options = FirefoxOptions()firefox_options.add_argument("--headless")  # Running Firefox in headless mode
firefox_options.add_argument(f"--lang={language_code}") # Set language
firefox_options.set_preference("general.useragent.override", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Firefox/91.0 Safari/537.36")  # User agent

# Path to geckodriver
geckodriver_path = os.path.join(current_dir, "geckodriver.exe")

# Ensure geckodriver exists
if not os.path.exists(geckodriver_path):
    logging.error(f"Geckodriver not found at path: {geckodriver_path}")
    sys.exit(f"Geckodriver not found at path: {geckodriver_path}")

# Initialize FirefoxDriver
service = FirefoxService(executable_path=geckodriver_path)
driver = webdriver.Firefox(service=service, options=firefox_options)
logging.info(f"Geckodriver started from: {geckodriver_path}")
Summary of Changes:
  • Import the FirefoxService and FirefoxOptions.
  • Set Firefox-specific options (firefox_options), such as headless mode and language settings.
  • Initialize the webdriver.Firefox instead of webdriver.Chrome with the appropriate service and options.
  • Ensure the path to geckodriver is correct.


That should be all you need to switch from Chrome to Firefox using Selenium and Geckodriver! The rest of your script (e.g., interacting with elements, taking screenshots, saving the page) should work with minimal modification because Selenium provides a consistent API for interacting with different browsers.
« Last Edit: March 15, 2025, 07:27:06 pm by afrocuban »

Offline Ivek23

  • Global Moderator
  • *****
  • Posts: 2830
    • View Profile
Re: Firefox Selenimu Script Discussion
« Reply #2 on: March 14, 2025, 11:56:15 am »
Thanks for the comprehensive explanation for Firefox browsers.

Thank you very much.
Ivek23
Win 10 64bit (32bit)   PVD v0.9.9.21, PVD v1.0.2.7, PVD v1.0.2.7 + MOD


Offline afrocuban

  • Moderator
  • *****
  • Posts: 623
    • View Profile
Re: Firefox Selenimu Script Discussion
« Reply #3 on: March 14, 2025, 12:04:39 pm »
Thanks for the comprehensive explanation for Firefox browsers.

Thank you very much.

You are more than welcome, Ivek. I never tried it, so I am not sure at all how Firefox would download pages (clicking "See more" pages, "Storyline" sections and other), and if final html code would be the same as downloaded with Chrome, so it might be frustrating to realize that there are differences actually in scraped hmtls with either.

P.S. In people script, I brought back career option to base function too, so just make sure the proper switch (ShouldParseCareer) is set not to parse it with bio function.

Offline Ivek23

  • Global Moderator
  • *****
  • Posts: 2830
    • View Profile
Re: Firefox Selenimu Script Discussion
« Reply #4 on: March 14, 2025, 05:07:16 pm »
I never tried it, so I am not sure at all how Firefox would download pages (clicking "See more" pages, "Storyline" sections and other), and if final html code would be the same as downloaded with Chrome, so it might be frustrating to realize that there are differences actually in scraped hmtls with either.

Notice

Does not work on Win 10 and with Firefox browser and geckodriver.
Ivek23
Win 10 64bit (32bit)   PVD v0.9.9.21, PVD v1.0.2.7, PVD v1.0.2.7 + MOD


Offline Ivek23

  • Global Moderator
  • *****
  • Posts: 2830
    • View Profile
Re: Firefox Selenimu Script Discussion
« Reply #5 on: March 15, 2025, 04:38:24 am »
I am sorry to hear that. If you could be a bit more specific maybe I'd get an idea what it might be.

This, as seen in the picture.
Ivek23
Win 10 64bit (32bit)   PVD v0.9.9.21, PVD v1.0.2.7, PVD v1.0.2.7 + MOD


Offline afrocuban

  • Moderator
  • *****
  • Posts: 623
    • View Profile
Re: Firefox Selenimu Script Discussion
« Reply #6 on: March 15, 2025, 08:31:48 am »
Ok, that's better. Let's move debugging to the other topic, you may call it Firefox Selenium?  Several ideas:
1. Did you rename any files, in this case did you rename Selenium_Chrome_Base_page_v4.py? If so, rename everywhere in the .psf too.
2. What says in the correspondent base.log file in \Tmp folder?

Offline Ivek23

  • Global Moderator
  • *****
  • Posts: 2830
    • View Profile
Re: Firefox Selenimu Script Discussion
« Reply #7 on: March 15, 2025, 10:11:16 am »
1. Did you rename any files, in this case did you rename Selenium_Chrome_Base_page_v4.py? If so, rename everywhere in the .psf too.

No, I didn't rename Selenium_Chrome_Base_page_v4.py or the other python files, because it would be too time-consuming to do it everywhere in the .psf files. Everything I did in all the python files, I only changed the settings on firefox and geckodriver.

2. What says in the correspondent base.log file in \Tmp folder?

That's all that was written in python_script_base_page.txt.
Quote
2025-03-15 10:08:08,399 - DEBUG - Starting the Python script.
2025-03-15 10:08:08,403 - DEBUG - Starting new HTTP connection (1): ipinfo.io:80
2025-03-15 10:08:08,641 - DEBUG - http://ipinfo.io:80 "GET /country HTTP/1.1" 200 27
2025-03-15 10:08:08,642 - DEBUG - Country code: SI, Language code: sl


Ok, that's better. Let's move debugging to the other topic, you may call it Firefox Selenium?

Ok.
Ivek23
Win 10 64bit (32bit)   PVD v0.9.9.21, PVD v1.0.2.7, PVD v1.0.2.7 + MOD


Offline afrocuban

  • Moderator
  • *****
  • Posts: 623
    • View Profile
Re: Firefox Selenimu Script Discussion
« Reply #8 on: March 15, 2025, 07:37:21 pm »

No, I didn't rename Selenium_Chrome_Base_page_v4.py or the other python files, because it would be too time-consuming to do it everywhere in the .psf files. Everything I did in all the python files, I only changed the settings on firefox and geckodriver.
When the driver is called in python script, is it called as chrome, or gecko? It would be good to share you script so we could visually see how you adjusted it.

Quote

That's all that was written in python_script_base_page.txt.
Quote
2025-03-15 10:08:08,399 - DEBUG - Starting the Python script.
2025-03-15 10:08:08,403 - DEBUG - Starting new HTTP connection (1): ipinfo.io:80
2025-03-15 10:08:08,641 - DEBUG - http://ipinfo.io:80 "GET /country HTTP/1.1" 200 27
2025-03-15 10:08:08,642 - DEBUG - Country code: SI, Language code: sl


This why I suspect driver isn't called at all... Try to test the script from the cmd and you will get more informative response. For the title search:

Quote
pyhton FullPathToTheScript titleIMDb "10 Things...." (with the double qoutes, or single quotes, it depends on your setting, try them both)

for the main page:

Quote
pyhton FullPathToTheScript "MovieURL" "FullPathToThe\downpage-UTF8_NO_BOM.htm" (with the double qoutes, or single quotes, it depends on your setting, try them both)
« Last Edit: March 15, 2025, 07:45:29 pm by afrocuban »

Offline Ivek23

  • Global Moderator
  • *****
  • Posts: 2830
    • View Profile
Re: Firefox Selenimu Script Discussion
« Reply #9 on: March 15, 2025, 08:44:54 pm »
Here are the Scripts adapted for firefox so that you can see my adaptation and maybe errors or I did something wrong or did not change anything.

Scripts adapted for firefox are attached.

Interesting, I also did a test with Chromedriver (without using the Chrome browser), and it works great there.
« Last Edit: March 15, 2025, 08:50:57 pm by Ivek23 »
Ivek23
Win 10 64bit (32bit)   PVD v0.9.9.21, PVD v1.0.2.7, PVD v1.0.2.7 + MOD


Offline afrocuban

  • Moderator
  • *****
  • Posts: 623
    • View Profile
Re: Firefox Selenimu Script Discussion
« Reply #10 on: March 15, 2025, 10:57:33 pm »

Interesting, I also did a test with Chromedriver (without using the Chrome browser), and it works great there.


I am so happy to hear it works!

Here are the Scripts adapted for firefox so that you can see my adaptation and maybe errors or I did something wrong or did not change anything.


The script looks mostly fine, but there is an issue in the following lines:

Quote
firefox_options_options.add_argument("--headless")  # Running Firefox in headless mode
firefox_options_options.add_argument(f"--lang={language_code}")
firefox_options_options.add_argument("--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36")
firefox_options_options.add_argument("--enable-unsafe-swiftshader") # Add this flag to use unsafe SwiftShader

Here, firefox_options_options is used, but the variable should be firefox_options. The correct lines should be:

Quote
firefox_options.add_argument("--headless")  # Running Firefox in headless mode
firefox_options.add_argument(f"--lang={language_code}")
firefox_options.add_argument("--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36")
firefox_options.add_argument("--enable-unsafe-swiftshader") # Add this flag to use unsafe SwiftShader

Once this change is made, the script should work as expected. Other than that, the syntax looks correct! Just make sure you have the necessary dependencies installed and the correct path to geckodriver.
« Last Edit: March 15, 2025, 11:02:35 pm by afrocuban »

Offline Ivek23

  • Global Moderator
  • *****
  • Posts: 2830
    • View Profile
Re: Firefox Selenimu Script Discussion
« Reply #11 on: March 16, 2025, 07:47:18 am »
I must have made a mistake, I just didn't notice it (I still have a lot of things to sort out for my mother's passing, so some details are missing and I don't notice them). I'll fix that and see if it works.
Ivek23
Win 10 64bit (32bit)   PVD v0.9.9.21, PVD v1.0.2.7, PVD v1.0.2.7 + MOD


Offline Ivek23

  • Global Moderator
  • *****
  • Posts: 2830
    • View Profile
Re: Firefox Selenimu Script Discussion
« Reply #12 on: March 16, 2025, 08:09:23 am »
One more thing I should mention is that my NOD32 antivirus program quarantines SeleniumPVDbScriptsConfig-v4.exe from the _Source_Code folder as well as from the Scripts folder due to Suspicious Object. So I have to bring them back from quarantine and specify that NOD32 skips them when scanning again. Once I do that, then the Suspicious Object cause is gone.
Ivek23
Win 10 64bit (32bit)   PVD v0.9.9.21, PVD v1.0.2.7, PVD v1.0.2.7 + MOD


Offline Ivek23

  • Global Moderator
  • *****
  • Posts: 2830
    • View Profile
Re: Firefox Selenimu Script Discussion
« Reply #13 on: March 16, 2025, 10:00:47 am »
I must have made a mistake, I just didn't notice it (I still have a lot of things to sort out for my mother's passing, so some details are missing and I don't notice them). I'll fix that and see if it works.

I fixed it now and it works perfectly.
Ivek23
Win 10 64bit (32bit)   PVD v0.9.9.21, PVD v1.0.2.7, PVD v1.0.2.7 + MOD