Personal Video Database

English => Development => PVD Python Scripts => Topic started by: afrocuban on February 08, 2025, 03:58:26 am

Title: Downloading with Selenium experiences and tips
Post by: afrocuban on February 08, 2025, 03:58:26 am

Share your tips here...

Title: Re: Downloading with Selenium experiences and tips
Post by: afrocuban on February 08, 2025, 04:07:38 am

I couldn't seem to download .htm and get "Storyline" section from the main page. Nothing helped that I tried. Even .mhtml couldn't catch it.


Then I tried it in a non-headless mode and succeeded. What was different? Scrolling! But not to the bottom of the page, but specifically to the storyline section:

Code: [Select]

                # Scroll down to the Storyline section
                storyline_section = driver.find_element(By.XPATH, "//*[@id='__next']/main/section[1]/section[1]/section[6]")
                driver.execute_script("arguments[0].scrollIntoView();", storyline_section)
                logging.info("Scrolled to the Storyline section.")


                # Adding a delay to ensure content loads
                time.sleep(3)

Adjusting the time helps depending of your connection and how old your computer is.

Now this might help to avoid downloading and parsing reference page... Yet to analyze.

But the point of the topic was to emphasize another way of getting dynamic conent of the page... Share your tips here too...