I couldn't seem to download .htm and get "Storyline" section from the main page. Nothing helped that I tried. Even .mhtml couldn't catch it.
Then I tried it in a non-headless mode and succeeded. What was different? Scrolling! But not to the bottom of the page, but specifically to the storyline section:
# Scroll down to the Storyline section
storyline_section = driver.find_element(By.XPATH, "//*[@id='__next']/main/section[1]/section[1]/section[6]")
driver.execute_script("arguments[0].scrollIntoView();", storyline_section)
logging.info("Scrolled to the Storyline section.")
# Adding a delay to ensure content loads
time.sleep(3)
Adjusting the time helps depending of your connection and how old your computer is.
Now this might help to avoid downloading and parsing reference page... Yet to analyze.
But the point of the topic was to emphasize another way of getting dynamic conent of the page... Share your tips here too...
