Scraping Hover Over Figure Using Python And Selenium

I am trying to scrape the data from http://fuelinsights.gasbuddy.com/Charts using Python and Selenium. The difficult part is that the data only appear when a point on the line gra

Solution 1:

You can't use the Xpath which you have mentioned above for locating the elements inside the svg tag.

Xpath which you can use to create a list of hover objects is:

//*[name()='svg']//*[name()='g'and @class='highcharts-markers']/*[name()='path']

I have written a java program for getting the text of all the tool-tip elements. You can use the logic and write a corresponding python code:

1.Get List of tooltip Elements

 List <WebElement> highChartElements= driver.findElements(By.xpath("//*[name()='svg']//*[name()='g' and @class='highcharts-markers']/*[name()='path']"));

2. Iterate through the list and use action class for moving and clicking on all the tooltip Elements

3. Get the text of the tooltip elements.

for(WebElement element:highChartElements){
        Actions action = new Actions(driver);
        action.moveToElement(element).click().perform();
        Thread.sleep(3000);
        List<WebElement> highChartToolTipTextElements= driver.findElements(By.xpath("//*[name()='svg']//*[name()='g' and @class='highcharts-tooltip']/*[name()='text']/*[name()='tspan']"));
        for(WebElement toolTipElement:highChartToolTipTextElements){
            System.out.println("The text for the elements is"+toolTipElement.getText());
        }
    }

Solution 2:

Thanks You ! 2 years later, i'm facing to an equivalent project and used your example to learn how to get the job done with python & Firefox. Perhaps the following code will be useful to some people.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains

driver = webdriver.Firefox(executable_path=r'path\to\the\geckodriver.exe')
driver.get('http://fuelinsights.gasbuddy.com/Charts')
element = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME, "highcharts-markers")))
test = driver.find_elements_by_xpath("//*[name()='svg']//*[name()='g' and @class='highcharts-markers']/*[name()='path']")

res = []
for el in test:
    hover = ActionChains(driver).move_to_element(el)
    hover.perform()
    date = driver.find_elements_by_css_selector(".highcharts-tooltip > text:nth-child(5) > tspan:nth-child(1)")
    price = driver.find_elements_by_css_selector(".highcharts-tooltip > text:nth-child(5) > tspan:nth-child(4)")
    res.append((date[0].text, price[0].text))

"res" contains :

('Saturday, May 30, 2020', '1.978 $/gal')
('Friday, May 29, 2020', '1.979 $/gal')
('Thursday, May 28, 2020', '1.977 $/gal')
('Wednesday, May 27, 2020', '1.972 $/gal')
('Tuesday, May 26, 2020', '1.965 $/gal')
.......