I've been trying to use Scrapy to recover the link of the first image of a query in Google Images.
For example, I've been trying to recover the first link for this specific query: Emma Watson - Google Images. To formulate the Xpath I've been using Xpath Helper which is a extension for Google Chrome. The First XPath I tried was the following:
.//*[@id='rg_s']/div[1]/a/@href
returned the following in the extension: Xpath Helper:
http://ift.tt/1IPmJSG
which actually is what I want. Then I'll scrap that url too and get the url for the Image. That's what I'm looking for. But for some reason I tried the Xpath on Scrapy Shell and returns []. Empty!
Tried with other XPath (to the same place):
.//div[@class='rg_di rg_el ivg-i'][1]/a[@class='rg_l']/@href
and still the same answer: []. I don't know what I'm doing wrong. Can you help me with this?
PS. What I use in Scrapy Shell is the following:
>response.xpath(".//*[@id='rg_s']/div[1]/a/@href")
# returned: []
>response.xpath(".//div[@class='rg_di rg_el ivg-i'][1]/a[@class='rg_l']/@href")
# returned: []
Something to add: When I tried to get the Title of the Page, it works.
>response.xpath(".//title/text()").extract()
# returns: [u'emma watson - Google Search']