dashwood.net -

Ryan Stefan's Micro Blog

Amazon as Keyword Research

Dec 182018

Amazon's qid in the url is the number of seconds since January 1st, 1979. They use it to detect whether a link is freshly searched thus signaling that that keyword is searched, or if its an old qid then that keyword was not used and someone just visited the link (I probably explained that like crap, but I've been coding for like 20 hours straight and my brain hurts). Here's the code:

import datetime

round((datetime.datetime.today()-datetime.datetime(1970,1,1)).total_seconds())

scrapy export to json:

scrapy crawl amazonproducts -o data3.json