SerpScrap

SEO python scraper to extract and analyze data from major search engine serps or text content of any other url. Extract data like title, url, type, text- and richsnippet of searchresults for given keywords. detect ads, automated screenshots. It might be usefull for SEO and research tasks.

Extract these result types

• ads_main - advertisements within regular search results
• image - result from image search
• news - news teaser within regular search results
• results - standard search result
• shopping - shopping teaser within regular search results

For each result of a resultspage get

• domain
• rank
• rich snippet
• site links
• snippet
• title
• type
• url
• visible url

Also get a screenshot of each result page. You can also scrape the text content of each result url. It is also possible to save the results as CSV for future analytics. If required you can also use your own proxylist.

Ressources

See http://serpscrap.readthedocs.io/en/latest/ for documentation.

Source is available at https://github.com/ecoron/SerpScrap

Contents

• Install
  • Chrome headless is recommended
  • lxml
  • cli encoding issues
• Result Data
  • serp_type
  • Related keywords
• Configuration
  • Permissions
  • Default configuration
  • Custom configuration
  • Headers
  • Proxy file
• Docker
• Example Usage
  • Simple Example
  • Simple example using phantomjs (deprecated)
  • Simple Example - custom phantomjs path (deprecated)
  • Image search
  • Url Scrape Example
  • Command Line
  • Example as_csv()
  • Example serp results and raw text of result urls
  • Example related
• Additional Ressources
  • Documentation
  • Tests
  • TFIDF

Usage

SerpScrap in your applications

import serpscrap

keywords = ['one', 'two']
scrap = serpscrap.SerpScrap()
scrap.init(keywords=keywords)
result = scrap.scrap_serps()

More detailes in the examples [1] section of the documentation.

Supported OS

• SerpScrap should work on Linux, Windows and Mac OS with installed Python >= 3.4
• SerpScrap requieres lxml
• Doesn’t work on iOS

Changes

Notes about major changes between releases

0.11.0

• Chrome headless is now the default browser, usage of phantomJS is deprecated
• chromedriver is installed on the first run (tested on Linux and Windows. Mac OS should also work)
• behavior of scraping raw text contents from serp urls, and of course given urls, has changed
• run scraping of serp results and contents at once
• csv output format changed, now it’s tab separated and quoted

0.10.0

• support for headless chrome, adjusted default time between scrapes

0.9.0

• result types added (news, shopping, image)
• Image search is supported

0.8.0

• text processing tools removed.
• less requirements

References

SerpScrap is using Chrome headless [2] and lxml [3] to scrape serp results. For raw text contents of fetched URL’s, it is using beautifulsoup4 [4] . SerpScrap also supports PhantomJs [5] ,which is deprecated, a scriptable headless WebKit, which is installed automaticly on the first run (Linux, Windows). The scrapcore was based on GoogleScraper [6] , an outdated project, and has many changes and improvements.

[1]	http://serpscrap.readthedocs.io/en/latest/examples.html

[2]	http://chromedriver.chromium.org/

[3]	https://lxml.de/

[4]	https://www.crummy.com/software/BeautifulSoup/

[5]	https://github.com/ariya/phantomjs

[6]	https://github.com/NikolaiT/GoogleScraper