Python-2.7安装Scrapy 1.0爬虫实例

Scrapy，Python开发的一个快速,高层次的屏幕抓取和web抓取框架，用于抓取web站点并从页面中提取结构化的数据。Scrapy用途广泛，可以用于数据挖掘、监测和自动化测试。
Scrapy吸引人的地方在于它是一个框架，任何人都可以根据需求方便的修改。它也提供了多种类型爬虫的基类，如BaseSpider、sitemap爬虫等，最新版本又提供了web2.0爬虫的支持。

使用python2.7.11


解压
#tar -xvf Python-2.7.11.tgz

cd Python-2.7.11


【
多版本python存在才进行修改

报错：
exceptions.ImportError: No module named _sqlite3

下载：

tar -zxvf pysqlite-2.8.1.tar.gz
cd pysqlite-2.8.1
python setup.py install


先修改Python-2.7.11目录里的setup.py 文件：
在下面这段的下一行添加’/usr/local/lib/sqlite3/include’,
sqlite_inc_paths = [ ‘/usr/include’,
                     ‘/usr/include/sqlite’,
                     ‘/usr/include/sqlite3’,
                     ‘/usr/local/include’,
                     ‘/usr/local/include/sqlite’,
                     ‘/usr/local/include/sqlite3’,
                     ‘/usr/local/lib/sqlite3/include’,
】

安装
#./configure  
#make all
#make install  
#make clean  
#make distclean

查看版本信息
#/usr/local/bin/python2.7 -V

建立软连接，使系统默认的 python指向 python2.7
#mv /usr/bin/python /usr/bin/python2.6.6
#ln -s /usr/local/bin/python2.7 /usr/bin/python

7.重新检验Python 版本
#python -V

解决系统 Python 软链接指向 Python2.7 版本后，因为yum是不兼容 Python 2.7的，所以yum不能正常工作，我们需要指定 yum 的Python版本

#vi /usr/bin/yum  

将文件头部的
#!/usr/bin/python

改成
#!/usr/bin/python2.6.6


下载pip



python get-pip.py

[root@testserver4 ~]# python get-pip.py
Collecting pip
  Downloading pip-7.1.2-py2.py3-none-any.whl (1.1MB)
    100% |████████████████████████████████| 1.1MB 350kB/s
Collecting wheel
  Downloading wheel-0.26.0-py2.py3-none-any.whl (63kB)
    100% |████████████████████████████████| 65kB 5.0MB/s
Collecting argparse (from wheel)
  Downloading argparse-1.4.0-py2.py3-none-any.whl
Installing collected packages: pip, argparse, wheel
Successfully installed argparse-1.4.0 pip-7.1.2 wheel-0.26.0
/tmp/tmpcBdh5G/pip.zip/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90:
InsecurePlatformWarning: A true SSLContext object is not available.
This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail.
For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.

报错：
出现：zipimport.ZipImportError: can't decompress data; zlib not available错误
解决办法重新编译一下Python源码安装包，如下：
tar zxvf Python-2.7.11.tgz
cd Python-2.7.11
./configure
vi Modules/Setup
在这里把454行左右的 找到
#zlib zlibmodule.c -I$(prefix)/include -L$(exec_prefix)/lib -lz
去掉注释
zlib zlibmodule.c -I$(prefix)/include -L$(exec_prefix)/lib -lz
make
make install


报错：ImportError:cannot import name HTTPSHandler
解决：
yum install -y openssl openssl-devel
然后重新编译python


pip install urllib3

使用pip安装:
pip install Scrapy

提示成功
Successfully installed Scrapy-1.0.3 characteristic-14.3.0 lxml-3.5.0 pyasn1-modules-0.0.8 service-identity-14.0.0

卸载软件包
pip uninstall Scrapy

列出软件包清单
pip list

更新pip
pip install -U pip

报错
/usr/lib/python2.6/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90:
InsecurePlatformWarning: A true SSLContext object is not available.
This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail.
For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.

首先安装python-devel libffi-devel openssl-devel
yum install python-devel libffi-devel openssl-devel

之后在安装pyopenssl ndg-httpsclient pyasn1
pip install pyopenssl ndg-httpsclient pyasn1

报错
Collecting Twisted>=10.0.0 (from Scrapy)
  Could not find a version that satisfies the requirement Twisted>=10.0.0 (from Scrapy) (from versions: )
  Some externally hosted files were ignored as access to them may be unreliable (use --allow-external Twisted to allow).
No matching distribution found for Twisted>=10.0.0 (from Scrapy)

安装Twisted
wget

安装Twisted
下载好Twisted后，进入到下载目录，解压：
[root@codebreaker ~]#tar -jvxf Twisted-15.5.0.tar.bz2
解压完成后进入相应目录：
[root@codebreaker ~]#cd Twisted-15.5.0
执行安装：
[root@codebreaker Twisted-15.5.0]#python setup.py install
安装完成后进入python，测试Twisted是否安装成功
[root@codebreaker Twisted-15.5.0]# python
>>> import twisted
如果没有错误发生，说明Twisted已经安装成功了


报错：
error: command 'gcc' failed with exit status 1

yum install python python-dev* python-lxml* libxml2-dev* libxslt-dev*


建立项目：
scrapy startproject itzhaopin

路径 /root/Twisted-15.5.0/itzhaopin/itzhaopin
.
├── itzhaopin
│   ├── itzhaopin
│   │   ├── __init__.py
│   │   ├── items.py
│   │   ├── pipelines.py
│   │   ├── settings.py
│   │   └── spiders
│   │      └── __init__.py
│   └── scrapy.cfg
scrapy.cfg: 项目配置文件
items.py: 需要提取的数据结构定义文件
pipelines.py:管道定义，用来对items里面提取的数据做进一步处理，如保存等
settings.py: 爬虫配置文件
spiders: 放置spider的目录


例子代码（以下配置内容以源码配置为准）：



定义Item

在items.py里面定义我们要抓取的数据：

from scrapy.item import Item, Field

class TencentItem(Item):
    name = Field()
    catalog = Field()
    workLocation = Field()
    recruitNumber = Field()
    detailLink = Field()
    publishTime = Field()


解释：
name = Field()                # 职位名称  
catalog = Field()             # 职位类别  
workLocation = Field()        # 工作地点  
recruitNumber = Field()       # 招聘人数  
detailLink = Field()          # 职位详情页链接  
publishTime = Field()         # 发布时间  


实现Spider
Spider是一个继承自scrapy.contrib.spiders.CrawlSpider的Python类，有三个必需的定义的成员
name: 名字，这个spider的标识
start_urls:一个url列表，spider从这些网页开始抓取
parse():一个方法，当start_urls里面的网页抓取下来之后需要调用这个方法解析网页内容，同时需要返回下一个需要抓取的网页，或者返回items列表
所以在spiders目录下新建一个vi spider/tencent_spider.py:

import re
import json


from scrapy.selector import Selector
try:
    from scrapy.spider import Spider
except:
    from scrapy.spider import BaseSpider as Spider
from scrapy.utils.response import get_base_url
from scrapy.utils.url import urljoin_rfc
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor as sle


from itzhaopin.items import *
from itzhaopin.misc.log import *


class TencentSpider(CrawlSpider):
    name = "tencent"
    allowed_domains = ["tencent.com"]
    start_urls = [
        ""
    ]
    rules = [ # 定义爬取URL的规则
        Rule(sle(allow=("/position.php\?&start=\d{,4}#a")), follow=True, callback='parse_item')
    ]

    def parse_item(self, response): # 提取数据到Items里面，主要用到XPath和CSS选择器提取网页数据
        items = []
        sel = Selector(response)
        base_url = get_base_url(response)
        sites_even = sel.css('table.tablelist tr.even')
        for site in sites_even:
            item = TencentItem()
            item['name'] = site.css('.l.square a').xpath('text()').extract()
            relative_url = site.css('.l.square a').xpath('@href').extract()[0]
            item['detailLink'] = urljoin_rfc(base_url, relative_url)
            item['catalog'] = site.css('tr > td:nth-child(2)::text').extract()
            item['workLocation'] = site.css('tr > td:nth-child(4)::text').extract()
            item['recruitNumber'] = site.css('tr > td:nth-child(3)::text').extract()
            item['publishTime'] = site.css('tr > td:nth-child(5)::text').extract()
            items.append(item)
            #print repr(item).decode("unicode-escape") + '\n'

        sites_odd = sel.css('table.tablelist tr.odd')
        for site in sites_odd:
            item = TencentItem()
            item['name'] = site.css('.l.square a').xpath('text()').extract()
            relative_url = site.css('.l.square a').xpath('@href').extract()[0]
            item['detailLink'] = urljoin_rfc(base_url, relative_url)
            item['catalog'] = site.css('tr > td:nth-child(2)::text').extract()
            item['workLocation'] = site.css('tr > td:nth-child(4)::text').extract()
            item['recruitNumber'] = site.css('tr > td:nth-child(3)::text').extract()
            item['publishTime'] = site.css('tr > td:nth-child(5)::text').extract()
            items.append(item)
            #print repr(item).decode("unicode-escape") + '\n'

        info('parsed ' + str(response))
        return items


    def _process_request(self, request):
        info('process ' + str(request))
        return request


实现PipeLine

PipeLine用来对Spider返回的Item列表进行保存操作，可以写入到文件、或者数据库等。
PipeLine只有一个需要实现的方法：process_item，例如我们将Item保存到JSON格式文件中：

vi pipelines.py

from scrapy import signals
import json
import codecs

class JsonWithEncodingTencentPipeline(object):

    def __init__(self):
        self.file = codecs.open('tencent.json', 'w', encoding='utf-8')

    def process_item(self, item, spider):
        line = json.dumps(dict(item), ensure_ascii=False) + "\n"
        self.file.write(line)
        return item

    def spider_closed(self, spider):
        self.file.close(
)


设置：
vi settings.py

# Scrapy settings for itzhaopin project
#
# For simplicity, this file contains only the most important settings by
# default. All the other settings are documented here:
#
#    
#

BOT_NAME = 'itzhaopin'

SPIDER_MODULES = ['itzhaopin.spiders']
NEWSPIDER_MODULE = 'itzhaopin.spiders'

# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'itzhaopin (+)'

ITEM_PIPELINES = {
    'itzhaopin.pipelines.JsonWithEncodingTencentPipeline': 300,
}

LOG_LEVEL = 'INFO'



创建目录misc
#vi log.py
from scrapy import log

def warn(msg):
    log.msg(str(msg), level=log.WARNING)

def info(msg):
    log.msg(str(msg), level=log.INFO)

def debug(msg):
    log.msg(str(msg), level=log.DEBUG)

    
#vi __init__.py (必须要这个文件,内容为空)


到现在，我们就完成了一个基本的爬虫的实现，可以输入下面的命令来启动这个Spider:

#scrapy crawl tencent

爬虫运行结束后，在当前目录下将会生成一个名为tencent.json的文件，其中以JSON格式保存了职位招聘信息。

部分内容如下：
{"recruitNumber": ["1"], "name": ["SD5-资深手游策划（深圳）"], "detailLink": "", "publishTime": ["2014-04-25"], "catalog": ["产品/项目类"], "workLocation": ["深圳"]}
{"recruitNumber": ["1"], "name": ["TEG13-后台开发工程师（深圳）"], "detailLink": "", "publishTime": ["2014-04-25"], "catalog": ["技术类"], "workLocation": ["深圳"]}


参考：

http://blog.csdn.net/HanTangSongMing/article/details/24454453
http://blog.siliconstraits.vn/building-web-crawler-scrapy/
http://blog.csdn.net/olanlanxiari/article/details/48086917