python网页文本爬虫

Python爬虫

1，p ython爬虫介绍：、

网络爬虫（又被称为网页蜘蛛，网络机器人），是一种按照一定的规则，自动的抓取万维网信息的程序或者脚本。各大搜索引擎都用爬虫缓存各种url，提供搜索服务。高级爬虫技术难度是很高的，要考虑很多，比如连接优化，代理服务器，大数据量下爬取优化，站点爬取规则设计，但是基础爬虫重点只是实现信息抓取保存和处理，爬取规则通常很简单。

以小说网站爬取为例，首先需要掌握python基础，比如urllib使用，python进行字符串操作，复杂一点使用正则表达式。还有就是基本的程序逻辑。具备这三点就能开始爬小说。

爬虫代码示例

首先贴上完整代码：

importurllib

import urllib2

importos

import time

import sys

defgetHtml(url):

page = urllib.urlopen(url)

html = page.read()

returnhtml.decode('gbk').encode('utf-8') + '\r\n'

definterstr(src, begin, end):

index1 = src.find(begin)

if index1 is -1:

return None

index1 += len(begin)

tmp = src[index1:]

index2 = tmp.find(end)

if index2 is -1:

return None

dst = tmp[:index2]

returndst

defgetTitle(html):

title = interstr(html, 'title = " ', '";')

if title is None:

return None

return title

defgetNextPage(html):

pageNum = interstr(html, 'next_page = "', '.html";') bookID = interstr(html, 'bookid = "', '";') ifpageNum is None or bookID is None:

return None

nextPage = (url + bookID +pageNum) returnnextPage

defgetContent(html):

data = interstr(html, '

', '

') if data is None:

return None

data = data.replace('', '') data = data.replace(' ', '\n')

data = data.replace('

', '')

return data + '\n'

defforstr(src, begin, end):

tmpSrc = src

strList = []

while True:

indexBegin = tmpSrc.find(begin)

ifindexBegin is -1:

break

indexBegin += len(begin)

tmp = tmpSrc[indexBegin:]

indexEnd = tmp.find(end)

ifindexEnd is -1:

break

tmpString = tmp[:indexEnd]

strList.append(tmpString)

tmpSrc = tmp

returnstrList

if __name__ == '__main__':

book =sys.argv[1]

url = sys.argv[2]

nextPage = url + (book)

html = getHtml(nextPage)

listSrc = interstr(html, '', '