需要cookie,使用插件:EditThisCookie,可以方便的获取格式化的cookie。
参考博文:/p/773c58406bdb
众所周知,使用常规方法爬取淘宝的难度是很高的,所以使用selenium+浏览器几乎成了爬取淘宝最理想的方法。然而现在淘宝搜索需要用户登录才可以,据说是为了收集匹配用户信息。
而每次使用selenium时(+chrome),chromedriver会新打开一个浏览器窗口,所以添加cookie便显得很必要了。
这里看一下selenium的add_cookie
方法
与普通的在headers里添加{'Cookies':' '}
不一样的是,此方法需要按照cookie的name,value,path,domain格式逐个cookie添加,如下图
淘宝的cookie很多,所以需要逐项添加
而手动复制的话很不方便,所以这里使用到一个chrome插件
EditThisCookie
它有个导出功能,当你登录完淘宝后点击导出便会得到一个list格式的字符串
cookies=[{"domain": ".","expirationDate": 1563853177.468827,"hostOnly": false,"httpOnly": false,"name": "BAIDUID","path": "/","sameSite": "no_restriction","secure": false,"session": false,"storeId": "0","value": "C0E2BCF5DCEB680B2453F78AC4CD0956:FG=1","id": 1},{"domain": ".","expirationDate": 1546756058.11283,"hostOnly": false,"httpOnly": false,"name": "BDORZ","path": "/","sameSite": "no_restriction","secure": false,"session": false,"storeId": "0","value": "FFFB88E999055A3F8A630C64834BD6D0","id": 2},{"domain": ".","expirationDate": 3648534764.317173,"hostOnly": false,"httpOnly": false,"name": "BIDUPSID","path": "/","sameSite": "no_restriction","secure": false,"session": false,"storeId": "0","value": "CC0841B75FADCE2D56D45D685391FED0","id": 3},{"domain": ".","hostOnly": false,"httpOnly": false,"name": "H_PS_PSSID","path": "/","sameSite": "no_restriction","secure": false,"session": true,"storeId": "0","value": "","id": 4},{"domain": ".","expirationDate": 2493441539.633233,"hostOnly": false,"httpOnly": false,"name": "MCITY","path": "/","sameSite": "no_restriction","secure": false,"session": false,"storeId": "0","value": "-%3A","id": 5},{"domain": ".","expirationDate": 3648534764.317246,"hostOnly": false,"httpOnly": false,"name": "PSTM","path": "/","sameSite": "no_restriction","secure": false,"session": false,"storeId": "0","value": "1501051117","id": 6},{"domain": ".","expirationDate": 2492662056,"hostOnly": false,"httpOnly": false,"name": "bdime","path": "/","sameSite": "no_restriction","secure": false,"session": false,"storeId": "0","value": "0","id": 7}]
稍加修改就可以作为python的list来导入cookie了
for item in cookies:driver.add_cookie(item)
之后使用selenium就可以正常搜索了
作者:Gcider
链接:/p/773c58406bdb
来源:简书
著作权归作者所有。商业转载请联系作者获得授权,非商业转载请注明出处。