-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathscrapy.log
executable file
·136 lines (134 loc) · 8.87 KB
/
scrapy.log
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
2019-04-08 14:44:46 [scrapy.utils.log] INFO: Scrapy 1.6.0 started (bot: kingofsat)
2019-04-08 14:44:46 [scrapy.utils.log] INFO: Versions: lxml 4.3.2.0, libxml2 2.9.9, cssselect 1.0.3, parsel 1.5.1, w3lib 1.20.0, Twisted 18.9.0, Python 3.7.2 (default, Mar 12 2019, 18:57:13) - [GCC 5.4.0 20160609], pyOpenSSL 19.0.0 (OpenSSL 1.1.1b 26 Feb 2019), cryptography 2.6.1, Platform Linux-4.15.0-46-generic-x86_64-with-debian-stretch-sid
2019-04-08 14:44:46 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'kingofsat', 'LOG_FILE': 'scrapy.log', 'NEWSPIDER_MODULE': 'kingofsat.spiders', 'SPIDER_MODULES': ['kingofsat.spiders']}
2019-04-08 14:44:46 [scrapy.extensions.telnet] INFO: Telnet Password: 496a5ebd168de3c8
2019-04-08 14:44:46 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.memusage.MemoryUsage',
'scrapy.extensions.logstats.LogStats']
2019-04-08 14:44:46 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2019-04-08 14:44:46 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2019-04-08 14:44:46 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2019-04-08 14:44:46 [scrapy.core.engine] INFO: Spider opened
2019-04-08 14:44:46 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-04-08 14:44:46 [py.warnings] WARNING: /home/bulute/.virtualenvs/TurksatScrape/lib/python3.7/site-packages/scrapy/spidermiddlewares/offsite.py:61: URLWarning: allowed_domains accepts only domains, not URLs. Ignoring URL entry https://tr.kingofsat.net/tvsat-turksat4a.php in allowed_domains.
warnings.warn(message, URLWarning)
2019-04-08 14:44:46 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2019-04-08 14:44:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://tr.kingofsat.net/tvsat-turksat4a.php> (referer: None)
2019-04-08 14:44:48 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tr.kingofsat.net/tvsat-turksat4a.php>
{'channel': 'Al Jazeera Satellite Channel', 'V-PID': '5308', 'A-PID': '5408'}
2019-04-08 14:44:48 [scrapy.core.engine] INFO: Closing spider (finished)
2019-04-08 14:44:48 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 234,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'downloader/response_bytes': 25608,
'downloader/response_count': 1,
'downloader/response_status_count/200': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2019, 4, 8, 11, 44, 48, 511190),
'item_scraped_count': 1,
'log_count/DEBUG': 2,
'log_count/INFO': 9,
'log_count/WARNING': 1,
'memusage/max': 51552256,
'memusage/startup': 51552256,
'response_received_count': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2019, 4, 8, 11, 44, 46, 665656)}
2019-04-08 14:44:48 [scrapy.core.engine] INFO: Spider closed (finished)
2019-04-08 14:47:40 [scrapy.utils.log] INFO: Scrapy 1.6.0 started (bot: kingofsat)
2019-04-08 14:47:40 [scrapy.utils.log] INFO: Versions: lxml 4.3.2.0, libxml2 2.9.9, cssselect 1.0.3, parsel 1.5.1, w3lib 1.20.0, Twisted 18.9.0, Python 3.7.2 (default, Mar 12 2019, 18:57:13) - [GCC 5.4.0 20160609], pyOpenSSL 19.0.0 (OpenSSL 1.1.1b 26 Feb 2019), cryptography 2.6.1, Platform Linux-4.15.0-46-generic-x86_64-with-debian-stretch-sid
2019-04-08 14:47:40 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'kingofsat', 'LOG_FILE': 'scrapy.log', 'NEWSPIDER_MODULE': 'kingofsat.spiders', 'SPIDER_MODULES': ['kingofsat.spiders']}
2019-04-08 14:47:40 [scrapy.extensions.telnet] INFO: Telnet Password: fc15726b4f3e5e08
2019-04-08 14:47:40 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.memusage.MemoryUsage',
'scrapy.extensions.logstats.LogStats']
2019-04-08 14:47:40 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2019-04-08 14:47:40 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2019-04-08 14:47:40 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2019-04-08 14:47:40 [scrapy.core.engine] INFO: Spider opened
2019-04-08 14:47:40 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-04-08 14:47:40 [py.warnings] WARNING: /home/bulute/.virtualenvs/TurksatScrape/lib/python3.7/site-packages/scrapy/spidermiddlewares/offsite.py:61: URLWarning: allowed_domains accepts only domains, not URLs. Ignoring URL entry https://tr.kingofsat.net/tvsat-turksat4a.php in allowed_domains.
warnings.warn(message, URLWarning)
2019-04-08 14:47:40 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2019-04-08 14:47:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://tr.kingofsat.net/tvsat-turksat4a.php> (referer: None)
2019-04-08 14:47:40 [scrapy.core.scraper] ERROR: Spider error processing <GET https://tr.kingofsat.net/tvsat-turksat4a.php> (referer: None)
Traceback (most recent call last):
File "/home/bulute/.virtualenvs/TurksatScrape/lib/python3.7/site-packages/scrapy/utils/defer.py", line 102, in iter_errback
yield next(it)
File "/home/bulute/.virtualenvs/TurksatScrape/lib/python3.7/site-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output
for x in result:
File "/home/bulute/.virtualenvs/TurksatScrape/lib/python3.7/site-packages/scrapy/spidermiddlewares/referer.py", line 339, in <genexpr>
return (_set_referer(r) for r in result or ())
File "/home/bulute/.virtualenvs/TurksatScrape/lib/python3.7/site-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr>
return (r for r in result or () if _filter(r))
File "/home/bulute/.virtualenvs/TurksatScrape/lib/python3.7/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
return (r for r in result or () if _filter(r))
File "/home/bulute/WorkSpaces/Scrape/kingofsat/kingofsat/spiders/kingsat.py", line 41, in parse
yield channels.append(channels2)
AttributeError: 'dict' object has no attribute 'append'
2019-04-08 14:47:40 [scrapy.core.engine] INFO: Closing spider (finished)
2019-04-08 14:47:40 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 234,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'downloader/response_bytes': 25608,
'downloader/response_count': 1,
'downloader/response_status_count/200': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2019, 4, 8, 11, 47, 40, 691313),
'log_count/DEBUG': 1,
'log_count/ERROR': 1,
'log_count/INFO': 9,
'log_count/WARNING': 1,
'memusage/max': 51638272,
'memusage/startup': 51638272,
'response_received_count': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'spider_exceptions/AttributeError': 1,
'start_time': datetime.datetime(2019, 4, 8, 11, 47, 40, 120608)}
2019-04-08 14:47:40 [scrapy.core.engine] INFO: Spider closed (finished)