solution for python scrapy ERROR: Spider must return request, item, or None, got'Tag'

created at 10-20-2021 views: 1

Today I wrote a crawler and encountered a pit, prompt

[scrapy.core.scraper] ERROR: Spider must return request, item, or None, got'Tag' in <GET https://www.

In fact, the reason is very unexpected. I used item in the code, and scrapy used item to pass data. As a result, this problem was caused.

for item in soup.select(".job-list-item"):
            uu=item.select_one("a").get('href').split("?")[0]
            if uu is not None:
                item['wz']=uu
                yield item

If you change the yield to return, it does not report an error, but the data cannot be transmitted, you need to change it to the following code

for itema in soup.select(".job-list-item"):
            uu=itema.select_one("a").get('href').split("?")[0]
            if uu is not None:
                item['wz']=uu
                yield item

python

created at 10-20-2021

TwoPunch

Please log in to leave a comment.

solution for python scrapy ERROR: Spider must return request, item, or None, got'Tag'

Related:

Table of Conents