feat: lesson about using a framework

[honzajavorek]

This PR introduces a new lesson to the Python course for beginners in scraping. The lesson is about working with a framework. Decisions I made:

• I opted not to use type hints to make the examples less cluttered and to avoid the need to explain type hints to people who didn't ever use them
• The logging section serves two purposes - first, it adds logging :) and second, it conveniently provides code for the whole program at the end of the lesson
• I had a hard time to come up with exercises, because most of the simple stuff I came up with was too simple and would result in shorter and simpler code without the framework 😅
  • I decided to have one classic scenario (listing & detail) just to let the student write their first Crawlee program. It's a bit challenging regarding traversal over the HTML to get the data, but it shouldn't be challenging regarding Crawlee.
  • I introduced one scenario where the scraper needs to jump through several pages (even domains) to get the result. Such program would be hard or at least very annoying to write without framework.
• As always, I focused on having the examples based on real world sites which are somewhat known and popular globally, but also don't feature extensive anti-scraping protections.

Crawlee feedback

Regarding Crawlee, I didn't have much trouble to write this lesson, apart from the part where I wanted to provide hints on how to do this:

requests = []
for ... in context.soup.select(...):
    ...
    requests.append(Request.from_url(imdb_search_url, label="..."))
await context.add_requests(requests)

I couldn't find good example in the docs, and I was afraid that even if I provided pointers to all the individual pieces, the student wouldn't be able to figure it out.

Also, I wanted to link to docs when pointing out the fact that enqueue_links() has a limit argument, but I couldn't find enqueue_links() in the docs. I found this which is weird. It's not clear what object is documented, or what it is, feels like some internals, not as regular docs of a method. I probably know how come it's this way, but I don't think it's useful this way and I decided I don't want to send people from the course to that page.

One more thing: I do think that Crawlee should log some "progress" information about requests made or - especially - items scraped. It's so weird to run the program and then just look at the program as if it hanged, waiting if something happens or not. E.g. Scrapy logs how many items per minute I scraped, which I personally find super useful.

[vdusek]

I got your point for avoiding type hints. However, in the case of the handler:

    @crawler.router.default_handler
    async def handle_listing(context):
        ...

It leaves the reader without any possibility for code completions or static analysis when working with the context object.

In my opinion, type hints should be included here. We have been using them across all docs & examples.

Just a suggestion for you to reconsider, not a request.

Other than that, good job 🙂, and the code seems to be working.

[vdusek]

I think this is fine, but if you would want more reasons you can check out this PR.

[honzajavorek]

Thanks for the review! I see your point and I will indeed reconsider adding the type hint, at least for the context. It would be easier decision if the type name wasn't 28 characters long, but you're right about the benefits for people with editors like VS Code, where we could assume some level of automatic code completions.

[mnmkng]

Suggested change

	1. We perform imports and specify an asynchronous `main()` function.
	1. Inside, we first create a crawler. The crawler objects control the scraping. This particular crawler is of the BeautifulSoup flavor.
	1. In the middle, we give the crawler a nested asynchronous function `handle_listing()`. Using a Python decorator (that line starting with `@`), we tell it to treat it as a default handler. Handlers take care of processing HTTP responses. This one finds the title of the page in `soup` and prints its text without whitespace.
	1. The function ends with running the crawler with the product listing URL. We await the crawler to finish its work.
	1. The last two lines ensure that if we run the file as a standalone program, Python's asynchronous machinery will run our `main()` function.
	1. We perform imports and specify an asynchronous `main()` function.
	2. Inside, we first create a crawler. The crawler objects control the scraping. This particular crawler is of the BeautifulSoup flavor.
	3. In the middle, we give the crawler a nested asynchronous function `handle_listing()`. Using a Python decorator (that line starting with `@`), we tell it to treat it as a default handler. Handlers take care of processing HTTP responses. This one finds the title of the page in `soup` and prints its text without whitespace.
	4. The function ends with running the crawler with the product listing URL. We await the crawler to finish its work.
	5. The last two lines ensure that if we run the file as a standalone program, Python's asynchronous machinery will run our `main()` function.