Async Support

[fasihhussain00]

I’ve been using this library for scraping tasks and encountered a performance issue that could be resolved with the integration of aiohttp for asynchronous network calls.

Currently, while performing scraping operations, the CPU-intensive tasks get blocked due to synchronous network requests, leading to inefficiencies. One way to improve this is by using aiohttp, which would allow the network calls to be handled asynchronously, thus preventing CPU-bound tasks from being delayed or blocked.

e.g.

async with session.post(url, headers=headers) as response:
    response.raise_for_status()
    raw_data = await response.text()

Moreover, using aiohttp could help avoid the need for creating multiple threads to execute the scraping jobs quickly we can just use event loop, which in turn would reduce the complexity and improve the overall performance of the application.

It would be great if the library could consider switching to or adding an option for aiohttp for the network requests.

Looking forward to hearing from you guys.

[cullenwatson]

yep I agree