This web crawler would have to allow the user to specify the URLs to be crawled, the depth of recursive crawling, any filters that should be applied (e.g., use of a particular referrer, or limitation ...

A Web Crawling Code of Conduct Unfortunately, and somewhat understandably, it’s easier to focus on the murky underbelly of Web crawling.

Playwright, a Python library, allows developers to essentially control the browser with a few lines of Python code to automate cross-browser testing for single-page web apps and Progressive Web Apps.