Up coming, you need to make your mind up how you desire the crawler to handle a variety of Internet technologies. There is certainly an ongoing discussion about the intelligence of internet search engine crawlers. It's actually not solely clear If they're full-blown headless browsers or just glorified curl https://www.youtube.com/watch?v=EVeTP0aV8gY