Either way, hidden inputs' names tend to be more stable than classes. Some will be available in other places or formats, but sometimes they are unique. In Amazon products, we can see that there are many more. Many forms use these to include internal IDs or security tokens. Hidden inputs allow developers to include input fields that end-users cannot see nor modify. And remember, store an HTML file locally if you are going to test it several times.
#Scale webscraper code
We won't need to run a headless browser - i.e., Puppeteer - if we find everything, saving time and memory consumption.ĭisclaimer: we won't include the URL request in the code snippets for every example. It is tiresome, but sometimes we must explore the original HTML to avoid running Javascript. You will see the content once the Javascript and network requests have (maybe) modified it. Before copying all those selectors, we recommend taking a couple of minutes to look for hidden inputs, metadata, and network requests.īeware of doing this with Chrome DevTools or similar means. We can open any product page on Amazon, for example, and a quick look will show us the product's name, price, availability, and many other fields. We will be using Chrome's DevTools, but other browsers have similar tools. For that, the easier way we know is to inspect the target page using a browser. Explore before Codingīefore we start coding, we have to understand the page's content and structure. Once we load statically from a file, we can test as many times as possible without any networking or blocking problem.
0 kommentar(er)
