Этот материал находится в платной подписке. Оформи премиум подписку и смотри или слушай Web Crawling with Nodejs (H&M, Amazon, LinkedIn, AliExpress), а также все другие курсы, прямо сейчас!
Купить сейчас
  • Урок 1. 00:03:43
    What is the web crawling and web scraping and how is it different ?
  • Урок 2. 00:04:57
    Legality of web scraping and web crawling
  • Урок 3. 00:01:19
    Tools we will be using during development
  • Урок 4. 00:02:33
    Methods of web crawling and web scraping
  • Урок 5. 00:06:13
    Finding hidden API using Chrome Dev Tools
  • Урок 6. 00:07:35
    Testing hidden API inside Postman, and finding other section API endpoints
  • Урок 7. 00:01:46
    Initializing NPM + some info about Nodejs Request and Needle
  • Урок 8. 00:02:21
    Creating our HTTP request with needle inside Nodejs
  • Урок 9. 00:03:25
    Adding User-Agent header to get past denial in nodejs
  • Урок 10. 00:04:06
    Creating MongoDB cluster for saving data
  • Урок 11. 00:05:21
    Connecting to MongoDB cluster from Nodejs
  • Урок 12. 00:04:54
    Saving data to MongoDB
  • Урок 13. 00:07:36
    Getting all products in MongoDB using a loop with offset variable and pagesize
  • Урок 14. 00:05:25
    Finding hidden API using Chrome Dev Tools
  • Урок 15. 00:04:26
    Making API request from Postman with correct headers
  • Урок 16. 00:05:50
    Making API request from Nodejs using Fetch API
  • Урок 17. 00:04:59
    Getting many items using a for loop and sleep function
  • Урок 18. 00:03:04
    Saving AliExpress products to MongoDB
  • Урок 19. 00:01:09
    Intro to project
  • Урок 20. 00:04:13
    Why are we using HTTP requests and not Puppeteer?
  • Урок 21. 00:01:27
    Initializing NPM + installing jest, cheerio and needle npm packages
  • Урок 22. 00:02:40
    Writing our reuseable httpRequest module for our testing and crawling
  • Урок 23. 00:05:12
    Creating our test HTML file (check resources for URL)
  • Урок 24. 00:03:27
    Setting up testing and intro to testing
  • Урок 25. 00:04:17
    Writing our first test for our HTML parser
  • Урок 26. 00:05:38
    Getting title from product page and making our test pass
  • Урок 27. 00:02:51
    Getting the price from product page
  • Урок 28. 00:06:41
    Making our second test and getting product links from page
  • Урок 29. 00:08:39
    Writing out our actual webcrawling in 6 minutes!
  • Урок 30. 00:03:08
    Setup so we only crawl only unique product ID's
  • Урок 31. 00:05:22
    Adding a new test case for different layout + outtro
  • Урок 32. 00:00:32
    Intro to project
  • Урок 33. 00:00:34
    Initializing project with puppeteer and cheerio packages
  • Урок 34. 00:02:44
    Opening Puppeteer browser and navigating to URL
  • Урок 35. 00:06:05
    Login to Linkedin using Puppeteer
  • Урок 36. 00:04:48
    Getting profile links on a LinkedIn profile
  • Урок 37. 00:04:33
    Building web crawler loop for Puppeteer