Webscrape and Automate


Web Development

Updated Nov 3rd, 2021

Automate Automate Automate. Use Python’s “Beautiful Soup” or Node’s “Puppeteer.”

Check a website’s terms-of-service to make sure you are in compliance.

Task One: Bring in the data. Companies will pay you to do this

Task Two: Run analytics on the data once you have brought it in.

Project Idea 2: Scrape the listings on cargurus.com for the best “cost-per-remaining-month” given a certain filter.

Project Idea 1: Automatically pull in data from external sources (using scraping or APIs) into one “Personal Analytics Dashboard.” Pull in metrics for health, personal finance, social, professional, thoughts, resources. Could have reddit-style upvoting. Integrate with text or email notifications.

Windows

Windows Task scheduler may be needed to run scripts every X hours