Growth hacking is an innovative marketing strategy that focuses on rapid experimentation and testing to create customer growth. In growth hacking, the only metric that ultimately matters is growth, and it aims to do so rapidly, so growth hacking techniques are excellent for SMBs with limited budgets.
Web scraping is one technique you can use in growth hacking, and the concept of web scraping has existed even before growth hacking was a thing. In web scraping, you’re simply extracting data from websites to be organized into databases for your reference. An example would be compiling a list of stores selling the same product as you and at what price, or a list of competitors running sales on their pages.
Those are just situational examples of web scraping, but it can be used for many different things, depending on the type of data you want to collect. Web scraping is thus an incredible useful technique for growth hacking when you need to quickly build your own database for reference.
How is web scraping done?
There’s a very simple DIY method which requires only minimal coding experience, as it involves setting up a robot.txt file with instructions for a web crawler on what data to scrape from websites. You can also consider the best web scraping tools available, and figure out if you’d like a free, open-source software or something more fit for SMB usage.
Some of the great and useful scraping tools you can check out are:
- ScrapingBee: Has numerous features such as classic and premium proxies to prevent being blocked while scraping.
- DiffBot: Relatively expensive but offers automatic extraction APIs for changing rulesets.
- ScrapeBox: Desktop software with many features.
- ScreamingFrog: Low cost, but runs a little slow on large scraping jobs.
- Scrapy: Free and open-source, which means free but not always stable.
These all have their pros and cons, so check out the list we linked above.
So one way you would use a scraping tool would be to use a keyword research tool for identifying your largest competitors for target keywords. You’d then run the scraping tool of your choice on each website, export the results as a CSV file, and use a formula to compare how many page titles are targeting major keywords, for example, or how many pages your competitor has built up around specific keywords.
With those results in hand, you can figure out how to replicate them, or monitor your competitors for sales promotions they’re running, etc.
Another example would be finding popular content and keywords to build your own content around. For example, you could scrape popular trends from aggregated content websites, and stay up-to-date with important news and insights to help you shape your SEO and marketing outreach.
Examples of web scraping in various industries
Real estate agencies also make use of web scraping, usually for building databases of properties for sale or rent. The real estate agency could scrape MLS listings and use the data to directly populate their website, which allows them to act as the property agents when someone inquires about a listing on their site.
Data analytics company hiQ Labs was scraping publicly available data from LinkedIn, for the purpose of fueling their insights into various work sectors. LinkedIn sent a cease and desist notice to hiQ Labs, claiming that hiQ Labs was in violation of the Computer Fraud and Abuse Act (CFAA), but the Ninth Circuit Court of Appeals upheld a previous district ruling that scraping publicly available data did not violate the CFAA. You can read more about that case and the legality of web scraping in this article.
