Price comparison websites have become part of our everyday lives. Whatever we are looking to purchase, we naturally seek out the best deals going. And most of us turn to comparison websites for everything from broadband and car insurance to flights and hotels. But have you ever wondered how these companies get the information for their websites? And what technology is required to set up a comparison website?
Types of comparison sites
The biggest comparison sites, such as Go Compare, let consumers compare deals across a range of products and services, including utilities, financial services, and multiple insurance categories. Others such as Idealo are focused on shopping and let you compare prices on products from thousands of retailers. Travel comparison websites are extremely popular but are increasingly looking to boost their green credentials. For example, Skyscanner recently launched an electric and hybrid car hire search facility on their website. There is also a range of comparison sites that focus on a single sector. One example is Bonusfinder UK, which searches for the best online gambling bonuses from the top casino and sports betting sites in the UK. Users can compare welcome bonuses and get in-depth reviews of the best sites all in one place.
Retailer data feeds
The key objective of these sites is to grab data from multiple sources and then present the best rates to their online visitors. There are several ways in which this can be done. The first and most common way is to set up direct feeds from the merchants. This method ensures accurate and up-to-date data and reduces the risk of errors. Most online retailers are open to partnering with price comparison engines as they offer access to a huge number of customers and help them boost revenues. This practice is so common that many of the biggest retailers have bespoke Application Programming Interface (API) systems in place to provide these kinds of feeds directly from the source. If you are planning to start a price comparison site, it makes commercial sense to reach out to the retailers and form partnerships.
Some retailers may charge a fee for this service to cover the additional costs of serving live API data feeds. The data received may also come in different formats depending on the merchant, so comparison site operators must be prepared for this when converting the data into web content.
Third-party data suppliers
In some cases, comparison sites may use data from third-party companies that specialise in gathering data from retailers and then selling it to comparison sites. This can save a lot of time and can also prove more cost-effective depending on how much data is supplied.
Web scraping
Another popular method of gathering data is through web scraping. This involves using automated software to lift data directly from merchant websites. The scraped data is then processed and presented as content on the comparison site. Many comparison sites hire data service companies to carry out this work for them. While bigger companies may develop their own web scraping systems to reduce long-term costs, the problem with this method is that many online retailers now use technology to prevent their data from being scraped because the data requests can put excess strain on their servers.
In practice, web comparison sites may use all these methods to gather the data they need. The more sources they have, the better the set of results they can give their customers.