Web Scraping Housing Prices in Real-time: the Covid-19 Crisis in the UK

Abstract

While official statistics provide lagged and aggregate information on the housing market, extensive information is available publicly on real-estate websites. By web scraping them for the UK on a daily basis, this paper extracts a large database from which we build timelier and highly granular indicators. One originality of the dataset is to provide the sellers’ perspective, allowing to compute innovative indicators of the housing market such as the numb er of new posted offers or how prices fluctuate over time for existing offers. Matching selling prices in our dataset with transacted prices from the notarial database using machine learning techniques allows us to measure the negotiation margin of buyers – an innovation to the literature. During the Covid-19 crisis, these indicators demonstrate the freezing of the market and the “wait-and-see” behaviour of sellers. They also show that prices have been increasing in rural regions after the lockdown but experienced a continued decline in London.