There are plenty of houses in England, and the rest of the Uk, that aren't built in red brick. You will find this material used more in towns and cities from the 19th century onwards - elsewhere you will find local stone and brick and other materials.
The red colouring is a result of firing rather than anything special in the building materials. Most bricks go to some shade of red when they are heated. In fact red brick is a common material in many countries, not just the UK.
In Victorian England there was a need for large scale urban housing due to the Industrial Revolution, which took place earlier and on a bigger scale in the UK. This led to the typical sight of a long terrace of red houses that you still see today. After World War II there was again a need for quickly built, affordable mass housing, and also for new higher education institutions - even today you can hear the term 'redbrick' used to describe one of the 'new' postwar universities as opposed to the traditional stone structures of Oxford, Cambridge etc.