Ongoing talks between the U.K. and the European Union could break down at any point, according to a former member of the Bank of England.