The Korn Ferry survey of retail bosses found 40% were pessimistic compared with only 27% before the vote