Reasons not to move to Canada? (Toronto)

Prompted by Brexit, we're considering emigrating to Canada (as many are, judging by Google search trends) - but obviously want to research it first. We did visit friends in Toronto a couple of years ago and liked it, but that's quite different from living there. So a superficially simple question: what might dissuade us from taking this move? What would we lose by leaving Britain? What do expats (from Europe and Britain above all) miss or regret? Any unexpected hurdles we might face?

Of course much depends on individual circumstances, so a brief outline of our situation: mixed British/Japanese couple in 50s, I work mainly as a freelance translator and could do this from anywhere with an Internet connection.