Montréal is a completely safe city to visit. Downtown is incredibly safe at all times, only danger on streets late at night is drunk American students screaming on the streets or puking in the corner! It is a large city, therefore there are somewhat significant numbers of pan-handlers that one may encounter. Finally, when exploring the city, there may also be an encounter with a narrow-minded Quebec or French supporter -- I have been shouted at for not knowing French by a bum out in Ahuntsic borough.
And then had a interesting encounter with a well-dressed woman in the nice area near Ave Parc and Laurier who took my help in loading her car with her unwieldy package of food and wine but insisted on speaking French in a condescending manner even when I said I don't speak French, and never saying thanks either in French or English!
Then there was an incident in the metro late at night where one guy shouted obscenities in English inter-mixed with the word "immigrant" (I do look like an immigrant, of course but am a tourist here!) - I never felt in danger, and ignoring such people usually works.
These are rare incidents, and I am only listing them here to point out that Montréal is a large city and they have a few whose insecurities may lead them to cause some discomfort to travelers or new-comers who look different or speak a different language. But to repeat, this is a very safe city for tourists, and while downtown is incredibly safe, outer boroughs are also safe and worth exploring.