This is something I have seen an increasing number of people try to claim lately, is it true?
Lolita is for everybody who likes it, if you like it, wear it. It doesn't matter what gender, race, height, religion, weight or whatever someone can try and nit pick over, you can wear Lolita if that is what you want to do.
The thing that irritates me the most about this myth is that for many of the people who first got into Lolita fashion at around the same time as me, many of the more famous models such as Mana Sama and Aya at the time were men, and while they are not as prominent as they once were they were still a big part of forming the fashion we know today.
Most annoyingly I keep seeing people assuming that if a guy wears Lolita then it must be because he is fetishising it.
If you get annoyed at people assuming you're wearing it because of a fetish then you have no right to assume someone else is just because they're not the same gender as you.