It can be that either you are having a too high rank or it is possible you got some character in your usage-rate which people don't like. So I am going to ask you if you go to your display, you see your top 3 used characters, which 3 characters are they?
If it would be: Masked Man, Ohnoki (Tsuchikage), Kiba or Deidara it probably has something to do with those. Or the same thing can be said about a lot of other characters people just have no strategy in how to fight against them.
Which also might be a reason why people will avoid you. That or you are right, you are a disease. Another thing that might be it, they want a specific somebody to join their lobby so they just kick everyone else.