Probably every vegan on the planet has heard, if not directly then at least second hand from other vegans, the "argument" from meat eaters that vegans care more about animals than they do about other humans. I have heard a few different responses to this, most of which were very good. One of the best, probably, is that caring about animals in no way diminishes one's compassion for other humans, and animal rights activism and human rights activism are not mutually exclusive. The other one I like, which is probably a little less effective but perhaps a bit more satisfying to say to an obnoxious omnivore, is that the majority of meat eaters don't actually do anything to help other humans either.
The truth of the matter is, at least in my case, animal issues are more important than human issues. It's not because I don't care about humans, or because I like animals better. For me, it's all about the level of suffering. I believe that, when it comes to suffering, all beings, human or nonhuman, should be given equal consideration. And the truth is, even in the very worst cases, there are no humans who are in situations as horrible as those that many animals are facing. Yes, there are human rights violations happening all over the world. Yes, there are people with diseases which could have been prevented. Yes, there are people living in abject poverty. And yes, all of these things are terrible.
However, if I had to choose between working to end any of these things and working for animal rights, I would choose animal rights every time. Because there are no humans being locked in battery cages or gestation crates. There are no humans being subjected to horrifying medical experiments. There are no humans being skinned, or boiled, or eviscerated while they are still alive simply because we want something that comes from them. The torment that animals face in our society everyday is far greater, and on a far grander scale, than any injustice endured by humans in our current culture.
So yes, it is true that as animal activists we can still work for human charities, and that is a very admirable thing. It is also true that most of the people who claim that vegans care more about animals than people probably do very little to help their fellow humans. I, personally, don't feel that we should have to justify ourselves though. We shouldn't have to pretend to care equally about human rights causes, if we don't. I think that we should have every right to focus on the greatest injustices first, without apology. And I believe that is exactly what animal rights activists are doing.