the problematics of language

This morning, a friend of mine was relating a story about being accosted by a man on the train who first asked him for change and then threateningly asked him to empty his pockets. We live in a big city; it's not pleasant, but everybody has at least one story like that. (My most alarming one involves getting thwacked lightly on the shoulder by a passing man asking for change. Physical contact? Yikes.) So I understood the need to vent, and I tried to let him get it out even though I was grumpy and not really feeling particularly sympathetic, but then he said this: "I think he was in a gang."
And I asked why, and he said, "Well, he asked me for money."
What? What does that have to do with supposed gang membership?
I haven't been sleeping well, which I blame the escalation and incoherence of the following conversation on. I said that I didn't see any connection whatsoever between gangs and asking for change. When I pressed my friend for more reasons he might make such an assumption, he told me the man had been wearing baggy low-slung jeans and a baggy shirt. Really? Just like two-thirds of the guys I see on the train every day. Maybe they're all in gangs!
I was upset because I think there's a very problematic tendency to place poor people, people of color, young men, into the category of "gang member" without any actual evidence of said gang membership. I don't dispute that gang violence exists (somebody was shot and killed a block away from the Hard Music, Hard Liquor concert I went to two weeks ago, for instance), but it's such an incredibly loaded (and racist) term that I avoid using it in any instance where it is not an absolutely concrete fact. I tried to explain this a little bit, that especially young black men are automatically shoved into this category and that it's used to justify the way they are subsequently treated, and my friend told me that his accoster wasn't black.
The whole thing just left me feeling icky. I felt like I had just participated in something so overwhelmingly classist and racist that I was unsettled for much of the afternoon. My inability to articulate what I felt was a very important point, his offensive (to me, at least) assumption based on nothing that I could see, and my own racism in assuming that the man was black based on what I had been told: triple ick.