Yeah it's definitely a depression or anxiety thing to some degree.
With older friends, it's not so much drifting away as that I don't want to follow them around all con, ya know? They're pretty much all couples - I was in a relationship when I met most of them too, so it made sense - and there's only so much third wheeling a guy can do. I was in a party at ALA last year where I actually counted, and it was literally 15 couples and me by myself. That's a punch in the gut.
I'm not one of those ronery /r9k/ guys either, I'd be content just hanging around with a bunch of single friends living it up or whatever, but being the odd man out all the time fucking sucks.
There's also a lot of awkwardness in that a lot of the friends in question are girls, and we have longtime kinda flirty dynamics that make their BFs/husbands uncomfortable as fuck. I try as hard as I can not to do it, but several of these girls are WAAAAAAAY too touchy feely or make too many lewd jokes and whatever else, and it's not really inappropriate.
I was rolling with a group at Fanime last night, which I know through a girl, and she was drunk and all up on me right in front of her BF, even to the point of grabbing my crotch and stuff like that. That's not cool at all, and all I could do was make eye contact with him and give him my best "dude, I'm not reciprocating, I don't know WTF your GF is doing" look. She's not the only one like that either, and it makes me minimize time around people like that.