Do what's best for you, your future and those you care about. No one is going to look out for your best interests aside from you in this career, not the agency, not the union and certainly not those who would get salty about you taking a desk or filing an ERR. They got paid to train you, it wasn't some privilege bestowed upon you by them out of the goodness of thier hearts. They weren't doing you a favor. That's toxic controller bullshit, and the workforce is full of it.
What your current coworkers will think of you shouldn't be a consideration in future career moves and improving your life situation. If getting off the boards so you can take the wife out on a Saturday night or make little Timmy's tee ball games or not missing out on social events with your friends because your schedule is shit outweighs the shift premiums, or a few years tacked on at the twilight of your career is less important than the next 25 years of nights weekends and holidays do whats good for you. There's absolutely no shame in that and anyone who says otherwise is an inconsolable prick.