Just being uncomfortable around the men in your office isn't a good enough reason to be able to fill an empty spot in another office. You have to go into detail with the specifics or you are stuck where you are. Of course we know going into specifics will make the problem worse.