Town councillors have issued a vote of no confidence in their county council leadership over a threatened community centre.