As a teacher, there is no end of work to be done. You could take work home every evening and weekend for the rest of your career and still find things that to be planned and graded, better ways of presenting what you already have, and new information that needs to be incorporated. So, in the small amount of free time that you have, there are only two reasons to pick your extra curricular activities based on:
1) Am I passionate about it?
2) Will it encourage personal growth?
Being a teacher can be very draining. At the end of the day, there is very little time to replenish your energy. Unfortunately, you still have to live life. What I would encourage, is picking activities in your life that will be a boon. Most of the time, those activities do require energy, but they are also rejuvenating.
A while back I chose to join a Spanish Language Meetup. It is a group of people of varying ages, ethnic backgrounds, and social status who get together once a month to speak in Spanish over dinner. Last night, on a Friday night after a full week of school, I was exhausted. What I know, however, is that this meetup incorporates many of the things that I am passionate about including, communicating, meeting new people, the Spanish language, and experiencing culture. I went despite my yearning for a quiet night of television, and it was such a blessing. My conversational Spanish is getting so much better. I talked to several native speakers during the evening and even joined a couple of them at a bar for pool afterwards. Talking and laughing in Spanish was the most natural thing in the world to me. In remembering how hard I worked to get to this skill level (for the sake of my students and their parents) it is something that I am very proud of.
If you're feeling argumentative today, you may say "No teach, sometimes you have to do things that you don't want to do and it is just a waste of time and energy." I will argue back and say that I don't really believe that. For example, if you don't want to go to your mother-in-law's house for Sunday brunch but you do anyway, it is because you are passionate about family. If you don't want to go to the grocery store but do anyway, it is because you value life. If you don't want to go to the gym, but do anyway, it is because health is important to you.
If you find yourself doing something and really wondering what the point is, cut it out, it's not making you a better person. If you find yourself missing something that made your life better, or that could, get it back. There are two reasons to use that precious little extra time we have in the day: because you are doing something you are passionate about, or because it is encouraging personal growth (and very often, it is doing both).