Multiple independent investigations have ascertained that well over half the detainees in GTMO were never associated with any state-listed terrorist organization. I don't know if the exchanged prisoners came from that group or not. But you'd hope so. It certainly would be possible to move hundreds of people out of GTMO without touching anybody affiliated with known terrorist organizations.
The majority of those guys were just standard enemy combatants. Just because you fight for a terrorist organization on the front lines doesn't necessarily make you a terrorist when the terrorist organization also happens to control a national government. At that point, it's entirely possible to just be a soldier. That's not to say these guys love the west. But really, they were soldiers for the same reason the majority of US soldiers are soldiers - it's a steady paycheck, and for many people, the best one they're likely to get. In a country like Afghanistan, even if you haven't been pressed into military service, there may not be a lot of other options to feed your family. So you fight. That doesn't make you a terrorist.