We consider three isogeny invariants of abelian varieties over finite fields: the Galois group, Newton polygon, and the angle rank. Motivated by work of Dupuy, Kedlaya, and Zureick-Brown, we define a new invariant called the weighted permutation representation which encompasses all three of these invariants and use it to study the subtle relationships between them. We use this permutation representation to classify the triples of invariants that occur for abelian surfaces and simple abelian threefolds.