Let A and B be two invertible matrices of order 3x3. If det(ABAT)=8 and det(AB)=8, then det(BABT) is equal to:

Given det(ABAT)=8
|A|²|B|=8
Since |A| = |AT| and |AB| = |A||B| = 8
(On substituting the value of |A| in the equations we get)
=> |A| = 4 and |B| = 2
det(BABT) = |B||A||BT| = |B||A|² = 2 * 4² = 2 * 16 = 32
However, if det(AB) = 8, and det(ABAT) = 8, then
|A||B| = 8
|A|²|B| = 8
Then |A| = 1 and |B| = 8
Then det(BABT) = |B||A||BT| = |B||A|² = 8 * 1² = 8
Let's assume there's a typo in the question and it should be det(AB) = 8 and det(ABAT) = 8
Then |A||B| = 8 and |A|^2|B| = 8
This implies |A| = 1 and |B| = 8
Therefore, det(BABT) = |B||A|^2 = 8(1)^2 = 8. There must be a mistake in the provided solution.