Matrices, linear algebra and vectors go together. At a univesity, those will usually be combined in a linear algebra class that usually depends on finishing the lower division calculus sequence (three semesters, typically), so if you're asking about courses to take at school then you'll need everything in and leading up to that. (arithmetic, algebra, geometry, trigonometry, complex numbers, etc.)
I suggest you take a calculus-based physics course, too. That will give you tools to deal with simulation of motion, collisions, force-fields and such. It will also give you practice in properties of 3D vectors that math courses won't. Mathematicians are more interested in properties of vectors that extend to an arbitrary number of dimension. The cross product only exists as a vector idea in 3-space, for example.
Quaternions are a small tide pool topic in game and graphics programming. If what I see on the web is representative, half the programmers using them don't understand them. There's nothing in 3D geometry you can solve with them that you can't solve with matrices and vector algebra, and the quaternion approach will usually be a bit slower. They do give a very cute way to describe a 3D rotation and translation as a single transformation in 4D, though, and you should learn about this use for interview questions.
There are a number of different non-calculus topics that you should know, and these are usually collected in a "discrete mathematics" course: Logic, sets, number theory, probability, graph theory and more. A BSCS degree will also apply a lot of that material in "analysis of algorithms" and "data structures" courses.
I don't know if that's everything, but that's some of the preparation that your competition for an entry-level job at a major game dev company will have.