You need some solid Linear Algebra:
Vector Space, dual vector space, unitary and hermitian matrices, eigenvectors and eigenvalues, trace…
Mind that you should learn these things with mathematical approach, for example, vectors are elements of vector space which has certain axioms, and not 3D arrows, like pupils learn in school.
Since book has this approach (matrix mechanics, rather than wave mechanics), you don’t need too strong analysis, you can just trust that some things are working that way, but if you want to understand it fully, i recommend taking some analysis course as well, to be able to understand decomposition in eigenfunctions. Integrals and derivatives are MUST, however.
What are the prerequisites for reading this? What level of mathematics and background of classical physics?
You need some solid Linear Algebra: Vector Space, dual vector space, unitary and hermitian matrices, eigenvectors and eigenvalues, trace… Mind that you should learn these things with mathematical approach, for example, vectors are elements of vector space which has certain axioms, and not 3D arrows, like pupils learn in school. Since book has this approach (matrix mechanics, rather than wave mechanics), you don’t need too strong analysis, you can just trust that some things are working that way, but if you want to understand it fully, i recommend taking some analysis course as well, to be able to understand decomposition in eigenfunctions. Integrals and derivatives are MUST, however.