Three are 6 ways of arranging the categories along the shelf: MHS, MSH, HMS, HSM, SMH, SHM, where M=math, H=history, S=science.
We can take each arrangement of categories and work out the probability of each arrangement.
Take MHS as an example. If M and H are organised to be kept together, then S automatically is kept together.
There are 13 books, 5 being math books. So the probability of selecting a math book in the first position on the shelf is 5/13, leaving 12 books, 4 of which are math. The probability of selecting another math book is 4/12. So the probability of the first 5 books being all math is 5/13.4/12.3/11.2/10.1/9=1/1287.
There are now 8 books left, 2 of which are history books. So the probability of selecting a history book next is 2/8, then 1/7 for the second history book. That's 2/8.1/7=1/28. The science books will all be together. Combine the math and history probabilities: 1/1287.1/28=1/36036.
Now let's take MSH. Next to math we have science and the probabilities are 6/8.5/7.4/6.3/5.2/4.1/3=1/28. When we combine the probabilities we again get 1/36036. And so it goes on for the other permutations of the categories.
So, adding the combined probabilities we get 6/36036=1/6006=0.01665%.