Let’s assume that interest is compounded monthly. 3.75% APR is 3.75/12% monthly=0.3125%=0.003125, and 30 years is 360 months.
The first monthly payment will accrue interest for the whole 360 months, and the second payment will accrue interest for 359 months and so on. If the initial amount borrowed is P, the growth of P is P(1.003125)³⁶⁰=3.0748P approximately. This is offset by the monthly payments=
800(1+0.003125+0.003125²+...+0.003125³⁵⁹)=800(1.003125³⁶⁰-1)/0.003125=531153.51 approx.
The two amounts must be the same for the mortgage debt to be reduced to zero, so 3.0748P=531153.51.
Therefore P=531153.51/3.0748=$172,744 approx. Rounding this off to the nearest $1000, the maximum mortgage they could afford is about $173,000.
(For continuous compound interest the growth factor on the principal is e^(0.0375*30)=3.0802 approx, which is not very different from 3.0748. But if we use continuous compound interest P=$172,441 approx. Another possible difference arises from when in the month the $800 is paid. P could be 172,744*1.003125=$173,284.)