The formula for calculating the magnitude of the perpendicular distance from a point P(X,Y) to a line Ax+By+C=0 is:
|AX+BY+C|/√(A2+B2). See later for derivation of this formula.
The directed distance depends on the sign of C and is given by (AX+BY+C)/√(A2+B2). C determines whether the positive or negative square root in the denominator applies. If C≥0, the sign is negative, otherwise positive. The sign of AX+BY+C is also important when deciding the direction. In this question, P1 is on the opposite side of the line to the origin so the direction is positive; P2 is on the same side of the line as the origin so the direction is negative; P3 is on the opposite side of the line to the origin so the direction is positive.
We need to apply the formula where A=5, B=-12, C=-26, X={3,-4,9} and Y={-5,1,0} depending respectively on {P1,P2,P3}.
dP1=|5×3-12×(-5)-26|/√(52+122)=|15+60-26|/13=49/13; since C<0 the directed distance is +49/13.
dP2=|5×(-4)-12×1-26|/√(52+122)=|-20-12-26|/13=|-58|/13=58/13; since C<0 the directed distance is -58/13.
dP3=|5×9+0-26|/√(52+122)=|45-26|/13=|19|/13; since C<0 the directed distance is +19/13.
DERIVATION OF FORMULA
Line: Ax+By+C=0. Line has slope -A/B so perpendicular has negative inverse slope=B/A.
Perpendicular: y-Y=(B/A)(x-X), y=(B/A)(x-X)+Y, which can be written Bx-Ay+AY-BX=0.
As a system of simultaneous equations, we can find Q where the perpendicular meets the line:
Ax+By=-C
Bx-Ay=BX-AY
A2x+ABy=-AC
B2x-ABy=B2X-ABY
(A2+B2)x=B2X-ABY-AC, x=(B2X-ABY-AC)/(A2+B2);
ABx+B2y=-BC
-ABx+A2y=-ABX+A2Y
y(A2+B2)=A2Y-ABX-BC, y=(A2Y-ABX-BC)/(A2+B2).
Q(x,y)=((B2X-ABY-AC)/(A2+B2),(A2Y-ABX-BC)/(A2+B2)).
d=√((Px-Qx)2+(Py-Qy)2) where the subscripts represent the x and y coordinates of the points P and Q.
d=√(X-(B²X-ABY-AC)/(A²+B²))²+(Y-(A²Y-ABX-BC)/(A²+B²))²),
d=√(X(A²+B²)-(B²X-ABY-AC))²+(Y(A²+B²)-(A²Y-ABX-BC))²)/(A²+B²).
Px-Qx=X(A²+B²)-(B²X-ABY-AC)=A²X+ABY+AC=A(AX+BY+C);
Py-Qy=Y(A²+B²)-(A²Y-ABX-BC)=B²Y+ABX+BC=B(AX+BY+C).
Let p=AX+BY+C, then:
d=√((Ap)²+(Bp)²)=p√(A²+B²)/(A²+B²) rationalised form of:
p/√(A²+B²)=(AX+BY+C)/√(A²+B²). Note that p can be negative, depending on the coordinates of P and the value of C.
So the formula for the (absolute) distance d=|AX+BY+C)|/√(A²+B²).