Assuming that each line shares common direction cosines defined by the equations, and adding to the given equations L²+M²+N²=1 (I’ve used uppercase versions of the direction cosines to avoid ambiguities), we have three equations with three unknowns, which is sufficient to find the individual direction cosines. Without this assumption there would be insufficient information to define the line vectors.
So we have:
(1) 3L+M+5N=0⇒M=-(3L+5N);
(2) 6MN-2NL+5L=0⇒-6(3L+5N)N-2NL+5L=0, -20NL-30N²+5L=0, L(1-4N)=6N², L=6N²/(1-4N)⇒M=-(18N²/(1-4N)+5N)=N(2N-5)/(1-4N);
(3) L²+M²+N²=1⇒
36N⁴/(1-4N)²+(18N²/(1-4N)+5N)²+N²=1,
36N⁴+N²(2N-5)²+N²(1-4N)²=(1-4N)²,
36N⁴+4N⁴-20N³+25N²+N²-8N³+16N⁴=1-8N+16N²,
56N⁴-28N³+10N²+8N-1=0.
This quartic can be solved using Newton’s Method and approximations of the solution as initial values: -0.35 and 0.15 (from a graph and using the Intermediate Value Theorem).
N₁=-0.3754, N₂=0.1130,
L₁=0.1397, L₂=0.3380,
M₁=-0.9837, M₂=0.8630 (approx figures).
Thus there are two solutions (only), presumably one for each line.
(L₁,M₁,N₁) defines the unit vector for one line and (L₂,M₂,N₂) defines the unit vector for the other line. The dot product of these gives the cosine of the angle θ between them, so θ=arccos(L₁L₂+M₁M₂+N₁N₂)=arccos(-0.8442)=147.58° or 2.5758 radians.