2x+y=5(cosø-sinø); 2xy=2(2cos^2ø-7sinøcosø+3sin^2ø)=
4cos^2ø-14sinøcosø+6sin^2ø=
4+2sin^2ø-14sinøcosø=
4+2sinø(sinø-7cosø).
So 2x+y-2xy is 5cosø-5sinø-2sin^2ø+14sinøcosø-4.
By substituting ø=0 we can see that this expression evaluates to 1, not 5; and if we substitute ø=(pi)/2, it evaluates to -11. Therefore in neither case do we have any form of consistency, implying an equation, not an identity.
However, the expression rises no higher than about 3.4362 at ø=26.76 degrees so it cannot be solved.
By treating the equations for x and y in terms of ø as simultaneous equations, it is possible to define sinø and cosø in terms of x and y: sinø=(x-2y)/5, cosø=(3x-y)/5, tanø=(x-2y)/(3x-y).
And we can write y in terms of x: y(1-2x)=5-2x, so y=(5-2x)/(1-2x).
I think the question should have read: prove that 2x^2+y^2-2xy=5. This is proved by taking (sinø)^2+(cosø)^2=1 and substitute for the trig functions:
((x-2y)^2+(3x-y)^2)/25=(10x^2-10xy+5y^2)/25=(2x^2-2xy+y^2)/5=1 or 2x^2+y^2-2xy=5.