This is the expansion of (ax+by+c)(dx+ey+f) where a to f represent numerical constants (integers), obeying the following arithmetic rules:
ad=6, be=-5, cf=-7 (coefficients of x^2, y^2, constant)
The only numbers we have for a, b, c, d, e or f must be in the set {1 1 1 2 3 5 6 7}; no other numbers can be used and the arithmetic rules must always apply. This is because the only factors of 5 are 1 and 5; the only factors of 7 are 1 and 7; but the pairs of factors of 6 are 1 and 6 or 2 and 3. I haven't yet considered the signs plus and minus.
We also have the sums of two products:
ae+bd=13 (coefficient of xy), af+cd=11 (x), ce+bf=36 (y).
The largest of these is 36, and there's only one way of using 4 numbers in the set to get 36 using each one once and that is 7*5+1*1. So from this we can find b, c, e and f.
The following table shows all 8 combinations for these coefficients:
bf+ce
| Ref |
b |
c |
e |
f |
| 1 |
1 |
-7 |
-5 |
1 |
| 2 |
1 |
-5 |
-7 |
1 |
| 3 |
-7 |
1 |
1 |
-5 |
| 4 |
-5 |
1 |
1 |
-7 |
| 5 |
-1 |
7 |
5 |
-1 |
| 6 |
-1 |
5 |
7 |
-1 |
| 7 |
7 |
-1 |
-1 |
5 |
| 8 |
5 |
-1 |
-1 |
7 |
The remaining two coefficients are a and d and the possible pairs of factors are 1 and 6, or 2 and 3. Here's a table to show all 8 possible combinations:
a and d
| Ref |
a |
d |
| 1 |
1 |
6 |
| 2 |
6 |
1 |
| 3 |
-1 |
-6 |
| 4 |
-6 |
-1 |
| 5 |
2 |
3 |
| 6 |
3 |
2 |
| 7 |
-2 |
-3 |
| 8 |
-3 |
-2 |
The next bit is tedious. We need to try out all the combinations of b, c, e and f against all those of a and d! Yes, that's 64 in all. What we're looking for is the sum of the products ae+bd=13. I'll spare you the details and just show you the results. Ref 1 from the first table with ref 8 from the second gives the required result and also satisfies af+cd=11. So we have the values of a to f: -3, 1, -7, -2, -5, 1, and the factorisation is: (-3x+y-7)(-2x-5y+1). This can also be written: (3x-y+7)(2x+5y-1), which is ref 5 from the first table with ref 6 from the second. Combinations in red have complementary signs to those shown in black.