a^2+b^2+3ab=a^2+b^2+2ab+ab=(a+b)^2+ab. If this is a perfect square then we can write (a+b)^2+ab=n^2.
Therefore ab=n^2-(a+b)^2 and n^2=(n-a-b)(n+a+b). This would be satisfied if a=n-a-b, and b=n+a+b.
So b=n-2a and a=-n, so b=n+2n=3n.
(a,b)=(-n,3n). a^2+b^2+3ab=n^2+9n^2-9n^2=n^2.
EXAMPLE: n=2: a=-2, b=6 and 4+36-36=4.