REVISION COMPLETE!
Integrate by parts.
Let u=ax and dv=xe-bx²dx, then du=adx and v=-e-bx²/(2b).
∫udv=uv-∫vdu=-axe-bx²/(2b)+(a/(2b))∫e-bx²dx. The integral is an arbitrary Gaussian integral related to the error function erf. It evaluates to √(π/b) (the proof of this is lengthy and usually involves switching to polar coordinates, so this proof has been omitted) when taken over the whole real domain (-∞,∞). Because of symmetry, if the limits are [0,∞), then the integral (effectively the area beneath the positive half of the bell-shaped curve) is ½√(π/b). You can find details of the proof on-line in text and videos.
So (a/(2b))∫e-bx²dx=(a/(2b))(½√(π/b))=a√(πb)/(4b2).
We still have to evaluate -axe-bx²/(2b) between the same limits, which creates a problem, because, as x→∞, e-bx²→0. However, the exponential factor is much more effective than the linear factor, so this term approaches zero quite rapidly. (When x=1, it's -a/eb, when x=2, it's -2a/e4b, so for large x the term becomes insignificant.).
Therefore the integral is a√(πb)/(4b2).
ADDENDUM
THE ARBITRARY GAUSSIAN INTEGRAL
Let I=∫e-bx²dx, I²=[∫e-bx²dx]²=[∫e-bx²dx][∫e-by²dy]=∬e-b(x²+y²)dxdy. The limits of integration are (-∞,∞). x and y are placeholders, hence the odd appearance of the derivation of the double integral.
Let x=rcosθ, y=rsinθ, then r²=x²+y², tanθ=y/x.
I²=∬e-br²rdrdθ, where the limits are r=(-∞,∞) and θ=[0,2π].
But the function f(r)=re-br²is an odd function, because f(-r)=-f(r), so the positive and negative areas beneath the curve for f(r) would cancel, that is, the integral over the domain (-∞,∞) would be zero. If we take the integral over [0,∞) and double it we would get the absolute area, the sum of the magnitudes of the positive and negative halves of f(r). So the intervals for r and θ would be [0,∞) and [0,π].
To evaluate the double integral start with the inner integral and let u=e-br², so du=-2bre-br²dr, re-br²dr=-du/(2b).
Integrate this and we get -u/(2b)=-e-br²/(2b), which after applying the limits [0,∞) becomes 0-(-1/(2b))=1/(2b).
The outer integral now becomes ∫dθ/2b=θ/(2b) for limits [0,π]=π/(2b)
Double this and we get π/b. I²=∬e-br²rdrdθ, therefore I=√(π/b) or √(πb)/b when rationalised.