First find the general solution to y"-3y'=0.
Solve the quadratic r2-3r=0=r(r-3) so r=0 and 3⇒y=Ae0x+Be3x=A+Be3x, where A and B are constants. This is the characteristic solution, yc. To find the particular solution yp we "guess" that yp=ae3x+bsin(x)+ccos(x) where a, b and c are constants. But we already have e3x in yc, so we adjust this to yp=axe3x+bsin(x)+ccos(x).
yp'=3axe3x+ae3x+bcos(x)-csin(x);
yp''=9axe3x+3ae3x+3ae3x-bsin(x)-ccos(x).
yp''-3yp'=9axe3x+3ae3x+3ae3x-bsin(x)-ccos(x)-9axe3x-3ae3x-3bcos(x)-3csin(x),
yp''-3yp'=3ae3x-(b+3c)sin(x)-(c+3b)cos(x).
Therefore 3ae3x-(b+3c)sin(x)-(c+3b)cos(x)=8e3x+4sin(x).
So 3a=8, b+3c=-4, c+3b=0⇒a=8/3; c=-3b, b-9b=-4, b=½, c=-3/2.
The complete solution is yc+yp=A+Be3x+8e3x/3+½sin(x)-3cos(x)/2.