Start by using the conservation of momentum equation between two particles:
m1v1i+m2v2i=m1v1f+m2v2f, or
(1) m1dv1=-m2dv2 where m1 and m2 are the masses of A and B respectively, and dv1 and dv2 the change in velocity of the particles.
Then we have coefficient of restitution, e=relative velocities of particles after collision/rel velocities before collision.
When one object is immovable (2) e=|final velocity/initial velocity| with an obvious change of direction.
What we'll get when we apply these equations is two simultaneous equations from which we can calculate the final velocities of the particles. From that we'll know at what speed B will strike the wall and hence we can calculate what its speed will be after it bounces off the wall.
From (1): m1=0.75, m2=0.6, dv1=v1f-v1i=v1f-6, where v1f is the final velocity of m1 after colliding with m2 and v1i=initial velocity of m1=6. dv2=v2f-v2i=v2f-(-4)=v2f+4 (we treat v1i as positive so v2i=-4 because m2 is travelling in the opposite direction. We need to be careful about direction of velocity. It's so easy to make a mistake, so I may need to go over this later to make sure I got it right!)
Therefore, 0.75(v1f-6)=-0.6(v2f+4); 0.75v1f-4.5=-0.6v2f-2.4; 0.75v1f+0.6v2f=4.5-2.4=2.1.
Now to apply e=4/5=-(v1f-v2f)/(v1i-v2i)=-(v1f-v2f)/(6-(-4))=(v1f-v2f)/10. (The minus indicates the change in direction.)
So, v2f-v1f=10*4/5=8. Therefore v1f=v2f-8, so we can substitute in the other equation:
0.75(v2f-8)+0.6v2f=2.1; 0.75v2f-6+0.6v2f=2.1; 0.75v2f+0.6v2f=8.1. (All this needs checking later.)
So v2f=8.1/1.35=6, making v1f=6-8=-2.
When particle B with velocity 6 strikes the wall, 2/3=|v2f'|/6, where v2f' is the recoil velocity. So |v2f'|=4 and v2f'=-4 because, as in the first collision with A, the particle changes direction. Since v1f=-2, B will catch up with A because its velocity is greater.
Now we apply the conservation of momentum and coefficient of restitution equations again making the relevant substitutions for the initial velocities.
We can write the momentum equation using v1i=-2 and v2i=-4. We have to calculate new values for v1f and v2f.
0.75(v1f+2)=-0.6(v2f+4); 0.75v1f+0.6v2f=-1.5-2.4=-3.9.
e=4/5=(v2f-v1f)/(-2+4)=(v2f-v1f)/2; v2f-v1f=8/5=1.6. So v2f=1.6+v1f.
0.75v1f+0.6(1.6+v1f)=-3.9; 0.75v1f+0.96+0.6v1f=-3.9; 1.35v1f=-4.86, v1f=-4.86/1.35=-3.6 m/s.
So v2f=1.6-3.6=-2 m/s.
Particle A is moving in the opposite direction at 3.6 m/s while B is moving in its original direction at 2 m/s.
(I've checked the workings and they seem to be correct.)
