This may be ambiguous. "I have 4 cakes" suggests the current situation now. "I gave you 3 of them" suggests a past action that results in my having 4 cakes right now. Therefore I must have had 7 cakes when I gave you 3 of them. So I have 4 cakes right now.
"I HAD 4 cakes" and "I gave you 3 of them" suggests that I have 1 cake right now.
Similarly, the text "I gave you 2 of them" implies that I originally had 6 and now I have 4, not 2.