How long should it take for me to lose the weight pregnancy?

The first time I had a baby I gained 55 upon release from the hospital I lost 30 lbs. (mostly water and a ten pound baby and placenta) then over the next 3 months I continued to lose another 20 or 30 lbs. on top of that....with my second which was directly conceived 6 months post first baby, I gained only 25 lbs., and had a 8 1/2 lb baby and lost 20 lbs. immediately, but the last 5 lbs. stuck to me for a while, when I got pregnant the third time nearly 1 year later, I have so far gained 20 lbs and I am 20 weeks along....is it harder for the 3rd pregnancy to lose the weight? My sister had this problem but she is a completely different build etc....I plan on wearing a compression garment and working out post partum at a gym.