Does a cake with pudding need to be put in the fridge?

Hi! I made a white cake with vanilla pudding in the middle and then frosted the cake with ercream icing. we left it overnight in a cake caddy. when it was time to be eaten, the pudding taste was a little off. should we have put it in the fridge? Also, can pudding be frozen and thawed at a later time? Lastly, how long is icing good for after being thawed? Thanks.