It’s complicated to answer, but yes it is… in some situations at least… if doing video too, those need to be in sync.  Using video or not you need to handle where different clock sources can be sending at slightly different rates (or have a different reference for what a rate is)… that case will give you underflow, or overflow in reference to each other.. underflow you might be able to get away with just silence for a frame or partial frame, overflow you need to do something with or a buffer will fill.  There are multiple approaches to deal with this. All of the approaches end up with a balance trying to have the effect barely perceivable, but still effective.  Trimming out packets that are silence (effectively shortening the silence periods) is one way for example… a much more expensive way that has even better sound results would be to insert or remove individual samples (requires some approach to keep things in phase)… On video you have similar issues where you might want to drop or “freeze” for a frame…  Lots of research on the best ways to do this.