Shale swelling is the same thing as clay swelling; when a fluid enters a reservoir that contains certain types of clay (such as smectite), the clays will expand, thereby decreasing the porosity and permeability of the reservoir in that aera and dmagaing the formation.
A good paper on the topic: Swelling Clays in Hydrocarbon Reservoirs: the Bad, the Less Bad,
and the Useful Z. (John) Zhou and David H.S. Law, Alberta Research Council, Edmonton, Alberta,