Those vertical gaps between bricks where there is no mortar let the wall breathe. They're called weep holes but mice and other critters can enter through them. Don't caulk them up! Just push these into them. Click on image for link.
Here is a beehive created between outer walls where just one weep hole was used for entry!