Blow compressed air in the hole to remove dust and debris. Place a tube of exterior-grade colored silicone caulk into a caulking gun, or use a squeezable tube of caulking compound. Insert the tip of the caulking tube into the hole. Squeeze the caulk into the hole to fill it flush with the existing brick.