User-agent: * Disallow: /share/ Disallow: /oauth/ # Certain social media sites are whitelisted to allow crawlers to access page markup when links to /images are shared. User-agent: Twitterbot Allow: /share/image/ User-agent: facebookexternalhit Allow: /share/image/ User-agent: LinkedInBot Allow: /share/image/