Here is another PSA from Gary Illyes of Google. In short, if you serve a 4xx status code with your robots.txt file, then Google will ignore the rules you have specified in that file.
SEO Consultant at G-Squared Interactive focused on Google algorithm update recovery, technical SEO audits, and SEO training. Columnist at Search Engine Land.