How to create a Robots.txt file in ASP.NET Core

A robots.txt file tells the crawlers where they are allowed to scrape the website. You can tell the search engines which links you don’t want to index. You can indicate where a sitemap is located.

A robot’s text file is not mandatory, but it’s recommended because you make crawling easier for robots. For example, maybe your sitemap is located elsewhere than usual.

A robot’s text file is pretty straightforward. Look for an example in this website robot’s file.

User-agent: *
Disallow:

Sitemap: https://programmingcsharp.com/sitemap_index.xml

You mention the crawler’s user agent and what is allowed or disallowed to crawl.

How to configure robots.txt file in ASP.NET Core

In ASP.NET, there are many alternatives to configure your robots.txt file. You can create it by hand and place the file in wwwroot folder. Documentation for writing a robot’s file can be found on the Google website.

Another approach is to use a library to do this for you. I personally use the RobotsTxtCore library.
RobotstxtCore library is a middleware that will convert your C# code to a valid robots.txt file.

First, install the NuGet package:

Install-Package RobotsTxtCore

After that, you should add the service. Look at this example:

services.AddStaticRobotsTxt(b =>
{
    b
        .AddSection(section => section
        .AddUserAgent("Googlebot")
        .Allow("/"))
        .AddSection(section => section
        .AddUserAgent("Bingbot")
        .Allow("/"))

    .AddSitemap("http://yourwebiste.com/sitemap.xml");

    return b;
});

I specify that Google and Big bots are allowed to crawl everything. I also specify the path to the sitemap file.

The final step is to add the middleware in the pipeline:

app.UseStaticFiles(new StaticFileOptions
{
    OnPrepareResponse = ctx =>
    {
        const int durationInSeconds = 60 * 60 * 24 * 7;
        ctx.Context.Response.Headers[HeaderNames.CacheControl] =
            "public,max-age=" + durationInSeconds;
    }
});
app.UseRouting();

app.UseRobotsTxt();

From now on, you can access the path to the robots.txt file.