Steps to reproduce:

I am not familiar with Webserver for Chrome but your problems still sounds familiar to me. The route /counter is a client-side route. Your webserver does not know about it. This problem applies to any SPA (Single Page App) framework (e.g. Angular).
You have to configure your webserver so that all e.g. extension-less requests lead to returning index.html. There, the SPA framework will take control and do its client-side routing logic. A quick look in Webserver for Chrome showed me that it has a setting for that:

See also this GitHub issue in the Webserver for Chrome sample.
Hope this helps.
BTW - here is an example for a SPA-configuration of a professional webserver (nginx). Maybe it helps to understand the problem and points you to possible solutions.
Thank you very much Rainer. The solution worked.
Thanks for answering @rstropek - this is really helpful! "How to host on a static-files webserver" would make a great page for https://learn-blazor.com/ :)
@SteveSandersonMS Great idea, here it is
Most helpful comment
@SteveSandersonMS Great idea, here it is