Bug
Should be able to use a custom tab label template in a sub tab but not in a parent tab.
Having a mat-tab-group within a mat-tab-group with the child mat-tab-group having mat-tab items with a custom tab template (using ng-template) puts the custom tab content on the parent tab too.
StackBlitz repro: https://stackblitz.com/edit/angular-7njwul
Not possible to use a custom tab label template on child tabs without breaking the parent tabs.
Browsers - all
Material: 5.2.4 (+?)
Angular 5
Nope
According to material guidelines (https://material.io/archive/guidelines/components/tabs.html#tabs-usage) tabs should not be nested.

Agreed that's what the guidelines say, but I'd point out it does say "should" and not "must". It would be nice to see this fixed.
Fortunately, if you use a custom label in both parent and child, it works right. Until it's fixed, it's an acceptable workaround.
@julianobrasil Ah, thanks for the tip Juliano :)
Confirmed that this is still a bug in v9. Relatively low priority since nesting tabs is discouraged by the spec.
From the tabs placement docs in the 2018+ Material Design spec:
Caution Avoid nesting a tab within another tab.
Here's an updated demo of this issue.
This demo shows that this is still a problem with the MDC-based tabs.