Django-oauth-toolkit: Impossible to swap models

[gbataille]

Hey guys,

So I'm swapping the OAuth models on an application that is already live. All sorts of nice things there, but I'm getting around. I have however 2 comments

• I haven't found much documentation on this subject. I think it's important to mention that since multiple models are linked together, it's a good idea to swap them all if you start to swap one. I started with just changing AccessToken but it created all sorts of complexities

• More importantly, I think for a brand new application, it's not possible to swap the models anymore. Indeed, with the new 1.1 datamodel, AccessToken references RefreshToken through source_refresh_token and RefreshToken references AcessToken through access_token. In your app this is ok because this is done over a few migration that creates the 2 tables with only one FK, and then add the second FK afterwards.
  But on new applications that try to swap the model, it will try to create the full table in one go and fail. I had to manually hack the migration and split the table creation manually.
  --> I don't have a great solution for you, but tables that cross references themselves cyclically is bad news I guess

[phillbaker]

I think this comment: https://github.com/jazzband/django-oauth-toolkit/issues/605#issuecomment-397863421 offers a potential solution via using the run_before attribute in the migrations that create the replacement models?

[gbataille]

Hey @phillbaker,

no run_before does not solve anything. As the data model is crafted today, you cannot deploy one table before the other. What needs to happen is (for example, can be the other way around)

• deploy the AccessToken swapped model WITHOUT the source_refresh_token column
• deploy the RefreshToken swapped model
• add the source_refresh_token column to the swapped AccessToken model table.

You basically need to manually amend the auto-generated migration

This is an extract of what I ended up with

operations = [
    migrations.CreateModel(
        name='AccessToken',
        fields=[
            ('id', models.BigAutoField(primary_key=True, serialize=False)),
            ('expires', models.DateTimeField()),
            ('scope', models.TextField(blank=True)),
            ('created', models.DateTimeField(auto_now_add=True)),
            ('updated', models.DateTimeField(auto_now=True)),
            ('token', models.TextField(unique=True)),
            ('application', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, to=settings.OAUTH2_PROVIDER_APPLICATION_MODEL)),
            ('user', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, related_name='common_accesstoken', to=settings.AUTH_USER_MODEL)),
        ],
        options={
            'abstract': False,
            'swappable': 'OAUTH2_PROVIDER_ACCESS_TOKEN_MODEL',
        },
    ),
    migrations.CreateModel(
        name='RefreshToken',
        fields=[
            ('id', models.BigAutoField(primary_key=True, serialize=False)),
            ('token', models.CharField(max_length=255)),
            ('created', models.DateTimeField(auto_now_add=True)),
            ('updated', models.DateTimeField(auto_now=True)),
            ('access_token', models.OneToOneField(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='refresh_token', to=settings.OAUTH2_PROVIDER_ACCESS_TOKEN_MODEL)),
            ('application', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.OAUTH2_PROVIDER_APPLICATION_MODEL)),
            ('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='common_refreshtoken', to=settings.AUTH_USER_MODEL)),
            ('revoked', models.DateTimeField(null=True)),
        ],
        options={
            'abstract': False,
            'swappable': 'OAUTH2_PROVIDER_REFRESH_TOKEN_MODEL',
        },
    ),
    migrations.AlterUniqueTogether(
        name='refreshtoken',
        unique_together=set([('token', 'revoked')]),
    ),
    migrations.AddField(
        model_name='accesstoken',
        name='source_refresh_token',
        field=models.OneToOneField(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, to=settings.OAUTH2_PROVIDER_REFRESH_TOKEN_MODEL, related_name='refreshed_access_token'),
        preserve_default=False,
    )
]

[agateblue]

Same issue here. Trying to swap those models in my project via:

class Application(oauth2_models.AbstractApplication):
    pass


class Grant(oauth2_models.AbstractGrant):
    pass


class AccessToken(oauth2_models.AbstractAccessToken):
    pass


class RefreshToken(oauth2_models.AbstractRefreshToken):
    pass

raises this error when applying migrations:

oauth2_provider.RefreshToken.access_token: (fields.E304) Reverse accessor for 'RefreshToken.access_token' clashes with reverse accessor for 'RefreshToken.access_token'.
    HINT: Add or change a related_name argument to the definition for 'RefreshToken.access_token' or 'RefreshToken.access_token'.
oauth2_provider.RefreshToken.access_token: (fields.E305) Reverse query name for 'RefreshToken.access_token' clashes with reverse query name for 'RefreshToken.access_token'.
    HINT: Add or change a related_name argument to the definition for 'RefreshToken.access_token' or 'RefreshToken.access_token'.
oauth2_provider.RefreshToken.application: (fields.E304) Reverse accessor for 'RefreshToken.application' clashes with reverse accessor for 'RefreshToken.application'.
    HINT: Add or change a related_name argument to the definition for 'RefreshToken.application' or 'RefreshToken.application'.
users.RefreshToken.access_token: (fields.E304) Reverse accessor for 'RefreshToken.access_token' clashes with reverse accessor for 'RefreshToken.access_token'.
    HINT: Add or change a related_name argument to the definition for 'RefreshToken.access_token' or 'RefreshToken.access_token'.
users.RefreshToken.access_token: (fields.E305) Reverse query name for 'RefreshToken.access_token' clashes with reverse query name for 'RefreshToken.access_token'.
    HINT: Add or change a related_name argument to the definition for 'RefreshToken.access_token' or 'RefreshToken.access_token'.
users.RefreshToken.application: (fields.E304) Reverse accessor for 'RefreshToken.application' clashes with reverse accessor for 'RefreshToken.application'.
    HINT: Add or change a related_name argument to the definition for 'RefreshToken.application' or 'RefreshToken.application'.

[fengyehong]

Have the same problem when trying to swap AccessToken model, don't know how to solve it.

[Alir3z4]

I have the same exact problem. Anyone has been able to find a solution for this yet ?

[Alir3z4]

This is how I've fixed this.

I defined the models as follow:

# oauth/models.py

class Application(models.Model):
    pass


class Grant(models.Model):
    pass


class AccessToken(models.Model):
    pass


class RefreshToken(models.Model):
    pass

Then did makemigrations. Then, inherited the classes from OAuth abstract models:

# oauth/models.py

class Application(oauth2_models.AbstractApplication):
    pass


class Grant(oauth2_models.AbstractGrant):
    pass


class AccessToken(oauth2_models.AbstractAccessToken):
    pass


class RefreshToken(oauth2_models.AbstractRefreshToken):
    pass

Set the swapable models to point to these.

```python

settings.py

OAuth

OAUTH2_PROVIDER_APPLICATION_MODEL = "oauth.Application"
OAUTH2_PROVIDER_ACCESS_TOKEN_MODEL = "oauth.AccessToken"
OAUTH2_PROVIDER_REFRESH_TOKEN_MODEL = "oauth.RefreshToken"
OAUTH2_PROVIDER_GRANT_MODEL = "oauth.Grant"
````

Once again, did makemigrations and all goes good.

[faxioman]

We can conclude that the only working "out-of-the-box" swappable model is the Application model (which is the only covered by documentation).
Probably, would be better to document this behaviour.

[n2ygk]

@Alir3z4 I'm trying the above and am still seeing the E.305 reverse accessor errors:

oauth.AccessToken.application: (fields.E304) Reverse accessor for 'AccessToken.application' clashes with reverse accessor for 'AccessToken.application'.
    HINT: Add or change a related_name argument to the definition for 'AccessToken.application' or 'AccessToken.application'.
etc.

What did I miss? Thanks.

[n2ygk]

Has this broken since release 1.1? https://gitmemory.com/issue/jazzband/django-oauth-toolkit/634/471959496

[n2ygk]

See https://docs.djangoproject.com/en/3.0/topics/db/models/#abstract-related-name
Trying out a fix now...

[ashiksl]

I tried the above steps. Still facing the same problem( Error fields.E305) . Is there any workaround to fix this issue.

[n2ygk]

After spinning around a lot with E304's and so on, I got this to work but I don't believe this is truly a swappable set of models (but maybe it is). What I did:

1. Defined my models that extend the built in ones. I only cared to extend AccessToken but had to make RefreshToken and Application as well due to the cross-references among them.

from django.db import models
from oauth2_provider import models as oauth2_models


class MyAccessToken(oauth2_models.AbstractAccessToken):
    """
    extend the AccessToken model with the external introspection server response
    """
    class Meta(oauth2_models.AbstractAccessToken.Meta):
        swappable = "OAUTH2_PROVIDER_ACCESS_TOKEN_MODEL"

    introspection = models.TextField(null=True, blank=True)


class MyRefreshToken(oauth2_models.AbstractRefreshToken):
    """
    extend the AccessToken model with the external introspection server response
    """
    class Meta(oauth2_models.AbstractRefreshToken.Meta):
        swappable = "OAUTH2_PROVIDER_REFRESH_TOKEN_MODEL"


class MyApplication(oauth2_models.AbstractApplication):
    class Meta(oauth2_models.AbstractApplication.Meta):
        swappable = "OAUTH2_PROVIDER_APPLICATION_MODEL"

1. Manually fixed up a migration to be similar to the one in oauth2_provider to work around circular references from MyAccessToken.source_refresh_token by deferring adding it until later.

2. My 'oauth' app in settings.INSTALLED_APPS:

INSTALLED_APPS = [
    ...
    'oauth2_provider',
    'oauth',
    ...
]

1. Set my models in setting.OAUTH2_PROVIDER_...:

OAUTH2_PROVIDER_APPLICATION_MODEL = "oauth.MyApplication"
OAUTH2_PROVIDER_ACCESS_TOKEN_MODEL = "oauth.MyAccessToken"
OAUTH2_PROVIDER_REFRESH_TOKEN_MODEL = "oauth.MyRefreshToken"
OAUTH2_PROVIDER_GRANT_MODEL = "oauth2_provider.Grant"

1. Started with a totally empty set of migrations and then do a migrate. This basically ends up with the following tables:

...
| oauth2_provider_grant          |
| oauth_myaccesstoken            |
| oauth_myapplication            |
| oauth_myrefreshtoken           |
...

This is really not a swappable model as far as I understand what that means. But it was a way to extend the AccessToken model which I can than override the validator class for:

OAUTH2_PROVIDER = {
    # here's where we add the external introspection endpoint:
    'RESOURCE_SERVER_INTROSPECTION_URL': OAUTH2_SERVER + '/as/introspect.oauth2',
    'RESOURCE_SERVER_INTROSPECTION_CREDENTIALS': (
        os.environ.get('RESOURCE_SERVER_ID','demo'),
        os.environ.get('RESOURCE_SERVER_SECRET','demosecret')
    ),
    'SCOPES': { k: '{} scope'.format(k) for k in OAUTH2_CONFIG['scopes_supported'] },
    'OAUTH2_VALIDATOR_CLASS': 'oauth.oauth2_introspection.OAuth2Validator',  # my custom validator
}

I'm doing the above because I want to use some locally-added claims from my external OAuth2/OIDC AS introspection endpoint. This is kind of non-standard but my AS lets me configure added response fields.

[armando-herastang]

@n2ygk Could you elaborate on the migration file you customized?. I am trying to follow your workaround for this issue but I can't seem to get it done. The migration structure you did would be a good addition to this issue discussion.

[n2ygk]

@armando-herastang sure. Here it is. It's basically the same as the 0001 migration in DOT; I've just added an extra field to MyAccessToken.

# Generated by Django 3.0.3 on 2020-04-03 20:33

from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion
import oauth2_provider.generators

class Migration(migrations.Migration):
    initial = True

    dependencies = [
        migrations.swappable_dependency(settings.OAUTH2_PROVIDER_ACCESS_TOKEN_MODEL),
        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
        migrations.swappable_dependency(settings.OAUTH2_PROVIDER_REFRESH_TOKEN_MODEL),
        migrations.swappable_dependency(settings.OAUTH2_PROVIDER_APPLICATION_MODEL),
    ]

    operations = [
        migrations.CreateModel(
            name='MyApplication',
            fields=[
                ('id', models.BigAutoField(primary_key=True, serialize=False)),
                ('client_id', models.CharField(db_index=True, default=oauth2_provider.generators.generate_client_id, max_length=100, unique=True)),
                ('redirect_uris', models.TextField(blank=True, help_text='Allowed URIs list, space separated')),
                ('client_type', models.CharField(choices=[('confidential', 'Confidential'), ('public', 'Public')], max_length=32)),
                ('authorization_grant_type', models.CharField(choices=[('authorization-code', 'Authorization code'), ('implicit', 'Implicit'), ('password', 'Resource owner password-based'), ('client-credentials', 'Client credentials')], max_length=32)),
                ('client_secret', models.CharField(blank=True, db_index=True, default=oauth2_provider.generators.generate_client_secret, max_length=255)),
                ('name', models.CharField(blank=True, max_length=255)),
                ('skip_authorization', models.BooleanField(default=False)),
                ('created', models.DateTimeField(auto_now_add=True)),
                ('updated', models.DateTimeField(auto_now=True)),
                ('user', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, related_name='oauth_myapplication', to=settings.AUTH_USER_MODEL)),
            ],
            options={
                'abstract': False,
                'swappable': 'OAUTH2_PROVIDER_APPLICATION_MODEL',
            },
        ),
        migrations.CreateModel(
            name='MyAccessToken',
            fields=[
                ('id', models.BigAutoField(primary_key=True, serialize=False)),
                ('token', models.CharField(max_length=255, unique=True)),
                ('expires', models.DateTimeField()),
                ('scope', models.TextField(blank=True)),
                ('created', models.DateTimeField(auto_now_add=True)),
                ('updated', models.DateTimeField(auto_now=True)),
                ('userinfo', models.TextField(blank=True, null=True)),
                ('application', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, related_name='oauth_myaccesstoken_related_app', to=settings.OAUTH2_PROVIDER_APPLICATION_MODEL)),
                # ('source_refresh_token', models.OneToOneField(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='oauth_myaccesstoken_refreshed_access_token', to=settings.OAUTH2_PROVIDER_REFRESH_TOKEN_MODEL)),
                ('user', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, related_name='oauth_myaccesstoken', to=settings.AUTH_USER_MODEL)),
            ],
            options={
                'abstract': False,
                'swappable': 'OAUTH2_PROVIDER_ACCESS_TOKEN_MODEL',
            },
        ),
        migrations.CreateModel(
            name='MyRefreshToken',
            fields=[
                ('id', models.BigAutoField(primary_key=True, serialize=False)),
                ('token', models.CharField(max_length=255)),
                ('created', models.DateTimeField(auto_now_add=True)),
                ('updated', models.DateTimeField(auto_now=True)),
                ('revoked', models.DateTimeField(null=True)),
                ('access_token', models.OneToOneField(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='oauth_myrefreshtoken_refresh_token', to=settings.OAUTH2_PROVIDER_ACCESS_TOKEN_MODEL)),
                ('application', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='oauth_myrefreshtoken_related_app', to=settings.OAUTH2_PROVIDER_APPLICATION_MODEL)),
                ('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='oauth_myrefreshtoken', to=settings.AUTH_USER_MODEL)),
            ],
            options={
                'abstract': False,
                'swappable': 'OAUTH2_PROVIDER_REFRESH_TOKEN_MODEL',
                'unique_together': {('token', 'revoked')},
            },
        ),
        migrations.AddField(
            model_name='MyAccessToken',
            name='source_refresh_token',
            field=models.OneToOneField(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='oauth_myaccesstoken_refreshed_access_token', to=settings.OAUTH2_PROVIDER_REFRESH_TOKEN_MODEL),
        ),
    ]

[armando-herastang]

@n2ygk . Thanks for the quick response, but I am still getting the same issue. I want to do the same thing you did. I want to add a field that I will populate in a custom Oauth2Validator I wrote. I ended up using the migration you suggested, just included my additional field on the MyAccessToken, but I am still getting this when I run python manage.py migrate:

The field oauth2_provider.AccessToken.source_refresh_token was declared with a lazy reference to 'oauth.myrefreshtoken', but app 'oauth' isn't installed.
The field oauth2_provider.Grant.application was declared with a lazy reference to 'oauth.myapplication', but app 'oauth' isn't installed.
The field oauth2_provider.RefreshToken.access_token was declared with a lazy reference to 'oauth.myaccesstoken', but app 'oauth' isn't installed.
The field oauth2_provider.RefreshToken.application was declared with a lazy reference to 'oauth.myapplication', but app 'oauth' isn't installed.

[n2ygk]

Do you have your 'oauth' app in INSTALLED_APPS? Here's my complete:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'cat_manager',  # my app
    'rest_framework_json_api',
    'rest_framework',
    'debug_toolbar',
    'corsheaders',
    'oauth2_provider',
    'oauth',  # my oauth2_provider extension
    'django_filters',
    'django_extensions',
    'simple_history',
    'django_s3_storage',
]

Make sure you also have this in settings (I'm not sure the grant one is needed):

OAUTH2_PROVIDER_APPLICATION_MODEL = "oauth.MyApplication"
OAUTH2_PROVIDER_ACCESS_TOKEN_MODEL = "oauth.MyAccessToken"
OAUTH2_PROVIDER_REFRESH_TOKEN_MODEL = "oauth.MyRefreshToken"
OAUTH2_PROVIDER_GRANT_MODEL = "oauth2_provider.Grant"

[armando-herastang]

@n2ygk . I do, although I do have this app inside a couple of folders, and I do have it in the INSTALLED_APPS like this:

   ...
    'api.apps.oauth'
   ...

And it's name on apps.py:

class OauthConfig(AppConfig):
    name = 'api.apps.oauth'

Then, on my settings file:

OAUTH2_PROVIDER_ACCESS_TOKEN_MODEL= "oauth.MyAccessToken"
OAUTH2_PROVIDER_APPLICATION_MODEL= "oauth.MyApplication"
OAUTH2_PROVIDER_REFRESH_TOKEN_MODEL= "oauth.MyRefreshToken"
OAUTH2_PROVIDER_GRANT_MODEL= "oauth2_provider.Grant"

I notice oauth.My.... here, but I can´t change it to api.apps.oauth because I get:

String model references must be of the form 'app_label.ModelName'.

I'm sorry, but maybe I am missing something. Thanks for the help

[n2ygk]

@armando-herastang

String model references must be of the form 'app_label.ModelName'.

This is a really annoying feature of this stuff that I wasted a lot of time looking at. It looks like a typical string-style module import but if you dig into the code, you'll see it does a simple split(".") and the [0] entry is the app name and the [1] entry is the model name, I believe expected to be in app/models.py. Try moving your oauth extension up to top-level.

[danlamanna]

What is the status of this issue? I'm having a really rough time trying to swap out the ACCESS_TOKEN_MODEL.

Is what @faxioman said correct?

We can conclude that the only working "out-of-the-box" swappable model is the Application model (which is the only covered by documentation).
Probably, would be better to document this behaviour.

If not, is there a set of reproducible steps that allow one to override the access token model?

[armando-herastang]

@danlamanna I am sorry, but I wasn't able to do it either.

[SpencerPinegar]

https://github.com/wq/django-swappable-models

Could someone add this alpha stealth django feature to the project? It seems it would allow these circular references to be handled out of the box and in the future all the models would easily be customizable. Looking for some feedback before this gets undertaken so custom access keys are easier to implement for others in the future.

[SpencerPinegar]

https://github.com/jazzband/django-oauth-toolkit/issues/871