Erpnext: Bench Migrate of JSON in Custom App will not update fields

[mariotruss]

Description of the issue

We have a custom "seibertmedia" app in which we save json for DocType configs. Before you have implemented the mutliple phone/email fields, we created similar fields ourselves and added them to Standard DocType "Contact". Now, as you have implemented the features yourself, we migrated our data from our to your fields, but are unable to delete our custom fields. We tried doing so by updating the json file (deleting the field in the json) in our app. Migrated the DocType with our updated json. This does not change the DocType config. They remain visible in the UI.

Steps to reproduce the issue

1. Add custom fields to JSON, export customizations and add them custom app.
2. Export customizations.
   
3. Delete fields from JSON in app again.
   
4. Export customizations again.
5. Bench migrate ERPNext.

Observed result

After bench migrate of JSON.

Expected result

Deleted fields remain deleted.

Conversation with Frappe

@hrwX: If you want to delete the fields, you can delete it from Custom Fields doctype
Mario: Before v12, we were able to delete and add just by changing the JSON. Did that change?
Himanshu: No it wasnt changed, Ill try and check the same on my end
Mario: I mean, it kind of weird to have to add changes on more than one end.
Himanshu: Ill check it why isnt it removing the fields

Additional information

ERPNext: v12.1.8 (master)
Frappe Framework: v12.0.17 (master)

[AndyOverLord]

I am having the same problem with bench migrate not working properly, I have two sites on one bench, after I change the fixture in one site, I ran bench --site all migrate but another site still have the deleted custom fields in customize form UI.

[iMoshi]

Just noticed this as well.

I'm making edits on my local server and testing, and then pushing the changes to my git repo. I then pull the changes on a different server, and notice that I'm not getting any changes of the Doctype on it.

Edit: doctype gets updated after bench restart :)

[Alchez]

I'm able to replicate this on my servers as well.

If custom fields are deleted from a DocType through Customize Form, and then exported, a migrate will not sync and delete the fields from the server.

[AndyOverLord]

Finally, this topic is spotted by someone.

Agreed with @Alchez.

If custom fields are deleted from a DocType through Customize Form, and then exported, a migrate will not sync and delete the fields from the server.

I have a rough idea on how to fix this, for each doctype that has custom fields on it, we should just pull the custom fields out and then compare with the fields in custom_field.json file. If any custom field is not in the custom_field.json, then it should get deleted.

I am not sure if Frappe supports an API to pull out the custom field already, but I would assume it's doable if we comparing custom fields with doctype fields.

Can anyone please confirm my assumption above is doable?

Thank you.

[Alchez]

@AndyOverLord, that seems like a reasonable process, yeah.

Regarding the doability of that change, I would just say that this used to work just fine upto v11, so a controller for the custom fields either already exists or was removed / broken at some point between v11 and the current v12 branch.

I think it might be better to locate and restore that functionality.