JabRef 4.0-dev
Linux 4.4.0-92-generic amd64
Java 1.8.0_131
Steps to reproduce:
Group name in the interface stays the same. If you check the group name by selecting an item and going to its Bibtex source you can see the change there.

When you close and expand the group, the change should also be visible. No idea though why it is not visible immediately. The same issue occurs if you assign an icon or color to the group
Might be related to this:
https://github.com/JabRef/jabref/issues/3112
These bug still exists, I will work on it.
When I press edit group that show org.jabref.model.groups.AbstractGroup#name instead of org.jabref.gui.groups.GroupNodeViewModel#displayName, why? What's the difference between them?
The displayname is a formatted version of name. For example, if name contains latex commands like Hell\'{o} then displayname is the unicode version Hell贸.
This should be fixed in the latest development version. Could you please check the build from http://builds.jabref.org/master/. Thanks! Please remember to make a backup of your library before trying-out this version.
Can confirm this has been fixed in:
JabRef 5.0-dev--snapshot--2019-01-05--master--d281ece4b
Windows 10 10.0 amd64
Java 1.8.0_191
Most helpful comment
These bug still exists, I will work on it.