Go: Proposal: reflect: DeepEqual with predicates map

[e11ni]

Summary:
Add one more DeepEqual function (proposed name: DeepEqualWithPredicates) that has one more argument predicates map[string]func(interface{}, interface{}) bool. Predicates is map from type name to compare method for this type. During recursive value traversal if value has type existing in the predicates map then user-defined comparator is used instead of just ==.

Motivation:
There are many cases when you need to compare deep objects but often you cannot use out of the box reflect.DeepEqual. DeepEqual doesn't allow you to compare float numbers with specific precision (with fixed or relative margin) and doesn't allow to skip some mutable fields of structs that are not represent the real state of the object. There already were some proposals to fix at least the second problem (issues/20444), but current proposal solution is more general and covers both cases with float comparisons and class comparison. Behavior of DeepEqual will not be broken by this change (it will reuse DeepEqualWithPredicates with nil predicates map what will lead to the same behavior as without proposed change).

Details:
What should be done to make this change:

• Add of type definition type PredicateMap map[string]func(interface{}, interface{}) bool
• Add one more function:func DeepEqualWithPredicates(x, y interface{}, predicates PredicateMap) bool {
• Update function deepValueEqual to get one more argument predicates PredicateMap and use it for types that in map
• Update function DeepEqual to reuse DeepEqualWithPredicates as DeepEqual became particular case of DeepEqualWithPredicates (with predicates = nil)
• Add tests for DeepEqualWithPredicates
• Add documentation for DeepEqualWithPredicates

Opens:
If you will be ok with this proposal I need some your advices:

• Not sure what key type of predicates map would be better. In text above and in my prototype I use string (I get it using v.Type().Name()), but probably it should be some another type representation, as in case of Pointer v.Type().Name() is empty (workaround: "*" + v.Elem().Type().Name())
• I'm not sure, but probably reflect is not the best place for DeepEqual function as it uses reflection but not provide any reflection abilities to user. I would propose to add package "compare" and move this func to it, but unfortunately it is not a backward compatible change.

[mvdan]

Have you had a look at third-party libraries like https://github.com/google/go-cmp?

[e11ni]

Yes, I have. In particular this library is really powerful, it is great. But on my mind it would be good to have some abilities to extend DeepEqual functionality out of the box. It covers many use cases without any dependencies to third-party.

[ianlancetaylor]

https://golang.org/doc/faq#x_in_std

[e11ni]

Thank you, I understand it. Ok, just want to ensure that you read the proposal. For me it doesn't look as big change that will cost a lot of maintenance. This just provide user more power in usage of functionality that already in go std lib.

and provide key functionality that many Go programs require

Deep equal comparison is really key functionality, but in most cases it is not correct to compare floats on just "==". So why deep equal with predicates cannot be considered as key functionality? Lack of this functionality in std lib cause many third party implementations.

[mvdan]

Note that the cost in adding new APIs isn't only the implementation and maintenance; we're also adding more complexity for Go developers to learn. Adding new APIs, particularly tricky ones like this one, isn't easy.

For example, there have been similar proposals in the past, which weren't accepted: https://github.com/golang/go/issues/20444

I think a proposal to change DeepEqual or add a more powerful version of it needs to be really well thought out. For example:

• What experience do you have with third party alternatives? Why are they not enough?
• What rough percentage of users needing to do deep comparisons require more than just DeepEqual?
• Why is this API (with a map parameter) better than other designs, like go-cmp's with Equal methods?

[dsnet]

Stepping back a bit, I've always found it somewhat odd that DeepEqual was in the reflect package in the first place. The purpose of reflect is to provide a programmatic way to interact with arbitrary Go types and is supposed to provide a dynamic API over the functionality that the Go language allows you to do statically.

DeepEqual doesn't quite fit that purpose. While it obviously uses reflect under the hood for its implementation, the language functionality that it most closely models is that of the == operator in the Go language. However, it's behavior is neither identical (since it descends into pointers) nor a superset of the == operator (since it is extended to work on slices and maps).

As such, I'm not sure extending the functionality of DeepEqual is good idea, especially in the reflect package. As pointed out by @mvdan, what does reflect.DeepEqualWithPredicates provide that the cmp package doesn't?

[e11ni]

What experience do you have with third party alternatives? Why are they not enough?

Actually I do not see any problem to use third-party. Just have observed that some functionality that already in std lib can be extended.

What rough percentage of users needing to do deep comparisons require more than just DeepEqual?

Tough question.. As I already told there are two cases:

1. Case when you have floats in your structure.
2. Struct cannot be compared by deep comparing of all its' fields, and you need to define your own comparator for this struct.

In my practice there were 2 of 3 cases when I cannot use DeepEqual (in our tests actually not in product code) and it was one with floats and one with struct. And one case when I successfully used DeepEqual with data that fits for it.

Why is this API (with a map parameter) better than other designs, like go-cmp's with Equal methods?

1. Equal method design forces user to implement the method Equal for the struct to use DeepEqual. It is not convenient when you want to compare some structs that are defined in third-parly packages. But in that case you still might workaround it with additional code.
2. Equal method design doesn't cover case with floats, to cover it go-cmp uses options.
3. Equal method (if it is implemented) breaks case when you really want to perform complete deep equal. Go-cmp use AvoidEqualMethod to solve this issue.

As pointed out by @mvdan, what does reflect.DeepEqualWithPredicates provide that the cmp package doesn't?

It provides the same that go-cmp but out of the box.

In general, I agree with you that there is no strong need for this change. And if you suspect that this will introduce additional code complexity and difficulties for users, let's close this issue. Thank you for considering this.

[ianlancetaylor]

Thanks.