Scryer-prolog: Remaining conformity issues

[triska]

Thanks to @notoria and #643 being resolved via #647, we can now easily go through the test cases for syntactic ISO conformance (http://www.complang.tuwien.ac.at/ulrich/iso-prolog/conformity_testing), see the video on how to quickly through the cases with Emacs and conformity_testing_copyline.el.

I have now done this:

It is looking good!

I am now going through the test cases again with the most recent commit.

Remaining issues I found:

None.

[UWN]

@triska: Thx for #191 and #192 which is fixed. Where else do you see issues?

[triska]

I used the Emacs definitions to go through all test cases in the order they appear. When I arrive at 52, I get:

?- /*52*/ catch((read(Goal_0),Goal_0),error(Error,_),true),nl.
op(9,xfy,e).
1.2e 3 = e(X,Y).

   Error = permission_error(create,operator,e).
?- caught: error(syntax_error(incomplete_reduction),read_term/3:1)

However, the same query works exactly as expected in a fresh instance of Scryer Prolog:

?- /*52*/ catch((read(Goal_0),Goal_0),error(Error,_),true),nl.
op(9,xfy,e).
1.2e 3 = e(X,Y).

   Goal_0 = op(9,xfy,e).
   X = 1.2, Y = 3.

It appears that there are interferences between the test cases, likely involving operator definitions.

[triska]

Similar issues after 63!

[UWN]

There is no order in the test cases except for cases with /**/.

The comment /**/ is replaced by the last preceding entry not containing /**/ .

In other words, each test case requires a fresh new instance.

Otherwise, test cases at the end would be very hard to reproduce, you would need to go through all preceding cases.

[triska]

Until now, my impression was that the Emacs definitions you gave in https://github.com/mthom/scryer-prolog/issues/52#issuecomment-419738188 were suitable to test the cases one after the other by simply pressing F12! In fact, I thought that was supposed to be the key appeal of these definitions.

Could you please consider making running through the cases easier than having to start a new process for each case? It seems that one way to do this would be to insert additional lines, possibly within HTML comments, that reset the operator table when necessary, as we step through the cases.

If that is not possible, I will automate this in a different way, possibly using Scryer Prolog's library(sgml) to reason about the HTML file.

[UWN]

The key appeal is to permit simple verification of single test cases without any extra time spent on installing a harness. Just think of Scryer: how long would we have waited for testing otherwise. Please keep in mind that before there was a quite ambitious test suite which required quite some effort for its installation and parameterization. Effectively, only two systems took that effort seriously: SICStus (which financed it) and GNU. All other systems effectively ignored it. The entry burden was too high for most. Thus: No harness.

[pmoura]

Thus: No harness.

A test harness capable of testing plain Prolog code, Prolog module code, and Logtalk code, supporting most Prolog compilers, and available for several years now:

https://logtalk.org/tools.html#testing

A Prolog standards conformance suite that can be run with this test harness containing, as of this writing, 147 test sets with +1800 tests, actively maitained:

https://github.com/LogtalkDotOrg/logtalk3/tree/master/tests/prolog

Used by multiple Prolog implementers and responsible of uncovering hundreds of compliance bugs. For a recent example of its usefulness, see e.g.

https://github.com/tau-prolog/tau-prolog

The entry burden was too high for most.

https://logtalk.org/testing.html#testing-backend-prolog-compilers

[triska]

@UWN: Please consider #73, it states /**/ X=[a|b]..

I take it that /**/ means that the last preceding entry not containing /**/ must be placed here, which is #72 reading: op(999,xfy,'|').. By #72, this should yield a permission error. However, #73 is expected to succeed with X=[a|b]., how is this possible? Is this the expected form of the test cases?

[triska]

@mthom: Thank you a lot for working on these issues!

One remaining issue I found is #126.

I have updated the table above with the newest results.

[UWN]

@triska: #73 and other test cases including #72 do make sense quite a lot. They show what happens after #72. That is, can list syntax somehow be affected by that operator declaration or not.

By #72, this should yield a permission error. However, #73 is expected to succeed with X=[a|b]., how is this possible?

72 has been handled in the case before. Now we look what happens thereafter.

[triska]

A remaining issue I found is 213. I have updated the table above with the latest results.

[triska]

I have found no remaining issues. Well done Mark, thank you a lot!