Scryer-prolog: Adding external database to Scryer-prolog

[jukowski]

Hi Mark,

I did some first attempts at integrating a mysql database into Scryer-prolog. The aim is not to import records into prolog’s heap. Instead, it should backtrack through records coming from an external database and unify free variables – basically something like SWI-Prolog’s ODBC library.

Here is what I tried until now. I added my own System-Call for connecting and querying a database. The current problem is that I don’t know where to keep the database handle and result iterator(s).

I first tried adding a new field to the MachineState structure but it seems like the structure gets destroyed and recreated multiple times. I wasn’t sure if my handle and iterator(s) would survive these. Then I tried to add it to the Machine structure and pass it as parameter to about 5-6 other functions until I can access it in system_calls.rs. This works, but I wonder if this is a good idea architecturally. I had to change signatures of methods I don’t fully understand… That’s not a good idea. What would you recommend?

There is a related issue #231. I need neither dynamic linking nor passing prolog data structures without copying… I just need a way to add a new stateful relation, without making a mess out of your code.

[triska]

This sounds great!

However, is there any reason to work at such a low level, i.e., by adjusting the engine code? I think a good architectural guideline is to provide simple interface predicates that allow talking to external processes via a stream (for instance, a network connection), and to formulate as much as possible within Prolog.

See also https://github.com/mthom/scryer-prolog/issues/231#issuecomment-556548042

As I see it, the key selling point of a Prolog implementation is to be able to write Prolog code, and hence as much as possible should also be implemented in Prolog to advertise this key feature.

If necessary, you can keep track of a handle within Prolog using the blackboard to set global variables. See the predicates bb_put/2 (non-backtrackable version) and bb_b_put/2 for the backtrackable version (which restores values on backtracking) in library(iso_ext). Use bb_get/2 to fetch the value.

[jukowski]

I don’t think it’s a good idea to reimplement the communication protocol with mysql in Prolog. Yes, Prolog is a great language – but not because it can do IO. Prolog will have enough opportunities to shine at reasoning on the data I get from mysql ;)

I don’t want to adjust the engine code but I don’t see any other way how I can use existing communication libraries from Rust. It would be great if there were a Plugin trait that one could implement and register it directly as a “system call” instead of adding code to the system_calls.rs file. In this way, the struct implementing the trait could also take care of state management.

If I understand correctly, with bb_put/bb_get I can store constants such as integers or strings but not a Rust database handler object.

BTW, I want to use Scryer-prolog as an embeded library. I added a lib.rs file and that works for me already. That’s why registering a trait would also make more sense for me.

[mthom]

It's true that MachineState often has its contents reset. IndexStore is deliberately consistent, though, and it's available to most downstream Rust functions.

[triska]

Regarding the following statement:

If I understand correctly, with bb_put/bb_get I can store constants such as integers or strings but not a Rust database handler object.

bb_put/2 lets you store any Prolog term, for example:

?- bb_put(x, hello(there)).
true.
?- bb_get(x, T).
T = hello(there).

So, one way to solve your issue could be to represent your handle as a Prolog term, and then to use the built-in Prolog features like bb_put/2 to reason about these handles, in tandem with selected interface predicates that you can also supply. A similar approach could be useful for OpenSSL bindings etc. I found it a good guideline to keep as much as possible in the Prolog part of the implementation so that this part is stressed, tested and improved.

And one additional comment about this statement:

Yes, Prolog is a great language – but not because it can do IO.

Prolog is a great language for IO! In fact, Prolog was designed for natural language processing. Scryer Prolog makes a serious effort to truly support this use case by compactly representing strings as lists of characters, so that they are amenable to efficient and pure processing with library(pio) and Prolog's built-in grammar mechanism, definite clause grammars.

[jukowski]

@mthom: thanks for the hint. It works and does not require changing method signatures. What is your project strategy for supporting this kind of extensions? I suspect you do not want a mysql dependency for your project and don’t want to have core project issues intermingled with extension project issues...

@triska: I will definitely use your advice about bb_get/bb_put for representing handlers on the prolog side. But I don’t feel motivated to reimplement and then maintain a communication protocol to mysql.

What are your reservations about using Rust’s library base? I think the library base is a HUGE asset for the scryer project. Let the boring stuff be implemented in C & Rust. Prolog should be the go-to language for capturing and reasoning about domain knowledge whatever the source. IMHO the project should also have a library interface so that inferred knowledge can be served to e.g. Web applications.

[triska]

@jukowski, regarding the statement:

I don’t feel motivated to reimplement and then maintain a communication protocol to mysql.

Yes, I fully agree! I am certain there are existing libraries to handle the interaction, and those can be used. I mentioned sockets and pipes as one obvious and very simple interface, and since making such streams available to Prolog programs would also be useful for many other programs, and it is in fact already on the TODO list of Scryer Prolog.

The point, and I think this is a good general guideline of development thrust for a Prolog implementation, is to move as much of the stress as possible to the Prolog side. With "stress", I mean that the features that Scryer Prolog actually provides are used.

So for instance, instead of making a low-level change to the core engine just to store states over backtracking, rather make the handle available as a Prolog term and use the available predicates like bb_put/2, because they are already there, and using them will, over time, make their implementation stronger, from which many other programs will also benefit! In addition, having the handle available in Prolog allows further reasoning in a way that fits the overall goal of a Prolog system, i.e., to allow writing nice and elegant Prolog programs.

In that regard, code is like an organic muscle: If it is used, over time it becomes stronger.

We see from experience with other Prolog implementations that, if too much code is moved to the low-level side, then it becomes too complex to maintain, and these systems often never get around to doing what they actually set out to do, which is improving how actual Prolog code runs. This is because the work on low-level components has hindered development of functionality that would have been more important overall, because many (Prolog) programs would benefit from it.

What are your reservations about using Rust’s library base? think the library base is a HUGE asset for the scryer project.

Yes, absolutely. I have zero reservations about Rust's library base, please do use it.

However, please think about how the existing Prolog features can be used to implement what you want, because the Prolog features are what will matter to users of Scryer Prolog. So, if you are considering changes to the core engine just to implement specific features (using a library or not), I highly recommend to think about how such changes can be implemented so that the existing Prolog functionality is stressed.

[jukowski]

We both agree, that it is a bad idea to extend the prolog engine every time you need to fallback to Rust code for implementing certain functionality. We also agree that such fallbacks cannot be avoided e.g. no way I would implement mysql connector in native prolog. I guess the only feasible answer to my question to @mthom: “What is your project strategy for supporting this kind of extensions?” is FFI. That’s also what SWI-Prolog does (https://github.com/SWI-Prolog/packages-odbc).

I don’t think it’s a good idea to mirror SWI-Prolog’s FFI. In fact, the system_call function from system_calls.rs is already a type of FFI. It’s also battle-tested, as it has a non-trivial number of low-level Prolog functions already implemented in it. For now, it’s the best bet. Maybe one could generalize it in a trait that takes two arguments:
1) machine : &mut Machine – which already contains core_repo, indices, call_policy, cut_policy
2) current_input_stream : &mut PrologStream

[mthom]

@jukowski Well, there's always TCP/IP sockets. You could write a client to connect with Scryer, and transmit information through text. There's some overhead involved, but at least it removes the need for further extensions to Scryer. I mention sockets because I'm going to implement a sockets library very soon.