Javacpp-presets: [question] Any plans about "The Java configuration files" generator

[archenroot]

Hi,

to adopt new larger projects it could be handy to have kind of config files generator, do you plan such tool, or you think it is overkill?

Thx.

[saudet]

You mean like a GUI?

[archenroot]

@saudet no, I mean even command line generator. In the section of creating new presets there is stated:
Finally the configuration file need to be created:

package org.bytedeco.javacpp.presets;

import org.bytedeco.javacpp.annotation.*;
import org.bytedeco.javacpp.tools.*;

@Properties(target="org.bytedeco.javacpp.zlib", value={@Platform(include="<zlib.h>", link="[email protected]"),
    @Platform(value="windows", link="zdll", preload="zlib1")})
public class zlib implements InfoMapper {
    public void map(InfoMap infoMap) {
        infoMap.put(new Info("ZEXTERN", "ZEXPORT", "z_const", "zlib_version").cppTypes().annotations())
               .put(new Info("FAR").cppText("#define FAR"))
               .put(new Info("OF").cppText("#define OF(args) args"))
               .put(new Info("Z_ARG").cppText("#define Z_ARG(args) args"))
               .put(new Info("Byte", "Bytef", "charf").cast().valueTypes("byte").pointerTypes("BytePointer"))
               .put(new Info("uInt", "uIntf").cast().valueTypes("int").pointerTypes("IntPointer"))
               .put(new Info("uLong", "uLongf", "z_crc_t", "z_off_t", "z_size_t").cast().valueTypes("long").pointerTypes("CLongPointer"))
               .put(new Info("z_off64_t").cast().valueTypes("long").pointerTypes("LongPointer"))
               .put(new Info("voidp", "voidpc", "voidpf").valueTypes("Pointer"))
               .put(new Info("gzFile_s").pointerTypes("gzFile"))
               .put(new Info("gzFile").valueTypes("gzFile"))
               .put(new Info("Z_LARGE64", "!defined(ZLIB_INTERNAL) && defined(Z_WANT64)").define(false))
               .put(new Info("inflateGetDictionary", "gzopen_w", "gzvprintf").skip());
    }
}

This file is supposed to be created by hand or generated somehow?.... sorry it is not much clear to me.

I would like to focus on PointCloud, CuDNN and NCCL libraries to be mapped to JavaCPP project, so would like to understand what I am supposed to do.

Thanks for this great project and any hints.

[saudet]

Yes, we need to write that manually. BTW, there are already presets for cuDNN:
https://github.com/bytedeco/javacpp-presets/blob/master/cuda/src/main/java/org/bytedeco/javacpp/presets/cudnn.java

[archenroot]

cuDNN - yes I found it later while examining all repos/presets you maintain....

regarding those config files is could this be automated, I mean I can write some procedure for generation, but if some rules must be defined, is it possible or overkill?

I can imagine something generic could be done with explicit binding file (optional) for specific cases to handle. Similar pattern like XJS for XSD to Java with binding file.

I mean I don't have skillset (yet) to create this config file, it is really not clear to me what to do and how.... maybe it is easier than it looks to me...

[saudet]

Ideally with a good enough "parser" we wouldn't need to write anything, but
it is very hard to do. You are welcome to try though.

[archenroot]

As expected, I was studying over weekend few materials about mapping c and cpp to java, and it is really overkill. I can imagine it will could be good task for some automatic machine or AI to learn the neural network about how to convert those 2.
I experiment on AI frameworks in general, but lets assume....

AI design proposal (naive one :-) ):
There could be some general rules if possible to low down the number of variations (to prevent "infinite" computations) or other hard computation borders.
The framework will try all possible combinations (java code generated to compile) and test always the result (constructed java code) against the test executed at original in source (c++), the results could be:

• cannot compile
• can compile, cannot run
• can run, but result is different than in original
  - can run, result is the same

The 3 failing strategies could be marked somehow based on some kind of percentage either automatically or by the user additionally in a way:

• this combination will never work
• this combination could work only in special case
• this combination will mostly work, but cannot be sure
• this combination will always work

Actually each execution will create those hard borders to make next project generation more accurate and also faster (combination will never work won't be instantiated at all).

What do you think about this way. I understand I talk now about completely new layer on AI to be developed, but I think that could be the way for future, lets keep it as vision only in the moment...

[saudet]

Trying all combinations, you're talking about exponential complexity here...

[archenroot]

Agree, but you have the Parser already where are some "known borders" to limit the complexity. I don't want to convert c++ to java, but generate properly the binding, so we talk about mapping procedures.

It could also work in small blocks, not to generate for whole application all combinations and test one by one. But generate all possible mappings for each method and test in direction "from core to functional border".

This might require parse the c++ code first to get graph method/object dependency, which could be used for incremental JNI mapping generation (from core base object/methods to higher level (dependant on core/lower level objects) and therefore incremental testing. So you first test mapping to few base objects, another iteration goes deeper into the object forest. Therefore going from simple core you have low combinations and with increasing number of those you also limit them as previous rounds restricted some....

The good outcome is that with increasing accuracy of mapping, it could at the same moment generate accurate testing scenarios for any kind of mapped function combination.

NOTE: when some low level objects/methods are not available to start with testing as they are not exposed to external world, they could be cut from source code and smaller version of same application with very limited number of objects/methods could be compiled and JNI generation tested.

I have ~300 CPU and about 700GB ram power with 70TB storage on Hadoop private cluster with Spark and 10GBe interconnects, could give it a try in larger scale. :D

I of course don't want to waste your time, but I am interested in discussion about what is possible and could make sense.

[archenroot]

Also what is the number of items we talk about to generate all combinations for in gross number?
The idea is to generate the full combination map once. But for example with first mapping string to char tested on simple function you will mark dozen of Integer to char as invalid, so they won't be used for compilation and testing next time.

[saudet]

It would be quite experimental, but it's an interesting idea for sure. IMO
though, we would first need to make the C++ parser better, using probably
something like Clang, but there are other options. Would you like to start
with that?

[archenroot]

I don't know Clang much, but will look at it first and come back with some
ideas, or maybe some very basic sample on single object -> 1 method example.

2017-08-21 23:42 GMT+02:00 Samuel Audet [e-mail]:

It would be quite experimental, but it's an interesting idea for sure. IMO
though, we would first need to make the C++ parser better, using probably
something like Clang, but there are other options. Would you like to start
with that?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/bytedeco/javacpp-presets/issues/451#issuecomment-323860855,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAhyKD61WTTuCI-ejnf7RzYM3pcwz6v_ks5safnRgaJpZM4O64Ex
.

[saudet]

Thanks! The problem with Clang though is that we'd need to create presets for that first:
https://github.com/bytedeco/javacpp/issues/51
IDEs (Eclipse, NetBeans, CLion, etc) usually come up with their own parsers in Java and we could use those as well...

[archenroot]

Ok, I have done some research on this and found following interesting articles:

1. Using LUA as "middleware" - just as example of hacker way :-)
   libclang - LiaJIT FFI - msgpack - zeromq-INPROC:// - msgpack - java

1. Use native Java port of Clang - looks promising but I didn't find the code linke anywhere, maybe it is under Oracle proprietary license, I will write the authors....
   https://www.youtube.com/watch?v=EpFJlARXO74
   http://llvm.org/devmtg/2017-03//assets/slides/clank_java_port_of_c_cxx_compiler_frontend.pdf

Update: I found it, it is included in Oracle Developer Studio, so I could give it try to extract and decompile:
http://www.oracle.com/technetwork/server-storage/developerstudio/overview/index.html

1. JNA way - so not much efficient due to overhead
   by utilizing https://github.com/nativelibs4java/BridJ/tree/master/Examples/BasicExample
   there is project https://github.com/pkukielka/jclang

[saudet]

Clank, that's interesting! It must be part of the NetBeans source code base
somewhere...

[archenroot]

Yes, you are right, it is in the Netbeans code... in main tree and under prototypes as well, the person on commits is mostly Maria Dalmatova [e-mail], so I will write here and those 2 guys (creators) email about some additional information if any available (papers, etc.), status and future plans about this.

[archenroot]

No responses so far, but what about SyntaxNet, this is anyway experimenting with possible solutions. What about utilize neural networks?

[saudet]

That sounds like a research project, not anything that looks like it could be useful anytime soon :(

[archenroot]

No response so far... let's leave this open for possible future enhancements, I am pretty busy with some AI project in the moment, so will not have finally much time to work on this, but still the way to go :-)
Thanks for your comments and hints.

[archenroot]

Not sure if right place, but I created initial try for discussion here: https://github.com/jnr/jnr-ffi/issues/144

Let's see, if no response happen I will try to come up with working prototype for basic getuuid function as we discussed here and theoretical documentation... again, I am quite busy now with our startup, but will try do my best.

[saudet]

Thanks, but JNR's goals do not include C++. They are only concerned about C APIs, not C++. They are not interested by C++.

[archenroot]

@saudet aha, shame on them :-)

[saudet]

BTW, Clank is available here now: https://github.com/java-port/clank

[saudet]

BTW, your idea of going about it by trial and error sounds a lot like how cgo does it:
https://golang.org/src/cmd/cgo/doc.go
However, the usual catch: They only try to support C, not C++.