Hhvm: Switch extensions from IDL to HNI

[simonwelsh]

This is more a longish-term task list issue to help track migrating extensions from IDL to HNI. I'll try to keep it updated as the migration progresses and include hints/types/recommendations as I come across them.

As part of this I'm also tidying up some of the API where possible, trying to move away from Variant if only a single type's expected/returned and getting everything working with ZendParamMode.

I'm using 512ecad as an example of what to do.

Existing idl files:

• [x] apache (6c21455, @simonwelsh, #1607)
• [x] apc (ffc1187, @nigelchanyk, #3627)
• [x] array (b44deaf & 404343a & 567b2fc7, @nigelchanyk, #3648)
• [x] asio (4b1b667, @sgolemon)
• [x] bcmath (8dfad03, @simonwelsh, #1534)
• [x] bzip2 (a464ad2, @simonwelsh, #1547)
• [x] calendar (96b7814, @tstarling, #2276)
• [x] class (3fe2a77, @simonwelsh, #1625)
• [x] closure (5aa8a2d, @sgolemon, #3980)
• [x] collections (in progress, @sgolemon) Pair: 23c7b9c, Vector: ed39e9ae87960511e608eb74ad475f4cef571c25 , Map & Set: 8b617670b916a2c6fbd64c51cf9b5d5fa791879e
• [x] constants (@sgolemon)
• [x] ctype (6076b94, @simonwelsh, #1612)
• [x] curl (30213e3, @simonwelsh, #1692)
• [x] datetime (78fa029, @nigelchanyk, #3863)
• [x] debugger (3248bb2, @bauerj, #3491)
• [x] domdocument (9fd3df2, @nigelchanyk, #3745)
• [x] error (3cb170ad, @simonwelsh, #1724)
• [x] fb
• [x] file (f9c593e, @Majkl578 + @nigelchanyk, #3911)
• [x] filter (abfcbff, @nigelchanyk, #3705)
• [x] ftp (96b7814, @tstarling, #2276)
• [x] function (9283735, @Majkl578, #3942)
• [x] get text (96b7814, @tstarling, #2276)
• [x] iconv (a578aa7, @stuartcarnie, #2464)
• [x] icu (cef9819, @sgolemon)
• [x] icu_ucnv (439131a, @sgolemon)
• [x] icu_ucsdet (e4011b6, @sgolemon)
• [x] icu_uspoof (1a70b15, @sgolemon)
• [x] image (5fc0301a, @sgolemon)
• [x] imap (df92ec0, @simonwelsh, #2731)
• [x] into (1d1c427, @sgolemon)
• [x] ipc (0378d04, @nigelchanyk, #3922)
• [x] json (f654c6d, @simonwelsh, #1688)
• [x] ldap (544140b, @nigelchanyk, #3931)
• [x] mailparse (fba842b, @bauerj, #3505)
• [x] math (a2aaaf8, @simonwelsh, #1684)
• [x] mb (54dc590, @nigelchanyk, #3675)
• [x] mcrypt (eb1f20b, @simonwelsh, #1760)
• [x] memcache (64d5962e, @simonwelsh, #2795)
• [x] memcached (d882070, @simonwelsh, #2421)
• [x] misc (69014845a478c525417f050340b04b471f515767, @bauerj, #3525)
• [x] mongo (96b7814, @tstarling, #2276)
• [x] mysql (2bcb8e0, @simonwelsh, #2027)
• [x] network (da6328e, @simonwelsh, #2340)
• [x] oauth (96b7814, @tstarling, #2276)
• [x] openssl (c9e36a6, @simonwelsh, #2472)
• [x] options (a792d54, @sgolemon)
• [x] output (cc8db5c4, @simonwelsh #2760)
• [x] pdo (fb1d616, @AlphaStream, #3923)
• [x] phar (1914e07, @simonwelsh, #1548)
• [x] posix (dc520e8, @nigelchanyk, #3933)
• [x] preg (140bed1, @simonwelsh, #1637)
• [x] process (77753ab, @nigelchanyk, #3934)
• [x] reflection (aa173f6, @simonwelsh, #1484)
• [x] server (09362b7, @nigelchanyk, #3948)
• [x] session (e3fcbc6, @simonwelsh, #2332)
• [x] simplexml (ad7eed3, @sgolemon)
• [x] soap (b9af980, @nigelchanyk, #3949)
• [x] socket (5721a9b, @nigelchanyk, #3951)
• [x] spl (80d98a4, @SiebelsTim, #4037)
• [x] sqlite3 (dfb54cd, @nigelchanyk, #3965)
• [x] stream (c08a34d, @nigelchanyk, #3966)
• [x] string (767dbf8, @SiebelsTim + @nigelchanyk, #3721)
• [x] thread (d2ec8bf, @nigelchanyk, #3967)
• [x] thrift (5224d2e, @nigelchanyk, #3968)
• [x] url (beb5d54, @simonwelsh, #1604)
• [x] variable (4d5a3113, @sgolemon)
• [x] xml (5b6b219, @nigelchanyk, #3971)
• [x] xmlreader (d0f7d24, @nigelchanyk, #3972)
• [x] xmlwriter (2263737f5, @aloiret, #3044)
• [x] xsltprocessor (9946ea7, @aloiret, #3059)
• [x] yaml (96b7814, @tstarling, #2276)
• [x] zlib (a9c43f1, @simonwelsh, #1635)

My current observations: (please correct if these are wrong)

• You can't use closures inside a systemlib.php
• Type hint parameters and return types (see the aforementioned commit for examples)
• Feel free to move C++ methods/functions into PHP if it makes sense and is easy to do so.

[staabm]

doing all at once could make it really hard to review..?

[simonwelsh]

Also, would be really boring to do ;)

I was thinking of using this more as organising/tracking and then doing one extension at a time (for instance, I'm currently doing ext/reflection) and keeping a record of things that I've discovered if others want to help.

[staabm]

ah got it now... ;-).

[scannell]

Thanks for working on this @simonwelsh!

[JoelMarcey]

This is awesome. Making sure @ptarjan and @sgolemon are a permanent part of this thread too...

[ptarjan]

Wow, awesome! I'd love if the IDL machinery died.

@jdelong and I were chatting about this, and migrating them is a great time to clean them as they move over.

1. Move anything that can be done in PHP, into PHP (constant declaration, wrapper functions, anything that doesn't call C++)
2. Remove any hand-rolled type checking
3. Check the zend tests for any low hanging incompatibility fruit

[Pomyk]

I started working on mongo, but it will take some time ;)

[awakmu]

Hi all,

What is an IDL? What it's differents with HNI?

And I know about HNI from the last release note: Hhvm Native Interface -> Possibility to call c++ function from PHP.
But I still not sure about these two and can't found any documentation about them.

-Aris

[elgenie]

Thanks for taking this on @simonwelsh . Are you using a script that reads in .idl.json and produces the .php declarations? If so, is that script checked in as part of these conversion commits?

[simonwelsh]

@elgenie No, I'm doing it all by hand, basing of the PHP docs, then what the function actually returns. I've found that the IDL files aren't accurate enough to convert directly, so just deciding to skip the segfaults from mismatching types.

[WizKid]

tools/docskel/docskel.php uses the Zend docs to create a HNI skeleton. At least I think that was the script I used when I created that start of mysqli. But @sgolemon should know more.

[simonwelsh]

@WizKid Sweet, that'll save me copy+pasting the descriptions. Still have to manually do the signatures, given how many edge cases the signature in the docs doesn't cover. Or things like bzopen with completely undocumented (but tested) features (it can take an open file resource instead of a filename).

[WizKid]

@simonwelsh Yea it is far from perfect but it usually give you a good start.

[igorastds]

What's a reason for having Bountysource record for this one? Is it actual? (I can help with porting..)

[scannell]

@igorastds: There is a lot of otherwise relatively thankless work to do here that, once complete, will allow us to clean up the extensions story. It is to encourage community assistance with that.

[dewyatt]

These all-encompassing bug bounties are very off-putting.

[scannell]

@dewyatt, we've provided the feedback to Bountysource that it would be better to have an integrated way to do this without creating 70 additional open issues on the project. Unfortunately there's no support for that at the moment.

[staabm]

@simonwelsh do we need to add XSL https://github.com/facebook/hhvm/commit/72ed6498a7c5d28ca6c6b9a79ba07d00f7f9fb06 to the todo-list?

[simonwelsh]

@staabm Already have. Has put the amount merged back below 15% :(

[staabm]

@simonwelsh gd is missing in this list and was done by sarah in https://github.com/facebook/hhvm/commit/5fc0301a343de843370d811b02bb8d4109fd454b

[simonwelsh]

@staabm This is a list of the IDL files. GD used image.idl.json.

[mcuadros]

@simonwelsh what do you think about port yaml ext to native? instead of ZendCompat? worth it?

[lowjoel]

@simonwelsh #2471 will allow closures in systemlib.

[simonwelsh]

50% of the IDL files are now gone :cake: :smile:

[Orvid]

Just a poke to point out this hasn't been updated in almost a year. Also, I just noticed that this has a $1000 bounty on it on bountysource.

[JoelMarcey]

@sgolemon's got this! She has asio and simplexml in the pipeline; next up is collections.

[simonwelsh]

Only the remainder of collections left :D

[simonwelsh]

Woot woot :)