Td: Flying with emscripten

[isopen]

This is a test drive tdsource->tdbin->tdjs
Will be supplemented as it proceeds.

debian 9.3 (amd64)

download sdk package
strings /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | grep GLIBCXX - fantastic (GLIBCXX_3.4.21)
./emsdk update
./emsdk install latest
./emsdk activate latest
source ./emsdk_env.sh
git clone https://github.com/tdlib/td.git
cd td
mkdir build
cd build
emcmake cmake -DCMAKE_BUILD_TYPE=Release ..
emmake make -j 4
...
52% - in process
...
100% - success
emmake make -j 4 (test - ok)

If openssl is not found. Try this flags with emcmake cmake.
(https://packages.debian.org/en/sid/amd64/libssl-dev/filelist)
-DOPENSSL_INCLUDE_DIR=/usr/include
-DOPENSSL_SSL_LIBRARY=/usr/lib/x86_64-linux-gnu/libssl.so
-DOPENSSL_CRYPTO_LIBRARY=/usr/lib/x86_64-linux-gnu/libcrypto.so

emmake make -j 2
[  1%] Building C object sqlite/CMakeFiles/tdsqlite.dir/sqlite/sqlite3.c.o
[  2%] Building CXX object tdutils/CMakeFiles/tdutils.dir/td/utils/port/Fd.cpp.o
In file included from ...td/sqlite/sqlite/sqlite3.c:11465:
In file included from /usr/include/stdio.h:27:
/usr/include/features.h:364:12: fatal error: 'sys/cdefs.h' file not found
#  include <sys/cdefs.h>
Perhaps it "This is probably caused by trying to build a 32 bit binary on a 64 bit system.
(https://www.quora.com/How-do-I-fix-fatal-error-sys-cdefs-h-file-not-found-include-sys-cdefs-h)"
try: apt install gcc-multilib g++-multilib
just in case. it looks like it needs something else.
try:
dpkg --add-architecture i386
apt update
apt openssl:i386 libssl-dev:i386

emcmake cmake .....
emmake make -j 2
In file included from ...td/sqlite/sqlite/sqlite3.c:11465:
/usr/include/stdio.h:79:9: error: unknown type name '__gnuc_va_list'
typedef _G_va_list va_list;
...
In file included from ...td/sqlite/sqlite/sqlite3.c:20258:
In file included from /usr/include/openssl/rand.h:14:
In file included from /usr/include/openssl/ossl_typ.h:19:
/usr/include/openssl/e_os2.h:13:11: fatal error: 'openssl/opensslconf.h' file not found
# include <openssl/opensslconf.h>
13 errors generated.

debian 9.3 (i386)

gcc version 6.3.0 (https://gcc.gnu.org/projects/cxx-status.html#cxx14)
cmake .. - ok
...
-- Performing Test HAVE_STD14
-- Performing Test HAVE_STD14 - Success
...
emcmake cmake .. - err
-- Found ccache
-- Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the system variable OPENSSL_ROOT_DIR (missing:  OPENSSL_LIBRARIES OPENSSL_INCLUDE_DIR) 
-- Performing Test HAVE_STD14
-- Performing Test HAVE_STD14 - Failed
-- Performing Test HAVE_STD1Y
-- Performing Test HAVE_STD1Y - Failed
CMake Error at CMakeLists.txt:104 (message):
  No C++14 support in the compiler.  Please upgrade the compiler.
...
CMakeLists.txt (comment on these lines to see what awaits us further)
#if (NOT HAVE_STD14)
#  message(FATAL_ERROR "No C++14 support in the compiler. Please upgrade the compiler.")
#endif()
Oops. emcmake cmake ..
Looking for pthread_create
-- Looking for pthread_create - not found
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - no
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - not found
...
Generally what is that crap...

I found these lines
td/telegram/td_emscripten.cpp
// Just for testing.
// Will be completly rewritten

[@levlam, @arseny30] Tried to compile via Emscripten? What can you recommend?

[UnblockerUsername]

I can not collect it either too.
Need explanations for professionals.:0
@levlam help please

[levlam]

We will publish build instructions for Emscripten, when they will be production ready. It shouldn't be hard to compile TDLib for Emscripten, but to actually use TDLib in a browser, additional Websocket to TCP proxies are needed. They IPs need to be hard coded to TDLib source code, which is acceptable only for testing purposes.

[UnblockerUsername]

@isopen Collect openssl via emscripten. readme

@levlam Thank you. You can close ticket. I was able to compile td_wasm.js, td_wasm.wasm

[levlam]

I prefer the issue to stay open until we publish our instructions on building TDLib with Emscripten and its usage from browser.

[isopen]

@levlam Thanks
TDLib works in the browser.

[Bannerets]

@isopen How did you build tdlib? I can't link openssl properly.

I used this dockerfile https://github.com/Bannerets/tdlib-binaries/blob/07ee8ac58bd65952935458b51553be300953dea5/misc/docker/emscripten/Dockerfile
Any suggestions?

I got these errors:

WARNING:root:emcc: cannot find library "ssl"
WARNING:root:emcc: cannot find library "crypto"
warning: unresolved symbol: AES_cbc_encrypt
warning: unresolved symbol: AES_encrypt
warning: unresolved symbol: AES_ige_encrypt
...

Full log

[isopen]

@Bannerets Build script Dockerfile.wasm

[jwktje]

@isopen It looks like you're doing some solid research. Just starting out myself and would love to use that build script. Do you have a quick tip on how to use that dockerfile?

Update: I think i'm figuring it out. Never mind for now :)
Update 2: It's failing for me on:
[48%] Building CXX object CMakeFiles/tdcore.dir/td/telegram/AuthManager.cpp.o
The command '/bin/bash -c make -j 4 && rm -rf *' returned a non-zero code: 137
Any specifics I'd need to know?
Update 3: If anyone has the same 137 error as me, I solved it by increasing Docker's resources under Preferences -> Advanced.

[jwktje]

@isopen Would love your help on this. I need to specify to Emscripten to take the following option when building:
-s EXTRA_EXPORTED_RUNTIME_METHODS='["cwrap", "allocate", "intArrayFromString"]'

Where in the dockerfile would I specify this? I tried adding it to the last line like so:
&& emmake make -s EXTRA_EXPORTED_RUNTIME_METHODS='["cwrap", "allocate", "intArrayFromString"]' -j 4

[isopen]

Try it export EMMAKEN_CFLAGS = "-s ..."

[jwktje]

No luck sadly. When console.logging the tdlib JS glue it is not including the extra functions. Any other way you could think of making this work? I'd wanna eventually use this code:

function execute(query) {
        // Create a pointer using the 'Glue' method and the String value
        var ptr  = allocate(intArrayFromString(JSON.stringify(query)), 'i8', ALLOC_NORMAL);

        // Call the method passing the pointer
        var retPtr = tdlib._td_execute(client, ptr);

        // Retransform back your pointer to string using 'Glue' method
        var resValue = Pointer_stringify(retPtr);

        // Free the memory allocated by 'allocate' 
        _free(ptr);  

        return JSON.parse(resValue);
    }

Update: @isopen tried adding the flag in all sorts of variations and locations in the file but can't get emscripten to include the runtime methods. What do you think?

[arseny30]

@jwktje You don't need to explicitly call intArrayFromString. It is much simpler to use cwrap, which is already exported by default.

In our code it looks like this:

    this.td_functions = { 
      td_create: this.TdModule.cwrap('td_create', 'number', []), 
      td_destroy: this.TdModule.cwrap('td_destroy', null, ['number']), 
      td_send: this.TdModule.cwrap('td_send', null, ['number', 'string']), 
      td_execute: this.TdModule.cwrap('td_execute', null, ['number', 'string']),
      td_receive: this.TdModule.cwrap('td_receive', 'string', ['number']), 
      td_set_verbosity: this.TdModule.cwrap('td_set_verbosity', null, [ 
        'number' 
      ]), 
      td_get_timeout: this.TdModule.cwrap('td_get_timeout', 'number', []) 
    };
// ...
   this.td_functions.td_send(this.client, JSON.stringify(query));   

Also in your example it probably should be tdlib_.intArrayFromString not just intArrayFromString.

[jwktje]

@arseny30 Thanks for helping. Sadly no luck yet. cwrap is also undefined for me and not exported.
The docs also mention this i think. Check this page

It also mentions needing to use the flag to include the function. But this doesn't seem to work for me in the dockerfile by @isopen

Did you build your wasm file differently @arseny30 ?

[arseny30]

@jwktje Ouch. Seems like we already added cwrap into CMakeLists.txt, but only in our private repository. Will be fixed with the next release of tdlib.

For now you may try to manually apply this diff to CMakeList.txt and then rebuild tdlib from scratch.

-  set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Os -s ALLOW_MEMORY_GROWTH=1 -s USE_ZLIB=1 -s MODULARIZE=1 -s WEBSOCKET_URL=\"'wss:#'\" -s EXTRA_EXPORTED_RUNTIME_METHODS=\"['FS']\"")
-  set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Os -s ALLOW_MEMORY_GROWTH=1 -s USE_ZLIB=1 -s MODULARIZE=1 -s WEBSOCKET_URL=\"'wss:#'\" -s EXTRA_EXPORTED_RUNTIME_METHODS=\"['FS']\"")
+  set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Os -s ALLOW_MEMORY_GROWTH=1 -s MEMFS_APPEND_TO_TYPED_ARRAYS=1 -s USE_ZLIB=1 -s MODULARIZE=1 -s WEBSOCKET_URL=\"'wss:#'\" -s EXTRA_EXPORTED_RUNTIME_METHODS=\"['FS','cwrap']\"")
+  set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Os -s ALLOW_MEMORY_GROWTH=1 -s MEMFS_APPEND_TO_TYPED_ARRAYS=1 -s USE_ZLIB=1 -s MODULARIZE=1 -s WEBSOCKET_URL=\"'wss:#'\" -s EXTRA_EXPORTED_RUNTIME_METHODS=\"['FS','cwrap']\"")

[Bannerets]

I get [Error : 0 : RSA_size != 256] error after checkDatabaseEncryptionKey. Any suggestions?

I used this tdlibParameters:

    api_id: ...,
    api_hash: '...',
    database_directory: 'db',
    use_message_database: false,
    use_secret_chats: false,
    system_language_code: 'en',
    application_version: '1.0',
    device_model: 'Browser',
    system_version: 'Unknown'

Tested in both Chrome and Firefox.
TDLib compiled with patch by Arseny and dockerfile by @isopen.

[levlam]

Something is completely broken. The error means that built-in public RSA keys of Telegram servers are invalid, which isn't the case. Rebuilding OpenSSL from scratch may help.

[isopen]

When using openssl1.0.2p all oki doki
[ 0][t 0][1541869303.285000086][PublicRsaKeyShared.cpp:23][!Td][&r_rsa.is_ok()] [Error : 0 : RSA_size != 256] -----BEGIN RSA PUBLIC KEY-----
RSA_size == 360
@levlam RSA_new?

[Bannerets]

Found that RSA_size(rsa) equals to 252.

If I comment these lines,

  if (RSA_size(rsa) != 256) {
    return Status::Error("RSA_size != 256");
  }

I receive authorizationStateWaitPhoneNumber update and see websocket connections in console, but get [Session.cpp:1106][!HandshakeActor] Failed to open connection: [Error : 0 : Unknown fingerprints : [172.29.2.0:443] to DcId{2} from [0.0.0.0:0]] warnings, as I expected.

@isopen I can't build tdlib with openssl-1.0.2p, can only with openssl-1.0.2a (seems that it requires many changes). Do you have dockerfile with openssl 1.0.2p? Or even prebuilt binaries?

[jwktje]

I got to the exact error that @Bannerets got stuck on. But I just concluded that I was dumb and doing stuff wrong and gave up haha. But this makes me motivated to keep trying.

[isopen]

@Bannerets openssl-1.0.2p https://github.com/tdlib/td/issues/18#issuecomment-416009253
For some versions, you can ignore fatal errors while building. Because only .a static files are important.

[levlam]

@Bannerets Yes, it will be unable to connect to Telegram DCs without correct RSA keys.

Between, we use OpenSSL 1.1.0 in our build for Emscripten. As a fast test, you can try this prebuilt OpenSSL 1.1.0f:
crypto.zip

[Bannerets]

Unfortunately, with your prebuilt openssl I got this error:

[ 98%] Linking CXX executable td_wasm.js
ERROR: linking module flags 'PIC Level': IDs have conflicting behaviors

I built openssl 1.1.0f myself but faced with the same RSA_size != 256 error.

Added some debugging prints:

#include <openssl/err.h>
//...
Result<RSA> RSA::from_pem(Slice pem) {
  // ...
  // ...
  if (!PEM_read_bio_RSAPublicKey(bio, &rsa, nullptr, nullptr)) {
    return Status::Error("Error while reading rsa pubkey");
  }

  LOG(INFO) << "RSA_size: " << RSA_size(rsa); //---> 252
  LOG(INFO) << "RSA_check_key: " << RSA_check_key(rsa); //---> 0

  unsigned long errcode = ERR_get_error();
  LOG(INFO) << "ERR_get_error: " << errcode; //---> 67764371
  char errstring[256];
  ERR_error_string(errcode, errstring);
  LOG(INFO) << errstring; //---> error:040A0093:rsa routines:RSA_check_key_ex:value missing

  RSA_print_fp(stdout, rsa, 0);
  /*--->
    Public-Key: (2014 bit)
    Modulus:
        2f:70:db:79:75:9c:fe:cf:75:9c:fe:cf:9a:41:da:
        f4:9a:41:da:f4:81:35:a6:f9:81:35:a6:f9:0e:c9:
        ba:97:0e:c9:ba:97:fc:f6:c5:68:fc:f6:c5:68:d6:
        34:86:49:d6:34:86:49:d6:d4:9f:4e:d6:d4:9f:4e:
        ae:97:d9:5c:ae:97:d9:5c:3c:0a:10:d8:3c:0a:10:
        d8:ed:d6:92:6a:ed:d6:92:6a:db:7d:45:7f:db:7d:
        45:7f:65:fc:d6:ff:65:fc:d6:ff:11:df:91:c0:11:
        df:91:c0:97:62:5f:6c:97:62:5f:6c:25:55:69:34:
        25:55:69:34:6b:34:f0:11:6b:34:f0:11:a0:90:19:
        6e:a0:90:19:6e:49:af:7e:b6:49:af:7e:b6:64:94:
        ca:5f:64:94:ca:5f:5b:6d:d2:76:5b:6d:d2:76:b6:
        0e:5d:f6:b6:0e:5d:f6:10:60:7e:f2:10:60:7e:f2:
        23:6c:25:5f:23:6c:25:5f:40:83:a9:67:40:83:a9:
        67:f6:4c:1d:f4:f6:4c:1d:f4:4f:b2:6a:2a:4f:b2:
        6a:2a:ee:e6:4a:d1:ee:e6:4a:d1:85:cd:e7:4a:85:
        cd:e7:4a:6a:bf:59:ba:6a:bf:59:ba:73:e7:13:0f:
        73:e7:13:0f:21:79:25:1f:21:79:25:1f
    Exponent: 65537 (0x10001)
  */

  if (RSA_size(rsa) != 256) {
    return Status::Error("RSA_size != 256");
  }

Correct modulus is:

So, first 4 bytes (C1 50 02 3E) are lost.
But RSA::from_pem(Slice pem) called with correct pem.
It seems, PEM_read_bio_RSAPublicKey(bio, &rsa, nullptr, nullptr) works incorrectly.

I checked this code:

char n_str[] = "AEEC36C8FFC109CB099624685B97815415657BD76D8C9C3E398103D7AD16C9BBA6F525ED0412D7AE2C2DE2B44E77D72CBF4B7438709A4E646A05C43427C7F184DEBF72947519680E651500890C6832796DD11F772C25FF8F576755AFE055B0A3752C696EB7D8DA0D8BE1FAF38C9BDD97CE0A77D3916230C4032167100EDD0F9E7A3A9B602D04367B689536AF0D64B613CCBA7962939D3B57682BEB6DAE5B608130B2E52ACA78BA023CF6CE806B1DC49C72CF928A7199D22E3D7AC84E47BC9427D0236945D10DBD15177BAB413FBF0EDFDA09F014C7A7DA088DDE9759702CA760AF2B8E4E97CC055C617BD74C3D97008635B98DC4D621B4891DA9FB0473047927";

BIGNUM *n_new = BN_new();
BN_hex2bn(&n_new, n_str);

char* number_str = BN_bn2hex(n_new);
LOG(INFO) << "bn2hex: " << number_str;

It outputs FFC109CBFFC109CB5B9781545B9781546D8C9C3E6D8C9C3EAD16C9BBAD16C9BB0412D7AE0412D7AE4E77D72C4E77D72C709A4E64709A4E6427C7F18427C7F1847519680E7519680E0C6832790C6832792C25FF8F2C25FF8FE055B0A3E055B0A3B7D8DA0DB7D8DA0D8C9BDD978C9BDD97916230C4916230C40EDD0F9E0EDD0F9E2D04367B2D04367B0D64B6130D64B613939D3B57939D3B57AE5B6081AE5B6081CA78BA02CA78BA026B1DC49C6B1DC49C7199D22E7199D22E47BC942747BC9427D10DBD15D10DBD153FBF0EDF3FBF0EDFC7A7DA08C7A7DA08702CA760702CA76097CC055C97CC055C3D9700863D970086D621B489D621B4897304792773047927 (first 4 bytes are lost), so, problem is definitely somewhere in openssl BigNum.

[levlam]

@Bannerets, could you also output original string:

char n_str[] = "AEEC36C8FFC109CB099624685B97815415657BD76D8C9C3E398103D7AD16C9BBA6F525ED0412D7AE2C2DE2B44E77D72CBF4B7438709A4E646A05C43427C7F184DEBF72947519680E651500890C6832796DD11F772C25FF8F576755AFE055B0A3752C696EB7D8DA0D8BE1FAF38C9BDD97CE0A77D3916230C4032167100EDD0F9E7A3A9B602D04367B689536AF0D64B613CCBA7962939D3B57682BEB6DAE5B608130B2E52ACA78BA023CF6CE806B1DC49C72CF928A7199D22E3D7AC84E47BC9427D0236945D10DBD15177BAB413FBF0EDFDA09F014C7A7DA088DDE9759702CA760AF2B8E4E97CC055C617BD74C3D97008635B98DC4D621B4891DA9FB0473047927";

LOG(INFO) << "Original: " << sizeof(n_str) << ' ' strlen(n_str) << ' ' << n_str;

BIGNUM *n_new = BN_new();
BN_hex2bn(&n_new, n_str);

char* number_str = BN_bn2hex(n_new);
LOG(INFO) << "bn2hex: " << strlen(number_str) << ' ' << number_str;

[Bannerets]

@levlam

Original: 513 512 AEEC36C8FFC109CB099624685B97815415657BD76D8C9C3E398103D7AD16C9BBA6F525ED0412D7AE2C2DE2B44E77D72CBF4B7438709A4E646A05C43427C7F184DEBF72947519680E651500890C6832796DD11F772C25FF8F576755AFE055B0A3752C696EB7D8DA0D8BE1FAF38C9BDD97CE0A77D3916230C4032167100EDD0F9E7A3A9B602D04367B689536AF0D64B613CCBA7962939D3B57682BEB6DAE5B608130B2E52ACA78BA023CF6CE806B1DC49C72CF928A7199D22E3D7AC84E47BC9427D0236945D10DBD15177BAB413FBF0EDFDA09F014C7A7DA088DDE9759702CA760AF2B8E4E97CC055C617BD74C3D97008635B98DC4D621B4891DA9FB0473047927

bn2hex: 512 FFC109CBFFC109CB5B9781545B9781546D8C9C3E6D8C9C3EAD16C9BBAD16C9BB0412D7AE0412D7AE4E77D72C4E77D72C709A4E64709A4E6427C7F18427C7F1847519680E7519680E0C6832790C6832792C25FF8F2C25FF8FE055B0A3E055B0A3B7D8DA0DB7D8DA0D8C9BDD978C9BDD97916230C4916230C40EDD0F9E0EDD0F9E2D04367B2D04367B0D64B6130D64B613939D3B57939D3B57AE5B6081AE5B6081CA78BA02CA78BA026B1DC49C6B1DC49C7199D22E7199D22E47BC942747BC9427D10DBD15D10DBD153FBF0EDF3FBF0EDFC7A7DA08C7A7DA08702CA760702CA76097CC055C97CC055C3D9700863D970086D621B489D621B4897304792773047927

Original is ok. But, hm, length is the same. I found that there are much more differences. Every four bytes are deleted. diff.

[levlam]

@Bannerets Yes, bytes 8k...8k+3 are always identical to bytes 8k+4...8k+7, which is clearly wrong. It is a clear sign that something is wrong with OpenSSL build.

[levlam]

@Bannerets
It seems that we have used emconfigure ./Configure linux-generic32 or emconfigure ./Configure linux-generic32 no-shared for OpenSSL configuration. Could you try them?

[levlam]

@Bannerets Maybe just change of linux-generic64 to linux-generic32 will resolve the problem.

[Bannerets]

linux-generic64 to linux-generic32 changes nothing, but change of emmake ./Configure to emconfigure ./Configure helped, it works. Big thanks, @levlam.

TDLib works in browser correctly, I was able to sign in.

Final dockerfile
https://github.com/Bannerets/tdlib-binaries/blob/bb632d800f2c19930ceadef74b1333417eba9eac/misc/docker/emscripten/Dockerfile

[isopen]

@Bannerets Building openssl emscription is still rather unstable.

[levlam]

TDLib 1.4.0 includes an example of TDLib usage in a browser. You can also use NPM package tdweb if you don't want to build TDLib yourself.