Protobuf: [Javascript] Incorrectly generated import paths when using `google/api` or `google/type`

[dominikeinkemmer]

/cc @zchee

What version of protobuf and what language are you using?
Version: v3.6.0
Language: Javascript

What operating system (Linux, Windows, ...) and version?
Docker Image: node:9

What runtime / compiler are you using (e.g., python version or gcc version)

What did you do?
Steps to reproduce the behavior:
When using imports for google/api/.. or google/type/... the imports generated for javascript resolve to incorrect paths
Example:

import "google/api/annotations.proto";
import "google/type/date.proto";

What did you expect to see
Correctly resolved import paths

What did you see instead?
Wrongly resolved import paths:
generated output (these directories don't exist):

var google_api_annotations_pb = require('../../../../../../google/api/annotations_pb.js');
var google_type_date_pb = require('../../../../../../google/type/date_pb.js');

We resolved these imports by generating these files ourselves (from https://github.com/googleapis/googleapis repository) and copying them to the correct folder but inside of annotations_pb.js another incorrect import is generated so it seems that this problem is carrying through to any imports down the road:

// inside of annotations_pb.js 
var google_api_http_pb = require('../../../../../../google/api/http_pb.js');

We wondered how we could resolve this? Generating the files ourselves and copying them is totally ok but the wrongly generated imports inside these files are an issue we can not resolve on our end unfortunately, so any help is appreciated.

Anything else we should know about your project / environment
We use grpc_tools_node_protoc and grpc_tools_node_protoc_ts plugins to generate typescript and grpc code.

Command we execute:

# Variables:
GRPC_GATEWAY_REPO=github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis
GOOGLE_API_REPO=github.com/googleapis/googleapis
VENDOR_DIR=vendor
PROTOC_OPTION=-I. -I$(VENDOR_DIR) -I$(VENDOR_DIR)/$(GRPC_GATEWAY_REPO) -I$(VENDOR_DIR)/$(GOOGLE_API_REPO)
TYPESCRIPT_OUTPUT=gen/typescript

# Command:
grpc_tools_node_protoc $(PROTOC_OPTION) --js_out=import_style=commonjs,binary:$(TYPESCRIPT_OUTPUT) --grpc_out=$(TYPESCRIPT_OUTPUT) --plugin=protoc-gen-grpc=$(shell which grpc_tools_node_protoc_plugin)

# followed by this command for typescript generation:
protoc $(PROTOC_OPTION) --plugin=protoc-gen-ts=`which protoc-gen-ts` --ts_out=$(TYPESCRIPT_OUTPUT)

[dominikeinkemmer]

I checked the code for the js_generator.cc regarding the import paths. I'm not familiar with C code but if I understood this part correctly then:
(https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/compiler/js/js_generator.cc#L174)

• it first checks for google/protobuf and resovles these imports to google-protobuf/
• it checks how many / are in the import string and resovles it to ./ for 0 and otherwise aggregates as many ../ as there are / in the import name.

As I said, I'm not familiar with C, but is there any chance that inside this method the import path could be compared with the path of the file that has this import?

e.g. "foo/bar/test.proto" has an import for import "foo/bar/api.proto if it would be possible to compare the path of foo/bar/test.proto to the import string, it should be possible to check if they are in the same directory and also return ./ in this case (which might resolve the issue we have I think).

If this is somehow possible I think it might shorten a lot of imports in general and make the generated javascript look cleaner overall?

Sorry if I miss something critical in my assumptions 🙇

[Fadelis]

I having the same issue. Is there any news about this? Maybe there's some module which has these common proto files compiled in js?

[dominikeinkemmer]

@TeBoring any updates for this? :)

[dominikeinkemmer]

@TeBoring I created a PR, maybe you can have a look at it?

[kritzware]

Any updates for this issue? I'm experiencing the same thing.

[dominikeinkemmer]

@kritzware I just updated my PR, please keep an eye on the PR for updates :)

https://github.com/protocolbuffers/protobuf/pull/5703

[clehene]

@damienmg so there's no actual npm package that contains annotations? Typically the language specific libraries have the equivalent of a "google-common-protos" (https://github.com/googleapis/api-common-protos) - in this case it should be https://www.npmjs.com/package/google-proto-files - but I see it doesn't actually contain any generated js

[rmilejcz]

Figured I would share, I'm currently handling this by manually compiling the google proto files via protoc:

protoc \
  --js_out=import_style=commonjs,binary:$JS/rpc-client/protos/google/api  \
  --proto_path=/usr/local/include/google/api  \
  annotations.proto, http.proto

This works well enough, but with the big annoying caveat that annotations.proto ends up throwing the same error as the file that relies on it:

could not resolve ./google/api/http_pb.js from protos/annotation_pb.js

Which I resolve by running this ugly little sed command:

sed -i '.old' 's/\.\/google\/api\/http_pb\.js/\.\/http_pb\.js/g' \ 
$JS/rpc-client/protos/google/apiannotations_pb.js
rm $JS/rpc-client/protos/google/apiannotations_pb.js.old

Transforming require('./google/api/http_pb.js') to require('./http_pb.js')

I store the whole thing as a command in my .bash_profile though I've only ever needed to run it once:

protoc-gen-google(){
  protoc \
    --js_out=import_style=commonjs,binary:$JS/rpc-client/protos/google/api  \
    --proto_path=/usr/local/include/google/api \
    annotations.proto, http.proto

    sed -i '.old' 's/\.\/google\/api\/http_pb\.js/\.\/http_pb\.js/g' \ 
    $JS/rpc-client/protos/google/apiannotations_pb.js

    rm $JS/rpc-client/protos/google/apiannotations_pb.js.old
}

Obviously this would need to be tweaked to fit other use cases and it's a terrible hacky solution but it is doing the trick for now until we have something more robust available to us.

[nathanhleung]

Figured I would share, I'm currently handling this by manually compiling the google proto files via protoc:

protoc \
  --js_out=import_style=commonjs,binary:$JS/rpc-client/protos/google/api  \
  --proto_path=/usr/local/include/google/api  \
  annotations.proto, http.proto

Ended up doing the same thing, except compiled the whole library so I wouldn't have to deal with this problem again:

protoc \
            --proto_path="$protoDir" \
            --proto_path="$protoLibDir" \
            --plugin="protoc-gen-grpc-web=$projectDir/bin/protoc-gen-grpc-web-1.0.6-darwin-x86_64" \
            --js_out=import_style=commonjs:$protoTsOutDir \
            --grpc-web_out=import_style=commonjs+dts,mode=grpcwebtext:$protoTsOutDir \
            $protoDir/*.proto \
            $protoLibDir/google/**/*.proto