Protobuf.js: How to use TypeScript with reflection-based protos?

[kellycampbell]

protobuf.js version: 6.8.6

I haven't been able to figure out how to use TS with reflected protos based on the documentation in the README. It would be helpful to have a more complete example on the wiki or something.

proto file:

syntax = "proto3";
package hello;

message HelloMessage {
    string message = 1;
}

Commands used to generate ts types:

npx pbjs -t json-module -w commonjs -o bundle.js hello.proto
npx pbjs -t static-module hello.proto | npx pbts -o bundle.d.ts -

I've tried a bunch of different things, but nothing really works as far as having good type info in the editor and still works at runtime. Here's some of the things I've tried:

import * as protobuf from "protobufjs";

// this gets the defs from bundle.d.ts
import { hello } from './bundle';
// but those are way different from the bundle.js and are only good for typing, not runtime
let root = require('./bundle.js');

console.log("root = ", root);

// Finds HelloMessage type ok, but it's type = any
let HelloMessage = root.lookupType("hello.HelloMessage");
let msg = HelloMessage.create({ message: "hello world"});

console.log("msg = ", msg);
console.log(msg.message);

var buffer = HelloMessage.encode(msg).finish();
console.log("encoded = ", buffer);

// reflected $type is undefined
console.log("$type = ", HelloMessage.$type);

// try casting, msg2 has hello.HelloMessage type, but HelloMessage is still type any
let msg2 = HelloMessage.create({ message: "hello world"}) as hello.HelloMessage;
console.log("msg2 = ", HelloMessage.encode(msg2).finish());

// try using the bundle.d.ts definition
// TypeError: Cannot read property 'HelloMessage' of undefined
// because bundle.d.ts doesn't match bundle.js at all
// let msg3 = hello.HelloMessage.create({ message: "hi" });
// console.log("msg3 = ", hello.HelloMessage.encode(msg3).finish());

// try casting the type?
// This works at runtime, but create and encode aren't there at compile time
let HiMsg = <hello.HelloMessage> HelloMessage;
let msg4 = HiMsg.create({ message: "create does not exist on type HelloMessage" });
console.log("msg4 = ", HiMsg.encode(msg2).finish());

Any guidance or better examples would be greatly appreciated.

[dcodeIO]

The following should give you correct type information, unless hello isn't populated on root because it is lowercase, which might be the case:

import { hello } from './bundle';
let root = require('./bundle.js');
let HelloMessage = root.hello.HelloMessage;

If hello isn't there, does it work if you rename the package to Hello or unwrap HelloMessage from it?

[kellycampbell]

Thanks for your example. Unfortunately, it doesn't work at runtime.

let HelloMessage = root.hello.HelloMessage;
                              ^
TypeError: Cannot read property 'HelloMessage' of undefined

Renaming the package to Hello does work. This isn't a very usable option for us though as we can't rename the packages on our existing protos. What did you mean specifically by "unwrap"?

[dcodeIO]

What did you mean specifically by "unwrap"

Basically just putting it into top level.

As a workaround, this might work:

let hello = <root.hello>root.lookup("hello");
hello.HelloMessage ...

[kellycampbell]

I finally had time to figure this out and of course it's really simple. This works:

import { Hello } from './bundle';
let msg = Hello.HelloMessage.create({message: 'hi'});

I couldn't understand where the requirement for capitalization came from. For some reason I thought it was something in TS.

Finally after diving into the code with a debugger, I found that it's some code during fromJSON in root.js. This code was added for #576 commit 99ad9cc08721b834a197d4bbb67fa152d7ad79aa with this comment:

// Add uppercased (and thus conflict-free) nested types, services and enums ...

@dcodeIO: Can you explain what these conflicts are? Is it just the member functions and properties like decode, encode, etc?

In your comment on the other issue you said:

Well, now that you mention it, it's theoretically possible to expose everything uppercase (there are no other uppercased properties) on the reflection objects so that a .d.ts from a static module could be used with reflection. encode, decode are the same anyway.

It seems to me that namespaces don't really need this restriction since they don't really have any methods or properties. What am I missing?

[dcodeIO]

The restriction comes from exposing those properties on the reflection objects directly for convenience.

It seems to me that namespaces don't really need this restriction since they don't really have any methods or properties. What am I missing?

Here's namespace, with properties such as toJSON, addJSON, get, getEnum, add, remove, define, resolveAll, lookup etc.

[kellycampbell]

I guess I was only thinking of the TS namespace, not the reflected JS Namespace. Even so, couldn't those names just be marked as reserved? I doubt many people will have a namespace of "getEnum" or most of those others, and it would be easy enough to throw an error if they did.

[dcodeIO]

Hmm, also extends ReflectionObject with properties such as options, name, parent, resolved, comment, filename, root, fullName. I think I didn't tackle this yet because it doesn't just work for everything.

[kellycampbell]

OK, I was able to put pbjs into an infinte loop by having a namespace named parent, so I see the issue. I'll try to come up with a suitable patch in the next couple of days.

[ppacher]

Hi, any progresss on this? I'm trying to use pbts and a json-module but I always get errors when using google.protobuf.Any ("Cannot read property protobuf of 'undefined'"). If this should be working please point me to an example of how to use TypeScript definitions with a json-module (I need reflection to be working).

Cheers, patrick

[cusher]

For anyone looking for a workaround to this issue (i.e. you are trying to use nested proto packages that have names starting with lowercase letters), what I've done in my project for now is modify Root.prototype._handleAdd (in protobufjs/src/root.js) to change the bit of code that says:

if (exposeRe.test(object.name))
    object.parent[object.name] = object; // expose namespace as property of its parent

to simply remove the if condition. Or you could modify exposeRe to include lowercase letters.

Keep in mind this is a little "risky" for reasons mentioned in the comments above, but if you're reasonably confident your package names won't run into such conflicts, this should hopefully let you get by until whenever this gets addressed with a more robust solution.

(Note: you'll probably want to create a shim to modify _handleAdd on the prototype rather than actually editing root.js itself).

[Akimyou]

May be we need change the README and point this issue for user who use pbts for TypeScript.

This usage is unmatched, be careful about it.

$> pbjs -t json-module -w commonjs -o bundle.js file1.proto file2.proto
$> pbjs -t static-module file1.proto file2.proto | pbts -o bundle.d.ts -

```proto
// awesome.proto
package awesomepackage;
syntax = "proto3";

message AwesomeMessage {
string awesome_field = 1; // becomes awesomeField
}

```ts
import { AwesomeMessage } from "./bundle.js";

// example code
let message = AwesomeMessage.create({ awesomeField: "hello" });
let buffer  = AwesomeMessage.encode(message).finish();
let decoded = AwesomeMessage.decode(buffer);

This is my usage.

pbjs -t json-module -w commonjs -o awesome.json.js awesome.proto && pbjs -t static-module awesome.proto | pbts -o awesome.json.d.ts -

Becareful about package name, as @kellycampbell said

// awesome.proto
package AwesomePackage;
syntax = "proto3";

message AwesomeMessage {
    string awesome_field = 1; // becomes awesomeField
}

```ts
import { AwesomePackage } from './awesome.json.js'

const { AwesomeMessage } = AwesomePackage
let message = AwesomeMessage.create({ awesomeField: "hello" })
let buffer = AwesomeMessage.encode(message).finish()
let decoded = AwesomeMessage.decode(buffer)

How can U do if U have to use an lowercase proto package name?

- You can try as @cusher said
- You can write a wrap function for root
- You can just use type directly

You can write a wrap function for root

```ts
import * as root from './awesome.json.js'

const rootWrap = fixLowercaseProtoPkgNames<typeof root>(root, ['awesomepackage'])
const { AwesomeMessage } = rootWrap.awesomepackage

function fixLowercaseProtoPkgNames<T> (root: T, pkgNames: string[]): T {
  const temp: any = root
  for (let name of pkgNames) {
    if (!temp[name] && temp.nested && temp.nested[name]) {
      temp[name] = temp.nested[name]
    }
  }
  return temp as T
}

You can just use type directly

import * as protobuf from 'protobufjs'
import { ad_event } from './ad_event'

const curRoot = require('./ad_event') as protobuf.Root

type Action = typeof ad_event.example.Action
type StatusMessage = typeof ad_event.example.StatusMessage

const Action: Action = curRoot.lookupEnum('ad_event.example.Action').values as any
const StatusMessage: StatusMessage = curRoot.lookupType('ad_event.example.StatusMessage') as any

console.log(Action.ActionNull)
console.log(StatusMessage.create().toJSON())

Hope can give U some help.