Protobuf: Building iOS Framework with custom public protobuf message classes

[laszloagardi]

I would like to build an iOS framework with protobuf message classes accessible through public headers. I ran into the following problem.

Headers imported like #import "google/protobuf/Any.pbobjc.h" in the protobuf runtime cannot be accessed when added to a framework's public headers as public headers are just copied to the framework Headers directory without keeping the directory structure. These headers are included by the #import "GPBProtocolBuffers.h" directive in the generated Objective-C file.

One could use a Copy Files build phase to copy these headers into the Headers directory of the framework but then the files imported into these files like #import "GPBProtocolBuffers.h" will not be found.

I suggest changing header imports from:
#import "google/protobuf/Any.pbobjc.h"
to:
#import "Any.pbobjc.h"
which still works in the xcode project but also makes building frameworks possible.

[thomasvl]

So there's a secondary problem, and that is the generated code also needing to import the WellKnownTypes or even the base headers themselves. We're trying to figure out how to full support this. Doing imports of just the name can lead to header collisions if some two things end up wanting to use the same file name (which we've see with larger proto trees where things are otherwise grouped in directories to avoid it).

[laszloagardi]

Could the well known types file names could also be prefixed with GPB like all others?
Like: #import "GPBAny.pbobjc.h"

[thomasvl]

For those, yea, that could be an option, but the problem can have in protos needed from other places. Two services should end up with a "user.proto" (or something like that).

[laszloagardi]

I think that is a more general problem and is out of this problem's scope. Services should prefix their classes in general as it is expected in Objective-C. There is a protobuf option for this:
option objc_class_prefix = "PREFIX";
The generated Objective-C files should also be prefixed.

[thomasvl]

That options is documented as only prefixing the symbols and not the files themselves.

If you aren't using CocoaPods/Frameworks, the directory structure that would exist on disk for those duplicate names also works for the includes. It's just when frameworks are used that things start to fail. :(

The limited mangling we already do of the filenames (underscores to camelcase), also causes problems for some folks: https://github.com/google/protobuf/issues/864. So for those build systems, having the output files depend on the content of the file would make that even harder to integrate.

[thomasvl]

Early snapshot (not fully tested), for protobufs being in a framework is in https://github.com/thomasvl/protobuf/tree/framework_includes.

[esteluk]

Is there a suggested workaround for this one in the meantime?

[dnkoutso]

@thomasvl the generated code for some of my .protos does not include the #if #else #endif style import headers only the Protobuf ones do.

Is there a file option that was added to trigger this output for frameworks? Yes, I am using the pre-released version from HEAD master.

[thomasvl]

You should only need the conditional imports when importing the headers that are from the core library, imports between your own protos should be fine since they aren't in the Protobuf framework created by the pods. Can you post a snippet of what isn't working for you?

[dnkoutso]

Sure here it is:

// Generated by the protocol buffer compiler.  DO NOT EDIT!
// source: redacted/common/validation.proto

// This CPP symbol can be defined to use imports that match up to the framework
// imports needed when using CocoaPods.
#if !defined(GPB_USE_PROTOBUF_FRAMEWORK_IMPORTS)
 #define GPB_USE_PROTOBUF_FRAMEWORK_IMPORTS 0
#endif

#if GPB_USE_PROTOBUF_FRAMEWORK_IMPORTS
 #import <Protobuf/GPBProtocolBuffers_RuntimeSupport.h>
#else
 #import "GPBProtocolBuffers_RuntimeSupport.h"
#endif

// This works if it's switched to #import "ValidationProtos_objc/Validation.pbobjc.h"
#import "redacted/common/Validation.pbobjc.h"

etc...

My podspec:

Pod::Spec.new do |s|
  s.name                  = 'ValidationProtos_objc'
  s.version               = '2.20160602.172449-alpha'
  s.summary               = 'Podspec for ValidationProtos_objc protocol buffer bindings.'
  s.homepage              = 'REDACTED'
  s.license               = { :type => 'Proprietary', :text => "REDACTED" }
  s.author                = { 'REDACTED' => '[email protected]' }
  s.source                = { :git => 'REDACTED' }
  s.ios.deployment_target = '8.0'

  s.dependency 'Protobuf', '3.0.0-beta-3'

  s.source_files = 'sources/ValidationProtos_objc/**/*.{pbobjc.h,pbobjc.m}'

  s.header_mappings_dir = 'sources/ValidationProtos_objc'

  s.pod_target_xcconfig = { 'GCC_PREPROCESSOR_DEFINITIONS' => '$(inherited) GPB_USE_PROTOBUF_FRAMEWORK_IMPORTS=1' }

  s.requires_arc = false
end

I had to remove some stuff but can get the idea. You can replace "redacted" with just another folder name, nothing changes semantically.

Also the .proto definition contains package redacted.validation;

[thomasvl]

Sorry, yes, this case isn't handled yet. The current fixes just take care of using protos out of a framework. You used case is the one I was worried about, another pod exposing protos. We'll need to do some work for this.

The easy case is we just need an option to say what framework all the protos will be in, and generate accordingly.

The twist is if someone needs to be able to expose a proto from one framework, and then have another framework expose another proto that depends on the first (to use a Message or Enum for one of their Message's fields). i.e. -

_A.proto_ from _Framework Alpha_ –

message A {
  // ...
}

_B.proto_ from _Framework Beta_ –

import "A.proto"
message B {
  optional A an_a = 1;
  // ...
}

When this generates, A.pbobjc.h/m doens't need anything, but B.pbobjc.h/m will need to know the framework name for A (_Alpha_), but also the framework name for B (_Beta_) (it needs both incase C.proto also exists with B). As you can guess, this can quickly cascade.

[dnkoutso]

Makes sense, I think a file option would help here as you mentioned perhaps something like:

option objc_framework_import_name = "ValidationProtos_objc"

Thanks for re-opening this.

[thomasvl]

re option - the problem is it is data recorded in the proto file that might not be correct for all developers. i.e. - you might want to package it differently from another developer. So one of you two has to edit the file. If the file is part of something like a grpc api, then you both end up editing it to meet your needs, and keeping it up to date as that grpc api updates becomes a pain.

[jcanizales]

you might want to package it differently from another developer

But there's never going to be a _need_ to do that, right?

[thomasvl]

You mean aside from the fact that @dnkoutso is already wanting to expose protos from his framework's public interface to other developers; so it just takes another developer using his framework and doing the same thing?

[jafara]

I have the same problem. Would love to at least get basic support for this. I think the proposal would work for most cases.

[thomasvl]

fyi - draft of the simple case (all generated source in one framework) is in https://github.com/thomasvl/protobuf/tree/third_party_framework.

[thomasvl]

First round of tests on that cl seem to say it covers that simple case, so I've got a PR running through travis to land that.

[dnkoutso]

Thank you @thomasvl! This is a great step.

Is the plan to support dependencies (imports) between protos through passing a protoc command line argument, like a mapping for which proto maps to what framework name?

[thomasvl]

I see two options, no particular order:

1. add another command line option to say where where all imports protos should come from
2. add another option to provide some input file that lets one do full mappings

The first seems like it would fail to be useful in too many cases (if you have two A & B, A depends on B, and both depend on a proto from another framework, it couldn't be expressed).

So that leans me in the direction of the second. So the question becomes how do we express this? listing out the files and the framework they come from? Do we instead hope they have good proto packages and try to use those for the mapping? I don't have high hopes for proto packages, so path based seems like the best choice. The other question is format for the file. TextFormat works, but the map<> formatting of TextFormat isn't that great.

[dnkoutso]

What about a .yaml file or a .json file?

[thomasvl]

protoc (and the generators) don't have anything to parse yaml or json (they only have proto json format parsing), otherwise, yes, those could be better. :(

[dnkoutso]

sounds like it will need to be something custom then...could it be something like along the the lines of proguard mapping.txt file? I am running out of ideas here, so forgive me!

my_first.proto -> value
my_second.proto -> value2

This will require custom parsing but it appears we're already here...

[thomasvl]

Yea, something like that is what I'm currently leaning toward.

Re out of ideas - yea, I'm mentioning it just to see if anyone had something to chime in with. :)

[jcanizales]

You'll have less frameworks than .proto files, so let's invert that map for the sake of people's fingers:

my_framework -> file1.proto, file2.proto
another_framework -> file3.proto, file4.proto

[dnkoutso]

Good point.

[thomasvl]

There's also man 3 regex, but I'm not sure I want to deal with hoping that exists for protoc on all platforms (especially Windows).

[thomasvl]

I should add, I was only pondering regex because I was trying to think of a way to say something like _All the protos in this directory are this framework_. They do come with their own complexity, so it isn't a great option.

[jcanizales]

What about globbing instead of letting people use regex? Every system _should_ have a way to expand programmatically the files that match dir/*.proto.

[thomasvl]

My initial thought/fear with Windows and the whole \ vs / problem. But I might end up facing that no matter what, I'll have to see. Maybe fnmatch() would work; but I don't know that it really has a way to do things really recursively down directories.

[jcanizales]

I thought Windows was fine with _either_ \ or /. But I might be making that up.

[rgosh]

@jcanizales @thomasvl I'd like to add that it seems that the issue is now fixed as the new '3.0.0-beta-3.1' version which contains fix for the problem is now referenced from the main CocoaPods trunk. I successfully built the resulting project.

But I think the same issue prevents one from building the accompanying 'BoringSSL' repo. As I was able to build 'Protobuf' project I got the same header compile errors while building 'BoringSSL':

Its repo has support for CocoaPods as one can see here:

https://boringssl.googlesource.com/boringssl

It has 'version_for_cocoapods_3.0' tag which is referenced from the main CocoaPods trunk spec but one may find that this .podspec was last updated 2 month ago and probably tested with version of CocoaPods older than '1.0.0':

https://github.com/CocoaPods/Specs/commit/6575121ba043047aa5fe87510a57f601dc56db44

Looks like this problem is being addressed by the following PR:

https://github.com/grpc/grpc/pull/6849

[thomasvl]

@mesmerizingr Not sure I'm following your comment. I don't think any part of protobufs references openssl (or boringssl).

[rgosh]

@thomasvl ah, yes, you're right. This's because I'm having it as a requirement for 'gRPC' repo which also requires 'Protobuf'.

[thomasvl]

@jcanizales comment about multiple files per framework makes sense, so I'm planning on going with something like:

framework_name: file.proto, file2.proto
framework2_name: dir/file.proto

We'll allow a framework_name to be listed multiple times to break up lines. And support comments (incase folks want them).

We'll still honor the generate_for_framework_named option just landed to put everything that wasn't listed into that framework.

[thomasvl]

https://github.com/google/protobuf/pull/1695 I think should close this out. Running it through travis now.

[dnkoutso]

@thomasvl thats awesome, I will be able to test it for you in our environment, we have a pretty long list of objc generated code from our protos including multiple dependencies between them.

[thomasvl]

@dnkoutso thanks! Let me know how it goes.

[dnkoutso]

It will be a few days but I will ensure I update this thread once we have a run through.

[gabrielchiron]

I have the same problem, but i have 3.13 en Podfile.
Unity generated all proyect but this is the error :(

In file included from /Users/babri/Documents/ohmybrand/app/app/Pods/Protobuf/objectivec/google/protobuf/Any.pbobjc.m:13: In file included from /Users/babri/Documents/ohmybrand/app/app/Pods/Headers/Private/Protobuf/GPBProtocolBuffers_RuntimeSupport.h:34: In file included from /Users/babri/Documents/ohmybrand/app/app/Pods/Headers/Private/Protobuf/GPBProtocolBuffers.h:44: /Users/babri/Documents/ohmybrand/app/app/Pods/Headers/Private/Protobuf/GPBWellKnownTypes.h:44:10: fatal error: 'google/protobuf/Any.pbobjc.h' file not found #import "google/protobuf/Any.pbobjc.h" ^ 1 error generated.