Bevy: iOS Support

[cart]

It should be possible to run Bevy Apps on iOS

[MichaelHills]

Hi, what’s involved in setting up Bevy on iOS? My understanding is that it uses winit which should already support iOS?

[MichaelHills]

I followed this guide to just get a Rust lib working from an ios project. Then I tried to substitute a bevy example app in. During cargo lipo --release it fails to build bevy-glsl-to-spirv.

It looks like there's no prebuilt binary in that repo so it tries to build glslang and then cmake fails

  -- Check for working C compiler: /usr/bin/cc -- broken
...
Building C object CMakeFiles/cmTC_97014.dir/testCCompiler.c.o
      /usr/bin/cc   -fPIC -m64 -m64 -mios-simulator-version-min=7.0 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator13.2.sdk -fembed-bitcode  -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk   -o CMakeFiles/cmTC_97014.dir/testCCompiler.c.o   -c /Users/mikeh/rust-ios-example/rust/target/x86_64-apple-ios/release/build/bevy-glsl-to-spirv-51cc402e67486116/out/build/CMakeFiles/CMakeTmp/testCCompiler.c
      clang: warning: using sysroot for 'MacOSX' but targeting 'iPhone' [-Wincompatible-sysroot]

Apparently that last line is a key sign something is off. I take it the build for glslang doesn't work properly to target aarch64-apple-ios or x86_64-apple-ios.

[MichaelHills]

I tried a couple of things, but I'm new to rust and new to cross-compiling to iOS, though slowly figuring things out. The approach of spawning the glslValidator process probably isn't going to fly on iOS. MoltenVK apparently can build the iOS static lib libglslang.a so I considered trying to interface with that. The bevy-glsl-to-spirv repo says TODO to switch to shaderc-rs so I figured I'd have a crack at that but had trouble cross-compiling for iOS.

I ended up stubbing out the code in crates/bevy_render/src/shader/shader.rs just to see how far I could get. I also had to disable the optional bevy audio as it didn't compile either (forgot why).

I managed to finally get something to build, on the simulator gfx-rs / gfx-backend-metal failed with Target OS is incompatible: library was not compiled for the simulator so I ran it on my phone instead to which I got

EventLoop cannot be run after a call to UIApplicationMain on iOS.

Stopping for tonight. Here's my diff so far to get where i got to...

diff --git a/Cargo.toml b/Cargo.toml
index 90b2368d..3254fd6c 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -13,7 +13,8 @@ readme = "README.md"
 exclude = ["assets/**/*", "tools/**/*", ".github/**/*", "crates/**/*"]

 [features]
-default = ["bevy_audio", "bevy_gltf", "bevy_wgpu", "bevy_winit", "png", "hdr", "mp3"]
+#default = ["bevy_audio", "bevy_gltf", "bevy_wgpu", "bevy_winit", "png", "hdr", "mp3"]
+default = ["bevy_gltf", "bevy_wgpu", "bevy_winit", "png", "hdr"]
 profiler = ["bevy_ecs/profiler", "bevy_diagnostic/profiler"]
 wgpu_trace = ["bevy_wgpu/trace"]

@@ -22,10 +23,10 @@ png = ["bevy_render/png"]
 hdr = ["bevy_render/hdr"]

 # Audio format support (MP3 is enabled by default)
-mp3 = ["bevy_audio/mp3"]
-flac = ["bevy_audio/flac"]
-wav = ["bevy_audio/wav"]
-vorbis = ["bevy_audio/vorbis"]
+#mp3 = ["bevy_audio/mp3"]
+#flac = ["bevy_audio/flac"]
+#wav = ["bevy_audio/wav"]
+#vorbis = ["bevy_audio/vorbis"]

 serialize = ["bevy_input/serialize"]

@@ -56,7 +57,7 @@ bevy_ui = { path = "crates/bevy_ui", version = "0.1" }
 bevy_window = { path = "crates/bevy_window", version = "0.1" }

 # bevy (optional)
-bevy_audio = { path = "crates/bevy_audio", optional = true, version = "0.1" }
+#bevy_audio = { path = "crates/bevy_audio", optional = true, version = "0.1" }
 bevy_gltf = { path = "crates/bevy_gltf", optional = true, version = "0.1" }
 bevy_wgpu = { path = "crates/bevy_wgpu", optional = true, version = "0.1" }
 bevy_winit = { path = "crates/bevy_winit", optional = true, version = "0.1" }
diff --git a/crates/bevy_render/Cargo.toml b/crates/bevy_render/Cargo.toml
index ea29b856..53fe2af4 100644
--- a/crates/bevy_render/Cargo.toml
+++ b/crates/bevy_render/Cargo.toml
@@ -24,7 +24,7 @@ bevy_window = { path = "../bevy_window", version = "0.1" }

 # rendering
 spirv-reflect = "0.2.3"
-bevy-glsl-to-spirv = "0.1.7"
+#bevy-glsl-to-spirv = { path = "/Users/mikeh/repos/glsl-to-spirv" } #"0.1.7"
 image = { version = "0.23", default-features = false }

 # misc
diff --git a/crates/bevy_render/src/shader/shader.rs b/crates/bevy_render/src/shader/shader.rs
index f963c0c9..dd448e74 100644
--- a/crates/bevy_render/src/shader/shader.rs
+++ b/crates/bevy_render/src/shader/shader.rs
@@ -1,6 +1,6 @@
 use super::ShaderLayout;
 use bevy_asset::Handle;
-use bevy_glsl_to_spirv::compile;
+//use bevy_glsl_to_spirv::compile;
 use std::{io::Read, marker::Copy};

 /// The stage of a shader
@@ -11,25 +11,26 @@ pub enum ShaderStage {
     Compute,
 }

-impl Into<bevy_glsl_to_spirv::ShaderType> for ShaderStage {
-    fn into(self) -> bevy_glsl_to_spirv::ShaderType {
-        match self {
-            ShaderStage::Vertex => bevy_glsl_to_spirv::ShaderType::Vertex,
-            ShaderStage::Fragment => bevy_glsl_to_spirv::ShaderType::Fragment,
-            ShaderStage::Compute => bevy_glsl_to_spirv::ShaderType::Compute,
-        }
-    }
-}
+//impl Into<bevy_glsl_to_spirv::ShaderType> for ShaderStage {
+//    fn into(self) -> bevy_glsl_to_spirv::ShaderType {
+//        match self {
+//            ShaderStage::Vertex => bevy_glsl_to_spirv::ShaderType::Vertex,
+//            ShaderStage::Fragment => bevy_glsl_to_spirv::ShaderType::Fragment,
+//            ShaderStage::Compute => bevy_glsl_to_spirv::ShaderType::Compute,
+//        }
+//    }
+//}

 fn glsl_to_spirv(
     glsl_source: &str,
     stage: ShaderStage,
     shader_defs: Option<&[String]>,
 ) -> Vec<u32> {
-    let mut output = compile(glsl_source, stage.into(), shader_defs).unwrap();
+    //let mut output = compile(glsl_source, stage.into(), shader_defs).unwrap();
     let mut spv_bytes = Vec::new();
-    output.read_to_end(&mut spv_bytes).unwrap();
-    bytes_to_words(&spv_bytes)
+    //output.read_to_end(&mut spv_bytes).unwrap();
+    //bytes_to_words(&spv_bytes)
+    spv_bytes
 }

 fn bytes_to_words(bytes: &[u8]) -> Vec<u32> {

[MichaelHills]

Looks like cross-compiling iOS stuff with cmake-rs is currently broken https://github.com/alexcrichton/cmake-rs/issues/96 so that's probably the source of some of my woes

[MichaelHills]

Success! I managed to get the 3d_scene demo loading in portrait mode (albeit a bit warped) on my phone.

• I pinned cmake-rs to 0.1.43 because that's the version which had the working iOS support
• Dropped bevy-glsl-to-spirv in favour of using shaderc-rs directly from crates/bevy_render/src/shader/shader.rs
• Created a main.swift to call out to my bevy_main() function directly which was the 3d_sprite main function renamed like this

#[no_mangle]
pub extern fn bevy_main() {

In xcode I had to add the static lib / header and search paths as per this site and I also had to link against lc++. Basically atm I build my "game" as a static lib and then call out to it manually from in an ios xcode project.

Trying to figure out how to get it into landscape mode now.

[cart]

Whoa that's huge! Thanks for the hard work here. People are going to love this!

[MichaelHills]

No worries. :) When I saw the post on Bevy on HN I thought it looked pretty awesome. I previously had started making a PC game in UE4 but life got busy. Decided I wanted to prototype it out as a mobile game instead and learn some Rust/ECS while I'm at it.

How should I go about getting stuff landed? I think the switch to shaderc-rs in general can be done now (tested a couple of examples still work). Did you want that to be done inside of bevy-glsl-to-spirv? I've currently just implemented it directly in crates/bevy_render/src/shader/shader.rs while hacking this all up.

In order to get iOS integration to "just work" the following issues need to be solved properly:

• Land using shaderc-rs in bevy
• Get that iOS support commit in cmake-rs fixed up and re-landed there
• In shaderc-rs I had to edit CMakeLists.txt in the glsl and shaderc submodules (see below), so maybe... get those landed there?
• Figure out why coreaudio-sys v0.2.5 panics during compile (that was why I had to disable audio)

shaderc

diff --git a/glslc/CMakeLists.txt b/glslc/CMakeLists.txt
index acf6fb0..4ec0067 100644
--- a/glslc/CMakeLists.txt
+++ b/glslc/CMakeLists.txt
@@ -40,6 +40,7 @@ shaderc_add_asciidoc(glslc_doc_README README)

 if(SHADERC_ENABLE_INSTALL)
   install(TARGETS glslc_exe
+    BUNDLE DESTINATION ${CMAKE_INSTALL_BINDIR}
     RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR})
 endif(SHADERC_ENABLE_INSTALL)

glslang

diff --git a/StandAlone/CMakeLists.txt b/StandAlone/CMakeLists.txt
index 2cf2899c..f84fdd2b 100644
--- a/StandAlone/CMakeLists.txt
+++ b/StandAlone/CMakeLists.txt
@@ -50,11 +50,13 @@ endif(WIN32)

 if(ENABLE_GLSLANG_INSTALL)
     install(TARGETS glslangValidator EXPORT glslangValidatorTargets
+            BUNDLE DESTINATION ${CMAKE_INSTALL_BINDIR}
             RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR})
     install(EXPORT glslangValidatorTargets DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake)

     if(ENABLE_SPVREMAPPER)
         install(TARGETS spirv-remap EXPORT spirv-remapTargets
+            BUNDLE DESTINATION ${CMAKE_INSTALL_BINDIR}
             RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR})
         install(EXPORT spirv-remapTargets DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake)
     endif()

Also here's the change to shader.rs

diff --git a/crates/bevy_render/src/shader/shader.rs b/crates/bevy_render/src/shader/shader.rs
index f963c0c9..2e710cd0 100644
--- a/crates/bevy_render/src/shader/shader.rs
+++ b/crates/bevy_render/src/shader/shader.rs
@@ -1,7 +1,6 @@
 use super::ShaderLayout;
 use bevy_asset::Handle;
-use bevy_glsl_to_spirv::compile;
-use std::{io::Read, marker::Copy};
+use std::{marker::Copy};

 /// The stage of a shader
 #[derive(Hash, Eq, PartialEq, Copy, Clone, Debug)]
@@ -11,14 +10,14 @@ pub enum ShaderStage {
     Compute,
 }

-impl Into<bevy_glsl_to_spirv::ShaderType> for ShaderStage {
-    fn into(self) -> bevy_glsl_to_spirv::ShaderType {
-        match self {
-            ShaderStage::Vertex => bevy_glsl_to_spirv::ShaderType::Vertex,
-            ShaderStage::Fragment => bevy_glsl_to_spirv::ShaderType::Fragment,
-            ShaderStage::Compute => bevy_glsl_to_spirv::ShaderType::Compute,
-        }
-    }
+impl Into<shaderc::ShaderKind> for ShaderStage {
+   fn into(self) -> shaderc::ShaderKind {
+       match self {
+           ShaderStage::Vertex => shaderc::ShaderKind::Vertex,
+           ShaderStage::Fragment => shaderc::ShaderKind::Fragment,
+           ShaderStage::Compute => shaderc::ShaderKind::Compute,
+       }
+   }
 }

 fn glsl_to_spirv(
@@ -26,10 +25,19 @@ fn glsl_to_spirv(
     stage: ShaderStage,
     shader_defs: Option<&[String]>,
 ) -> Vec<u32> {
-    let mut output = compile(glsl_source, stage.into(), shader_defs).unwrap();
-    let mut spv_bytes = Vec::new();
-    output.read_to_end(&mut spv_bytes).unwrap();
-    bytes_to_words(&spv_bytes)
+    let mut compiler = shaderc::Compiler::new().unwrap();
+    let mut options = shaderc::CompileOptions::new().unwrap();
+    if let Some(shader_defs) = shader_defs {
+        for def in shader_defs.iter() {
+            options.add_macro_definition(def, None);
+        }
+    }
+
+    let binary_result = compiler.compile_into_spirv(
+        glsl_source, stage.into(),
+        "shader.glsl", "main", Some(&options)).unwrap();
+
+    binary_result.as_binary().to_vec()
 }

 fn bytes_to_words(bytes: &[u8]) -> Vec<u32> {

[MichaelHills]

Stumbled across this issue in CMake https://gitlab.kitware.com/cmake/cmake/-/issues/20956 which suggests that the whole BUNDLE DESTINATION thing might be due to some CMake default that doesn't make sense when building libs, and I don't need to try and get BUNDLE DESINATION added to shaderc and glsl.

[naithar]

Basically atm I build my "game" as a static lib and then call out to it manually from in an ios xcode project.

Godot works in same way, though it implements it's own main which gets used by iOS application.

Obj-C can be used too, but this would require only main.mm file like this:

@import UIKit;

extern "C" void bevy_main(void);

int main(int argc, char * argv[]) {
    bevy_main();
}

And additional values in Other C Flags in Build Settings: -fmodules -fcxx-modules.

I've tested with 2d -> sprite.rs example, but got empty window.
I guess asset loading would require some work for iOS, but correct (?) texture handle was returned Handle<Texture>(82f42690-2c58-4843-a88a-aa066583f97d), so I'm not sure.

• Figure out why coreaudio-sys v0.2.5 panics during compile (that was why I had to disable audio)

This PR (https://github.com/RustAudio/coreaudio-sys/pull/33) seems to be addressing this issue, but I wasn't able to make it work even with this changes. But maybe it's just me as I don't really have much experience with rust.

[MichaelHills]

@naithar oh cool glad to see someone else getting it working too. Yeah I didn’t get the sprite example working, I also assumed it was related to asset loading but I haven’t looked into that yet.

Interesting I’ll try pulling that audio PR later to see if it works.

[MichaelHills]

By the way were you able to get the simulator working? I’m still getting the issue with Target OS is incompatible: library was not compiled for the simulator but supposedly there should be simulator metal support https://github.com/gfx-rs/gfx/pull/2873 so not sure yet what the problem with that is.

[naithar]

No, I've got some error about unsupported architecture from dependency. I'll try to look more into it after I figure out asset loading :)

Godot uses 13.0 sdk version for simulator, since this the only simulator version supporting Metal. But while it does build, MoltenVK fails and application doesn't launch in the end.
So cargo lipo might be using lower sdk version that actually required for bevy.

[MichaelHills]

Looks like the ios audio support is still in-progress as per your link, I tried and failed to get that branch to work as well. I'll leave audio disabled for now and might have a try at getting the simulator going.

[naithar]

So compiled binary of bevy faces the same problem - not displaying sprite at specified path (icon.png).
Providing a full path fixed this for macOS, but haven't for iOS.

[MichaelHills]

I played around with the paths on iOS on my phone, I added icon.png to my xcode project and examined the various paths...

Bundle.main.bundlePath -> /var/containers/Bundle/Application/6AD264E9-6A09-47CE-B20A-56F906BE56CD/rust-ios.app
std::env::current_exe() -> /var/containers/Bundle/Application/6AD264E9-6A09-47CE-B20A-56F906BE56CD/rust-ios.app/rust-ios
Bundle.main.path(forResource: "icon", ofType: "png") -> /var/containers/Bundle/Application/6AD264E9-6A09-47CE-B20A-56F906BE56CD/rust-ios.app/icon.png

I tried std::fs::canonicalize(std::env::current_exe().unwrap().join("..").join("icon.png")).unwrap(); to produce /private/var/containers/Bundle/Application/6AD264E9-6A09-47CE-B20A-56F906BE56CD/rust-ios.app/icon.png and pass that to the asset loader. Logs in the loader showed it was successfully loading the file. I guess you already checked the texture handle... but still nothing shows up on screen.

I tried adding some color to the texture to see if it'd help, it didn't. I also tried just the color by itself and no texture, but that doesn't seem to work even on mac.

        .spawn(SpriteComponents {
            material: materials.add(ColorMaterial {
                color: Color::RED,
                texture: texture_handle.into(),
            }),
            ..Default::default()
        });

If we can get a red square rendering, that'd be a good first step. I edited the fragment shader to show only red, worked on mac, didn't work on iOS.

[naithar]

It seems like iOS doesn't really handle/load assets in any way.
Also compiled binaries doesn't really like relative paths too, as I've reported in #344

I'm looking for a way to debug rust on iOS, so problem could be traced, but I've got no progress yet.

[MichaelHills]

In crates/bevy_render/src/render_graph/nodes/pass_node.rs I added some logs, basically can_draw_indexed() returns false because one of the "bind groups" is None. I don't know what a bind group is though.

                                    println!("maybe draw indexed can_draw={} index_buf={} bgs={} vbs={}", draw_state.can_draw(), draw_state.index_buffer.is_some(), draw_state.bind_groups.iter().all(|b| b.is_some()), draw_state.vertex_buffers.iter().all(|v| v.is_some()));
                                    for bg in draw_state.bind_groups.iter() {
                                        println!("bg is some {}", bg.is_some());

                                    }
                                    if draw_state.can_draw_indexed() {

logs are

1 visible entities
maybe draw indexed can_draw=false index_buf=true bgs=false vbs=true
bg is some true
bg is some false
bg is some true

[MichaelHills]

Oh nevermind, it was because I had left this in the frag shader

# ifdef COLORMATERIAL_TEXTURE
layout(set = 1, binding = 1) uniform texture2D ColorMaterial_texture;
layout(set = 1, binding = 2) uniform sampler ColorMaterial_texture_sampler;
# endif

I removed it and now it looks like it is issuing the draw call. Still no red square. :(

[naithar]

I've put some logs into fn load_asset(&self, load_request: &LoadRequest) -> Result<TAsset, AssetLoadError> and that's the result.

loading untyped at /private/var/containers/Bundle/Application/DF750ECE-8232-4398-B02E-F9749CCA6A03/BevyTest.app/icon.png
loading: Handle<Texture>(7841c797-5cf9-401d-b1ce-67245a08cd42)
loading asset: LoadRequest { path: "/private/var/containers/Bundle/Application/DF750ECE-8232-4398-B02E-F9749CCA6A03/BevyTest.app/icon.png", handle_id: HandleId(7841c797-5cf9-401d-b1ce-67245a08cd42), handler_index: 1, version: 0 }
file exists: true
bytes read: [... very big array ...]

So iOS actually seems to be loading assets correctly in the end if full path is given.

I removed it and now it looks like it is issuing the draw call. Still no red square. :(

Maybe iOS handles drawing differently or shaders are not compiled so it fails to draw anything?

Edit:
Seems like shaders are also compiled correctly, so I guess something might be wrong with shader itself or something is not supported on iOS.

[MichaelHills]

I finally got my red square working. WIthout the image/texture to determine the sprite size, you have to set the size manually:

        .spawn(SpriteComponents {
            material: materials.add(ColorMaterial {
                color: Color::RED,
                // texture: texture_handle.into(),
            }),
            sprite: Sprite { size: Vec2::new(100.0, 100.0) },
            ..Default::default()
        });

so I think the shaders are fine... but still something going wrong in image or texture loading, not sure what yet.

[MichaelHills]

in crates/bevy_render/src/texture/image_texture_loader.rs

        println!("do dyn img");
        let dyn_img = image::load_from_memory_with_format(bytes.as_slice(), img_format)?;
        println!("got dyn img");

It seems to never finish load_from_memory_with_format (at least I don't see the log afterwards), so this explains why the texture loading is not working.

Now the whole "it has a handle" thing is actually because you get a handle back but the png is loaded into a texture on a background thread. Eventually the handle will resolve to a valid texture, but only once it has loaded.

Discovered this initially by adding logs to crates/bevy_sprite/src/sprite.rs

pub fn sprite_system(
    materials: Res<Assets<ColorMaterial>>,
    textures: Res<Assets<Texture>>,
    mut query: Query<(&mut Sprite, &Handle<ColorMaterial>)>,
) {
    println!("sprite system");
    for (mut sprite, handle) in &mut query.iter() {
        let material = materials.get(&handle).unwrap();
        println!("got material with color {} {} {}", material.color.r, material.color.g, material.color.b);
        if let Some(texture_handle) = material.texture {
            println!("got texture handle");
            if let Some(texture) = textures.get(&texture_handle) {
                println!("got texture size {} {}", texture.size.x(), texture.size.y());
                sprite.size = texture.size;
            }
        }
    }
}

and noticed that on iOS it never printed the texture size but on mac it did.

Edit: finally got the error message out from the image loading... Format error decoding Png: CgBI chunk appeared before IHDR chunk

I think somehow the PNG is getting corrupted when added to the iOS project? It's smaller...

ios bytes=12244
mac bytes=15713

[MichaelHills]

Well, I found our problem! You need to disable XCode "Remove text metadata from PNG". I disabled everything though just to be sure... I guess the image loading library can't handle whatever XCode does to it.

My phone is now showing a red bevy bird. 👍

[cart]

Very exciting! Great work here!

[naithar]

Adding std::env::set_current_dir(std::env::current_exe().unwrap().parent().unwrap()); works as workaround for iOS and macOS .app not loading assets at relative path.

[naithar]

@cart is there a reason for AssetServer not using get_root_path method for loading assets at relative paths?

[MichaelHills]

@naithar how did you create a mac .app bundle?

[MichaelHills]

Nevermind found this https://stackoverflow.com/questions/40309538/creatin-a-minimal-macos-app-bundle

So I made

sprite.app/Contents/MacOS/sprite
sprite.app/Contents/Resources/assets/branding/icon.png
sprite.app/Contents/Info.plist

where Info.plist is

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>CFBundleExecutable</key>
    <string>sprite</string>
    <key>CFBundleSignature</key>
    <string>MYAP</string>
    <key>CFBundlePackageType</key>
    <string>APPL</string>
    <key>CFBundleVersion</key>
    <string>1.0.0</string>
    <key>CFBundleIdentifier</key>
    <string>com.myself.myapp</string>
    <key>CFBundleDisplayName</key>
    <string>sprite</string>
    <key>CFBundleName</key>
    <string>sprite</string>
</dict>
</plist>

as per https://github.com/bevyengine/bevy/issues/344 my app doesn't work because I haven't tweaked the asset path yet.

[MichaelHills]

@cart is there a reason for AssetServer not using get_root_path method for loading assets at relative paths?

Oh I see, get_root_path is used in load_asset_folder and filesystem_watcher_system but not in load / load_untyped. It's also not a public function so probably can't try to use it ourselves.

Also, this code path of get_root_path doesn't seem very useful?

        match std::env::current_exe() {
            Ok(exe_path) => exe_path
                .parent()
                .ok_or(1)
                .map(|exe_parent_path| exe_parent_path.to_owned()),
            Err(err) => Err(2),
        }

[naithar]

how did you create a mac .app bundle?

I've simply used Godot's .app template for iOS and tweaked it a little 😃

Also, this code path of get_root_path doesn't seem very useful?

Well, that's basically what I'm passing into set_current_path atm, so binary run by double clicking, .app and iOS application could work correctly. But I guess there is a reason for bevy not using this for asset loading

[MichaelHills]

I've simply used Godot's .app template for iOS and tweaked it a little 😃

Oh I see. :)

Btw I was going to file a bug on the image crate but it turns out the XCode PNG optimization converts it to a different file format https://iphonedevwiki.net/index.php/CgBI_file_format which is not valid PNG anymore, so actually there's no bug.

I'll open a PR here though for better error reporting. Took many hours to track down the issue which some simple error reporting could've highlighted immediately.

Edit: looks like there already is error reporting on asset load failure... odd that I never saw any logs though, will investigate

                Err(err) => {
                    asset_server
                        .set_load_state(result.handle.id, LoadState::Failed(result.version));
                    log::error!("Failed to load asset: {:?}", err);
                }

[MichaelHills]

Okay so logging needs to be configured

diff --git a/Cargo.toml b/Cargo.toml
index 1d8a1bcf..8f437052 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -62,6 +62,9 @@ bevy_gltf = { path = "crates/bevy_gltf", optional = true, version = "0.1" }
 bevy_wgpu = { path = "crates/bevy_wgpu", optional = true, version = "0.1" }
 bevy_winit = { path = "crates/bevy_winit", optional = true, version = "0.1" }

+log = { version = "0.4", features = ["release_max_level_info"] }
+env_logger = "0.7.1"
+
 [dev-dependencies]
 rand = "0.7.2"
 serde = { version = "1", features = ["derive"]}
diff --git a/examples/2d/sprite.rs b/examples/2d/sprite.rs
index d91caefc..3199e72f 100644
--- a/examples/2d/sprite.rs
+++ b/examples/2d/sprite.rs
@@ -1,6 +1,9 @@
 use bevy::prelude::*;

 fn main() {
+    env_logger::init();
+    log::warn!("hi");
+
     App::build()
         .add_default_plugins()
         .add_startup_system(setup.system())

and then you see this :) much better!

$ RUST_LOG=warn cargo run --example sprite
    Finished dev [unoptimized + debuginfo] target(s) in 0.73s
     Running `target/debug/examples/sprite`
[2020-08-29T13:41:11Z WARN  sprite] hi
[2020-08-29T13:41:12Z ERROR bevy_asset::loader] Failed to load asset: LoaderError(Format error decoding Png: CgBI chunk appeared before IHDR chunk

    Caused by:
        CgBI chunk appeared before IHDR chunk)

[naithar]

I managed to finally get something to build, on the simulator gfx-rs / gfx-backend-metal failed with Target OS is incompatible: library was not compiled for the simulator so I ran it on my phone instead to which I got

EventLoop cannot be run after a call to UIApplicationMain on iOS.

Well, I have some progress on this:

This requires changes to gfx's Makefile and some other files:

Changes

Cargo.toml for bevy game will require this changes as well as cloning gfx before https://github.com/gfx-rs/gfx/commit/b373ebe0f057300fff76534afba25b261527db7a commit

[patch.crates-io]
gfx-backend-metal = { path = "./gfx/src/backend/metal" }
gfx-hal = { path = "./gfx/src/hal" }
gfx-warden = { path = "./gfx/src/warden" }

Changes for gfx:
Running make shader-binaries will also be required after this changes.

diff --git a/Makefile b/Makefile
index 3a46869c..a99511b7 100644
--- a/Makefile
+++ b/Makefile
@@ -32,6 +32,8 @@ else
        endif
        ifeq ($(TARGET),aarch64-apple-ios)
                EXCLUDES+= --exclude gfx-backend-vulkan --exclude gfx-backend-gl
+       else ifeq ($(TARGET),x86_64-apple-ios)
+               EXCLUDES+= --exclude gfx-backend-vulkan --exclude gfx-backend-gl
        else
                FEATURES_GL=gl
        endif
@@ -96,4 +98,9 @@ ifeq ($(UNAME_S),Darwin)
        xcrun -sdk iphoneos metal -c *.metal -mios-version-min=11.4 && \
        xcrun -sdk iphoneos metallib *.air -o gfx-shaders-ios.metallib && \
        rm *.air
+       # iOS Simulator
+       cd ./src/backend/metal/shaders && \
+       xcrun -sdk iphonesimulator metal -c *.metal -mios-simulator-version-min=13.0 && \
+       xcrun -sdk iphonesimulator metallib *.air -o gfx-shaders-ios-simulator.metallib && \
+       rm *.air
 endif
diff --git a/examples/Cargo.toml b/examples/Cargo.toml
index f9fc6e4e..415b0f5f 100644
--- a/examples/Cargo.toml
+++ b/examples/Cargo.toml
@@ -58,7 +58,7 @@ version = "0.6"
 features = ["x11"]
 optional = true

-[target.'cfg(any(target_os = "macos", all(target_os = "ios", target_arch = "aarch64")))'.dependencies.gfx-backend-metal]
+[target.'cfg(any(target_os = "macos", all(target_os = "ios")))'.dependencies.gfx-backend-metal]
 path = "../src/backend/metal"
 version = "0.6"
 optional = true
diff --git a/src/backend/metal/Cargo.toml b/src/backend/metal/Cargo.toml
index 4a7734aa..7a279c8d 100644
--- a/src/backend/metal/Cargo.toml
+++ b/src/backend/metal/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "gfx-backend-metal"
-version = "0.6.0"
+version = "0.6.1"
 description = "Metal API backend for gfx-rs"
 homepage = "https://github.com/gfx-rs/gfx"
 repository = "https://github.com/gfx-rs/gfx"
diff --git a/src/backend/metal/shaders/gfx-shaders-ios.metallib b/src/backend/metal/shaders/gfx-shaders-ios.metallib
index f20b7f71..f946086a 100644
Binary files a/src/backend/metal/shaders/gfx-shaders-ios.metallib and b/src/backend/metal/shaders/gfx-shaders-ios.metallib differ
diff --git a/src/backend/metal/shaders/gfx-shaders-macos.metallib b/src/backend/metal/shaders/gfx-shaders-macos.metallib
index b331b9ac..642ae185 100644
Binary files a/src/backend/metal/shaders/gfx-shaders-macos.metallib and b/src/backend/metal/shaders/gfx-shaders-macos.metallib differ
diff --git a/src/backend/metal/src/internal.rs b/src/backend/metal/src/internal.rs
index ca37701a..e662a07b 100644
--- a/src/backend/metal/src/internal.rs
+++ b/src/backend/metal/src/internal.rs
@@ -452,8 +452,10 @@ impl ServicePipes {
     pub fn new(device: &metal::DeviceRef) -> Self {
         let data = if cfg!(target_os = "macos") {
             &include_bytes!("./../shaders/gfx-shaders-macos.metallib")[..]
-        } else {
+        } else if cfg!(target_arch = "aarch64") {
             &include_bytes!("./../shaders/gfx-shaders-ios.metallib")[..]
+        } else {
+            &include_bytes!("./../shaders/gfx-shaders-ios-simulator.metallib")[..]
         };
         let library = device.new_library_with_data(data).unwrap();

diff --git a/src/warden/Cargo.toml b/src/warden/Cargo.toml
index 310056ce..8f7fa0a5 100644
--- a/src/warden/Cargo.toml
+++ b/src/warden/Cargo.toml
@@ -52,7 +52,7 @@ path = "../../src/backend/dx11"
 version = "0.6"
 optional = true

-[target.'cfg(any(target_os = "macos", all(target_os = "ios", target_arch = "aarch64")))'.dependencies.gfx-backend-metal]
+[target.'cfg(any(target_os = "macos", all(target_os = "ios")))'.dependencies.gfx-backend-metal]
 path = "../../src/backend/metal"
 version = "0.6"
 features = ["auto-capture"]

After this running cargo lipo --targets aarch64-apple-ios x86_64-apple-ios will produce a binary that could be used to run application on both simulator starting from iOS 13.0 and device with metal support.

[naithar]

After having #334, #324 and some workarounds for #344 doing iOS stuff should be a bit easier.

Edit:

Cargo.toml for bevy game will require this changes as well as cloning gfx before gfx-rs/gfx@b373ebe commit

Cloning gfx before this commit is required for bevy to compile - current version of wgpu on bevy is using old version of gfx and there are a breaking changes.

[MichaelHills]

Nice work on the simulator! I've actually started prototyping my game and learning how to use systems and Query etc. Using mac for dev, then deploy on phone when ready. I've hacked up some touch support so I can move my character (a sphere...) around.

Here's my diffs for the touch if you want to play with it, I'll try and make a PR when I have time.

diff --git a/crates/bevy_input/src/lib.rs b/crates/bevy_input/src/lib.rs
index 1893d86b..a9fa49fe 100644
--- a/crates/bevy_input/src/lib.rs
+++ b/crates/bevy_input/src/lib.rs
@@ -1,6 +1,7 @@
 mod input;
 pub mod keyboard;
 pub mod mouse;
+pub mod touch;
 pub mod system;

 pub use input::*;
@@ -12,6 +13,7 @@ pub mod prelude {
 use bevy_app::prelude::*;
 use keyboard::{keyboard_input_system, KeyCode, KeyboardInput};
 use mouse::{mouse_button_input_system, MouseButton, MouseButtonInput, MouseMotion, MouseWheel};
+use touch::{touch_input_system, TouchInput, TouchInputState};

 use bevy_ecs::IntoQuerySystem;

@@ -25,6 +27,7 @@ impl Plugin for InputPlugin {
             .add_event::<MouseButtonInput>()
             .add_event::<MouseMotion>()
             .add_event::<MouseWheel>()
+            .add_event::<TouchInput>()
             .init_resource::<Input<KeyCode>>()
             .add_system_to_stage(
                 bevy_app::stage::EVENT_UPDATE,
@@ -34,6 +37,11 @@ impl Plugin for InputPlugin {
             .add_system_to_stage(
                 bevy_app::stage::EVENT_UPDATE,
                 mouse_button_input_system.system(),
+            )
+            .init_resource::<TouchInputState>()
+            .add_system_to_stage(
+                bevy_app::stage::EVENT_UPDATE,
+                touch_input_system.system(),
             );
     }
 }
diff --git a/crates/bevy_input/src/touch.rs b/crates/bevy_input/src/touch.rs
new file mode 100644
index 00000000..1c681231
--- /dev/null
+++ b/crates/bevy_input/src/touch.rs
@@ -0,0 +1,103 @@
+use bevy_ecs::{Local, ResMut, Res};
+use bevy_app::{Events, EventReader};
+use std::collections::{HashMap, HashSet};
+
+
+/// A touch input event
+#[derive(Debug, Clone)]
+pub struct TouchInput {
+    pub phase: TouchPhase,
+    pub x: f64,
+    pub y: f64,
+    ///
+    /// ## Platform-specific
+    ///
+    /// Unique identifier of a finger.
+    pub id: u64,
+}
+
+/// Describes touch-screen input state.
+#[derive(Debug, Hash, PartialEq, Eq, Clone, Copy)]
+#[cfg_attr(feature = "serialize", derive(serde::Serialize, serde::Deserialize))]
+pub enum TouchPhase {
+    Started,
+    Moved,
+    Ended,
+    Cancelled,
+}
+
+#[derive(Default)]
+pub struct TouchSystemState {
+    touch_event_reader: EventReader<TouchInput>,
+}
+
+#[derive(Debug, Clone)]
+pub struct ActiveTouch {
+    start_x: f64,
+    start_y: f64,
+    cur_x: f64,
+    cur_y: f64,
+}
+
+impl ActiveTouch {
+    pub fn dx(&self) -> f64 {
+        self.cur_x - self.start_x
+    }
+    pub fn dy(&self) -> f64 {
+        self.cur_y - self.start_y
+    }
+}
+
+#[derive(Default)]
+pub struct TouchInputState {
+    pub active_touches: HashMap<u64, ActiveTouch>,
+    pub just_pressed: HashSet<u64>,
+    pub just_released: HashSet<u64>,
+    pub just_cancelled: HashSet<u64>,
+}
+
+
+/// Updates the TouchInputState resource with the latest TouchInput events
+pub fn touch_input_system(
+    mut state: Local<TouchSystemState>,
+    mut touch_state: ResMut<TouchInputState>,
+    touch_input_events: Res<Events<TouchInput>>,
+) {
+    touch_state.just_pressed.clear();
+    touch_state.just_released.clear();
+    touch_state.just_cancelled.clear();
+
+    for event in state
+        .touch_event_reader
+        .iter(&touch_input_events)
+    {
+        // println!("{:?}", event);
+
+        let active_touch = touch_state.active_touches.get(&event.id);
+        match event.phase {
+            TouchPhase::Started => {
+                touch_state.active_touches.insert(event.id, ActiveTouch {
+                    start_x: event.x,
+                    start_y: event.y,
+                    cur_x: event.x,
+                    cur_y: event.y,
+                });
+            },
+            TouchPhase::Moved => {
+                let prev = active_touch.unwrap();
+                let mut cur = prev.clone();
+                cur.cur_x = event.x;
+                cur.cur_y = event.y;
+                touch_state.active_touches.insert(event.id, cur);
+            },
+            TouchPhase::Ended => {
+                touch_state.active_touches.remove(&event.id);
+                touch_state.just_released.insert(event.id);
+            },
+            TouchPhase::Cancelled => {
+                touch_state.active_touches.remove(&event.id);
+                touch_state.just_cancelled.insert(event.id);
+            },
+        };
+    }
+}
diff --git a/crates/bevy_winit/src/lib.rs b/crates/bevy_winit/src/lib.rs
index b6261ae8..5659086f 100644
--- a/crates/bevy_winit/src/lib.rs
+++ b/crates/bevy_winit/src/lib.rs
@@ -4,6 +4,7 @@ mod winit_windows;
 use bevy_input::{
     keyboard::KeyboardInput,
     mouse::{MouseButtonInput, MouseMotion, MouseScrollUnit, MouseWheel},
+    touch::{TouchInput, TouchPhase}
 };
 pub use winit_config::*;
 pub use winit_windows::*;
@@ -192,6 +193,22 @@ pub fn winit_runner(mut app: App) {
                         });
                     }
                 },
+                WindowEvent::Touch(touch) => {
+                    let mut touch_input_events =
+                        app.resources.get_mut::<Events<TouchInput>>().unwrap();
+                    // TODO use convert* thing
+                    touch_input_events.send(TouchInput {
+                        phase: match touch.phase {
+                            event::TouchPhase::Started => TouchPhase::Started,
+                            event::TouchPhase::Moved => TouchPhase::Moved,
+                            event::TouchPhase::Ended => TouchPhase::Ended,
+                            event::TouchPhase::Cancelled => TouchPhase::Cancelled,
+                        },
+                        x: touch.location.x,
+                        y: touch.location.y,
+                        id: touch.id,
+                    });
+                },
                 _ => {}
             },
             event::Event::DeviceEvent { ref event, .. } => {

You can use it as follows:

The app setup...

        .init_resource::<TouchHandlerState>()
        .add_system(touch_handler.system())

And then the code

struct TouchHandlerState {
    touch_event_reader: EventReader<TouchInput>,
}

fn touch_handler(
    mut state: ResMut<TouchHandlerState>,
    touch_events: Res<Events<TouchInput>>,
    touch_input_state: Res<TouchInputState>,
) {
    for event in state.touch_event_reader.iter(&touch_events) {
        println!("{:?}", event);
    }
    // touch_input_state.active_touches;
    // touch_input_state.just_pressed;
    // touch_input_state.just_released;
    // touch_input_state.just_cancelled;
}

in my prototype game I didn't actually use the event stream and just did this to get a finger direction and move my character in the direction the held-finger has moved from the original touch position.

    let mut dx = 0.0;
    let mut dy = 0.0;
    for id in touch_input_state.active_touches.keys() {
        let t = touch_input_state.active_touches.get(&id).unwrap();
        dx = t.dx();
        dy = t.dy();
        break;
    }

[naithar]

Cool 👍
For iOS's touch processing this might be relevant: https://github.com/godotengine/godot/pull/39624 as winit seems to be using touchesBegan/touchesMoved methods for touch recognition too.
I'm not 100% sure about this being a problem for bevy, but might be something to take into account.

[MichaelHills]

Oh yeah click emulation. I also have no idea if winit supports pinch to zoom or other gestures which would be nice to have.

[naithar]

Hardly. From what I've seen it gives only basic touch info, just as iOS would without using gesture recognizers.
But this could be used for making all zoom, pitch, pan and whatever gestures required, so I don't really see a problem.

Godot's required this change for touch processing because newest iPhones send moved events to early which broke gui input handling due to the way it's made (emulation of mouse from touches).

[CleanCut]

Thank you for your awesome work on this, @MichaelHills & @naithar! I think a lot of us are watching your efforts with interest in anticipation of iOS support. 😄

One comment: I encourage just getting the basic touch info working, and forgetting about gestures/mouse-emulation. Gestures/mouse-emulation support (if any) would be best implemented as an optional driver or ui-level system anyway, not handled up at the raw input level. See the excellent Event Chaining as a Decoupling Method in Entity-Component-System post about such designs.

[naithar]

One comment: I encourage just getting the basic touch info working, and forgetting about gestures/mouse-emulation.

Yeah, that's what I think too.

Basically there are no stoppers for iOS support left:

• There are already PR for shader fix for iOS #324
• We also have a PR for touches #334 which seems to be as basic as it could be (which is great for now).
• I'm planning on making a PR for gfx to allow compilation for iOS Simulators. But I just want to test everything before I actually open it.
• Also wgpu crate update might be required for everything to work.

iOS could probably be hidden behind default_ios feature for now as coreaudio is not yet ready for iOS.
It would probably required to make a templates so iOS application could be tested easily.

[MichaelHills]

Ohhh I didn't realise https://github.com/bevyengine/bevy/pull/334 was a PR, I assumed it was the issue https://github.com/bevyengine/bevy/issues/300 I was looking at earlier. Cool. :) I didn't need to spend that time writing touch support haha, but it's good I'm learning how to use Bevy at the same time.

[naithar]

gfx simulator support should land soon-ish (If it didn't already) for both master and 0.6 version which is used atm by bevy.

cmake-rs could be bumped to 0.1.43 by project with

[dependencies]
cmake = "=0.1.43"

Modified shaderc is still required, so I guess something should be done on that part too in shaderc and glslang.

I've also modified Cargo.toml to simplify building:

index 56f70cc..8ccd284 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -8,17 +8,17 @@ edition = "2018"

 [dependencies]
 cmake = "=0.1.43"
-bevy = { version = "0.1.3" }
+bevy = { version = "0.1.3", default-features = false, features=["default_ios"]  }

 [patch.crates-io]
+bevy = { path = "./bevy", default-features = false, features=["default_ios"] }
+shaderc = { path = "./shaderc-rs/shaderc-rs" }
--- a/bevy/Cargo.toml
+++ b/bevy/Cargo.toml
@@ -17,6 +17,8 @@ default = [
     "bevy_audio", "bevy_gltf", "bevy_wgpu", "bevy_winit",
     "dynamic_plugins", "png", "hdr", "mp3", "x11"
 ]
+default_ios = ["bevy_gltf", "bevy_wgpu", "bevy_winit", "png", "hdr"]
+

[MichaelHills]

gfx simulator support should land soon-ish (If it didn't already) for both master and 0.6 version which is used atm by bevy.

Awesome.

Modified shaderc is still required, so I guess something should be done on that part too in shaderc and glslang.

Yeah either an update to cmake-rs or to shaderc/glslang is required due to the bundle destination thing referenced earlier.

+    BUNDLE DESTINATION ${CMAKE_INSTALL_BINDIR}

My opinion is that cmake should not default CMAKE_MACOSX_BUNDLE to ON https://gitlab.kitware.com/cmake/cmake/-/issues/20956#

Failing that I think it can be fixed in cmake-rs as I mentioned here https://github.com/alexcrichton/cmake-rs/issues/95 by adding cmd.arg("-DCMAKE_MACOSX_BUNDLE=FALSE");. This way cmake-rs can workaround cmake's bad defaults.

Haven't really received any response from the above projects though.

In the meantime... waiting for Naga to be ready so we can ditch shaderc. :)

[naithar]

Awesome.

It's already landed, so updating gfx-metal-backend to 0.6.2 should do the trick.

In the meantime... waiting for Naga to be ready so we can ditch shaderc. :)

Yeah, naga looks very promising. But at least now we can already do some stuff :)

While at it, it seems iOS will need some WindowMode that tries to save window aspect ratio or scales content based on its size. But maybe it' already there and I just don't know it :)

Edit:
This does look kinda related https://github.com/bevyengine/bevy/issues/195, but it seems to be only for UI

[MichaelHills]

I did a cargo update and now simulator works. 👍

[MichaelHills]

I tried to add a build script step to XCode so I don't need to manually run my cargo build script in the terminal. I couldn't get it to work though, have you had any luck with that?

[MichaelHills]

Also my staticlib with --release using opt-level = 'z' is like 192MB. Am I doing something wrong or is it meant to be that large?

[naithar]

Yeah, it's mostly okay.
Binary has lots of unused symbols and multiple archs. Once you upload a binary to AppStore it would shrink, since App Slicing would be used on Apple side.

I tried to add a build script step to XCode so I don't need to manually run my cargo build script in the terminal. I couldn't get it to work though, have you had any luck with that?

I didn't do it. But you should have some luck if you create a custom script target and then link .a file to library scheme as well as resulting application target dependencies. Godot should have some tutorials on this. I can probably make one, but not right now :)

[MichaelHills]

I managed to create an archive (disabled bitcode, otherwise get some error about Rust using llvm10 and apple using llvm11). It is 91MB. Not too bad I guess. Was hoping for smaller. :)

[naithar]

Once you upload it will be smaller - resulting app should be around 20-30MB in size depending on libraries used.
My test Godot library is about 600mb in size, but results in 28mb as application on TestFlight

[MichaelHills]

There are a couple of issues around the window on iOS. I've set the window resolution to 1334 by 750 which is the iPhone 6 resolution. However it doesn't reach the edge of the screen, stops a tiny bit short. i.e. black bars on the right and bottom. Increasing width did nothing.

I also can't get rid of the status bar. I configured it to off in the XCode project, but it seems to want to stay on. I tried to hook into the window created event, and access the winit window directly to call set_decorations(false) but that did nothing.

Edit: I've fixed my game to landscape in the XCode project, that works to prevent portrait mode.

[MichaelHills]

It looks like the window is fixed at 1280x720, happened to notice that when logging touch event x/y coordinates (adding virtual joysticks to my game). Haven't looked into where it's coming from, I suspect winit perhaps.

[CleanCut]

Hmm, I don't see anything in winit, but this part of bevy_window seems suspicious:

https://github.com/bevyengine/bevy/blob/612c2552a50e8009715e825504b9e8b74b56821e/crates/bevy_window/src/window.rs#L87-L99

[MichaelHills]

@CleanCut haha suspicious indeed. I'll check my code tonight, maybe I configured the window wrong.

[MichaelHills]

Thanks, was just my config. My code was like this, when I moved WindowDescriptor to be before the default plugins my settings actually took effect. I still can't get it to just "fullscreen" cleanly and map to the edges of the screen. Now things are a bit vertically stretched and the window appears to extend past the right and bottom of my phone. Will keep investigating.

     App::build()
         .add_default_plugins()
         .add_resource(WindowDescriptor {

Edit: The window is the correct size, and the touch events x/y confirm it. Maybe something wrong with the viewport... not sure.

Edit 2: I can fix it with this... the window size is 1334 by 750 though.

wgpu_render_pass.set_viewport(0.0, 0.0, 1112.0, 563.0, 0.0, 1.0);

[MichaelHills]

Argh nevermind, I had typo'd my ios vs not-ios cfgs and I had #[cfg(target = "ios")] not #[cfg(target_os = "ios")] and so it was doing the opposite of what I wanted.

It meant I was passing 1600x1000 to iOS, which still somehow created a window of size 1334x750 as per windows.get_primary().unwrap(); but when I logged winit's ScaleFactorChanged event it logged

ScaleFactorChanged(2, w=1600 h=1000)

so that's how I knew I messed up. Anyway after fixing this issue, I now have another problem. I have defaulted my game in XCode to landscape-only. When I do fullscreen, it's choosing a portrait mode window size of 750x1334 even though it's already in landscope mode.

[MichaelHills]

Finally figured it out. I think there's an issue in winit where if you start the iOS app in landscape, the view frame doesn't match the window bounds (view in portrait, window in landscape), so in layoutSubviews just need to detect this and update the frame. Now if you start in either portrait or landscape, and rotate the phone, everything works fine.

diff --git a/src/platform_impl/ios/view.rs b/src/platform_impl/ios/view.rs
index 5481b8ef..da195a0a 100644
--- a/src/platform_impl/ios/view.rs
+++ b/src/platform_impl/ios/view.rs
@@ -134,6 +134,18 @@ unsafe fn get_view_class(root_view_class: &'static Class) -> &'static Class {
                     height: screen_frame.size.height as f64,
                 }
                 .to_physical(scale_factor.into());
+
+                // If the app is started in landscape, the view frame and window bounds can be mismatched.
+                // The view frame will be in portrait and the window bounds in landscape. So apply the
+                // window bounds to the view frame to make it consistent.
+                let view_frame: CGRect = msg_send![object, frame];
+                let window_bounds: CGRect = msg_send![window, bounds];
+                if view_frame.size.width != window_bounds.size.width
+                    || view_frame.size.height != window_bounds.size.height
+                {
+                    let () = msg_send!(object, setFrame: window_bounds);
+                }
+
                 app_state::handle_nonuser_event(EventWrapper::StaticEvent(Event::WindowEvent {
                     window_id: RootWindowId(window.into()),
                     event: WindowEvent::Resized(size),

[MichaelHills]

@cart I'm assuming if I get this landed https://github.com/rust-windowing/winit/pull/1703 that I'll need you to also update cart-tmp-winit? I couldn't find where that repo was.

[cart]

Yeah if we need to consume upstream master winit, then we can update cart-tmp-winit. Currently theres no public repo for it, but I can definitely put one up as soon as we need it.

I would like to eventually move back upstream though. Maybe we can just ask them to cut a new version :smile:

[MichaelHills]

Audio might be a bit of a hurdle. Rodio uses cpal uses coreaudio-rs uses coreaudio-sys and the latter don't yet work on iOS.

Relevant PRs to make it work on iOS but based on the first one it looks like it's never been in a working state?

• https://github.com/RustAudio/coreaudio-rs/pull/69
• https://github.com/RustAudio/coreaudio-sys/pull/33

Over at cpal someone started but abandoned an OpenAL backend https://github.com/RustAudio/cpal/issues/224

Apple provides an OpenAL implementation but apparently it's been deprecated in macOS Catalina, so maybe iOS is next.

The OpenAL framework is deprecated and remains present for compatibility purposes. Transition to AVAudioEngine for spatial audio functionality.

Perhaps an AVAudioEngine backend to cpal would be the way forward?

Edit: What am I talking about, finishing off the CoreAudio stuff is probably the quickest route.

[simlay]

On the subtopic of Audio:

Relevant PRs to make it work on iOS but based on the first one it looks like it's never been in a working state?

I think the README on cpal is/has been incorrect. I stumbled into coreaudio-rs/sys and thought adding those PRs at it seemed like low hanging fruit. I ended up spending months slowly adding better objective-c support to rust bindgen. I think it's one PR merge short of being pretty useful.

This means that the bindings for coreaudio-sys will look a bit different for iOS targets than they do macOS if you want to use the objective-c features of bindgen (I think you might have to). One annoying thing about coreaudio-rs is that it rexports which is then used in cpal. So it seems that it might need a bit of work.

Funny enough, I actually don't know much about the CoreAudio framework or the internals of cpal well enough to actually implement this without a lot of learning. Given enough time, I'll probably get to it but if someone wants to take the reins, I'm all for it.

[MichaelHills]

Thanks for the hard work on iOS @simlay I've seen your PRs all over the place. :) Including in cmake-rs unfortunately got reverted :( https://github.com/alexcrichton/cmake-rs/issues/95 I tried to spark some discussion but didn't go anywhere. I think for Bevy at least, once naga is ready we can drop shaderc-rs and then I think the dependency on cmake-rs goes away (for now).

Back to CoreAudio, I imagine the differences wouldn't be that major between macOS and iOS. I think in cpal we could get away with some target_os switches where macOS and iOS diverge?

I might have a go, but I'm not sure where's the best place to start. If I were to try, do I need the 3 of your branches? rust-bindgen, coreaudio-rs, coreaudio-sys? Why did you need to make changes to rust-bindgen for iOS CoreAudio?

[simlay]

Including in cmake-rs unfortunately got reverted :( alexcrichton/cmake-rs#95 I tried to spark some discussion but didn't go anywhere.

I'd like to eventually work on this again. It's valuable for a bunch of iOS projects.

I might have a go, but I'm not sure where's the best place to start.

Yeah! @MichaelHills, that would be awesome! There are few rust iOS folks and I'd like there to be more! You can find me on discord in the Bevy server (is that what they're called?) along with most of the other rust discords as simlay#3120.

If I were to try, do I need the 3 of your branches? rust-bindgen, coreaudio-rs, coreaudio-sys?

Right now, I think so. I'm all for a better work flow.

• my rust-bindgen branch objc-category-inheritance might not be needed anymore as most of the core stuff is either on crates.io or on the master branch. I wish https://github.com/rust-lang/rust-bindgen/pull/1784 was easier/faster to merge, but it's best that a core rust-lang project be slow and stable rather than quick and breaking.

• my coreaudio-sys branch add-ios-support which is mostly just an update to the build.rs
• my coreaudio-rs branch ios-support. This branch might not actually be all that helpful. Some of the updates were for more modern rust (swapping try! for the ?, adding dyn on traits because). In hindsight, I should have not worried about the warnings and kept it to the smallest of changes.

Why did you need to make changes to rust-bindgen for iOS CoreAudio?

CoreAudio for iOS has a bunch of objective-c in the headers and so bindgen (which uses clang to get the AST from the headers), and so just to get it to compile it required adding -x objective-c as a clang parameter to bindgen, this resulted in an issue with the objective-c generics not generating working rust.

After that, I slowly added to make all the bindings more ergonomic:

• higher level "types" corresponding to the objective-c classes
• inheritance tree generation
• the bindgen names corresponding to objective-c classes
• up/down casting objective-c classes
• I hope the last of these is to fix the objective-c category inheritance.

It's possible I might have gone a bit far down the rabbit hole but these are still cool and useful features if you ask me.

I've spent so much time trying to get the objective-c Nullability stuff working but I've not had a ton of luck. :/.

I'm not going to claim to actually know enough CoreAudio to tell you that these features really help you all that much. For all I know, you could blacklist all of the objective-c and you'd be left with some bindings that are quicker to generate.

I don't know your knowledge level for iOS development stuff but I think making an example iOS project for coreaudio-sys or coreaudio-rs using xcodegen would be useful. I did something like this for uikit-sys. There's also some useful things in a blogpost I wrote about using uikit-sys as well.

Eventually, using cargo-dinghy in CI would also be a nice touch.

Anyway, I hope this is useful and gives some perspective.

[MichaelHills]

Success! I managed to play a wav directly with Rodeo. The API has changed a bit in Rodeo/CPAL latest versions compared to where bevy is at, so I don't know if something is broken with that integration now on the newer versions. (Short story is that the cpal stream seems to not be retained and gets dropped from memory and so the mixer weak reference is null.)

Here's what I did

• Rebased/squashed @simlay 's coreaudio-sys change on master and backed out the bindgen for the objective-c headers (wasn't needed)
• Manually fixed coreaudio-rs compile error, I think that code path is unused though
• No changes to rodio
• Hacked up cpal coreaudio by using the webaudio host as a "simple" starting point for a single output device, then hacked in a build_output_stream_raw implementation based off https://stackoverflow.com/a/14478420 and the original cpal macos stuff. I completely butchered this file, that's possibly why I had issues with bevy/rodio integration.

This might take a while to clean up the hacks. i.e. no mic input, and no device enumeration atm. There is a bunch of shared code with the macos coreaudio. The existing enumeration is macOS-specific. It might be hard to untangle this stuff.

I guess I'm initially thinking splitting up the cpal code from mod.rs / enumeration.rs to something like:

• mod.rs
• macos.rs
• macos_enumeration.rs
• ios.rs
• shared.rs

Then use target_os switch to pull in the correct implementation.

[MichaelHills]

I applied the audio_output.rs patch from here https://github.com/bevyengine/bevy/pull/310 and now I can finally play audio through Bevy and iOS. :)

[MichaelHills]

An update on where things are at:

• [x] With @simlay 's https://github.com/bevyengine/bevy/pull/539 we have an iOS example building out of the box! No local hacks required. No touch and no sound though.
• [x] @naithar already landed simulator metal support upstream and it's been released, so simulator works great
• [x] I landed a fix in winit so that starting the app in landscape mode the window size works fine (needs new winit release though)
• [ ] I'm working with @simlay with updates to coreaudio-sys https://github.com/RustAudio/coreaudio-sys/pull/33 and coreaudio-rs https://github.com/RustAudio/coreaudio-rs/pull/69 as part of the audio support. Meanwhile I've been reading up on coreaudio docs to understand best to how implement the cpal host properly and productionize my hack.
• [ ] https://github.com/bevyengine/bevy/pull/334 seems to have stalled. I posted my touch support diffs earlier in this thread if you scroll up, that's what I'm currently using. If someone wants to look at that PR and look at my diffs and take over touch support that'd be helpful. 👍
• [ ] I filed a crash issue over at winit https://github.com/rust-windowing/winit/issues/1705 but I can't seem to reproduce it on the latest bevy ios example even though I could reproduce it earlier. Once I get to rebasing my game to latest bevy I'll test it again.

Once touch and audio are working, I think we can call it as Bevy having iOS support? There might be bugs here and there and we can fix them as we go.

[cart]

334 seems to have stalled. I posted my touch support diffs earlier in this thread if you scroll up, that's what I'm currently using. If someone wants to look at that PR and look at my diffs and take over touch support that'd be helpful.

Lol I can't find the diffs you're talking about because apparently my brain doesn't work :smile: I think I'll address my own comments in that pr, maybe apply your diff, then merge it. Alternatively i could address my comments and you could create a follow up pr (that way you get proper attribution).

Once touch and audio are working, I think we can call it as Bevy having iOS support? There might be bugs here and there and we can fix them as we go.

Yup that sounds good to me! I'm so happy that this came together so quickly :heart:

[MichaelHills]

Lol I can't find the diffs you're talking about because apparently my brain doesn't work 😄 I think I'll address my own comments in that pr, maybe apply your diff, then merge it. Alternatively i could address my comments and you could create a follow up pr (that way you get proper attribution).

I had implemented my own touch support in parallel to that PR because I didn't know about that PR. My comment is buried further up this page https://github.com/bevyengine/bevy/issues/87#issuecomment-683432609 (had to click load hidden items).

In my version I had the following and avoided using Input since it didn't map well. However, in my game code I never used the just_*. I just used active_touches to power my on-screen virtual joysticks and local system state to keep track of which touch id was which joystick. Perhaps the just_* stuff would be useful for UI buttons and things like that.

#[derive(Default)]
pub struct TouchInputState {
    pub active_touches: HashMap<u64, ActiveTouch>,
    pub just_pressed: HashSet<u64>,
    pub just_released: HashSet<u64>,
    pub just_cancelled: HashSet<u64>,
}

I also didn't implement TouchMotion event. I wasn't super clear on what's the correct API. Doing both TouchInputState and TouchMotion event lines up the closest with existing mouse implementation. Naming-wise TouchInputState would be better as TouchInput to line up with Input.

Probably best to just clean up and merge that other one and then I can follow up with anything useful from my diff. e.g. having dx() and dy() helpers was useful for calculating distance moved from when the touch was initiated. Might want to call it delta and return a Vec2 instead or something, not sure.

[MichaelHills]

Just an FYI, my personal situation has changed and I'll be strapped for time to continue developing my game / Bevy iOS support for the time being. I intend to finish off the audio support but will hand over a WIP cpal branch if I need to. Happy to keep helping out with testing though!

Once all the other pieces land, I still plan to improve the iOS example project with the following:

• add assets file/directory reference to show how to use assets (purely because I did it the wrong way at first which led me down the bad path of Xcode compressing/destroying the PNGs)
• add some png texture
• add audio background track (and maybe a press a button to play a sound)
• some kind of touch support, maybe just adding camera pitch/yaw look around

[CleanCut]

@MichaelHills Thank you for all your hard work. I love being around great contributors like you who push projects forward in meaningful ways. I hope you are able to return to your game in the future!

[cart]

@MichaelHills I agree with @CleanCut. You've done a bunch of great work already. Don't feel pressured to put in more work than you want to!

[dvtkrlbs]

The new version of rodio is now released. I am working on PR. (this helps with audio not working on macOS and ios)

[naithar]

I've made some changes to original @MichaelHills touch support diff here: In my fork
I plan to make a PR if #334 wouldn't be moving anywhere if that's okay :)
I'll also run some more test just to be sure that everything is working as intended.

[MichaelHills]

Update on audio:

• [x] https://github.com/RustAudio/coreaudio-sys/pull/33
• [ ] https://github.com/RustAudio/coreaudio-rs/pull/72
• [ ] https://github.com/RustAudio/cpal/pull/485

[dvtkrlbs]

Another problem is it takes a long time for a new version of rodio to release on crates.io. So even if these issues gets fixed you wont be able use them for a long time (rodio makes new releases for every 4 months or so) so you need to maintain your own bevy-audio (since crates.io crates cant use git dependencies).

[MichaelHills]

Another problem is it takes a long time for a new version of rodio to release on crates.io. So even if these issues gets fixed you wont be able use them for a long time (rodio makes new releases for every 4 months or so) so you need to maintain your own bevy-audio (since crates.io crates cant use git dependencies).

I haven't needed to modify Rodio. Does that mean we're ok? Or at worst can just use [patch.crates-io] to change the cpal version if there's a significant version bump.

[MichaelHills]

I've made some changes to original @MichaelHills touch support diff here: In my fork
I plan to make a PR if #334 wouldn't be moving anywhere if that's okay :)
I'll also run some more test just to be sure that everything is working as intended.

@naithar Looks like no movement on #334, want to put up yours? Or give some diffs so that #334 can be landed with original attribution, and then put up another PR for enhancements?

[naithar]

@MichaelHills yeah, seems like it. I'll rebase and make a PR soon.

[naithar]

Well, I found our problem! You need to disable XCode "Remove text metadata from PNG". I disabled everything though just to be sure... I guess the image loading library can't handle whatever XCode does to it.

this shouldn't be required anymore after https://github.com/bevyengine/bevy/pull/693. Listing assets folder in Copy bundle resources is enough for assets to work. Tested with iOS device and macOS .app file.

Also there is no need for a workaround for set_current_dir as PR seems to be solving this too.

[francesca64]

Is there any interest in using cargo-mobile to simplify generating Xcode projects / building / running on device / etc?

[cart]

Medium-to-long term I might be interested in adopting higher level abstractions. Short term I'd rather keep it simple, encourage people to become familiar with the "native" mobile tooling, and maintain control over "official" templates. We will likely create an official bevy_template repo in the near future.