Flutterfire: [firebase_ml_vision] textRecognizer doesn't work on iOS (no cloudTextRecognizer workaround)

[Alecsplus]

Describe the bug
Same bug of issue 1276
https://github.com/FirebaseExtended/flutterfire/issues/1276

Also in my case I have this problem only on iOS devices.
Text is recognized only with cloudTextRecognizer and not with textRecognizer method (with the same image).

This is a really great problem, because cloud recognition needs billing.

What we can do to to solve it?

Thanks in advance

To Reproduce
use textRecognizer() on iOS devices do not works.
cloudTextRecognizer() works well

Expected behavior
textRecognmizer should give a similar result to cloudTextRecognizer

Additional context
only iOS. Android works well.

[iapicca]

Hi @Alecsplus
can you please provide your flutter doctor -v ,
your flutter run --verbose
and your pubspec.yaml
or if possible a reproducible minimal code sample.
Thank you

[chaser79]

The FirebaseVision.instance.textRecognizer() is not recognizing any text on iOS device. It works perfect on an android device.

To Reproduce
Use FirebaseVision.instance.textRecognizer() on an image to try and recognize text.

void _read() {
    progressDialog.show();
    _takePicture().then((String filePath) async {
        final FirebaseVisionImage visionImage =
            FirebaseVisionImage.fromFilePath(filePath);
        final TextRecognizer textRecognizer =
            FirebaseVision.instance.textRecognizer();
        textRecognizer.processImage(visionImage).then((visionText) {
          print(visionText.text);
          _processText(visionText.blocks);
          textRecognizer.close();
        }).catchError((error) {
          progressDialog.dismiss();
          PlatformErrorAlert(context: context, text: error).show();
        });
    }).catchError((error) {
      progressDialog.dismiss();
      PlatformErrorAlert(context: context, text: error).show();
    });
  }

The print(visionText.text); is empty when using an iOS device.

I expect it to return a VisionText with the recognized text like it does on an android device.

Flutter doctor output

Doctor summary:
[✓] Flutter (Channel master, v1.16.4-pre.22, on Mac OS X 10.14.6 18G103, locale en-GB)
    • Flutter version 1.16.4-pre.22 at /Users/lukechase/Apps/FlutterSDK
    • Framework revision e4a61dfdea (12 hours ago), 2020-03-29 21:01:02 -0400
    • Engine revision 361d5d5320
    • Dart version 2.8.0 (build 2.8.0-dev.17.0 1402e8e1a4)


[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.3)
    • Android SDK at /Users/lukechase/Library/Android/sdk
    • Android NDK location not configured (optional; useful for native profiling support)
    • Platform android-29, build-tools 29.0.3
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_212-release-1586-b4-5784211)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 11.3.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 11.3.1, Build version 11C504
    • CocoaPods version 1.9.1

[!] Android Studio (version 3.6)
    • Android Studio at /Applications/Android Studio.app/Contents
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.
    ✗ Dart plugin not installed; this adds Dart specific functionality.
    • Java version OpenJDK Runtime Environment (build 1.8.0_212-release-1586-b4-5784211)

[✓] VS Code (version 1.43.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.8.1

[✓] Connected device (1 available)
    • iPhone XR • 00008020-001D6D883C45002E • ios • iOS 13.3.1

! Doctor found issues in 1 category.

[zelliot]

On Android textRecognizer() and cloudTextRecognizer() both work perfectly.

On iOS neither textRecognizer() or cloudTextRecognizer() work.

I've done a bit of digging on this and I think it is something to do with the image. It seems to be rotated or flipped on iOS.

Testing with a photo of my Australian Passport taken within the app on iOS, textRecognizer doesn't work at all but with cloudTextRecognizer it seems to work but returns everything back to front i.e. "AIJAЯТ2UA"

Flutter doctor output:
[✓] Flutter (Channel stable, v1.12.13+hotfix.9, on Mac OS X 10.15.4 19E266, locale en-AU)
• Flutter version 1.12.13+hotfix.9 at /Users/elliot/dev/flutter
• Framework revision f139b11009 (9 days ago), 2020-03-30 13:57:30 -0700
• Engine revision af51afceb8
• Dart version 2.7.2

[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.2)
• Android SDK at /Users/elliot/Library/Android/sdk
• Android NDK location not configured (optional; useful for native profiling support)
• Platform android-29, build-tools 29.0.2
• Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
• Java version OpenJDK Runtime Environment (build 1.8.0_202-release-1483-b49-5587405)
• All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 11.4)
• Xcode at /Applications/Xcode.app/Contents/Developer
• Xcode 11.4, Build version 11E146
• CocoaPods version 1.9.1

[✓] Android Studio (version 3.5)
• Android Studio at /Applications/Android Studio.app/Contents
• Flutter plugin version 43.0.1
• Dart plugin version 191.8593
• Java version OpenJDK Runtime Environment (build 1.8.0_202-release-1483-b49-5587405)

[✓] VS Code (version 1.43.2)
• VS Code at /Applications/Visual Studio Code.app/Contents
• Flutter extension version 3.9.0

[✓] Connected device (1 available)
• ePhone • 47e705ac2711c7e3436bbe0fda4b9ab3beeafbef • ios • iOS 13.4

• No issues found!

[lurajon]

Hi!
I experience the same problem;

`void _initializeVision() async {
final File imageFile = File(path);

if (imageFile != null) {
  await _getImageSize(imageFile);
}

await GallerySaver.saveImage(path);

final FirebaseVisionImage visionImage =
    FirebaseVisionImage.fromFile(imageFile);

final TextRecognizer textRecognizer =
    FirebaseVision.instance.textRecognizer();

print(visionImage);
final VisionText visionText =
    await textRecognizer.processImage(visionImage);


String text = "";
print(visionText.text);
for (TextBlock block in visionText.blocks) {
  print(block.text);
  for (TextLine line in block.lines) {
      print(line.text);
      text += line.text + '\n';
      for (TextElement element in line.elements) {
        print(element.text);
        _elements.add(element);
      }

  }
}

if (this.mounted) {
  setState(() {
    recognizedText = text;
  });
}

}`

pubspec.yml

When I open a camerastream however, and use fromBytes

FirebaseVisionImage.fromBytes( _concatenatePlanes(image.planes), _buildMetaData(image, _rotationIntToImageRotation(imageRotation)), ),
Text are recognized on the stream.

Maybe the issue is somewhere in FirebaseVisionImage.fromFile(imageFile);?

[junnyea]

Facing the same problem here. Value for visionText is empty

VisionText visionText =
await textRecognizer.processImage(visionImage)

[hariprasadiit]

For me with original photo it doesn't work but, If I use flutter_image_compress and use the resulted compressed image, on device text recognition works well. I think it has something to do with rotation and EXIF information.

[TehRaab]

For me with original photo it doesn't work but, If I use flutter_image_compress and use the resulted compressed image, on device text recognition works well. I think it has something to do with rotation and EXIF information.

I am also facing the same problem. If i am to use flutter_image_compress, how do you determine how much rotation is needed? I have noticed that the iPad Mini 2 only works in landscape. I believe this is because the camera hardware is actually physically installed at +/-90 degree angle.

[hariprasadiit]

@TehRaab I think flutter_image_compress automatically rotates the images based on EXIF information

[nickykln]

Hello, any progress on that issue ?
@hariprasadiit, I have try the flutter_image_compress workaround with no success.
Could you please share the piece of code you wrote as a workaround.

Thanks a lot for your help,

Nicolas

[robwafle-scripd]

Hi! We are experiencing this issue too. And, we're starting to have some issues with Android as well. Usually, might get an exception or two when starting the android app saying something along the lines of "please wait, still downloading ocr model from cloud", but now it won't download at all.

Also, I am getting some warnings when building my app:
[!] FirebaseMLVisionTextModel has been deprecated in favor of MLKitTextRecognition

It looks like the MLKit is going through a bit of a transformation at the moment: https://firebase.google.com/docs/ml-kit/recognize-text

This appears to be the pod file name that will replace the current podfile:
pod 'GoogleMLKit/TextRecognition'

But, replacing with this line didn't help either.

[SimonDziak]

Android works perfectly for me. iOS text detection does not work. It detects weird texts and outputs wrong information if it pick up something.

[TehRaab]

Android works perfectly for me. iOS text detection does not work. It detects weird texts and outputs wrong information if it pick up something.

I've had this issue for several months now and still haven't found a solution. I've you find one, please let us know. I raised a separate issue back in June - https://github.com/FirebaseExtended/flutterfire/issues/2770