Essentials: Add Photometer

[Mrnikbobjeff]

Description:
Adding a photometer api to expose brightness level
API Declaration
class Photometer

public double Intensity { get; }
public EventHandler<LuminosityChangedEventArgs> IntensityChanged;

Additional Classes or Structs needed:

public class LightChangedEventArgs : EventArgs
    {
        public LightChangedEventArgs(LightData intensity) => Intensity = intensity;

        public LightData Intensity { get; }
    }

    public readonly struct LightData
    {
        public LightData(double lx) =>
            Illuminance = lx;

        public double Illuminance { get; }
    }

Documentation links for supported platforms:
UWP: https://docs.microsoft.com/en-us/windows/uwp/devices-sensors/use-the-light-sensor
Android: https://developer.android.com/guide/topics/sensors/sensors_environment
iOS is a bit hacky, xamarin does not expose IOKIT, thus we would need to pinvoke a bit.
IOS: http://iphonedevwiki.net/index.php/AppleISL29003

VS bug #735661

[mattleibow]

I think we should call this AmbientLightSensor...

[mattleibow]

I spotted this at the top of the wiki:

I/O Kit is a low-level framework communicating with hardware or kernel services. Although it is a public framework, Apple discourages developers from using it, and any apps using it will be rejected from App Store.

[mattleibow]

This does not mean we can't/won't add this API, but it moves lower down the list as it is not support on one of the core platforms. I am sure we will accept a PR, but to do the work, we will need a specific use case where this is essential.

[Mrnikbobjeff]

Sure, for the use case I have something. We are running a custom device with android installed for a uni project. This device has every available android sensor stiffen in them. Each of these stations should allow inspecting all sensor values plus some additional configuration. An ambient light sensor is essential for this as it allows us to calculate the current weather pattern from the light/humidity/barometer values. I have a fork with working implementations for Android and uwp. I will look into integrating iOS and create a proposal once I'm through

[Mrnikbobjeff]

If there is still a willingness to include this in XE I would create a pr soon. I have working UWP and Android implementations. iOS works as well, I still have memory leaks though because memory management with IOKit and PInvoke is a real bummer. Otherwise I'd just release them as a separate module if the use case is to fringe for Essentials :)

[mattleibow]

I think before we can accept this, we need to establish if the various stores will reject the app if we talk to those APIs.

If they don't, we can then evaluate if this is something that we want to support for just a few platforms, and not all.

[Mrnikbobjeff]

This can either be closed or accepted as non ios inclusive, which is pretty major. I have a working implementation for other platforms and for ios, but apple will reject any app using the light sensor.

[newky2k]

As this is not possible on all platforms, it's unlikely to be implemented and so we are going to close this off.

[LuckyDucko]

Sorry to revive, @mattleibow mentioned on the Brightness API #296 that I should make a proposal for some form of Light Sensor, and I was led here.

@Mrnikbobjeff mentioned that he had a working version of this API, minus iOS due to private headers.

To get around this limitation, others have used the EXIF data from a Video/Photo Stream(?). With this data, run through a simple enough calculation, we are able to get an approximation for lux on iOS.

Below is me explaining how I worked about getting the EXIF data, along with explanation on why its shoddy code 😆

First off, I would like to mention that this is the first time I have delved into this level of complexity within my code, so there is a lot to improve upon, and I may be missing something key. However, I managed to create an approximation of 'Brightness sensor' for iOS use that _shouldn't_ get booted for using iOS private API's.

I made a very ramshackle example using the Xamarin iOS manual camera controls repo as boilerplate for my tinkering. I did a very small addition to what is 90% their work, and without it I was going to drop this attempt entirely.

This is how I got the brightness values
Note: again, this is code I hacked away at haphazardly and doesn't represent anything close to what I imagine the proper code to look like

private double GetBrightness(CMSampleBuffer sampleBuffer)
{
    try
    {
        var metaData = sampleBuffer.GetAttachments(CMAttachmentMode.ShouldPropagate);
        var exif = metaData.ValueForKey(new NSString("{Exif}"));
        var fNumber = exif.ValueForKey(new NSString("FNumber"));
        var exposureTime = exif.ValueForKey(new NSString("ExposureTime"));
        var ISOSpeed = Regex.Replace(exif.ValueForKey(new NSString("ISOSpeedRatings")).ToString(), "[^.0-9]", "");
        var brightness = (50.0 * (Double.Parse(fNumber.ToString()) * (Double.Parse(fNumber.ToString())) / (Double.Parse(exposureTime.ToString()) * Double.Parse(ISOSpeed))));
        return brightness / 15000.0;
    }
    catch (Exception)
    { }
    return 0.5;
}

For a full context, I recommend looking at the below attached zip to see how this falls in with everything else.

This approach has some downsides

1. Its not a _real_ lux meter
2. The 15000.0 value is based off my testing by putting the camera under direct bright LED lights
3. There is a BrightnessValue part in this EXIF data, it may be more reliable. (I created a version using this, however, I couldn't work out the ranges fully (-9 to, 20?) and it may change about in the future, or perhaps that was just a hiccup)
4. It requires video access permissions (as far as i'm aware), this is the real bummer of this approach, making it alot less seamless than the lux meters on other platforms.

Link to my example.
Xamarin_iOS___Manual_Camera_Controls.zip

Shoddy iOS code over

This API has potential for fine tuning apps that are designed for a specific audience/Scenario.

For one, it allows app developers to control the 'Brightness' of their app independent of screen brightness.

An example I was interested in would be where an app that would be used on a car mount.
When in direct sunlight (Bright situation) 'Unimportant' interaction UI elements would 'dull' and 'important' visual readout elements become more vivid.

[mattleibow]

@LuckyDucko maybe duplicate this comment on that other issue so it doesn't get lost.

This is an interesting approach... I think this is worth having a discussion on this. Especially the idea of making "compound" APIs that are not native. Would we rather not have this API for iOS or "build it"

Also, do these calculations change between devices, cameras or settings?

[Mrnikbobjeff]

I read about using the camera to try to get a light level reading, but I really do dislike the approach as much now as I did back then. The camera permission is just the top of the ugly iceberg. We do not receive a light level in a scientific unit but rather a value which does not evenly scale based on an implementation detail iirc. Also, this prevents applications where I would like to read the light level while using the camera app or at least make them work suboptimally. There would be no common denominator between all other platforms supplying scientific units and iOS. You are correct that this will approach will not prevent app store approval on ios. In my personal opinion I would rather not expose ios at all if this would be a desired feature. This was also the killer the last time we had this ticket open.

[LuckyDucko]

@Mrnikbobjeff

These are valid concerns to raise, however

We do not receive a light level in a scientific unit but rather a value which does not evenly scale based on an implementation detail iirc.

This calculation is created (i believe) based on the following website. While this website does not mince its words concerning its accuracy to an actual lux meter, i believe
A) with a bit of tweaking and comparison with the iOS private Lux sensor, we could get an idea of how accurate this actually is.
B) people shouldn't expect ultra accurate and consistent results between platforms because
C) the readouts even between Android devices can be vary, with tests (even though 2016), showing that this method (or something similar) on iOS can provide results in the same ballpark as the Android.

this prevents applications where I would like to read the light level while using the camera app or at least make them work suboptimally.

This hasn't shown to be the case in my experience. I did run into a hiccup when constantly reading this data while also showing the camera, however, when i read even more data, or swapped things about, it seemed to go away? I believe my issue is just because of its ramshackle nature, and this shouldn't be a problem in the actual implementation.

@mattleibow

Also, do these calculations change between devices, cameras or settings

I couldn't answer you without more testing, however, i believe that these would change depending on the camera module, similar to how the android values would change depending on the lux meter, and how its been calibrated, on different models. This calculation should take into account different potential camera settings, such as shutter speed.

I think this API should have a focus on more of a 'accurate enough' reading. The sensors, as shown in the above link (on mobile, i apologise for not finding more sources), can only be relied upon to provide 'accurate enough' readings on Android, and i believe this 'Compound' API provides an 'accurate enough' reading for iOS as well.

I believe now is the best time to add at least the Android and UWP implementations as part of a 'Xamarin Essentials Brightness API's release'.
(I apologise for formatting, its midnight and i'm on mobile)