New Uses for Smartphone AI: A Short Commentary on Recording History and Privacy

This opinion piece is reactionary to recent announcements.

Having just attended the Huawei keynote here at the IFA trade show, there were a couple of new features enabled through AI that were presented on stage that made the hair on the back of my neck stand on end. Part of it is just an impression on how quickly AI in hand-held devices is progressing, but the other part of it makes me think to how it can be misused.

Let me cover the two features.

"Real-Time Multi-Instance Segmentation"

Firstly, AI detection in photos is not new. Identifying objects isn’t new. But Huawei showed a use case where several people were playing musical instruments, and the smartphone camera could detect both the people from the background, and the people from each other. This allowed the software to change the background, from an indoor scene to an outdoor scene and such. What this also enabled was that individuals could be deleted, moved, or resized. Compare the title image to this one, where people are deleted and the background moved.

What does this mean? People can be removed from photos. Old lovers can be removed from those holiday photographs. Individuals can easily be removed (or added) from the historical record. The software would automatically generate the background behind them (if it’s the original background), and the size of people could even be changed. This was not only photographs, but video. The image blow shows one person increased in size, but it could just as easily be something significant.

Now I know that these algorithms already exist on photo editing software on a PC, if you know how to use it. I know that the demo that Huawei showed on stage was more of a representative aspect to AI on a smartphone, but I could imagine something similar coming to a smartphone, and being performed on a smartphone, and the goal to make it as easy to use as possible on a smartphone. How we in future might interpret the actions of our past selves (or past others) may have to take into account the level of access (and ease of use) in the ability to modify images and video.

Detecting Health Rate with Cameras

The second feature was related to Health and AR. By using a pre-trained algorithm, Huawei showed the ability for your smartphone to detect your heart rate simply by the front facing camera (and assuming the rear facing camera too). It does this by looking at small facial movements between video frames, and works on the values it predicts per pixel to get an overall picture.

Obviously, it isn’t meant to be used as a diagnostic tool (at least, I hope not). I could imagine similar technology being used with IP cameras for a home security system perhaps, and when it detects an elderly relative in distress, it can perform the appropriate action. But it lends itself to abuse, if you are able to use it on other people unsuspectingly. Does that constitute an invasion of privacy? Does it work on these smartphones with 10x zoom? I’m not sure I’m qualified to answer those questions. 

A big part of me wants to see technology moving forward, with development and progression from generation to generation. But in seeing these two technology features today, there’s the tiniest part that doesn’t sit right, unless the correct security procedures are in place, such as edited images/videos have a signature marker, or only pre-registered people on a smartphone can have their heartbeat measured. Hopefully my initial fears aren't as serious as they first appear.