Algorithm to determine flat surfaces and camera orientation without specialized hardware

Question

Modern augmented reality platforms such as Google's ARCore and Apple's ARKit seem to only operate on mobile devices, I'm guessing, because their underlying algorithms require specialized hardware that is typically available on these devices (accelerometers, gyroscopes, etc.). Since this hardware isn't available on laptops/desktops, I'm guessing these libraries would never be able to work off of mobile platforms.

Having said that I'm wondering if the following AR/image processing "capabilities" can be achieved via algorithms that do not require such specialized hardware:

• Flat surface detection inside an image or video; and
• Orientation/angle of the camera view inside an image or video

Meaning, if I am given an image or video (set of sequential images), can I detect flat surfaces in that media without specialized hardware (as in, on a Linux box running on server or PC)? Same goes for orientation/angle determination (meaning, determining the (x,y,z) coordinates of the camera within the "domain" of the image/video and determining which angles its pointed towards)?

Answer 1

You are quite right, Augmented Reality benefits a lot from a combination of video/image analysis together with the data from motion sensors. Quote from Apple ARKit: Understanding World Tracking:

To create a correspondence between real and virtual spaces, ARKit uses a technique called visual-inertial odometry. This process combines information from the iOS device’s motion sensing hardware with computer vision analysis of the scene visible to the device’s camera. ARKit recognizes notable features in the scene image, tracks differences in the positions of those features across video frames, and compares that information with motion sensing data. The result is a high-precision model of the device’s position and motion.

So, having only the image/video certainly decreases the amount of useful and diverse information available to the algorithm. Now, you still have an option to supply your desktop machine with a data captured from accelerometer, gyroscope, etc. It's still totally fair, since you probably have not captured the video directly with your Linux machine.

Now, if you want to exclude any non-video information at all, you are back to classic and basic computer-vision. See, for example, the documentation for OpenCV detection of planar objects.