Sync ImageReader and SurfaceTexture with Android Camera2 output

Issue

ImageReader and SurfaceTexture is async from app side. SurfaceTexture.OnFrameAvailableListener and ImageReader.OnImageAvailableListener are coming in different time.

Now I will make an AR App. I calculate the object motion with the image from ImageReader and output the object motion information. On the other hand. Call updateTexImage to render background. But the question is the object motion has obviously latency behind background rendering.

The workflow is below:

Camera2->ImageReader->calculate object motion -> Render a virtual object with object motion information

Camera2->SufaceTexture->Render backgroud with updateTexImage

the upateTexImage and rendering-virtual-object is call in Render.onDrawFrame

So obviously the question is how to Sync ImageReader and SurfaceTexture with Android Camera2 output

Solution

The easiest option is not to use two data paths, and instead either do image analysis on the SurfaceTexture buffer (either in EGL or read back from GPU to CPU for analysis), or use the ImageReader buffer to draw everything with.

If that's not feasible, you need to look at the timestamps (https://developer.android.com/reference/android/graphics/SurfaceTexture.html#getTimestamp() and https://developer.android.com/reference/android/media/Image.html#getTimestamp()). For the same capture, the two paths will have the same timestamp, so you can queue up and synchronize your final drawing by matching them up.

Answered By - Eddy Talvala