There are encouraging advances in prosthetic vision for the blind, including retinal and cortical implants, and other “sensory substitution devices” that use tactile or electrical stimulation. However, they all have low resolution, limited visual field, and can display only few gray levels (limited dynamic range), severely restricting their utility. To overcome these limitations, image processing or the imaging system could emphasize objects of interest and suppress the background clutter. We propose an active confocal imaging system based on light-field technology that will enable a blind user of any visual prosthesis to efficiently scan, focus on, and “see” only an object of interest while suppressing interference from background clutter. The system captures three-dimensional scene information using a light-field sensor and displays only an in-focused plane with objects in it. After capturing a confocal image, a de-cluttering process removes the clutter based on blur difference. In preliminary experiments we verified the positive impact of confocal-based background clutter removal on recognition of objects in low resolution and limited dynamic range simulated phosphene images. Using a custom-made multiple-camera system, we confirmed that the concept of a confocal de-cluttered image can be realized effectively using light field imaging.
Grants
01/2016 – 01/2021, Active confocal imaging for visual prostheses, US Department of Defense, Vision Research Program, W81XWH-16-1-0033 (Co-PI)
02/2015-10/2015, Active confocal imaging for visual prostheses, Grant from the Promobilia Foundation, Stockholm, Sweden #14222 (PI)
Publications
Pamir, Z., Jung, J. – H., & Peli, E. (2022). Preparing participants for the use of the tongue visual sensory substitution device. Disability and Rehabilitation: Assistive Technology , 17(8), 888-896. Publisher’s Version
Peli, E., Jung, J. – H., & Qui, C. (2022). Motion parallax in object recognition. US Patent No. US11270451B . US patent.
Avraham, D., Jung, J. – H., Yitzhaky, Y., & Peli, E. (2021). Retinal prosthetic vision simulation: temporal aspects. Journal of Neural Engineering , 18 (4), 0460d9. Publisher’s Version
Avraham, D., Jung, J. – H., Yitzhaky, Y., & Peli, E. (2021). Simulating the effects of persistence and perceptual fading in retinal prosthetic vision. 12th The Eye and the Chip World Research Congress 2021.
Peli, E., & Jung, J. – H. (2020). Active confocal imaging systems and methods for visual prostheses . US Patent No. US10857036B2 . US patent.
Pamir, Z., Canoluk, M. U., Jung, J. – H., & Peli, E. (2020). Poor resolution at the back of the tongue is the bottleneck for spatial pattern recognition. Scientific Reports , 10 (1), 2435. Publisher’s Version pamir-etal-2020-scirep-tracingstudy-data_3.xlsx
Jung, J. – H., & Peli, E. (2018). Light-field background de-cluttering for visual prostheses. In Imaging and Applied Optics 2018 (3D, AO, AIO, COSI, DH, IS, LACSEA, LS&C, MATH, pcAOP) (pp. 3Tu3E.3) . Optical Society of America. Publisher’s Version
Han, S. ‘E., Qiu, C., Lee, K. R., Jung, J. – H., & Peli, E. (2018). Word recognition: re-thinking prosthetic vision evaluation. Journal of Neural Engineering , 15 (5), 055003 . IOP Publishing. Publisher’s VersionAbstract wordrecognition2018datasharing_1.xlsx
Qiu, C., Lee, K. R., Jung, J. – H., Goldstein, R., & Peli, E. (2018). Motion Parallax Improves Object Recognition in the Presence of Clutter in Simulated Prosthetic Vision. Translational Vision Science & Technology , 7 (5), 29-29. Publisher’s Version motion_parallax_data_sharing_2.xlsx [PDF]
Peli, E., & Jung, J. – H. (2018). Active confocal imaging systems and methods for visual prostheses . US Patent No. US10130513B2 . US Patent.
Jung, J. – H., Qiu, C., & Peli, E. (2017). Background de-cluttering for object recognition in video-based visual prostheses. 10th The Eye and The Chip World Research Congress . Detroit, MI.
Jung, J. – H., Pu, T., & Peli, E. (2016). Comparing object recognition from binary and bipolar edge images for visual prostheses. Journal of Electronic Imaging , 25 (6), 061619. Publisher’s Version
Jung, J. – H., Pu, T., & Peli, E. (2016). Comparing object recognition from binary and bipolar edge features. IS&T Human Vision and Electronic Imaging XXI. Publisher’s Version
Jung, J. – H., Aloni, D., Yitzhaky, Y., & Peli, E. (2015). Active Confocal Imaging for Visual Prostheses. Vision Research , 111 (June), 182-196. Publisher’s Version activeconfocalimaging_datasharing_2.xlsx (Best paper of the year by a trainee in SERI, Harvard Medical School)