Sony IMX500 & IMX501: The world’s first image sensors with in-built AI processing

Sony‘s image sensors are well-known in the technology industry, with applications in smartphones, DSLRs, as well as various industrial uses. Now, the Japanese company has announced a pair of new image sensors, which they’re calling the “world’s first intelligent vision sensors with AI processing”.

The Sony IMX500 and IMX 501 sensors feature built-in AI-based processing, which means that machine-learning computing tasks can be handled by the sensor on its own. As a result, external processors and memory hardware aren’t required—which could result in the potential for faster and more secure AI cameras.

Both models are 1/2.3″ 12.3MP sensors, with IMX500 samples shipping from April 2020 and Sony set to begin shipments for the IMX501 in June 2020. It’s worth noting that we are unlikely to see either of the new sensors appearing in smartphone or mobile cameras anytime soon. Instead, Sony says that the sensors will have a wide range of applications in cloud-based industrial and retail industries.

“The spread of IoT has resulted in all types of devices being connected to the cloud, making commonplace the use of information processing systems where information obtained from such devices is processed via AI on the cloud. On the other hand, the increasing volume of information handled in the cloud poses various problems: increased data transmission latency hindering real-time information processing; security concerns from users associated with storing personally identifiable data in the cloud; and other issues such as the increased power consumption and communication costs cloud services entail.”

So, what are the benefits?

Sony explains that the sensors’ high-speed AI processing addresses the issue of data transmission latency. When shooting a video with a conventional sensor, each frame is individually processed by AI—this can result in latency and poor real-time performance.

Instead, the new sensors will handle ISP and AI processing on its own, with the whole process done in a single video frame. As a result, AI-based real-time tracking in recorded videos can be done with “high precision”, which opens up potential use for retail and industrial applications.

For example, cameras with the new sensors can be installed at shop entrances to detect the number of entrants within a certain time period, while applications even extend to inventory count. Sony says that the AI models of the cameras can then be rewritten to suit a customer’s needs, whether it is to track entrants, identify customer behavior, and more.

And with the continued development of self-service, cashierless stores, Sony says that its new sensors will continue to enable such services.

“The AI capability makes it possible to deliver diverse functionality for versatile applications, such as real-time object tracking with high-speed AI processing. Different AI models can also be chosen by rewriting internal memory in accordance with user requirements or the conditions of the location where the system is being used.”

Additionally, Sony says that the onboard ISP and AI processing capabilities of the sensors help to address security and privacy concerns. If image data is sent to the cloud or external hardware for processing, this could potentially leave the data susceptible to malicious parties. When everything is handled on-board the sensor, only the metadata is output, which reduces security risk.

As mentioned, samples of the IMX500 are already being shipped for JPY10,000 (~RM405) while the IMX501 is set to ship in June for JPY20,000 (~RM811). We don’t have any official word on when we will see the sensors appearing in actual products, although we could see them appear sometime in 2020 or 2021.

For more details, read Sony’s full statement here.