Remote NetApp array over photonic link nears local access speed

A proof of concept (PoC) has shown that metro-distance access to a NetApp flash array is almost as fast as local access with GPU training time increases of less than 1 percent over 3,000 km.

The PoC also shows that cheaper power costs in rural locations means GPU training runs there, while using metro datacenter flash storage, can cut energy costs by up to 30 percent.

The scenario described here was conceived by the Innovative Optical and Wireless Network Global Forum (IOWN), an international industry consortium founded in January 2020. It develops next-generation communication and computing infrastructure based on photonics (light-based technology) rather than traditional electronics. NTT, Intel, and Sony founded the group. Now there are more than 170 member organizations, including major players from telecom, semiconductors, IT, and other sectors (e.g. Microsoft, Nvidia, Cisco, Nokia, Samsung, Fujitsu, KDDI, Orange, Red Hat, and more). 

Their pitch is that high energy costs in cities can limit the availability of GPU training datacenters. The data needed for GPU training is accumulated and stored in metro areas, and moving it to rural locations with cheap power is impractical. IOWN says the solution is to put the GPU servers in the remote rural location and pump the data out to them using IOWN's all-photonics network (APN) technology. This uses optical transport technologies to allow end-to-end disaggregated optical networking, connecting endpoints directly with optical paths. 

The PoC is described in a weighty "Green Computing with Remote GPU over APN (tsuzumi-7B)" document that outlines three situations tested: direct GPU server to remote storage array, GPU server access to local cache fed from the remote storage array (FlexCache), and GPU server access to a local cache fed by asynchronous replication (SnapMirror).

The networking links are 100G single-mode fiber (SMF), and NFS over RDMA or TCP is used while the GPU server trains a tsuzumi AI model. We've illustrated the setup:

With direct access from the GPU server to NetApp storage, the testers "demonstrated that the increase in training time when using tsuzumi on a remote GPU at distances from 100 km to 3000 km over APN is less than 1%." The GPU training time exceeded six days, while the data read time was less than 20 seconds with APN.

The use of a local cache fed by FlexCache or SnapMirror "in long-distance scenarios... leads to shortened read times."

The 27-page PoC document is a substantial read but adds the meat of the information we have summarized and much more on the energy cost side of things.