Smarter AI Needs More Memory And Storage

This is the second of two articles on Celesta Capital’s TechSurge Summit on February 13, 2023. In this article I will include discussions of a semiconductor panel and also some comments by Internet Legend, Vint Cerf on ethical use of technology and how to get AI to be more useful.

Jason DiLullo, President of Qatalyst Partners chaired a session at the TechSurge Summit talking about the semiconductor market. This panel included Sanjay Mehrotra, President and CEO of Micron,;Rene Haas, CEO of Arm and Ronnie Chatterji, the White House CHIPS Act Coordinator.

Sanjay had a number of things to say regarding the semiconductor industry and in particular the semiconductor memory industry. He said that semiconductors form the backbone of everything in the world today. 20-30% of global economy is powered by semiconductors, directly or indirectly. The semiconductor industry is growing faster than the GDP through the rest of this decade. Micron is the only US manufacturer of memory, DRAM and NAND flash.

Semiconductor memory growth in the next decade will outpace overall semiconductor growth. All the trends around AI and autonomous vehicles, all need memory and this will contribute to economic growth. Today about 85% of semiconductors are manufactured in Asia. 98% of memory silicon is made in Asia. The major Asian countries that provide the supply chain and semiconductor manufacturing are concentrated in South Korea, China, Japan and Taiwan. There is demand for a balanced supply chain, hence so much interest in various national government to promote local growth in semiconductor manufacturing. This is behind the CHIPS act and other such funding. This is an attempt to diversify the global supply chain.

Rene said that business cycles are an inherent feature in the semiconductor industry. Companies must make big bets for supply and demand. We were in a supply-based crunch during the pandemic, which has switched to a market driven by demand today. He thinks these business cycles will continue although overall demand for electronics in everything will drive demand higher for the foreseeable future.

Ronnie joined the US Commerce Department in 2021. His boss asked him to find out about the cause of automotive price increases. A lot of the price increases were driven by a diminished supply of semiconductor chips during the pandemic. Even very simple old technology chips were in short supply. He said that the CHIPS act will create new jobs, particularly for those without college degrees. He thinks that this is a key time to make investments in the semiconductor industry and that the development of Silicon Valley after WWII was tied to significant investments in technology in the area (e.g. NASA and various defense contractors).

Rene said that AI is the biggest driver of semiconductor growth, referring to generative AI such as Chat GPT. This is a big opportunity, but to enable it we will need more efficient data processing, something Arm is known for. He said that current worldwide data center energy use is about 1-2% of total produced power but in some countries, this is closer to 10%, which is not sustainable. New data centers are being built with fixed real estate and power availability, which will require more efficient electronics to meet growing demand.

Sanjay said that generative AI and other sophisticated AI will drive memory demand. He said that the Chat GPT servers requires five more memory than a standard server. Smarter AI need compute, memory (lots of storage and memory) and connectivity. All of this needs to come together for AI to become ubiquitous. Low power and high performance, high BW memory is important as well. The Chat GPT Supercomputer is among the top 5 in the world. He provided some details about the hardware needed for Chat GPT. It has 285,000 CPU cores, 10,000 GPUs and 400 GB/s connectivity. He also said that trends for memory and computer to come together through packaging or innovative solutions to bring memory and logic together in the coming years.

Ronnie said that the Chips and Science Act will fund $52.7 B. This is large for a government industry investment in the US. He moved to White House from the Commerce Department where he coordinates the CHPS act. The government money is leveraging money from private industry, up to $300B since the CHIPS act was proposed. He also said that we need a broader supply chain, including smaller companies. Private sector has more information than public sector on the supply chain.

Sanjay said that in addition to the $52B to bridge the cost gap with Asian semi manufacturing. The cost gap today is 35-45% for production in US versus Asia and primarily due to incentives from Asian governments. Another important element of CHIPS is the investment tax credits. Up to 25% ITC will also help to bridge the gap with Asia. Both the investment and tax credits are needed. Although memory wafers generate about 30% of all semi revenue WW they comprise about 60% of all semi wafers. The CHIPS needs to invest in memory as well as logic, sensors and other devices as well. Sanjay said that memory is under price pressure and that you must make your own plants to survive, you can’t build through 3^rd parties.

Rene said that IP companies and the IP industry as it is today didn’t exist 30 years ago. System on Chip (SoC) technology is running into road blocks with how much you can put into a monolithic die., particularly as we approach 2nm features. In the next 5-10 years he said we will see the disaggregation of the SoC where smaller specialized die made with different processes and different companies are combined in a package (these smaller die are often called chiplets). This will lead to lots of innovation and new IP opportunities. He said that chiplet packaging will be standard 4-5 years from now.

When asked about the trend for large hyperscalers to design their own hardware, Rene said that these companies have large internal workloads which require very specific functions and that these companies have the best information on how to solve their needs. As a consequence, the trend for hyperscalers to design their own chips will continue. They will use iP where it makes sense and also use off the shelf hardware.

Ronnie said that regional benefits from the CHIPS act can be immense. More people may be able to work where they live, rather than having to move. He stressed that there needs to be regional partnership with private companies.

Rene said that we need to get more ambitious and aggressive to solving the power efficiency problem for scaling up to meet the needs of future AI and other applications. For instance, he said that although there are goals to move to all EVs in 10 years, the power grid is not built to support this. Greater electronic power efficiency could reduce these power requirements, making this more achievable.

Sanjay said that the CHIPS act will create about 50,000 direct jobs in semiconductor industry and half a million indirect jobs. This requires that we bring in under-represented groups, blacks, Hispanics, people in rural areas. Within the next 10 years there will be a $100B Micron investment. Construction is a big part of these high paying jobs. He said that by 2030 Micron will have reduced its carbon emissions by 42%, versus 2018, in line with the Paris Accords. By 2025 their facilities will be 100% renewable energy powered.

Sanjay also encouraged investment in smart manufacturing technology. He said that their factories have over 500,000 sensors with lots of control points and many images of wafers. He said that they have 30PB of data that they would like to analyze and understand to create smarter manufacturing with faster equipment maintenance and better quality to meet demand.

Larina Yee from McKinsey and Co. interviewed Vint Cerf from Google. Vint was one of the people who helped to create the Internet. Regarding the Internet he said that all powerful technologies have the possibility to be used in a bad way. He had three ideas to countermand bad uses of the Internet (and other technologies). First keep people from doing bad things, and if you catch them doing bad things there should be consequences. Finally, having an ethic that you just shouldn’t do such things, and perhaps this would be the strongest countermeasure.

Vint went into the history of how the Internet came to be from the ARPAnet using packet switching networks. He was less enthusiastic about the general concept of the metaverse, but thought that there were useful capabilities with augmented reality where you combine the real world with information.

Vint very much liked John Hennessey’s statement that AI can amplify humans and that we can use computers to do things we can’t do ourselves or to save time. He too thought that a generative AI, like Chat GPT, if trained on curated factual materials could do a better job, but because computers work on algorithms, rather than reasoning like we do, it will get things wrong and even if trained on factual materials, AI can still get things wrong and he didn’t think generative AI is ready to use in business.

However, AI programs such as alpha fold, which predicts how proteins will fold, is enormously helpful. He also mentioned that making information available on things that don’t work could be very helpful and suggested that we should publish our mistakes as well as our successes. This is also a good lesson for entrepreneurs, if something doesn’t work find this out as soon as possible and change or adapt your goals if needed.

A big challenge for AI is to get the system to formulate a working model of what it knows. He gave an example. Any two-year old understands what a table is, but computers have a hard time understanding things about the real world. This is gained by interacting and working with the real world, something that John McCarthy worked on in the early years of AI research.

Sanjay Mehrotra of Micron, spoke about the importance of memory to the semiconductor industry and with Rene Haas and Ronnie Chatterji discussed topics such as the CHIPS act, chiplets and the IP industry. Vint Cerf spoke about an ethical Internet and how to make better AI.