Roblox is launching a generative AI that builds 3D environments in a snap

By Scott J Mulliganarchive page September 6, 2024

Collected at: https://www.technologyreview.com/2024/09/06/1103707/roblox-is-launching-a-generative-ai-that-builds-3d-environments-in-a-snap/

Roblox plans to roll out a generative AI tool that will let creators make whole 3D scenes just using text prompts, it announced today. 

Once it’s up and running, developers on the hugely popular online game platform will be able to simply write “Generate a race track in the desert,” for example, and the AI will spin one up. Users will also be able to modify scenes or expand their scope—say, to change a daytime scene to night or switch the desert for a forest. 

Although developers can already create similar scenes like this manually in the platform’s creator studio, Roblox claims its new generative AI model will make the changes happen in a fraction of the time. It also claims that it will give developers with minimal 3D art skills the ability to craft more compelling environments. The firm didn’t give a specific date for when the tool will be live.

Developers are already excited. “Instead of sitting and doing it by hand, now you can test different approaches,” says Marcus Holmström, CEO of The Gang, a company that builds some of the top games on Roblox.  “For example, if you’re going to build a mountain, you can do different types of mountains, and on the fly, you can change it. Then we would tweak it and fix it manually so it fits. It’s going to save a lot of time.”

Roblox’s new tool works by “tokenizing” the 3D blocks that make up its millions of in-game worlds, or treating them as units that can be assigned a numerical value on the basis of how likely they are to come next in a sequence. This is similar to the way in which a large language model handles words or fractions of words. If you put “The capital of France is …” into a large language model like GPT-4, for example, it assesses what the next token is most likely to be. In this case, it would be “Paris.” Roblox’s system handles 3D blocks in much the same way to create the environment, block by most likely next block. 

Finding a way to do this has been difficult, for a couple of reasons. One, there’s far less data for 3D environments than there is for text. To train its models, Roblox has had to rely on user-generated data from creators as well as external data sets. 

“Finding high-quality 3D information is difficult,” says Anupam Singh, vice president of AI and growth engineering at Roblox. “Even if you get all the data sets that you would think of, being able to predict the next cube requires it to have literally three dimensions, X, Y, and Z.”

The lack of 3D data can create weird situations, where objects appear in unusual places—a tree in the middle of your racetrack, for example. To get around this issue, Roblox will use a second AI model that has been trained on more plentiful 2D data, pulled from open-source and licensed data sets, to check the work of the first one. 

Basically, while one AI is making a 3D environment, the 2D model will convert the new environment to 2D and assess whether or not the image is logically consistent. If the images don’t make sense and you have, say, a cat with 12 arms driving a racecar, the 3D AI generates a new block again and again until the 2D AI “approves.”

Roblox game designers will still need to be involved in crafting fun game environments for the platform’s millions of players, says Chris Totten, an associate professor in the animation game design program at Kent State University. “A lot of level generators will produce something that’s plain and flat. You need a human guiding hand,” he says. “It’s kind of like people trying to do an essay with ChatGPT for a class. It is also going to open up a conversation about what does it mean to do good, player-responsive level design?”

The new tool is part of Roblox’s push to integrate AI into all its processes. The company currently has 250 AI models live. One AI analyzes voice chat in real time and screens for bad language, instantly issuing reprimands and possible bans for repeated infractions.

Roblox plans to open-source its 3D foundation model so that it can be modified and used as a basis for innovation. “We’re doing it in open source, which means anybody, including our competitors, can use this model,” says Singh. 

Getting it into as many hands as possible also opens creative possibilities for developers who are not as skilled at creating Roblox environments. “There are a lot of developers that are working alone, and for them, this is going to be a game changer, because now they don’t have to try to find someone else to work with,” says Holmström.