"Spatial intelligence" and "world model" are very hot research directions in academia and industry recently, and the key step to the combination of virtual-real and simulated world mechanism is to create a vivid virtual world.

However, it is still difficult to create 3D virtual worlds that are rich and full of detail, while being highly editable and physically realistic.

To solve these problems, a joint team from the Institute of Automation of the Chinese Academy of Sciences and the University of Science and Technology Beijing has proposed for the first time a new 3D scene generation framework, SceneX, which can quickly generate high-quality 3D virtual scenes with simple text descriptions.

Whether it is vast natural scenery, or vibrant city streets, the model can easily cope with.

picture

Website: https://zhouzq1.github.io/SceneX/

Paper: https://arxiv.org/abs/2403.15698

More importantly, SceneX is also flexible enough to adapt the scene to the user's imagination.

With SceneX, the construction of virtual worlds will also be easier, simpler and more efficient.

SceneX opens a new era of efficient and realistic scene generation

SceneX breaks traditional limitations in an innovative way to comprehensively improve efficiency and realism, mainly consisting of two core modules:

PCGHub: By integrating multiple programmatic generation modules and encapsulating them as standardized apis, PCGHub provides a flexible platform that solves the limitations of a single generation module due to inherent algorithms and rules, greatly expanding the diversity and flexibility of resource generation.

PCGPlanner: As an intelligent planner, PCGPlanner efficiently coordinates PCGHub resources to complete scenario generation. The fully automated process includes scene decomposition, terrain generation, asset generation and retrieval, asset placement and other stages. The modular design ensures the consistency and geometric consistency of the generated results.

SceneX compacts time to hours, compared with traditional methods that take weeks to complete large-scale scene modeling, while enabling precise control of the details of the scene through simple text instructions. SceneX's ability to collaborate across modes gives unprecedented flexibility and control to scene generation.

picture

Figure 1: SceneX can automatically generate large-scale 3D natural scenes or cities based on text commands. The resulting models feature fine geometry, realistic material textures and natural lighting effects that can be seamlessly applied to industrial processes

PCGHub: Diversified programmatic asset and layout generation platform

The diversity of scenes cannot be separated from the diversity of assets.

To this end, PCGHub provides a platform that integrates rich programmatic generation modules and 3D assets, designed to address the limitations of traditional approaches and enhance content realism.

PCGHub provides 172 programmatic assets covering multiple fields such as natural environment and architecture, and is equipped with a variety of flexible layout generators such as scattering, grid, linear, etc., so that developers can easily create rich and diverse 3D scenes.

Table 1: An overview of the functions of each element in PCGHub

These assets can be diversified by adjusting geometric and material parameters.

263 core parameters were extracted from 2,362 original parameters and packaged into standardized apis, each with detailed documentation, including functional descriptions and parameter specifications, for easy user invocation and extension.

In addition, PCGHub contains 11,284 high-quality 3D static assets, greatly enriching the diversity of the repository.

The launch of PCGHub provides an efficient and flexible solution for generating diverse and realistic scenes.

Figure 2: API documentation for a tree programmatic generation module, API functionality, and sample generation results

PCGPlanner: Intelligent scenario generation and layout planning

PCGPlanner leverages the resources provided by PCGHub for efficient automated scenario generation. The whole process consists of four key stages:

(1) Scenario decomposition: Analyze the scenario according to user needs and list the required assets;

(2) Terrain generation: Build the basic terrain and apply the appropriate materials;

(3) Object generation and retrieval: generate or import the assets required by the scene according to the demand;

(4) Asset placement: Arrange assets in the scene according to different layout types and programmatic generators.

picture

Figure 3: The SceneX framework transforms user text input into different 3D scenes through four stages: scene decomposition stage, terrain generation stage, object generation and retrieval stage, and asset placement stage

A series of experiments verified SceneX's comprehensive advantages in quality, efficiency and scene editing. The main achievements are as follows:

Users and experts alike praise

When scoring the quality of the scene, users and professional designers were invited to participate in the test.

The results showed that SceneX generated scenes were unanimously approved by users and experts, and the ratings were almost equal.

This means that both regular users and industry experts are raving about the scenes SceneX generates.

Table 2: Comparative analysis of average Aesthetics score (AS) and average aesthetics expert score (AES)

Figure 4: Visualization of personalized editing results in different scenarios

Figure 5: SceneX scene generation effect comparison

What you see is what you get

SceneX shows excellent semantic understanding and generation ability by evaluating the matching degree between prompt words and scenes.

Whether it is the quiet riverside scenery or the bustling city scene, SceneX can perfectly reproduce the imagination of users and achieve "what you see is what you get".

Figure 6: Large-scale scene and city generation results

Unparalleled efficiency

Timing is everything, and SceneX makes scene generation fast and accurate.

The results show that it takes only a few minutes to create a magnificent natural landscape, and it is also surprisingly fast to create a large city.

picture

Conclusion

SceneX framework shows the great potential of programmatic content generation combined with multi-agent workflow, and provides a new solution for large-scale scene generation.

Through its strong controllability and high-quality generation capabilities, SceneX opens up new possibilities in the fields of virtual world construction, game development, film and television production.

In the future, SceneX will further optimize the generation process and provide users with a more efficient and convenient scene generation experience.

Reference data: https://arxiv.org/abs/2403.15698