[3D Spatial Scanning] Matterport Digital Twins: The Ideal Solution for Home Renovations

With Matterport, you can transform every room in your home into a space ready for renovation planning without moving a single piece of furniture.

Imagine being able to visualize a completely renovated living room without having to move any furniture—that is exactly what Matterport aims to achieve.

Matterport is embracing the concept of artificial intelligence (AI), exploring how advanced technologies in 3D semantic understanding and inpainting can bring new possibilities to digital twins.

Initially, Matterport focused on creating static spatial models that realistically replicated real-world spaces, laying the foundation for virtual tours and various consumer applications. However, static reconstruction alone is insufficient for actually modifying these spaces, evaluating their potential uses, or managing day-to-day operations. Therefore, Matterport has been developing advanced property intelligence tools that leverage semantic understanding to provide deeper insights and valuable information about properties.

Now, backed by the latest breakthroughs in generative AI technology, Matterport is expanding its efforts to create new content and experiences within Matterport spaces, further enriching how users experience and perceive these digital environments.

By combining Matterport's 10 years of experience in machine learning and artificial intelligence with the powerful capabilities of new generative AI tools, the company is working through Project Genesis to bring new design and furniture ideas to life with the click of a button—and the first step is the ability to instantly renovate any space.
 

What is Defurnishing?

Defurnishing is a vital technology in digital image processing and 3D modeling that removes furniture and movable objects from images of a space, recreating the room in an empty state.

This technique plays a crucial role in applications where the potential of a space needs to be clearly visualized, such as interior design, real estate, and virtual staging.

Defurnishing is one of the features being developed for Matterport digital twins and is divided into three steps:

1. Reconstruction: First, the space is captured and reconstructed to create a digital twin.

2. Understanding: Next, semantic understanding is applied to the reconstructed space to specifically identify pixels (within images) or mesh surfaces (within Dollhouse view) belonging to the furniture to be removed.

3. Synthesis: Areas blocked by furniture are not directly captured during the digital twin scan. Therefore, removing furniture creates blank pixels in images and holes in the mesh. It is necessary to use inpainting on the "blank areas" in images and apply completion and texturing to the holes in the mesh.

 

 

Understanding Semantic Segmentation

Semantic segmentation is a key computer vision technique that divides an image into different regions and assigns a specific category to each. Its goal is to assign categories like "floor," "wall," "window," or "table" to each pixel, accurately identifying objects and defining boundaries to provide a comprehensive understanding of a scene.

While object detection focuses on objects within bounding boxes and image classification labels an entire image, semantic segmentation allows for finer analysis of a scene, enabling deeper interpretation. Semantic segmentation is a foundational technology in the computer vision field, utilized in various sectors such as autonomous driving, medical imaging, and robotics.

In recent years, it has also become a crucial element in virtual interior design. When a space is first captured, the available primary data depicts the overall structure and appearance of the space. Semantic segmentation plays a vital role in deepening the understanding of content in Matterport spaces, enabling precise manipulation of elements, such as moving, editing, indexing, and deleting.

To effectively modify any element of a Matterport space, detailed semantic segmentation that identifies the primary components of the space is required.


The Role of Segmentation in Defurnishing

To remove furniture from digital twin images or 3D structures, one must first identify the pixels/mesh surfaces belonging to the furniture. Removing these pixels/surfaces often results in missing information because areas behind or under furniture are not visible during digital twin capture.

Therefore, after removing furniture, it is necessary to generate reliable image/3D content to fill these gaps. This process is called "image inpainting."

Inpainting is an advanced technique used in image editing and restoration, aimed at filling in missing or damaged parts of an image to make the whole appear natural and complete. The main objective is to seamlessly connect these areas with the surrounding image, maintaining the structural integrity and visual continuity of the image.

Many inpainting methods rely on segmentation masks that accurately specify both the object to be removed and the area to be subsequently filled. Differences or artifacts in the furniture segmentation mask can significantly impact the inpainting results. For example:

• Removing a part of the building instead of furniture can lead to significant structural hallucinations. For instance, instead of filling in part of a floor or wall, a doorway leading to a non-existent room might be generated.

• If furniture segmentation is inaccurate and fails to mask part of an object correctly, unintended fake objects may be drawn instead of the required "blank area" (typically recognized as wall or floor depending on the perspective).

• If an actual piece of furniture is not segmented, a false negative occurs, leaving remnants of furniture in the final result.

Therefore, obtaining accurate semantic segmentation is critical for achieving high-quality defurnishing.


Matterport’s Semantic Segmentation Approach

1. Data
Matterport performs semantic segmentation on 360-degree panoramic images using equirectangular projection to capture as much visual information as possible within a single frame. Context plays a vital role in computer vision tasks, especially when using state-of-the-art neural network frameworks like Vision Transformers.

2. Custom Ontology
Initially, Matterport used parts of the ADE20K ontology, which includes 150 categories commonly found in architectural environments. However, this method alone could not fully meet specific needs.

Matterport plans aim to remove all removable furniture while keeping built-in furniture intact. Common public datasets sometimes group these different types of furniture into one generic category—for example, simply classifying both standalone and built-in closets as "closet."

Therefore, to meet specific needs, it is necessary to create a custom dataset that annotates fine distinctions in furniture, while considering several other task-specific elements.

3. Network Architecture
Matterport decided to leverage the capabilities of the Vision Transformer architecture, which has seen success in various AI applications, specifically choosing ViT-Adapter as the foundation for segmentation experiments. This model improves upon the original Vision Transformer (designed to generate a single feature vector from image input) to handle image-to-image tasks requiring feature maps instead of single vectors.

Although ViT-Adapter was not specifically trained on 360-degree equirectangular images, it performed excellently when processing this type of data. It also showed good results despite not being designed to solve the ontology differences mentioned above.

4. Deployment
Recently, Matterport shifted semantic segmentation and depth estimation into key processing steps of the pipeline. Consequently, semantic segmentation is now executed every time an image is captured. Inference is performed in the cloud, allowing the system to handle sudden traffic fluctuations, simplify maintenance, and enable smoother updates.

5. 3D Semantic Understanding
Matterport has a unique strength in semantic understanding in 3D space. By incorporating 3D context into semantic segmentation, the company can more deeply understand the spatial and semantic connections within a captured space. Matterport innovatively utilizes the 3D Dollhouse view and combines information from multiple perspectives to significantly improve prediction accuracy. This advanced method enables more accurate and meaningful modifications.

A typical example is the defurnishing scenario, which requires a complex and accurate understanding of both the 2D and 3D features of the environment.

Technical Challenges and Limitations of Defurnishing

Even the most state-of-the-art semantic segmentation models are not perfect, and generalizing them effectively to unknown new data remains difficult. Given this reality, Matterport must develop strategies to correct errors or create alternatives.

While supervised semantic segmentation methods generally produce the best results, defining and managing ontologies is a significant challenge. Since these ontologies change easily depending on the application, data annotation must be performed frequently when making major adjustments. Therefore, the more a model can be trained using self-supervised methods, the less time, effort, and cost are required to adapt the segmentation model to new ontologies. On the other hand, the design of these ontologies themselves presents many challenges. Using defurnishing as an example, Matterport's goal is to remove "standalone" furniture while keeping "built-in" fixtures.

Determining whether furniture qualifies as "built-in" is a complex task that typically requires a comprehensive set of rules to ensure consistency and reproducibility in judgment. Without clear guidelines, the quality of data annotation can decline, consequently affecting the performance of the segmentation model.

---

Future Outlook

Self-Supervised Learning
Having previously conducted research on self-supervised learning, and with image-based models currently achieving success, Matterport is at an ideal point to invest further in this field.

Self-supervised learning offers significant benefits. For example, it can minimize reliance on annotated data, speed up training processes, and improve performance on specific tasks.

Integration of 3D Context
Integrating 3D context into the workflow is a promising approach to further evolving the process. Currently, Matterport's data aggregation method is passive, using heuristic weighting for features projected from multiple viewpoints. By researching ways to integrate 3D context during the training phase, it may be possible to develop viewpoint-independent features and enhance the model's understanding capabilities.

Furthermore, Matterport is researching the potential of end-to-end 3D technologies, examining whether directly processing 3D representations for semantic understanding could improve results. This includes re-evaluating reconstruction methods. Adopting cutting-edge technologies like Neural Radiance Fields (NeRF) or other innovative approaches could fundamentally alter current methods, potentially leading to significant improvements in model understanding and performance.

Multi-Task Models
The concept of multi-task models capable of performing multiple tasks simultaneously has long attracted significant attention. However, because such models must be maintained as a single integrated system, a strategy that employs a shared backbone across multiple models becomes more attractive.

As Matterport evolves its technology, finding the right balance between the benefits and complexity of multi-task models will be key to improving workflows and outcomes.

Open-Vocabulary Models
Another noteworthy area of development is open-vocabulary models. Traditional models are constrained by fixed ontologies, which limits their ability to respond to diverse customer needs.

In contrast, open-vocabulary models are freed from such constraints and can identify a wider range of objects and concepts without being bound to pre-defined categories.

This adaptability is highly valuable to Matterport, enabling deeper semantic understanding across diverse spaces and applications. Adopting an open-vocabulary approach is expected to significantly enhance the capability to meet diverse customer-specific needs while strengthening the interoperability of Matterport assets with other tools.

---

Summary
Expanding semantic understanding of spaces enables many new applications across various industries. Since a single ontology cannot satisfy all customer needs, Matterport finds great value in open-vocabulary technologies and other approaches not constrained by rigid ontology frameworks.

Another goal is to improve compatibility with resources and various tools. To this end, we are developing multiple integrated functions, working toward ensuring the final empty space produced is accurate and visually consistent.

---

Matterport PRO3 is a professional 3D spatial scanner capable of capturing high-resolution 134-megapixel imagery. When paired with Matterport Capture 3D scanning software for interior environments, it can quickly scan spaces of various sizes with a single click, instantly generating high-precision 2D floor plans and 3D virtual spaces.

 

---

If you would like to know more about Matterport product information, please contact us below!

- Contact Us -

3DMart provides not only 3D printing but also three major services: "3D Printing Service," "3D Scanning Service," and "Spatial 3D Scanning Service."

Latest information here:
Facebook | Instagram | LinkedIn