The Scientific Reference Model—A Methodological Approach in the Hypothetical 3D Reconstruction of Art and Architecture

Abstract

Scientific practice relies on the rigorous documentation of procedures, methods, and outcomes, governed by principles like method verification, objectivity, and source disclosure. In the computer-based hypothetical 3D reconstruction of destroyed or never realised art and architecture, adhering to these principles faces challenges due to evolving software, methods, and data types, leading to a lack of standardised documentation and publication practices for 3D models. Consequently, the traceability, accessibility, and sustainability of research outcomes are compromised. Decades after the advent of computer-aided 3D visualisation in cultural heritage, there is a critical need to define applicable methodology and comprehensive documentation standards. Web-based platforms necessitate technical infrastructures and clear scientific methodologies to ensure understandable model creation and sustainable accessibility to 3D research data. The Scientific Reference Model proposes an accessible academic framework for this kind of 3D reconstruction, aiming to facilitate broad adoption. Developed and tested in research projects and educational contexts, this model aims to establish clear, accessible 3D models on the web, serving as foundational references for future research and knowledge dissemination.

1. Introduction

The documentation of the procedures, the decision-making processes, and the applied methods, as well as the resulting findings, forms one of the cornerstones of scientific practice. Over the centuries, scientific publication established itself with fixed basic principles, such as the verifiability of methods, objectivity, the disclosure of sources, the comprehensibility of reasoning, the accessibility of results, accuracy and reliability, and uniformity [1] (pp. 3–4).

In the field of the computer-based hypothetical 3D reconstruction of destroyed or never realised art and architecture, the application of the above basic principles faces yet unsolved challenges related to the new nature of research data and their derivation. Considering 3D modelling and the 3D model as a scientific interpretation, reasoning, and hypothesis, it is found that due to manifold and rapidly developing software applications, modelling methods and types, no application-related, standardised documentation and publication of 3D models has been established. In consequence, the results of the work are not traceable, cannot be found, are not accessible, and are therefore not sustainable.

Almost four decades after the spread of computer-aided 3D visualisation in the research and dissemination of cultural heritage, we observe an intensified examination of the question of what and how this should be documented and shared [2] (pp. 165–187). Web-based documentation and publication requires technical infrastructures and services as well as the definition of scientific methods in terms of a comprehensible model creation and sustainable accessibility to the research data (re-)presented in the form of 3D models. The Scientific Reference Model (SRM) aims at establishing a methodology with a low barrier for a broad application in hypothetical 3D reconstruction according to the minimal requirements of a scientific practice. As a result, it should ensure a comprehensible and re-usable 3D model, which can serve as a ‘source reference’ for the dissemination of knowledge and further research (see Figure 1). The SRM is an approach based on many years of practical experience in research and didactics in higher education developed at the Institute of Architecture (AI MAINZ) at the Hochschule Mainz. This is a trial to cut the Gordian Knot in the form of heterogeneous 3D data sets and the silo architecture in the (scientific) 3D community, enabling accessibility and re-usage, and giving credits to the model authors and the cultural heritage represented by the 3D models.

2. Reality-Based Model, Sourced-Based Model, and Modelling Methods

In the field of the 3D reconstruction of cultural heritage, we distinguish between the ‘reality-based model’, which relies on data acquisition by means of laser scanners, photogrammetry, and the registration of the raw data generated in the process, and the ‘source-based model’, which relies (mostly) on the interpretation of incomplete, heterogeneous (historical) text and image sources.

In the case of a reality-based model, the object is reconstructed in 3D modelling software using the raw data (point clouds) and/or orthophoto plans derived from them. Depending on the requirements of the project and the resulting modelling method, a surface-based (Boundary Representation) or an object-based model (Constructive Solid Geometry) is created. In the case of the latter, the challenge is that an analysis and interpretation of the structure behind the surface takes place via the pure surface representation of the raw data (point clouds). This creative effort of the model author leads to hypothetical assumptions, if no verification can be done by the invasive opening of the building elements (as in most cases).

Source-based modelling, on the other hand, is always based on subjective assumptions of the model author who, as a result of the creative processes of source analysis and interpretation, creates a ‘hypothetical 3D reconstruction’ of a possible actual state of the building at a given time. In most cases, the knowledge gaps must be closed by the authors of the reconstruction based on their own considerations. The adequate documentation of these creative processes is of great importance in order to appropriately map the verified knowledge and the hypothetical assumptions. This is a basic prerequisite for ensuring scientific rigour, which guarantees the traceability of the reconstruction results.

3. Scientific Reference Model

The Scientific Reference Model (SRM) aims to establish an applicable methodology within hypothetical 3D reconstruction that meets scientific requirements and enables the referencing and re-use of the results. In most cases, the SRM provides an initial model, the so-called ‘reference model’, which is made available for further re-use. This approach follows the insight that the majority of the hypothetical 3D reconstructions are used for (visual) mediation in high-quality renderings, animated films, (serious) games, augmented reality, virtual reality, 3D printing, etc., and thus have to meet a wide range of (technical) requirements in the final result. The SRM anticipates this dilemma by publishing the 3D model at an earlier stage, which guarantees the standardisation, interoperability, and sustainability of a core knowledge framework. The SRM is a reference model from which various derivatives can be derived for further applications.

The SRM method is based on several work packages (see Figure 2), developed within research projects at the Hochschule Mainz and applied in the educational courses at the Warsaw University of Technology [3,4]. In the following, the SRM for application in education is presented using the example of one of a total of nine wooden synagogues reconstructed in the summer term of 2022. The synagogue in Wołpa (Belarusian: Вoўпа) was modelled, documented, and published by Katarzyna Prokopiuk under the supervision of the authors of the paper. The following introduction of the SRM methodology for the Wołpa Synagogue will serve as our case study. This paper was first presented at the Cultural Heritage and New Technologies Conference in Vienna in November 2022.

3.1. Object Identification

The hypothetical 3D reconstruction is a research method used in object-oriented disciplines such as archaeology, art history, and architectural history. The examination of the object is the starting point of the process. The object identification clarifies the name, location, and historical context of the building. A special field of interest is the analysis of the information on the web. Wikipedia, as the largest collaborative knowledge project of humankind, provides the first and often last access to knowledge for the ‘Digital Natives’. The proposed SRM method confronts the model authors with the knowledge representation on the web, sharpening the criticism of digital sources and encouraging them to participate in the process of information creation in the sense of Open Science and Citizens Science. This includes the editing of existing articles and/or the creation of not yet existing articles on the objects as well as the uploading of vectorised drawings and augmented reality postcards under free licences on Wikimedia Commons (see Figure 3).

Dealing with information indexing in Wikidata sensitises the model authors to the topic of structured knowledge representation in the context of Linked (Open) Data technologies. Special attention is paid to the authority files and the persistent identifiers regarding the unambiguous labelling of the data records.

The example of the Wołpa Synagogue clearly shows that the majority of the (secondary) cultural heritage does not yet have any identifiers from the renowned initiatives of the German Integrated Authority File (GND), the controlled vocabularies of the Getty Foundation, or the Virtual International Authority Files. This is where the great advantage of the Wikimedia Foundation’s Open Science approach with the major heavyweight projects Wikipedia and Wikidata comes in [5,6]. Here, objects such as the synagogue in Wołpa can be recorded for the first time and given a unique identifier. In addition, clear human and machine-readable statements can be made about the object.

3.2. Sources

The hypothetical 3D reconstruction is mainly based on primary, secondary, and tertiary sources. Source collection involves the compilation, classification, and analysis of historical sources relating to the object and refers to all possible formats. This includes primarily pictures, plans, drawings (also own hypothetical sketches), scientific works (research results of object-oriented disciplines), and relevant analogue example objects, as well as, to a greater extent, visualisations from previous hypothetical 3D reconstructions, in rare cases also digital 3D models (see Figure 4).

In many cases, the exploration, interpretation, and correct classification of these sources requires the involvement of experts such as archaeologists, art historians, and architectural historians, which, however, cannot take place within an appropriate framework in many projects due to a lack of resources. The SRM method aims to promote a low-threshold application of hypothetical 3D reconstruction and to ensure the minimum requirements for the verification of the results. The basic prerequisite for this is the communication of the necessity of a comprehensive and structured source exploration as a result of which the information situation on the object and its components is revealed and the gaps in knowledge are supplemented by analogies and own hypothetical considerations, e.g., in the form of sketches. On the one hand, this requires an understanding of the classification of sources according to their type, the context in which they were created, their provenance, and the rights of use. On the other hand, the procedure trains a structured approach to recording and archiving sources. These are two essential components of a scholarly approach to historical objects that ensures a structured documentation of the state of knowledge.

3.3. Normative Structure

Assuming a broad application of hypothetical 3D reconstruction in the sense of Open Science, the vast majority of projects take place on the personal computer within an individual project and folder structure. This often random structure is difficult to follow even for the author after a few years. The SRM method emphasises the importance of a normative structuring of the project in terms of standardisation to ensure the basic requirement of sustainable documentation. A proposal is made for the project organisation and naming convention of the data, which should ensure a structured storage of sources, the argumentation, and hypothesis in the sense of traceability and reusability (see Figure 5).

A structure consisting of five folders is specified for the student projects. The first folder contains a handout that uses the case study of the former synagogue in Speyer in 1250 to describe in detail the procedure for 3D reconstruction, documentation, and publication according to the SRM method [7]. In addition, tutorials for the semantic enrichment of the 3D models in Sketchup using the add-on CityEditor (enabling CityGML export) are provided here. The second folder contains template files for the postcard with the augmented reality application as well as a Word template for the documentation of the sources used and the 3D reconstruction process. In the third folder, the structure for storing the sources is presented, separated into literature, plans, and image material. The fourth folder contains the 3D models created and the textures used. The fifth folder is for the accompanying results, in which an object presentation, documentation, the designed postcard, the edited Wikipedia article, and the semantically enriched 3D model in CityGML data exchange format can be stored.

3.4. Segmentation

The study of a building requires a structural analysis of its components. The semantic division depends on the research question and the requirements of the project. In practice, it is often divided according to structural and functional aspects. Here, a first classification of the building elements already takes place according to the established criteria, such as wall, ceiling, support, beam, roof, window, decorative element, etc.

This type of classification follows the approach of the object-oriented 3D modelling of Building Information Modeling [8] and is particularly suitable for the documentation of 3D reconstruction. By dividing the object into its individual components, the granularity of the documentation can be significantly increased. The increase in granularity is also accompanied by a higher documentation effort, which must be carefully weighed up depending on the requirements and purpose of the 3D reconstruction. On the one hand, controlled vocabularies, such as the Art & Architecture Thesaurus from the Getty Research Institute (Getty AAT) [9], can be used for the unambiguous classification of the objects. On the other hand, the sources used and the argumentation, as well as the degree of the hypothesis of the 3D representation, can be attached to the individual building elements. Segmentation is an important structural tool that supports handling, modelling, documentation, and re-use.

The SRM method presents a simple approach based on source indexing (see Figure 6). The sources are used to identify the building elements (objects), which are classified using Getty AAT. In the case of missing historical sources, the gaps must be documented by using analogue examples and own sketches. The decisive factor is that only building elements (objects) based on an analysis and interpretation of the sources (including hypothetically sketched assumptions) can be reconstructed. A 3D reconstruction can only remain traceable and reproducible if this requirement is fulfilled.

3.5. 3D Modelling

The 3D reconstruction is carried out in a freely selected 3D modelling software, depending on personal knowledge and/or the requirements of the project. Two modelling methods are primarily used: the surface-based Boundary Representation (see Figure 7) or an object-based Constructive Solid Geometry.

With regard to the interoperability and re-use of the SRM, great importance is attached to providing the 3D model in the approved data exchange formats. With the CityGML exchange format from the application area of digital 3D city models [10], individual surfaces and/or surfaces grouped into objects can be marked with attributes (the practical use of this format is described and illustrated in the context of the level of uncertainty in the following chapter). For object-based modelling within BIM-supporting software, the data exchange format IFC is used [11], which enables interoperable information storage in the 3D model. In both cases, the entire represented object can be marked at the highest hierarchical level, indicating the (historical) object-related information as well as the author of the digital 3D model and the declaration of rights to the 3D data set.

The SRM places the greatest emphasis on the following last two work packages: the documentation and publication of the 3D model, behind which all potential scientific value is concealed [12]. The 3D modelling serves as a means to an end for knowledge representation. The source-based depiction of the current state of knowledge in the form of pure 3D representation cannot be a satisfactory result for a scientific work. The 3D model becomes a serious carrier of information only if it undergoes semantic enrichment within the modelling software and if the result can be provided in an interoperable data exchange format (see Figure 1, the published SRM in IFC and/or CityGML format is the re-usable resource for further research and derivates).

3.6. Documentation

Although the scientific community repeatedly points out the invaluable importance of documentation [13,14], opinions are strongly divided on the practical approach to this topic [15] (pp. 30–31). Discrepancies appear in the interpretation of community guidelines regarding the principles for computer-based visualisation research [16] (pp. 7–8), [17] (pp. 8–9), which only provide general remarks, like the need for scientific transparency in described methods, claimed hypotheses, used sources, and their interpretation. Numerous reconstruction methods have been developed with documentation centred around various aspects, such as the workflow and methodology used [18], data modelling based on established standards [3], determining the level of uncertainty (hypothesis) [19], the extensive analysis of source materials [20], the argumentation of scientific decisions [21], or creating variants of possible solutions [22]. This diversity indicates the lack of an ideal method of documentation, which should be selected according to the project criteria.

Therefore, for the purposes of education, a simplified method of documentation was adopted based on a prepared template (see Figure 8). The basis of the record is the lowest division of the object element hierarchy, and each of these listed elements should have its individual form completed. The header of the table is the name of the element, adopted according to the Getty AAT terminology. The largest part of the table is taken up by a visual 3D representation of the element being documented, usually in the form of a screenshot from the modelling software. Below there is a list of the used sources, which are listed by name and illustrated by a graphic preview. It is possible to add in a list of sources as well as paradata, i.e., a sketch of the author of the reconstruction illustrating his analytical way of thinking. The last row in the table is for a textual argumentation of the reconstruction in the form of a few sentences and an evaluation of the level of uncertainty in relation to the available historical sources. It is also the place to provide possible hypotheses and an analytical description of the process.

The mentioned level of uncertainty has a particular value among the good practices to adopt in the light of a scientific methodology for hypothetical 3D reconstructions; it is therefore important to mention its definition. There have been several proposals concerning uncertainty scales, based not only on different visualisation techniques [23], but also on different parameters, such as quality, coherence, and the type of source [21,24,25]. A simplified scale was proposed, based on four levels corresponding to the methods used to reconstruct an object (see Figure 9).

Colour coding can be a useful way to visually communicate to a differentiated audience the level of uncertainty associated with digital reconstructions of hypothetical artefacts. For this simple scale, the colours chosen are the most recognisable ones, in line with studies on optics and colour perception [26,27,28]. The possibility of adjusting the scale based on the specific needs and branding guidelines or in the case of visual impairments should nevertheless be retained [25].

Not only at a cultural level, but also at a technical one, uncertainty data should be shareable and, hopefully, both human- and machine-readable. For this reason, each category of the proposed scale is also associated with a progressive value, represented by numbers 0–4 to be integrated into documentation. These features can be applied to the entire model or to each element composing it, according to its level of detail and semantic segmentation.

For instance, in a digital reconstruction of a historic building, there may be a high level of uncertainty associated with the reconstruction of a specific architectural feature, such as a decorative moulding, due to limited information about its original design and condition. At the same time, there may be a low level of uncertainty associated with the overall layout and dimensions of the building, which can be accurately reconstructed using a combination of historical records, on-site measurements, and other data sources. The definition of uncertainty at different levels of detail can present technical challenges: in order to allow interoperability, these data should also be integrated in the 3D data set and shared through standard exchange formats such as IFC for models based on Constructive Solid Geometry or City GML for models based on Boundary Representation (see Figure 10).

To reach this goal, an attribute related to uncertainty was added and structured according to the scale presented here. The elements of the model were grouped so that the attribute could be applied not only to the single elements, but also to the entire building. Once exported in IFC or CityGML format, the parameter related to uncertainty could still be visible at both levels. This approach is an opportunity to attach the uncertainty information and even the links to the sources used in the data exchange files (ISO standards, serious 3D, etc.): for this reason, it should be seriously considered in addition to the documentation templates seen above.

Initial assumptions about the simplicity of the use of the presented documentation method were verified during a voluntary survey with students, which denied ease and fast use and indicated the problem of the scientific transparency of sources provenance [4]. In view of the emerging new possibilities for documentation in promising web-based platforms, the continued use of templates in .docx format was questioned. One of these platforms is the “Infrastructure for Documentation of Virtual Reconstructions” (IDOVIR), which, with its extensive documentation of scientific decisions function, also enables the export of the final report in PDF format [29]. The use of this type of solution will be considered in the further development of the SRM.

3.7. Publication

The final step is to ensure access to the research results by publishing 3D models with prepared documentation in a way that ensures the traceability and reusability of data. In the light of the Semantic Web [30], publication of knowledge should take place in online, open access platforms and repositories. However, the choice of platform for publication requires the consideration of several factors, and the failure to meet these may limit the potential future use of the data.

Firstly, published files need to be accompanied by an appropriate set of metadata, which will increase data findability on the web [31]. In order to reduce the difficulty of describing the model, the SRM methodology seeks to use ready-to-use solutions and, although many metadata standards for digital heritage have already been developed [32,33], the field of source-based digital reconstructions remains unsettled. Nevertheless, attempts are being made to harmonise and develop a shared standard for the exchange of metadata. The current results already allow for the development of a prototype schema that can be used in practice [34].

The method of data storage by web-based platforms is also a significant matter. Special mention should be made here of the Resource Description Framework (RDF), which is standard for describing web resources and data interchange, developed and standardised with the World Wide Web Consortium [35]. Records in RDF format with the use of an ontology-based data model can improve the machine readability of the data. In addition, the research dissemination can be enriched by the network effect obtained by combining our data with external Linked (Open) Data resources [36].

The final factor is how to share the 3D data itself. The publication process should provide access to a broad public [37]. In the case of 3D models, this access is particularly difficult due to the fact that it is not a medium used on a daily basis. Furthermore, the intricate issues of licensing and the plurality of possible file formats additionally restrict access to these kinds of data. Therefore, the best solution is to use a platform with an integrated 3D viewer, which will allow viewers to interact with the model without the necessity of downloading the file. In terms of the reuse of 3D data in scientific research, it is compulsory to provide neutral formats that can be easily imported into most 3D software (OBJ, PLY, DAE, STL, etc.) [38]. An additional advantage is the provision of the previously mentioned standardised documentation formats such as CityGML or IFC due to their human and machine readability. Good practice indicates the inclusion of the 3D file in the original (native) format, which may allow insight into used methods or workflow and can enable further scientific work on the model.

Faced with all of these criteria, the SRM utilises a customised Virtual Research Environment (VRE) for the purpose of the human- and machine-readable publication of the 3D project results. Here, the long-proven VRE in hypothetical 3D reconstruction project was adapted according to the requirements of a 3D repository [3]. The developed publication platform stores metadata in RDF-Triple-Store, with a data model based on the CIDOC CRM referenced application ontology OntSciDoc3D [39]. In addition, the platform offers the possibility to share 3D models in several ways by offering multiple uploads of any 3D file format with the indication of the native file and the visualisation of the 3D model in a browser using a special module developed within the DFG 3D-Viewer project [40]. Using the 3D repository (see Figure 11), students were required to upload their models with metadata and paradata documentation, proclaimed by the London Charter [17]. In this way, the results of their work remain accessible both visually (3D models) and in terms of documentation, contributing to the development of the field of hypothetical 3D reconstruction towards Serious 3D and Open Science.

4. Conclusions

The presented Scientific Reference Model (SRM) is a methodological approach that covers the whole range of creating a 3D model of destroyed or never realised art and architecture. The aim of the proposed method is to raise awareness about the availability and reusability of the hypothetical 3D reconstruction, inter alia for further application (see Figure 12). For this, the 3D community must agree on common norms and a certain standardisation to enable the findability, exchange, and traceability of the models. Only then we will be able to speak of a serious handling of the digital 3D model in a scientific context.

The SRM method shows the need for documentation and for a web-based publication of the findings and results. It wants to mainstream hypothetical 3D reconstruction as a research tool and makes practice-oriented suggestions in the form of templates, forms, and presentations of services already available, such as the DFG 3D-Viewer, IDOVIR, Semantic Kompakkt, and, of course, Wikipedia and Wikidata.

The SRM is intended as a starting point for an upcoming discussion on methodology and standardisation in the field of computer-based hypothetical 3D reconstruction, which was successfully evaluated in courses at the Hochschule Mainz—University of Applied Sciences, Lodz University of Technology and Warsaw University of Technology in the summer semester of 2023 as well as in the ongoing EU project ‘CoVHer—Computer-based Visualisation of Architectural Cultural Heritage’ [41].

Each of the mentioned SRM steps ended with a questionnaire allowing the students to evaluate the practical use of the SRM method, in particular, with regard to the documentation and publication of 3D models. A total of 27 responses were collected [42]. The survey involved assessing the degree of alignment between a given term (“Time-consuming”, “Work-intensive”, “Practical”, “Unnecessary”, and “Unclear”) and various stages on a scale of zero to five, where zero indicated no alignment and five indicated full alignment.

Most steps of the methodology received similar ratings for each term. The lowest alignment was observed with the terms “Unnecessary” (mean values ranging from 0.22 to 1.67/5) and “Unclear” (mean values from 1.00 to 2.33/5). Neutral alignment was found with the term “Practical” (mean values from 2.67 to 4.13/5). The highest alignment was with the terms “Work-intensive” (mean values from 3.60 to 4.89/5) and “Time-consuming” (mean values from 3.89 to 4.78/5).

Participants were also asked to estimate the time spent on documentation and publishing the 3D model. On average, preparing the documentation took about 8.5 h, while publishing the model in the repository (including 3D model preparation) took about 2 h. Students were further asked to rate the dedicated repository for digital reconstructions on a scale of 1–5. Instructional materials received a high rating (4.04/5), while the comprehensibility of the data input form was rated moderately (3.45/5). The lowest scores were for intuitiveness (3.08/5) and the difficulty of use (2.66/5).

The survey shows that the theoretical framework of the methodology appears appropriate and is positively evaluated, but its practical implementation requires significant improvement due to the high effort and time demand and difficulty in using available documentation platforms and the lack of automatization of the documentation process. Further efforts should focus on enhancing the tools available to automate documentation and expedite the entire process.