An Online Dissemination Workflow for the Scientific Process in CH through Semantic 3D: EMtools and EMviq Open Source Tools

Abstract

This article explores the use of open source 3D tools to improve the transformation of the archaeological record into a virtual reconstruction. The goal of the research was to improve the dissemination of complete reconstructive Extended Matrix (EM) datasets, organized by epochs, to allow a “time travel” experience, by means of the EMviq online service “metaphor”. This article presents an incremental version (ver. 1.3.1) of EMtools (add-on for Blender 3D) and a renewed version of EMviq. These two original open source (GPL3) tools have been developed, on one hand, to facilitate the process of semantic enrichment and source-based 3D modeling of cultural contexts (EMtools) and, on the other hand, to visually inspect data within immersive virtual reality viewers, online (WebXR), or via mobile devices (EMviq). An application case is shown to illustrate the entire work-flow from the archaeological stratigraphic reading to the representation of the virtual reconstruction of what a context must have looked like at a given time in antiquity.

1. Introduction

Archaeology revolves around a few questions: what were humans like, their lives and places over the centuries?Digital archaeology and virtual reconstructions offer concrete tools to take contemporary humans back in time and “visit” the places they came from: their past. A real time machine built over 200 years, from splendid painted reconstructions to immersive virtual reality and open world video games.

In recent years, the interest of the scientific community in virtual reconstructive hypotheses (the visualization of a context as it must have appeared at a given time in antiquity) has grown [1,2,3,4,5,6]. A central aspect in the development of scientifically correct virtual reconstructive hypotheses is the possibility to manage and publish not only the visual result of the reconstruction, but also all the data used to obtain it and, in particular, the sources, reasoning and interpretations. This type of activity, however, requires complex cross-references of data, even very different from each other (written sources, photographs, photogrammetric 3D models, stylistic comparisons, etc.). In order to achieve this goal, therefore, both a theoretical framework and a series of innovative computer tools that can simplify, standardize and make the workflow that leads from the archaeological data to the reconstructive hypothesis efficient are necessary. This research stands upon the extended matrix (EM), a formal language designed to manage both the archaeological record and to annotate the reconstructive hypotheses as they emerge from the early stages of the investigation. In other words, within the EM approach, it is possible to transform the stratigraphic archaeological record into formalized reconstructive hypotheses depicting how a context must have appeared at a given time in the past.

Thus, the goal of the research was to improve the dissemination of complete reconstructive extended matrix (EM) datasets, organized by epochs, to allow a “time travel” experience, by means of the EMviq online service “metaphor”. This article presents an incremental version (ver. 1.3.1) of EMtools (add-on for Blender 3D) and a renewed version of EMviq. These two original open source (GPL3) tools have been developed, on one hand, to facilitate the process of semantic enrichment and source-based 3D modeling of cultural contexts (EMtools) and, on the other hand, to visually inspect data within immersive virtual reality viewers, online (WebXR), or via mobile devices (EMviq).

While the EM is the theoretical and methodological background, usable even outside a digital environment (an EM can be sketched out on a sheet of paper with a pencil as well as with a software tool) the EMF includes digital solutions and software platforms (which can substantially change in the future if relevant technological innovations occur). EM is about scientific-driven content creation, EMF is about technological-driven solutions. Extended matrix tools (EMtools) and an extended matrix visual inspection querier (EMviq) are part of the EMF (see Section 3.1). Every version of the EM formal language (the core of the system) drives its own EMF digital framework identified by the same number (the present EM 1.3 has its EMF 1.3). The versioning of these tools follows the EM convention: here we are presenting the EMtools and EMviq at their 1.3.x version. The software mentioned are just some of the possible practical applications of the extended matrix methodology. Thus, the development of these tools follows a bottom-up approach and began from real problems that emerge during the development of some case studies as reported by the archaeologists and 3D modelers involved. The software has been released under an open source GPL 3.0 license in order to be used by the scientific community and possibly modified to adapt them to their own case studies within the same methodological EM framework. The article, after a brief state of the art description (see Section 2), will describe the methodology behind the results (see Section 3), and, finally, will shortly present an example of the workflow on a real case study and technical details related to the operative aspects of the software (see Section 4).

2. Background and Context

The activities of study and documentation that go along with the archaeological excavation lead to a large amount of information (stratigraphic unit forms, drawings, photographs, excavation reports, etc.). This type of data has a great complexity because it describes both fragmentary archaeological artifacts and their temporal relations (chronological succession), but also the transformations to which they have been subjected over the centuries (reuse, adaptation, restoration, etc.). The archaeological record is traditionally organized within relational databases (based on tables) linked to a coded and interpreted bi-dimensional representation (archaeological drawing), and, in recent years, even to interactive 3D models [7,8]. At the same time, the activity of virtual reconstruction has a tradition that tries to reconcile philological scientific aspects (attention to sources) and the aesthetic quality of the representation [9]. In some cases, the search for aesthetic rendering has attracted the suspicion of a lack of a scientific approach and has reduced virtual reconstruction to an aesthetic challenge. The increasingly widespread introduction of advanced semi-automatic photogrammetry (SfM) tools in the last decade has offered new methods of representation of the archaeological context to complement the two-dimensional codified representation [10]. This innovation, among other aspects, has made the aesthetic quality standards of representation more similar between surveys of the present state of monuments and virtual reconstructive hypotheses by means of computer graphics techniques. Conversely, the reconstructive hypothesis, despite having in many cases a high aesthetic quality, did not have a semantic structure that reflected the detail of the archaeological record. In the last two decades, several contributors were limited to associating the “prevalent source” [11,12] used for the virtual reconstruction with a part of the reconstructive model (for example “stylistic comparison” as the prevalent source for a Corinthian capital), losing the connection with the granularity of the stratigraphic data as they emerge from excavations or from the analysis of the masonry of ancient architectural complexes. In fact, it is only in the last five years that tools have been developed to semantically model the reconstructive archaeological record.

As regards the creation of the reconstructive record and in particular the adoption of specific semantic tools to describe the scientific processes behind the reconstruction, in the last 10 years different approaches have been proposed, each with a specific focus; in some cases, starting from previous semantic tools such as CIDOC-CRM [2,13] or proposing a minimum set of information able to describe the reconstruction process along with an online service to annotate them [14,15] or in other cases proposing a new formal language (CONML) [16]. There are projects focused on the transparency of the data behind virtual reconstruction, by means of virtual reality experiences, based on closed software solutions such as Unity 3D [4]. Within the scientific community, the Extended Matrix project (Zenodo project page: [17], official web site: [18]), on which the present research is based, is attracting a growing interest. The extended matrix is an open formal language, entirely based on open source software and formats to ensure full repeatability and FAIR dissemination of the results. The EM can be used to keep track of virtual reconstruction processes and, more in general, of the interpretation processes during the scientific tasks of analysis and synthesis. It starts from standards already used for the collection of the stratigraphic record (physical and chronological description of artifacts) in the archaeological field work or in building archaeology. It is intended to be used by archaeologists and heritage specialists to document their scientific activities in a robust way. The EM allows us to record the sources used and the processes of analysis and synthesis that have led from scientific evidence to virtual reconstruction. It organizes the 3D archaeological record so that the 3D modeling steps are smoother, transparent and scientifically complete (for a detailed description of the formal language of the extended matrix, see [17,19,20]). There are also other attempts to involve stratigraphic aspects into the formalization of a virtual reconstruction that shares some basic aspects with the EM approach [21].

3. Methods

Bearing in mind that the stratigraphic reading is the most detailed and ductile formalization tool to model the archaeological record, in this article we will consider the extended matrix (EM), a system that is specifically designed to capture the complexity of the stratigraphic archaeological record and exploit it, with minimum loss of information, to validate a reconstructive hypothesis. The improvements brought by this approach concern both the methods through which the record is created, organized, and linked to the 3D representation, but also how it is shared and published.

In the next two sessions, a wider methodological context will be provided to specific domains of application:

• Annotation and representation of interpretative hypotheses are core aspects of semantically enriched 3D models, useful both to clarify ideas during the scientific process and to communicate them to the colleagues as well as to give contextualized feedback for each assumption/interpretation. The goal of the research was to improve the dissemination of the full reconstructive EM dataset, organized by epochs, to allow a “time travel” experience (see Section 3.1).
• An important practice of the EM methodology is the online dissemination workflow that focuses on visualizing the models and the scientific record, making the semantic layer searchable and accessible. The online publication needs robust technical pipelines to formalize (in a JSON container file format) and optimize the data architecture (see Section 3.2).

3.1. Organizing Reconstructive Data According to Temporal Horizons

Behind the management of the archaeological record, there is the need to semantically enrich the 3D models to explicitly connect them with the stratigraphic record. Thus, a geometrical representation for each unit, namely the proxy model, should be provided. This formal tool represents what in the GIS domain is represented by point, lines, and areas: an approximation of a given object that enables us to interact with the information that is connected to it (i.e., the visualization of stratigraphy according to the epochs they belong to or the visualization of the degree of certainty for the reconstructive hypotheses). The description of this behavior partially overlaps with the concept of 3D GIS, but added to this is the native ability to work with temporal, graph-shaped data (in our case, a GraphML format file, see below), and encoded paradata related to the scientific processes that led to the production of the digital assets. Thus, these distinctive features have driven the design of the EMtools 1.3.1 (see Section 4.1) to meet the data architecture of the EMviq 4D scene (stored in a JSON file), where proxy models as well as representation models are grouped and aligned to each other according to the “epochs” that the users are going visit within their “time travel”.

3.2. Visual Inspection and Queries of Multi-Temporal Environments

In order to perform efficient runtime queries on extended matrices within interactive 4D inspection tools, different steps need to be carried out [20]. Such procedures must be designed to create intermediate data structures for fast retrieval by a real-time application. Specifically, the extended matrix formalism requires three different steps to be performed on the involved EM: (1) timeline extraction; (2) proxy-graph extraction and (3) source-graph extraction. The first step is required to build a collection of periods that can be exploited at the run-time to switch between different representation models over time. The second step is required to build a semantic 3D model (proxy-graph) comprised by basic shapes (“proxies”) used to perform interactive queries (see Figure 1).

The third step creates an internal representation of EM source relationships (paradata) attached to each 3D proxy. These steps need to be computed every time the EM is modified or edited. By EM definition, each node n in the EM has a temporal property, so for timeline extraction we define an operator T as:

$$T\left(n\right)\to \tau$$

(1)

where $\tau$ is a specific period comprising a start and end pair (that also defines the period duration). We can use T to map each node in the EM to a collection of time-periods (${\tau }_1$, ${\tau }_2$, ${\tau }_3,\dots$). We also require a selector S to retrieve a representation model G depending on a given period:

$$S\left(\tau \right)\to G$$

(2)

S can be employed by the interactive tool/application to switch between different time-periods, by means of user input (e.g., a slider). A given context may present areas or portions having different pacing from a temporal perspective; for instance, a portion of the 3D scene that did not evolve during multiple time-periods. Temporal instancing [22] can be adopted to share sub-graphs of representation models (see Figure 2), with two major advantages: (A) a compact memory footprint during interactive inspection and (B) caching that minimizes the online transmission of 3D data for representation models (Web3D applications).

4. Results

As already said, this paper presents 3D tools developed for the upcoming Extended Matrix Framework (EMF) version 1.3 for transforming the archaeological record into a virtual reconstructive hypothesis. In this section, the improvements of the aforementioned tools are presented. EMtools [23] version 1.3 [24] now allows digital humanists to produce and disseminate a consistent dataset for the EMviq web-app (including paradata within a time travel experience) by means of an “EMviq scene exporter”. The EMviq itself has been enhanced with several features regarding the ways paradata are inspected and represented (see infra).

4.1. EMtools and an Example of Data Workflow

EMtools (Extend Matrix tools) is a Python based add-on for Blender 3D to facilitate the creation of 3D reconstructive hypotheses integrated with an Extended Matrix graph database (stored in an ISO standard GraphML format). The tool provides the concurrent preparation of a GraphML file in the yED (https://www.yworks.com/products/yed, accessed on 28 January 2023) free software according to the rules of the EM formal language ([19], p. 1). A specialist archaeologist operator edits the GraphML file in yED (OP1), while a modeling archaeologist (OP2) creates the proxy model inside Blender-EMtool.

Its purpose is to facilitate the processes of semantic enrichment and source-based ([19], p. 43) 3D modeling of cultural contexts.

The tool allows you to perform a variety of data management, 3D modeling, data retrieval and data-set publishing tasks. The tool is active inside Blender’s 3D view-port and is divided into different panels (see Figure 3a–c). A description (with examples) of the previous version of EMtools (ver 1.2) with its capabilities has already been published ([25], p. 20), while an in-depth description of the tool is in the online documentation (docs.extendedmatrix.org, accessed on 28 January 2023).

For the sake of this research, we will focus on the export manager panel (see Figure 3d), which represents the main improvement of the EM 1.3.1. It allows the export of Proxy models and representation models divided into sub-folders according to their chronological period along with the GraphML file and the JSON file describing the EMviq 4D scene. Through this tool, it is also possible to export in csv format the list of stratigraphic units with their chronological belonging and descriptions, sources and extractors to edit tables for publication or raw data export functionalities.

Besides that, the tool allows a hot reload of the GraphML file inside Blender (which affects any stratigraphic data previously loaded). The entire workflow is carried out following the EM five steps methodology [25]. The example provided in these sections is the reconstruction of room “G” of the Roman smithy of Montebelluna (details in the acknowledgments section) and will help to describe how the new features of the EMtools work in practice. Bearing in mind that the work process is not destructive (it is possible at any time to return to one or more previous steps), it is normally carried out by two operators: OP1 works on the EM graph db (GraphML), and OP2 crafts 3D models within Blender and connects the EM graph db with proxies within the 3D scene.

(1) The first step (see Figure 4) consists of the stratigraphic reading of the physical remains of the Roman smithy and its representation by means of proxy models within a scene in Blender 3D open source software. The actions that may occur within this step are: the creation in yED (software) of the EM concerning the remains of the context (the stratigraphic reading of the building); import of the results of the 3D survey (laser scanner combined with a UAV photogrammetric survey) and of the graphic documents related to the context; and import of the GraphML file (container of the EM) coming from yED.

(2) In the second step, in order to represent the stratigraphic units, the proxies have been modeled and linked to the stratigraphic units.

(3) In the third step (see Figure 5), a reconstructive hypothesis by OP1 is proposed in the EM, using yED, along with its representation by means of proxies, created by OP2 (the connection between stratigraphic units and proxies is provided). This process is not one-way: it is an iterative process of hypothesis making and visualization (thanks to the hot reload functionality of the GraphML inside Blender/EMtools). During this operation, a report of virtual activities that represents a textual discursive version concerning the reconstructive hypothesis is drawn up (it is divided according to the supposed evolution/construction phases of the context). The 3D proxy model is ready for a first review carried out by experts outside the team.

(4) In the fourth step (see Figure 6), a detailed model including materials and textures, the “representation model” (RM), is performed for each of the phases of life of the context that is worth being reconstructed, visualized and experienced by the users. Using the EMtools add-on, each 3D object model is associated to one or more chronological phases to which they belong to using the Period Manager.

(5) In the fifth step, it is time to publish the reconstruction. The “export manager” tool of EMtools will help the users to create tables for an article or a book chapter (csv export), an EMviq interactive web app (through ATON, see Section 4.2) or a desktop real-time environment (Unreal, Unity, Cry-engine or Godot, just to mention some examples). The export of the reconstructive hypothesis within an “EMviq” folder can be performed directly into a cloud folder, making it possible to directly publish the data-set through the EMviq web-app.

Summarizing, to enable the EMviq export, three prerequisites should be addressed in the previous aforementioned steps:

• Data preparation: representation models (RM) have been associated with one or more epochs (in step 4);
• An export set-up targeting specific platform texture and 3D models sizes has been configured;
• Export of the scene in a folder with a standardized name-space and tree of sub-folders managing possible reuses of the assets within different epochs.

4.2. EMviq

EMviq (Extended Matrix Visual Inspector and Querier) is a complete, interactive 4D visualization and run-time interrogation tool for extended matrices developed on top of the methods described in Section 3.2. The tool focuses on parsing and automatic extraction of run-time data from GraphML files (Extended Matrices designed through yED software). It targets real-time 3D visualization, performance and ease-of-use in order to establish a fast and robust pipeline within multi-disciplinary teams. Furthermore, it adopts a cloud-based workflow, allowing remote professionals to edit/modify extended matrices and automatically reload and rebuild proxy-graph and source-graphs at run-time. This also allows quick design iterations for extended matrices improving the workflow among local professionals.

The first desktop application was implemented on top of OpenSceneGraph framework (C++) including support for immersive VR inspection (HMD). The tool offered also advanced solutions to customize run-time queries behavior, such as spherical peeling [22], which allows us to spatially “carve” proxy-graph and/or representation models (scene-graph) in order to perform complex inspection within nested/occluded 3D geometries (see Figure 7).

Within the SSHOC European project (https://sshopencloud.eu/, accessed on 28 January 2023) a completely redesigned EMviq has been developed. The new tool is completely web-based, implemented as a web-app on top of the ATON (https://osiris.itabc.cnr.it/aton/, accessed on28 January 2023) framework [26], already employed in previous national and international projects involving cloud-based processing and visualization services for landscapes [27] or presentation of museum collections [28,29]. More specifically, the new EMviq tool inherits all the features offered by the framework (no installation required by final users, multi-device support, etc.), including building blocks for immersive VR inspection [30]. The tool also introduces run-time search functionalities within specific extended matrix fields (proxy ID, description, chronology, URLs), thus allowing professionals to easily filter, select or focus on subsets of proxies in the 4D space. In addition to filtering/search routines, it is possible in EMviq to present associated information to queried proxies. The proxies queried in this way are highlighted, and information concerning the layer name and description is displayed alongside the source-graph (see Figure 8). Thanks to a specially created recursive routine (see Algorithm 1), information concerning the description and the archaeological chronology of the children nodes is also displayed recursively (see Figure 9). The pseudo-code is as follows:

Algorithm 1: SourceGraph visitor

EMNodeVisitor (N)

1 printNode(N)

2 $C\leftarrow getChildren\left(N\right)$

3 if $empty\left(C\right)$ then

4         return

5 else

6         foreach $c\in C$ do

7         EMNodeVisitor (c)

This type of presentation was realized through the HTML5 $<details>$ and $<summary>$ tags. This element creates a disclosure widget in which information is visible only when the widget is toggled into an open state. A summary (or label) must be provided using the $<summary>$ element. They are dynamically generated by the recursive SourceGraph visit described in Algorithm 1, producing cascading widgets to toggle and explore different branches of the given SourceGraph.

Similarly to the previous desktop application, it was then integrated with NextCloud (https://nextcloud.com/, accessed on 28 January 2023) to provide a distributed workflow among remote professionals.

5. Conclusions and Future Work

The article presented EMtools (Blender add-on) and EMviq open source tools (GPL3) to facilitate the process of semantic enrichment and source-based 3D modeling of cultural heritage contexts as well as a complete workflow of transformation of the archaeological record into a virtual reconstructive hypothesis showing real application cases. The development of these tools, EMtools and EMviq, is still in progress and foresees the addition of an online tool for the editing of the extended matrix, with the aim of extending the current possibilities of collaborative work between specialists from different disciplines. New features will be added in Emviq that will allow the customization of information displays and the insertion of dynamic stratigraphic diagrams relating to the proxies queried. These tools can contribute to the scientific and professional community, both as ready-to-use tools and as a base for custom implementation (due to the GPL3 license), also providing a validated method to gather, analyze, implement, publish and share both the results and all the intermediate steps of the research, enabling solid reuse strategies by means of validated assets/workflows and software.