(PDF) Making 4D: principles and standards for virtual reconstruction in the humanities by the 4D Research Lab

Making 4D: principles and standards for virtual reconstruction in the humanities by the 4D Research Lab Tijm Lanjouw, Jitte Waagen The 4D Research Lab Report Series has been established as an instrument to promote trans- parency regarding the 4DRL virtual visualisation projects, workflow and pipeline develop- ment and technical experiments. The aim is to maximize knowledge sharing, meta- and paradata communication and clarification of author- and ownership of 4D Research Lab products. The 4DRL Report Series is published under a CC-BY license. Patricia Lulof, Jitte Waagen, Tijm Lanjouw Making 4D: principles and standards for virtual reconstruction in the humanities by the 4D Research Lab 4D Research Lab Report Series 1 Author(s): Tijm Lanjouw, Jitte Waagen Layout: Mikko Kriek Editor(s): Jitte Waagen, Tijm Lanjouw ISSN 2772-7734 July 2021 http://4dresearchlab.nl Introduction The 4D Research Lab (4DRL) is a digital technology lab with a focus on the application of 3D technology and visualization in the humanities. The 4DRL is multi- and interdisciplinary in approach, within a field including history, art-history, architecture history, archaeology, conservation and restoration sciences, and memory and heritage studies. The main objectives are improving research and education using virtual visualization and support the creation of 3D models based on thorough research of traditional sources as well as the latest 3D scanning techniques. The 4DRL is a place where scholars, 3D modellers and IT specialists come together and work together on innovative solutions for important contemporary research questions. As such, it operates on the intersection of fundamental research, applied science and science communication and valorisation. In this document, the first instalment of the 4DRL Report Series, we discuss our approach and important related concepts for the creation of 3D models. The use and creation of 3D models for scientific purpose, or simply within an academic context, raises the issue of transparency. Also, in a general context, concern has been raised about the absence of contextual information, and the potential impact of 3D models on the popular perception about the past. As an ethical code that all researchers should adhere to, the scientific process should be fundamentally transparent, and all models should come with para- and metadata. This not only provides the necessary caption and disclaimers to a digital representation, but also opens the models for further research and refinement. For 3D models of physical objects, a couple of related concepts are essential to understand the quality of these models. The first is ‘level of detail’ (LoD). LoD not only affects the visual detail, but also the accuracy and precision of these models. The second important concept is ‘degree of certainty’ (DoC). In contrast to LoD, DoC is a more subjective assessment of data or source material. It combines an assessment of the reliability of sources, of types of logical inference to reach a conclusion, and a classification of objective figures about accuracy and precision of data. 3 The 4DRL work aims to adhere to the standards set out in this document and provide models with para and metadata that includes a treatment of LoD and DoC (see principle statement in appendix I). The baseline format will be this report series (see project report template appendix II), but we intend to integrate such information with the models. Moreover, the report series is intended to solve other issues of concern from the academic 3D community, such as authorship and ownership of 3D models produced by the 4DRL. “It's Not Camelot, It's Only a Model” Everything that is not the actual phenomenon or object itself, is a model of it. Whether it is visualised or not, a model exists in our minds and is primarily a human, cultural product. As a specific class of models, 3D models are digital representations of real or imagined objects. The defining property is the use of the third dimension, which makes it stand apart from 2-dimensional, flat, representations. In a 2D representation, an object is just modelled from one point of view. In contrast, 3D models allow for an infinite number of perspectives. They exist in a simulated 3D space, akin to our own. Therefore, a 3D model is a kind of virtual reality. Since 3D models contain information on all sides of an object, they can be effective vehicles for comprehensive data storage and interactive visualisation, allowing data exploration and stimulating perception of visual and spatial properties. Models can be reality based or theoretical. Reality based models are models of measured or scanned objects, and are used for documentation, object analysis and preservation. Theoretical models are visualisations of hypotheses, and as such play a pivotal role in the scientific process of incremental understanding. In an idealised form this process is represented by a circle that leads from model to theory or hypothesis, to verification and experimentation and finally to model adjustment. Finally, these models can also play a role in communication of results of research to both a lay audience and a professional audience of peers. Theoretical models of historical or archaeological sites are often called ‘reconstructions’. Also found are related terms such as ‘restitution’, ‘restoration’ or ‘anastylosis’. These are partly synonymous, but also have different meanings depending on context of use. Without going into detail, we view these as aspects of the general activity of modelling historical objects and environments that have only partially been preserved. The level of interpretation required to create such models differs, between models, but also between smaller parts of models. To document and report these, we have developed a level of certainty scale, discussed in detail later in this document. Types of digital 3D models Digital 3D models exist in a plethora of types and formats. Frequently used data types are point clouds and meshes. Point clouds are nothing more than 3D coordinate measurements (XYZ), with some additional attribute data such as signal intensity or RGB colour. The majority of 3D models are polygon structures informally called 4 Figure 1. The difference between a surface model, solid model, and volumetric model of an egg. meshes. This latticework of points, lines and faces (‘polygons’) defines the geometrical shape of an object. These are essentially empty shells describing the exterior surface of an object and contain no information on the interior. Meshes are so frequently used as they are easy to edit, light on data storage and supported by hardware and software acceleration for fast rendering. Data on the interior is stored only with volumetric and solid models (fig. 1). Volumetric 3D models contain data points on the interior of an object, much like on the exterior. Associated values (i.e. readings from a scanner) can vary throughout the volume. This is the principle form of data used for CT scanning, but has many more applications in various types of physics simulations. Volumetric data can come in structured raster format (‘voxels’) as well as unstructured polygon format consisting of tetrahedrons (‘pyramids’) or hexahedrons (‘cubes’). Solid models are different in that they do not allow gradients or varying distributions throughout a volume, but they do store information on the interior of an object. This mostly involves a measure of volume or material properties. Solid models can be sectioned at any location, which will reveal a solid surface. Such models are mostly used in computer aided design (CAD) in the field of engineering and manufacturing. Volumetric and solid models are less frequently used in the domain of cultural studies, but can both have their specific analytical applications. Accuracy, precision, reliability, and resolution Accuracy, precision, reliability, and resolution are important concepts in modelling that apply to various types of data, both 3D and 2D. Accuracy refers to the deviation between a measured point in a model and reality. Precision is the number of points of measurement distributed over an object, expressed for instance as points per mm2, cm2 or m2. Measurements made with devices such as 3D scanners, but also Total Stations, and measuring tape, have both an accuracy and a precision. Theoretically, any model (2D/3D) derived from this data also has an accuracy and precision. But this is less easy to quantify in the same way, as it requires another base measurement to verify. For data used to display we prefer to use reliability and resolution rather than accuracy and precision (table 1). 5 Concept M D Definition Accuracy Deviation of a single point of measurement from reality. Or: average deviation of a group of measurements (e.g. 3D scanned object) to reality. Precision Average distance between different measurement points. Reliability Deviation of a representation from reality due to subjective choices. Resolution (Average) distance or size of points in a display model of data. E.g. cell size for raster data, average edge length in polygon data. Table 1. Accuracy, precision, reliability, and resolution. M = applies to measurement, D = applies to display Resolution refers to the display properties of the model. The resolution of raster data, familiar to most, is usually expressed in pixels or image points in the width and height of an image. The higher the resolution the more potential detail can be displayed. Likewise, with polygon data such as 3D meshes, the mesh can be denser or less dense, which could be described as the resolution of the mesh. However, contrary to raster images, the resolution for meshes is better expressed in the average edge length of the triangles rather than the total number of polygons. Mesh resolution is akin to precision in concept, as it both refers to the number of points distributed over a defined area. However, as meshes based on 3D measurements can be decimated to lower densities, we prefer to refer to it as mesh resolution to distinguish it from the precision of the original measurement. Reliability is a concept that we use to express the deviations of data due to choices during the process of acquiring or presenting data or information (in any kind of format: a drawing, point cloud, mesh, etc). Related to reliability is the concept of bias. An obvious example is the artist’s approach to representing reality. Pieces of art rarely intend to copy reality, but may involve exclusion of elements, rearrangement, addition of imagined elements, or modifying properties such as size, colour, and material. However, similar choices may be made in the course of a more objective documentation or capture of data. Contrary to the other concepts, reliability is based on a subjective assessment of the source by comparison, literature research and specialist judgment. The concept helps to classify data for which accuracy and precision are hard to express in exact units. Texture resolution Textures (also referred to as ‘texture maps’) are images that are wrapped around 3D models to increase the visual detail. Texture resolution is the number of pixels used in the images used as textures. This is usually expressed with two values: one for the number of pixels in the width of an image, and one for the number of pixels in its 6 height. It is customary to work with exactly square images as textures with values like 1024, 2048 or 4096. Although it is not strictly necessary, this is considered good practice as graphics processors are optimized to process image resolutions which are in a power of 2. Texture resolution is important also for the discussion of level of detail (LoD). Together with texture area, i.e. the real-world area covered by a texture, it determines the minimum size of features that can be represented on a texture. In image-based modelling procedures, the distance of the camera to the model determines the minimal size of features that can be captured. The closer the camera to the subject, the higher the amount of detail. Or, the closer the camera, the smaller the surface area in reality covered by a single pixel of an image. In the field of remote sensing using imaging techniques, the principle is referred to as ‘Ground Sample Distance’ (GSD). If a certain minimum level of detail is a concern, the actual area in reality that is covered by a single pixel on a photograph should be calculated ahead of a project. This way the optimal capturing distance can be chosen. Level of Detail (LoD) Level of detail is one of the defining properties of 3D models. The higher the level of detail, the smaller the features presented in the model. The level of detail should be set to accommodate the aims or research questions of a project. It is important as it will influence how the models can be used in analysis. An additional reason to specify the required level of detail before the start of a project, is that increasing the detail has a big impact on production resources. Two main approaches rise from a literature review of the use of LoD, which we define as the computer graphics (CG) approach and the modelling approach. The approach to CG, which we still use to represent 3D shapes as polygonal meshes, was defined in the 1970s (e.g. Clark 1976). From this perspective, LoD is nothing more than the number of polygons used to define a geometrical object. The lower the number of polygons, the more generalised the representation is. However, more polygons can be used to represent the same basic shape (fig. 2), i.e. a higher number of polygons does not by definition create more detailed models. Another potential problem when defining LoD purely based on polycount, is that the same number of polygons can result in a completely different representation of the same object (fig. 3) (Biljecki et al. 2016). This strongly affects the results of visual analysis or simulations that the models are potentially used for. The modelling approach to LoD originates in disciplines in which 3D models are created and used as part of a professional workflow or analysis. In this case, level of detail explicitly corresponds to the detail found in real world geometrical shapes: landscapes, cities, buildings, architectural elements, etc. Different frameworks have been developed in urban/landscape GIS modelling (Gröger et al. 2012; Biljecki et al. 2016) and Building Information Modelling (BIM wiki 2020). These have been transferred to historical architecture and heritage research (Andrews et al. 2015; Antonopoulou and Bryan 2017), and to a limited degree to archaeology (Agugiaro et 7 Figure 2. Different LoDs for cubes and spheres. The numbers refer to the number of triangles used to draw the shape. Figure 3. Illustration of the shortcomings of the original CityGML LoD concept. Both buildings on the left are LoD2 models of the same object with an identical number of polygons, revealing the ambiguity of the definition (after Biljecki et al. 2016, fig. 2). LoD and textures al. 2011). In a widely used standard like CityGML, LoD are defined by a combination of criteria, such as a modelling size tolerance, but also accuracy of the data, precision, type of data, and type of objects (e.g. exterior vs interior). This stacking of criteria of different kind in the same class, is not an ideal situation, since these are not necessarily correlated variables. For instance, an object can be represented by a highly detailed mesh, but inaccurate in global location. Most applications of LoD in archaeology and heritage emphasize the importance of modelling tolerance (Agugiaro et al. 2011; Andrews et al. 2015; Antonopoulou and Bryan 2017). The LoD classifications always refer to the actual geometric content present in the model. In 3D models, especially when a degree of photorealism is required, much detail is often applied through use of textures, adding colour, depth and other material information (see also below at ‘material accuracy’). Although smaller construction elements, such as bricks, planks and roof tiles can be modelled as geometrical objects, it is often not required, or not feasible, to include all detail in the geometric model. 8 LoD and textures The LoD classifications always refer to the actual geometric content present in the model. In 3D models, especially when a degree of photorealism is required, much detail is often applied through use of textures, adding colour, depth and other material information (see also below at ‘material accuracy’). Although smaller construction elements, such as bricks, planks and roof tiles can be modelled as geometrical objects, it is often not required, or not feasible, to include all detail in the geometric model. LoD in the 4DRL The 4DRL uses a general classification of level of detail with 4 classes (fig. 4), which roughly corresponds to the those proposed by Andrews (2015) for heritage models. We define a primary object, the main model, which is subdivided in components and subcomponents. The primary object can for instance be a city block, an individual house, or a particular window frame construction. In a street of houses that are individually modelled, the houses are the primary object. Our LoDs are a relative classification. If the main object is a smaller feature like a window, the LoD classification of its components and subcomponents also changes. This makes our LoD applicable to different scales or object types. LoD 1 consists of general outline models of the primary objects, which resembles their entire volume. No further detailing is added in this class. Which objects are primary is to be decided ahead of a modelling project. A logical choice is based on a rough assessment of structural units. So, houses or sheds can be considered primary objects, because they form separate structural units. LoD 2 is a general outline of primary components. So, from the perspective of the previous class which treats compound objects (such as houses) as a single entity, in this class a subdivision is made of all main structural elements. These structural elements are modelled as general outlines lacking detail. LoD 3 is an accurate outline of all components. This is a more elaborate model in comparison to class 2, and a finer grade of detail is added. This includes decorative or structural detail such as profiles and bevels. LoD 4 is a precise modelling of all components and subcomponents. If a building is the main object, this includes elaborate decorative elements and structural features such as individual planks in a shutter window. Additionally, ahead of a modelling project it should be specified whether a certain LoD is represented by geometrical content (3D) or by textures (2D). 9 Figure 4. Levels of detail. 10 Figure 5. Level of completeness Level of Completeness (LoC) In addition to LoD, we use level of completeness (LoC) as a complementary concept to indicate a list of modelled elements. For instance, a model of a street with houses can be modelled purely structurally, or with various environmental detail such as fences, street furniture, nature, people (fig. 5). LoC allows for addition or exclusion of specific modelled elements, without having to adjust the overall modelling tolerance of LoD. For instance, for houses it may be required to include the windows, shutters, and doors in the model, but not the chimneys and gutters, even though the dimensions of the latter would allow them to be included at this LoD. Such a list of modelled elements is especially vital for project planning and communication. Expression of material in 3D models In many 3D reconstruction models of archaeological or historical sites a degree of photorealism is required for aesthetic or functional reasons, for instance to increase immersion. In addition, certain analytical applications that involve the perception of space and colour may require accurate expression of material. Due to the way 3D computer graphics work, there is a separation between a geometric and a material component. The material component is commonly driven by image textures, which can add detail to a model that is not present in its geometry (see ‘LoD and textures’), and as such helps to limit file sizes, framerate, rendering speed, and modelling time. Photo textures are probably the most frequently used in archaeological visualisation since the advent of photogrammetry and other 3D scanning techniques. Although this is a quick and effective way to visualise data, photo textures are of limited use in accurate simulations of materials and light, since shadows and reflections are fixed on the image. For accurate material rendering in light simulations, additional textures are required driving individual material properties. Physically Based Rendering (PBR) materials denote materials that specify all the properties required for accurate representation of a certain material. PBR materials therefore generally include texture maps for colour, roughness, microsurface, metallic and optionally other material maps if applicable to the modelled material. 11 Common material properties that are used by 3D models/modelling programs are: • Diffuse colour (RGB) • Roughness/glossiness (inverts of each other, 0-1 value) • Microsurface detail: -Normal (microsurface face orientation variation, encoded in RGB channels) -Bump/height (microsurface height variation, 0-1 value) • Transparency/translucency (0-1 value) • Subsurface colour (RGB, for objects with a translucent outer layer which the interior shines through) • Metallic (value of either 0 or 1) Each of these values are driven by either procedurally (mathematically) generated texture maps, or by RGB texture maps. The latter are raster images, in commonly used file formats such as .png and .jpg. Generally, the option of creating and displaying procedural materials is only available in 3D modelling software. 3D viewers and game engines generally only display static texture maps because procedural materials are computationally expensive. There is a significant difference in workflow and time investment between different workflows to create and apply materials for 3D models. Ahead of the modelling project it should therefore be specified what kind of material properties are required, and whether these should be custom created or that textures from online repositories can be used to create the materials. Level of Material Realism (LMR) We specify different levels of material realism. The chosen level is determined by the general aesthetic or analytical goals of the project. The levels of material realism range from a simple colour coding of 3D models to full PBR rendering. Besides level of material realism, there are multiple approaches to texturing which influences production time. LMR, as shown in table 2 and illustrated in figure 6, has three base classes, and three subclasses, each step adding to material complexity. The most basic material is listed in the top left corner, and the most complex material in the bottom right. The LMR classes are based on the most commonly used material combinations: 1. just RGB (colour), 2. RGB and microsurface bump, 3. full PBR. The subclasses x.0 to x.2 further specify the use of texture: no texture, tiling texture, and non-tiling texture. The amount of effort spend on texturing depends on the complexity of the object. In a procedure called UV-unwrapping, 2D textures are mapped onto a 3D model. For simple objects, automatic methods often suffice, but for more complex geometry custom and manual unwrapping is required to avoid warping and visible seams. Another significant factor is whether textures from online repositories are used to make materials, or that materials are completely built from scratch. The latter can be 12 LMR/texture type x.0 – no texture x.1 – tiling texture x.2 non-tiling texture LMR 1 – just RGB Unique constant value for Tiling image texture Customized (photographed, (colour) each material applied to to drive RGB value of hand painted, image edited, object or object part object surface baked) texture to drive RGB value of object surface Application: simple co- Application: sym- lour symbology bology with material Application: detailed basic indication, basic photo photo-realism, 3D scanned realism, renders from a objects distance (>10m) LMR 2 – RGB and N.A. Tiling image textures to Customized (hand painted, micro surface bump drive RGB and normal/ image edited, baked) tex- (normal/height) height value of object ture to drive RGB value and surface normal/height value of ob- ject surface Application: basic pho- torealism with micro Application: like class 2.1, surface height varia- but with customized detail tion, medium distance to close-up renders (<10m) LMR 3 – full PBR Constant value for each Tiling image textures to Customized (hand painted, object, multiple material drive complete range image edited, baked) tex- properties (roughness, of material properties ture to drive complete range metallic, etc.) of object surface of material properties of object surface Application: simple ma- Application: High fidel- terials with no variation ity apps and renders Application: Material study across a surface (smooth with incidental close- and analysis. High fidelity mirrors, plastics etc.) up. Object/material not apps and renders with di- main focus. rected close-up. Main focus on object/material. Table 2. Level of material realism (LMR) and texture types. done by procedural generation using specialised software, or reality-based textures that are captured using 3D scanning techniques. 13 Figure 6. Level of material realism (LMR) and texture types visualized. Degrees of Certainty Due to the heterogenous nature of historical or archaeological sources, there is a strong variability in the degree of certainty in visualisations, ranging from absolute (empirical evidence) to an academic best guess (no direct evidence). Documenting and displaying the relative differences in certainty of 3D models is generally viewed as an essential element of a transparent reconstruction (Kensek et al. 2004; Hermon et al. 2006; Gkintzou et al. 2012; Apollonio and Giovanni 2015; Ferdani et al. 2019). 14 Terminology The literature review shows there is no generally accepted terminology with associated definitions. Next to ‘certainty’ and ‘uncertainty’, we find authors write about ‘reliability’ and ‘confidence’, when discussing the issue of uncertainty about virtual reconstructions. E.g. Reilly (1992) uses ‘degrees of confidence’ and Hermon et al. (2006) use uncertainty, reliability and confidence interchangeably in slightly different context, but do not define these terms. In view of the uncertainties arising from modelling, we consider it important to distinguish between ‘source data’ and ‘resulting model’. Source data, such as drawings, texts, photos, or 3D point clouds, can be reliable or accurate in the way these represent the real thing. This has been covered already under ‘Accuracy, precision, reliability and resolution’. On the other hand, uncertainty is an attribute of the virtual model, or model part, as opposed to an attribute of the source data. We can be certain or uncertain about for instance the original shape, or the location of an architectural element. Since this degree of certainty is a subjective classification by the researcher based on his or her assessment of the – reliability of – the source data, we can say that certainty is a derivative of reliability. Multiple sources can inform the shape and location of a single model of an object. Therefore, certainty is the reliability of the most reliable/accurate source, or the result after cross-comparison and evaluation of various sources for a single object. Last, confidence, another term often used in this context, can be regarded as roughly synonymous with certainty. Causes for uncertainty We define three major factors that contribute to the level of certainty of a virtual model: the reliability of the source data (subjective factors), the accuracy and precision of the source data (objective factors), and type of inference. The availability and reliability of source data is a main factor. Different types of sources (photos, paintings, 3D scans) have different reliabilities, but also within a category there can be strong variations in reliability as a result of technological (hardware and settings) or human (biases) factors. The accuracy and precision of the source data has been discussed above under a separate heading, ‘Accuracy, precision, reliability and resolution’. These only refer to objective ‘measurable’ properties of an object, and their deviation from reality. The type of inference is the way a source is employed to build an argument towards a model or hypothesis. This differentiation in three major factors ultimately allows us to define that we could have highly accurate data, such as high-resolution 3D scans, as sources for uncertain reconstructions, based on an indirect parallel. The different factors interact with each other in complex ways, but they can be represented in simplified form by a 3-axis graph (fig. 7). Not all types of source interaction can be directly shown in this graph. An example of this is mutual reinforcement: in general, the more sources that verify each other, the higher the degrees of certainty. Another example is the hierarchical position of an element within the model (cf. Niccolucci and Hermon 2004). For instance, reconstructions of archaeological sites generally start with excavated remains (e.g. wall bases), and work 15 Figure 7. Three-axis graph showing the relationship between 3 factors that play a role in cer- tainty of a model. their way upward from there, each step adding more uncertainties, creating more potential for alternative hypothesis. However, each element is only partly dependent on other elements in the chain, but also has its individual reliability based on unique sources for its shape or location. Aspects of uncertainty The concept of uncertainty can be applied to various aspects of models, relating to different properties of physical objects, i.e. shape, location, time, and material. Each of these four aspects possesses uncertainty, that scales independent from the others. So, a high level of certainty about an object’s shape or location does not guarantee a high level of certainty about its colour or temporal presence. Shape and location are both three-dimensional properties of objects that can be approximated in a virtual model using a type of geometric modelling in a coordinate system (a.k.a. 3D modelling). So, uncertainty about shape and location is a matter of metric accuracy, or: how well does the virtual model correspond to the physical dimensions and location of the real object? Time influences the certainty of historical reconstructions significantly. Archaeological or historical sites are often palimpsests. That means that new phases erase old, or parts of old, phases, and that a site may preserve traces of various phases that are hard to untangle. It is often not certain whether an element existed in one phase or another. This uncertainty can be expressed using different methods, such as probability, fuzzy logic, or binary expressions, though their application is further complicated by the likeliness of uneven distributions. This means that we may estimate the probability that an object was present in phase one at 25%, phase two at 65%, and in phase three at 10%. 16 The last property of objects that can be subject to uncertainty are material properties of objects, such as colour or roughness. This aspect determines for a large part the ‘look’ of objects, and has as significant impact on our perception of objects in reality and in virtual models. The certainty of these aspects can be visualized with an independent graph like shown on figure 7, and therefore each can receive a separate classification of uncertainty. Uncertainty classification There is no universally accepted standard for the criteria used to classify degrees of certainty. In a review of practices, all studies showcased their own unique system. Differences exists in identification and classification of relevant factors, and whether the system is quantified or not. A general division can be made between approaches that focus on source data reliability classification, and approaches that classify different types of logical inference (e.g. fig. 8) to arrive at a reconstruction. There exist either separate classifications of both types, or mixed variants. Source data classifications look at the reliability and accuracy of the source data, i.e. 3D scans, survey drawings, photographs, historical artistic rendering, text description. Although important and useful, as a single basis for an uncertainty classification they do not suffice. As has been argued above, uncertainty is a complex interaction of factors. Another issue with a classification of uncertainty with a focus on the sources rather than the models of objects, is that it does not accommodate for the different aspects of certainty (shape, location, time, and material). With inference we refer to the type of reasoning that supports the presence of a certain object. It is common in classifications to distinguish direct observation of evidence from various ways of filling in the gaps, such as structural or analogical reasoning. Classifications based on type of inference are more useful as a basis for uncertainty, since the focus is on the model rather than the source data. However, these should nevertheless be used in combination with an assessment of source data reliability and accuracy. In our system (table 3), types of inference play a central role. We classify direct observation, refitting, interpolation and extrapolation, direct parallel, indirect parallel and analogy. Direct observation only relates to in situ evidence. It can apply to any of the aspects defined earlier, i.e. location, shape, material, and time. Refitting involves restoration of elements found in situ, but not on their original location. Much analysis is based on structural logic to complete the missing elements, e.g. an arch is relatively reliably completed if two large fragments of it preserve. This is what we classify as interpolation and extrapolation. After that come reasoning based on parallel evidence, which are divided in two levels of reliability: direct and indirect parallel. A parallel object found on the site may be regarded as more reliable than a parallel object from a broader art-historical, temporal, or geographic context. The difference between these two is somewhat arbitrary and is in fact more gradual than a binary option. The last type of inference is analogy. Analogies are more general than parallels, which always involve specific objects. On the contrary, analogies 17 Figure 8. Example of use of a certainty classification using 4 classes based on knowledge type (after Ferdani et al. 2019, fig. 10). involve things like construction methods, or general stylistic traditions. The range of analogies and parallels available to the individual researcher or modeller is important. The role of expertise in using types of inference should not be underemphasised. An interpretation of evidence and reconstruction model using analogy is more reliably when made by an expert than a novice. The inference-based approach appears sufficient for a first-level classification, with 6 degrees of certainty based on inference type the most clear and useful towards the purpose of straightforward communicating certainty about a reconstruction. Going beyond 6 classes would, at least visually, create illegible and complicated images. This classification of uncertainty can apply to any scale, from the individual building block (i.e. a brick) to a complete house, depending on the aforementioned LoD. In the resulting models, one can introduce various grain sizes for attributing uncertainty, i.e. dealing with walls, windows and roofs separately, but also associate an uncertainty index with a complete house (this is analogous to how we view annotations must be incorporated, as they can refer to various grain sizes, i.e. granularity; in fact, uncertainty is a subclass of annotation). As a second-level classification, we aim for a system that does include source reliability, accuracy, precision, and inference type 18 CertaintyClass Inference type Variability Example Colour In-situ remains (e.g. Certain Direct observation None 89a5c1 scanned) Placing back of remains Quite certain Refitting Low 95bf8f found on site Interpolation / Missing part of an other- Moderately certain Limited eee3ab extrapolation wise complete object Not so certain Direct parallel Considerable Same type, direct relation d9cfc1 Quite uncertain Indirect parallel High Same type, indirect relation a77e58 E.g. constructional argu- Very uncertain Analogy Very high ba3f1d ment Table 3. Degrees of certainty based on inference types. in all aspects of certainty. A solution would be a quantifying approach, in which we score every individual intersection of factors affecting certainty and arrive at a mathematical description of certainty (cf. Niccolucci and Hermon 2004; Hermon and Nikodem 2008). The purpose of such as system would be to a) be more specific and systematic in order to render the documentation of any project comparable and quantifiable, and b) to be able to generate cumulative scores for (parts of) reconstructions that better approach their exact certainty status. Uncertainty information system To ensure association of the important meta- and paradata related to uncertainty, the model, sources, and the uncertainty classification should form a closed information system. Unlike GIS, this is generally not well integrated with 3D modelling programs out of the box and requires custom-made solutions. Both Hermon and Nikodem (2006) and Ferdani et al. (2019) used a system in which a database with sources and metadata are plugged into 3D modelling software (Blender) (fig. 9). Ferdani et al. (2019) use the Extended Matrix system developed by Demetrescu (2015), a type of graph database which extends the archaeological stratigraphic model of the Harris matrix to reconstruction modelling. Our current approach is limited to assigning certainty classes to 3D model parts as material data, with a colour associated to it. Sources are classified and described in excel table, in which the second-level classification is applied. The value of a more integrated database approach is acknowledged and is an important course of development in the near future. 19 Figure 9. Database schema proposed by Hermon & Nikodem (2006). Uncertainty visualisation When model parts are classified, or linked to a database, it is a simple task to visualise differences in certainty. Visualisation of data should support easy cognition and understanding. Historically, a small number of alternative visualisation styles have been used to draw the viewer’s attention to the interpretative uncertainty in virtual models, concisely reviewed by Kensek et al. (2004: 177-179). Traditional archaeological formats juxtaposing hypothetical model with documented remains are still frequently used. These are quite effective due to their simplicity and clarity. Another popular approach is to use the virtual model as a transparent overlay. A variation on this is to display a wire frame model over the actual remains. Wireframes are probably the most objective display mode for a 3D model, since it honestly reflects the actual model without any shading effects. However, highly detailed models displayed this way can be confusing and may prevent a good understanding of the model. For displaying the graded nature of degrees of certainty, only a few options are practically available to us. The first is a basic colour symbology (often called ‘false colour’) reflecting different degrees of certainty, already suggested by Paul Reilly (1992: 98). This remains the most popular format to this day and may be the most effective for visualising multiple classes, which is why we often use it in the 4DRL (fig. 10). An alternative is to vary the opacity of the model components, where highly opaque components are certain and nearly transparent components are very uncertain. This approach is perhaps more useful in displaying temporal uncertainty, as has been shown by Zuk et al. (2005). Another approach, not often applied to virtual 3D models, is a symbology with different graphical content, such as stripes, dots, etc, as is common in 2D mapping or CAD modelling. Hybrid systems combining colour and opacity or mixing rendering modes (wireframe/solid) are also discussed by Kensek et al. (2004: 179), but aside from this publication, not frequently used in practice. These can be used to effectively communicate uncertainty, especially when the classification includes different aspects of uncertainty. However, they are more complex to design properly to ensure it supports understanding and not cause confusion to the viewer. 20 Figure 10. Degrees of uncertainty visualised in model. Conclusion In this first paper in the 4DRL Report Series, we have set out to shed light on the concepts that are essential to the process of virtual reconstructions, LoD and DoC. These are central to understanding the quality of a 3D model, and therefore also its capabilities for visualization, communication, interactivity and, most importantly, scientific analysis. Following from this, it is therefore very important to have a good conception of the potential affordances of 3D reconstructions beforehand in order to ensure an outcome suitable for specific visualisation and research goals. We therefore hope to have offered a good overview of the possibilities and difficulties of central aspects to 3D modelling, which have their own complexity to them. Whereas LoD can be defined by a rather straightforward set of definitions determined by the current state of 3D technology, the theoretical and methodological aspects of a system of DoC are quite intricate and allow for different approaches. Therefore, although this paper presents and explains the current working principles and standards of the 4DRL, we will further research, develop and refine our methodologies, with the final goal of maximizing the transparency of our work. 21 Bibliography • Agugiaro, G., Remondino, F., Girardi, G., von Schwerin, J., Richards-Rissetto, H., Amicis, R., 2012. A Web-Based Interactive Tool for Multi-Resolution 3D Models of a Maya Archaeological Site. ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences. XXXVIII-5/W16. 10.5194/isprsarchives- XXXVIII-5-W16-23-2011. • Andrews, D., Bedford, J., Bryan, P. 2015. Metric Survey Specifications of Cultural Heritage. Historic England, Swindon. • Antonopoulou, S., Bryan, P. 2017. BIM for Heritage. Developing a Historic Building Information Model. Historic England, Swindon. • Biljecki, F., Ledoux, H., Stoter, J., 2016. An improved LoD specification for 3D building models. Computers, Environment, and Urban Systems, vol. 59, pp. 25-37. • Clark, J. H., 1976. Hierarchical geometric models for visible surface algorithms. Communications of the ACM, 19, 547–554. • Demetrescu, E., 2015. Archaeological Stratigraphy as a formal language for virtual reconstruction. Theory and practice. Journal of Archaeological Science. 57. 10.1016/j. jas.2015.02.004. • Denard, H., 2012. A new introduction to the London Charter, In: Bentkowska-Kafel, A., Denard, H., Baker, D. (eds), 2012. Paradata and Transparency in Virtual Heritage, Routledge, London, pp. 57-72. • Ferdani, D., Demetrescu, E., Cavalieri, M., Pace, G., Lenzi, S., 2019. 3D Modelling and Visualization in Field Archaeology. From Survey To Interpretation Of The Past Using Digital Technologies. Groma 4. DOI: 10.12977/groma26 • Gkintzou, C., Georgopoulos, A., Melón, V., Miranda, Á., 2012. Virtual Reconstruction of the Ancient State of a Ruined Church. Progress in Cultural Heritage Preservation. 551-567. 10.1007/978-3-642-34234-9_57. • Hermon, S., Nikodem, J., Perlingieri, C., 2006. Deconstructing the VR - Data Transparency, Quantified Uncertainty and Reliability of 3D Models. In: Ioannides, M., Arnold, D., Niccolucci, F., Mania, K., 2006. The 7th International Symposium on Virtual Reality, Archaeology and Cultural Heritage VAST, pp. 123-129. • Hermon, S. and Nikodem, J., 2008. 3D Modelling as a Scientific Research Tool in Archaeology, in: Posluschny, A., Lambers, K., Herzog, I. (eds.), 2008. Layers of Perception. Proceedings of the 35th International Conference on Computer Applications and Quantitative Methods in Archaeology (CAA), Berlin, Germany, April 2–6, 2007 (Kolloquien zur Vor- und Frühgeschichte, Vol. 10). Dr. Rudolf Habelt GmbH, Bonn, pp. 140 • Hermon, S., 2012. Scientific Method, chaîne opératoire and Visualization: 3D Modelling as a Research Tool in Archaeology. In: Bentkowska-Kafel, A., Denard, H., Baker, D. (eds), 2012. Paradata and Transparency in Virtual Heritage, Routledge, London, pp. 13-22. • Kensek, K., Dodd, L., Cipolla, N., 2004. Fantastic reconstructions or reconstructions of the fantastic? Tracking and presenting ambiguity, alternatives, and documentation in virtual worlds. Automation in Construction. 13. 175-186. 10.1016/j. autcon.2003.09.010. 22 • Niccolucci, F., Hermon, S. 2004. A fuzzy logic approach to reliability in archaeological virtual reconstruction. In: Niccolucci, F. and S. Hermon (eds.), 2004. Beyond the Artifact. Digital Interpretation of the Past. Proceedings of CAA2004, Prato 13–17 April 2004. Archaeolingua, Budapest, pp. 28-35. • Reilly, P., 1992. Three-dimensional modelling and primary archaeological data. In: Reilly, P., Rahtz, S., 1992. Archaeology and the Information Age. A global perspective, • pp. 92-108, Routledge, London and New York. • Gröger, G., Kolbe, T.H., Nagel, C., Häfele, K-H., 2012. OGCCity Geography Markup Language (CityGML) En-coding Standard. Open Geospatial Consortium. www. opengis.net/spec/citygml/2.0 • Zuk, T., Carpendale, S., Glanzman, W., 2005. Visualizing Temporal Uncertainty in 3D Virtual Reconstructions. Proceedings of the Conference on Virtual Reality, Archaeology, and Cultural Heritage (VAST). 99-106. 10.2312/VAST/VAST05/099-106. • BIM Wiki 2020. https://www.designingbuildings.co.uk/wiki/Level_of_detail_for_ BIM (accessed 21 April 2021). 23 Appendix I. Principle statement I Function of 3D models A 3D model or visualisation is both a research tool and a communication tool. We distinguish between process oriented and outcome-oriented 3D models. In the former the process of 3D modelling is used as a visual research tool, intended to answer specific research questions, and stimulating better insights regarding aspects such as manufacture, structural properties, spatial relations, visual properties, material, or social context of use. The latter refers to the use of 3D models as a medium for dissemination of research or opening up cultural heritage data resources. 3D visualisations are used to communicate the results of a process to either an academic or a lay audience, and/or simply give access to documented 3D data. II Academic rigour A published 3D model is to be held to standards generally accepted by the academic community. Our models strive to live up to the standards set out in the generally accepted principle documents: “The London Charter for the computer-based visualization of cultural heritage” (Denard 2012) and the “Principles of Seville, international principles of virtual archaeology” (International Forum of Virtual Archaeology 2011). The central tenet of these is that a model must in all cases be verifiable, that all the steps should therefore be reported on, the reasoning should be coherent and transparent, and sources should be critically assessed and cited. In order to ensure the highest quality of work, each 4D Research Lab project is planned with the following criteria in mind: All choices and steps made during the process of modelling are kept in a journal and reported on in a final report. The report is written using the ‘project report and setup template’. Sufficient knowledge regarding original manufacture, construction methods, typological ranges, material properties, social and cultural context should be acquired. Interdisciplinarity: ensure regular and fluid exchange of ideas and views between a number of specialists. The model should be build up ‘bottom up’, e.g. starting with the documented material evidence, rather than starting with a general theory which is projected on the data. All sources should be properly referenced, following regular academic practice for written works. These should be included at least as metadata and paradata (see ‘project report and setup template’) but preferably integrated in the model. 24 As most models are the result of interpretative choices, we strive to visualize these as ‘degrees of certainty’. We realise that not all projects request a product that includes all these criteria. We commit to inform our academic collaborators about the importance and advantages of working according to these criteria. At least, the distinction between actual registration and reconstruction should be made clear. Preferably, the level of certainty should be recorded according to our 6 gradations scale (for definitions see ‘degrees of certainty’). Potential variations or alternative reconstructions should be included. III Accessibility and sustainability A 3D model should be made accessible to a large audience and stored in a future-proof way. We are continuously looking for ways to make (online) accessible certain 3D projects respecting open source/open information principles, and aiming for a manner of long-term storage, adhering to standards of open access file formats, that promises the best perspectives in view of sustainability. 25 Appendix II. Project template and 4DRL Report Series Discussion? Project template for historical 3D reconstruction in the 4DRL Template for 3D visualisation projects and reports, v.1 This is a template for work produced by the 4D Research Lab. The template ensures that 3D (visualisation) projects are executed and documented to generally accepted academic standards. The standards adhered to are the “The London Charter for the computer-based visualization of cultural heritage” and the “Principles of Seville, international principles of virtual archaeology”. This template is an implementation of the main principles and ideals found in these documents. In an accompanying document (‘work quality principles’) we set out the most relevant principles for work produced by the 4D Research Lab. Basic project info Name project: Date (from – to): Author of report: Name(s) and function main project initiator(s)/client(s): Name(s) executor (s): …. 4D Research Lab Delivered product: Where to access main outcomes/product: Location and accessibility of project files: Aims and justification Includes the following: Brief overview of project context (e.g. research project, historical/archaeological period and topic) What is the specific aim of the 3D visualization/documentation? (Aims & Methods, principle 2 of LC; Purpose, principle 2 of Seville). E.g. for study/answer question/ testing a hypothesis, or dissemination of research results. In the latter case, also indicate audience: academic/lay public. If applicable: what is/are the research question(s)? Justification: considering the range of visualization and documentation methods, why is 3D visualization/documentation required? (justification, principle 2.1 of London Charter.) Documentation and research design Documentation of the process of modelling. In the course of documentation the terms metadata and paradata may be applied, if consistently, and respecting the following definition: Metadata: the starting data, i.e. the ingredients Paradata: all subsequent steps, i.e. the recipe 26 NB. The end stage of the modelling process results in a new product (ingredient) that receives its own metadata description when further distributed. 27 3D data recording, processing and analysis workflows Applicable for 3D recording projects using any type of scanning technology, such as laser, structured light, image based modelling/photogrammetry. Describe all steps of this process, including all the relevant settings in hard and software. 3D data acquisition procedure, include the following: Properties of 3D scanner: type, model, accuracy, precision (theoretical) Scanning setup/conditions: outside/inside/light conditions/atmospheric conditions/ physical support installation Properties of resulting point clouds/meshes/textures The software used for recording In case of UAS deployment, the flight form should be incorporated into the documentation Discussion of 3D data processing methods following initial capture: Software used Procedures used (e.g. decimation, statistical outlier removal, mesh production etc) Justification for procedures (e.g. in relation to research or visualization aims) Discussion of 3D data analysis: Software used Analytical techniques used (statistical, visual) Justification for chosen techniques Virtual restoration or reconstruction workflows Virtual restoration involves the digital modelling of gaps in 3D data similar to a way a restorer approaches a broken or fragmentary object. Virtual reconstruction involves the reconstruction of objects that do not exist anymore, but evidence of their existence can be found in various material, visual and textual sources. The aim is to document the sources and important steps in the reasoning that led to a certain restoration/reconstruction. The modeller/researcher should also document uncertainties and alternative possibilities where possible. Methodology Description of the general methodology or approach. These include the ‘rules’ the modeller/researcher adhered to during the process. 28 For instance, digital artefact restoration involves careful visual study of surfaces and surface curvatures, often aided by different surface representations in 3D visualization tools. These insights lead to hypothetical continuations of broken surfaces. In the case of building reconstruction, certain structural principles may guide or dictate the reasoning. Sources (principle 3 LC) List of all sources, digital and non-digital directly influencing the visualization: Written work Drawings Photos Previous reconstructions Specialists involved that contributed to the research (often the client) Source criticism: brief assessment of reliability and potential bias. Process description and results This is a step by step description of important decisions, taking head of the following aspects: Indicate alternative solutions to specific reconstruction problems. Indicate where incongruences arise, in particular with previous ideas or reconstructions. Indicate new insights resulting from the process of modelling. Essential: make sure to document the entire process during modelling in a document including screenshots, references to sources and a description of issues and choices. Conclusion Concise description of most important results. Outputs A suitable form has to be found for the dissemination of the results of the modelling work and the related research (LC principle 6: access). In many cases, it is just this report and the related files handed over to the client. In this case it is the client’s responsibility to further disseminate the results. 29 In other cases, the production of a form for dissemination is part of the project assignment. Examples are a paper in a journal, a blog on a website, a location based visualization, or a web based visualization. In the report, discuss the following: Format, type or medium of publication Discuss which results are disseminated, and how In the case of alternatives and uncertainties: how are these visualized/integrated? How are sources dealt with: as accompanying general text, or included in the model as annotations? In the latter, how are these annotations stored (hardcoded, or in a database)? Describe process of production of output. E.g. applications used, code written/ modified How to access outputs Classification of (un)certainty 1. Discussion: evidence vs variability 2. 4DRL (un)certainty classification 4D Research Lab degrees of certainty definition v.1 Definition For 3D visualizations we apply a classification of 6 classes, listed below. Classes are defined based on the principle of ‘potential variability’. Potential variability is a subjective assessment of the degree of possible variation within a model part. It is an indication of the reliability of sources or certainty of our knowledge about how a certain object or part looked like. Each project may include a specific application of this classification, with particular definitions for each class. An example is that class definitions may reference particular sources (parallels, depictions) used repeatedly for various elements in the model. Number of classes We believe in a limited number of ordinal qualitative classes, with the main aim of clarity and legibility. It is also to avoid the creation of small meaningless classes, that may start to resemble a quantitative scale; the qualitative arguments must always be the main focus. We do not recommend using subclasses of uncertainty (i.e. 2.1-2.5) for this reason. Ambiguity In case of typical issues such as ambiguity, for example contradicting sources, possibly leading to a weight-of-argument formula, this gets solved in the text/database. Scale 30 Our classification of uncertainty can apply to any scale, from the individual building block (i.e. a brick) to a complete house. In the resulting models, one can introduce various grain sizes for attributing uncertainty, i.e. dealing with walls, windows and roofs separately, but also associate an uncertainty index with a complete house (this is analogous to how we view annotations must be incorporated, as they can refer to various grain sizes, i.e. granularity; in fact, uncertainty is a subclass of annotation). Accumulation The various indications of uncertainty of individual building blocks can aggregate into a cumulative certainty, this is exactly why there are different grain sizes. Integration Simple indications of the certainty classes listed below may be incorporated into the model as colour codes. The justification and reasoning go into the report, the accompanying text and/or database. Modelling should take into account the integration of colour coding. To accomplish this, two general methods are proposed: 1) keep model parts with a different certainty classification apart as separate (mesh) objects in the 3D model. The colour coding must be applied as separate material to the corresponding model parts. 2) if model parts cannot be kept separate, but must be integrated in single objects, separate colour coded textures are made for these objects. 31