Isabella Di Lenardo

LLM-Powered Agents for Navigating Venice's Historical Cadastre

T. Karch; J. Saydaliev; I. di Lenardo; F. Kaplan

Cadastral data reveal key information about the historical organization of cities but are often non-standardized due to diverse formats and human annotations, complicating large-scale analysis. We explore as a case study Venice's urban history during the critical period from 1740 to 1808, capturing the transition following the fall of the ancient Republic and the Ancien Régime. This era's complex cadastral data, marked by its volume and lack of uniform structure, presents unique challenges that our approach adeptly navigates, enabling us to generate spatial queries that bridge past and present urban landscapes. We present a text-to-programs framework that leverages Large Language Models (LLMs) to translate natural language queries into executable code for processing historical cadastral records. Our methodology implements two complementary techniques: a text-to-SQL approach for handling structured queries about specific cadastral information, and a text-to-Python approach for complex analytical operations requiring custom data manipulation. We propose a taxonomy that classifies historical research questions based on their complexity and analytical requirements, mapping them to the most appropriate technical approach. This framework is supported by an investigation into the execution consistency of the system, alongside a qualitative analysis of the answers it produces. By ensuring interpretability and minimizing hallucination through verifiable program outputs, we demonstrate the system's effectiveness in reconstructing past population information, property features, and spatiotemporal comparisons in Venice.

Recognizing and Sequencing Multi-word Texts in Maps Using an Attentive Pointer

M. Zou; T. Dai; R. Petitpierre; B. Vaienti; F. Kaplan  et al.

Extracting and recognizing texts from historical maps presents significant challenges due to complex layouts, varied typographic conventions, and the entanglement of multiple sequences. In this paper, we present a modular neural framework for linking and ordering text segments together. This task goes beyond simple word recognition; it enables to recover the complete text sequences. Our solution, based on an Attentive Pointer, successfully manages the presence of distractor words. It leverages both positional and Bézier directional features. We demonstrate the effectiveness of our framework with two practical applications. First, we prove its scalability by applying it to the 1890s Ordnance Survey of London, retrieving 285,846 text sequences. Second, we validate the practical effectiveness of the sequenced placenames by geocoding them and showcasing their capability to automate city maps realignment. Our approach is scalable, trainable, and generic. It supports hierarchical integration and multimodal feature fusion by design, making it an extensible and modular framework for further advancements.

Segmentation and Clustering of Local Planimetric Distortion Patterns in Historical Maps of Jerusalem

B. Vaienti; I. di Lenardo; F. Kaplan

The advancement of computational tools for cartometric analysis has opened new avenues for the identification and understanding of stemmatic relationships between historical maps through the analysis of their planimetric distortions. The 19th-century Western cartographic depiction of Jerusalem serves as an ideal case study in this context. The challenges of conducting comprehensive onsite surveys—due to limited time and local knowledge—combined with the fascination surrounding the area’s representation, resulted in a proliferation of maps marked by frequent errors, distortions, and extensive copying. How can planimetric similarities and differences between maps be measured, and what insights can be derived from these comparisons? This paper introduces a methodology aimed at detecting and segmenting regions of local planimetric similarity across maps, corresponding to the portions that were either copied between them or derived from a common source. To detect these areas, the ground control points from the georeferencing process are employed to deform a common lattice grid for each map. These grids, triangulated to maintain shape rigidity, can be partitioned under conditions of geometric similarity, allowing for the segmentation and clustering of locally similar regions that represent shared areas between the maps. By integrating this segmentation with a filter on the intensity of distortion, the areas of the grid that are almost non-deformed, and thus not relevant for the study, can be excluded. To showcase the support this methodology offers for close reading, it is applied to the maps in the dataset depicting the Russian Compound. The methodology serves as a tool to assist in constructing the genealogy of the area’s representation and uncovering new historical insights. A larger dataset of 50 maps from the 19th century is then used to identify all the local predecessors of a given map, showcasing another application of the methodology, particularly when working with extensive collections of maps. These findings highlight the potential of computational cartometry to uncover hidden layers of cartographic knowledge and to advance the digital genealogy of map collections.

Paris and Jerusalem Maps Text Dataset

T. Dai; K. T. Johnson; R. G. Petitpierre; B. Vaienti; I. di Lenardo

This dataset contains 82 annotated map samples from diverse historical city maps of Jerusalem and Paris, suitable for map text detection, recognition, and sequencing.

London 1890s Ordnance Survey Text Layer

M. Zou; R. G. Petitpierre; I. Di Lenardo

This dataset contains a sample of 10,000 (3.5%) out of a total of 285,846 text sequences extracted from the 1891–1896 Map of London by the Ordnance Survey (OS). The methodology used for the automated recognition, linking, and sequencing of the text is detailed in the article Recognizing and Sequencing Multi-word Texts in Maps Using an Attentive Pointer by M. Zou et al., 2025. The map is drawn at a scale of five-feet to the mile (c.a. 1:1,056). The text on the map is an invaluable source of information about the Greater London in the late Victorian period. It includes the names of streets, squares, parks, watercourses and even some estates ('Poplars', 'The Grange', 'Arbutus Lodge'). In addition, the map contains many details of the function of buildings and economic activity, such as factories ('Sweet Factory', 'Crown Linoleum Works', 'Imperial Flour Mills', 'Lion Brewery'), warehouses or commercial infrastructure ('Warehouse', 'Jamaica Wharf', 'Rag Store'), offices ('Offices'), etc. The map also mentions public buildings such as schools ('School Boys, Girls & Infants', 'Sunday School'), hospitals or clinics ('St. Saviour's Union Infirmary', 'Beulah Spa Hydropathic Establishment', 'South Western Fever Hospital'), railway stations ('Clapham Station'), post offices, banks, police stations, etc. Other social venues are also mentioned, such as public houses, i.e. pubs ('P.H.'), clubs, casinos, and recreational areas (e.g. 'Cricket Ground'). Special attention is given to churches, with a regular count of the number of seats (e.g. 'Baptist Chapel Seats for 600'). In addition, the map provides details that can be of great interest in the study of everyday life in London at the end of the 19th century. For example, there are numerous mentions of 'Stables', 'Drinking Fountain'[s] (or simply 'Fn.') or 'Urinal'[s]. Fire protection infrastructure is highlighted, e.g. fire plugs ('F.P.') and fire alarms ('F.A.'). The map also includes information on elevation (e.g. '11·6') and flood levels (e.g. 'High Water Mark of Ordinary Tides').

Reconstructing Saint Mark’s Square in Venice: A 4D Model with Point Cloud Integration for Analyzing Historiographical Hypotheses

I. di Lenardo; B. Vaienti; P. Guhennec; Y. Ubelmann; I. Bianchi  et al.

This study explores the 4D reconstruction of part of the monumental heritage of St. Mark’s Square in Venice, contextualizing the modeling of its evolution in the broader field of digital historiography. The presented methodology is based on comparing the plausibility of proposed 3D models with current point cloud and historiographical hypotheses that have been compared to the problem of planimetric and volumetric simulation of buildings. Methods include integrating 3D modeling with temporal data to simulate architectural and urban evolution, providing a dynamic and problematic visualization of historical changes and identifying discordant historiographical assumptions. Aligning 3D model reconstructions with the current point cloud provides a better understanding of the plausibility of reconstructions proposed by historiography while identifying potential errors and inconsistencies in some historical documentary sources on which reconstructive approaches are often based. For the reconstructions proposed here, the paper provides an assessment of the level of uncertainty based on some recent notions related to quantifying average uncertainty. Although preliminary, these results provide a basis for testing hypotheses about Venice’s architectural past. Moreover, they open up a new, more generic perspective for the analysis of architectural–urban transformations, based on the notion of planimetric and volumetric constraints, and their comparison with current point cloud records. Using this methodology, we are able to make some solid points with respect to reconstructive hypotheses of the past and indicate which historiographical theories are more plausible than others.

The Skin of Venice: A Computational Approach to the City's Urban Structure

P. Guhennec

This thesis explores the how computational methods can be used to enrich the history of architecture. Using Venice as a case study, it establishes a rich and multifaceted information system, made up of temporal, stylistic, photographic, architectural, and administrative information taken from diverse sources. A complete point cloud of the city, obtained from aerial photogrammetry, is segmented using tailor-made algorithms to produce 3D models of each building in the city. Further methods of segmentation and reprojection are developed to convert those 3D point cloud models into a complete set of orthophotos of Venetian façades. This unique corpus of over 8,567 buildings and 14,167 façades --- alongside the supplementary information system --- are used as the starting point of a formalist reading of Venice. A series of experiments cross-interrogate the fabric of the city: the morphologies of footprints are mapped and analysed in relation to their date and location; the propagation of architectural motifs and typologies of façades are assessed across the city; the networks of lines of sight are reconstructed to question the visual centrality of buildings, identify strategic positions in the city; the long-mentioned homogeneity of the fabric is challenged and instances of rupture are identified. Each of these experiments is then used to contextualise and better understand the architectural solutions chosen and their relationship to topographical, cultural, and socio-political conditions. By displacing the focal point from well-studied centres of power and high-profile architecture to the more vernacular and so-called "minor" architecture, this thesis argues that a computational outlook can open new avenues of research, shedding light on the understudied urban fabric hitherto left in the shadows, and offer original ways of looking at the city at scale. Finally, this works attempts to prove that its large-scale visual hermeneutics is at the heart of a fruitful dialogue between a long historiographical tradition and digital methods.

Exploring cartographic genealogies through deformation analysis: case studies on ancient maps and synthetic data

B. Vaienti; I. Di Lenardo; F. Kaplan

This paper presents a new methodology for identifying copying relationships in maps through cartographic deformation analysis, and introduces a novel approach for the creation of synthetic datasets for evaluation. Challenging traditional approaches that compare historical maps to modern ones for an accuracy assessment, this study focuses on the comparison between old maps through a modern reference map to identify copying processes. Using a synthetic dataset of deformed maps, this research introduces a novel way to assess deformation analysis results. The methodology’s effectiveness is demonstrated through experiments on the synthetic dataset and on five maps of Jerusalem where copying relationships are known, revealing that cosine similarity, filtered with vectors’ norm and combined with their ratio, offers a good discriminative power between maps that are complete copies and those which are not. The methodology, with the selected metric, is applied to a selection of fifty 19th-century maps of Jerusalem and used to create a hierarchical clustering. This approach enables the identification of complete copies of varying quality within large map collections. This study not only advances cartometric methodologies but also lays the groundwork for future research on the automatic detection of map lineages, marking progress toward the genealogical analysis of large cartographic datasets.

Cartographic Stemmatology: Segmenting Local Deformation Similarities in Historical Maps of Jerusalem

B. Vaienti; I. Di Lenardo; F. Kaplan

The advancement of computational tools for cartometric analysis has opened new avenues for the identification and understanding of stemmatic relationships between historical maps through the analysis of their planimetric deformations. The 19th-century western cartographic depiction of Jerusalem serves as an exemplary case study in this context, since the difficulties in conducting comprehensive onsite surveys and the fascination towards the depiction of this area led to a proliferation of maps characterized by frequent errors, distortions, and widespread copying. How can similarities and differences between maps be measured, and what insights can be derived from these comparisons? This paper introduces a methodology aimed at detecting and segmenting regions of local similarity across maps, corresponding to the portions that were either copied between them or derived from a common source. To detect these areas, the Ground Control Points from the georeferencing process are employed to deform a common lattice grid for each map. These grids, triangulated to maintain shape rigidity, can be partitioned under conditions of geometric similarity, allowing for the segmentation and clustering of locally similar regions that represent shared areas between the maps. By integrating this segmentation with a filter on the intensity of deformation, the areas of the grid that are almost non-deformed, and thus not relevant for the study, can be excluded. Our analysis of fifty 19th-century maps of Jerusalem, the most extensive dataset computationally explored to date, reveals the effectiveness of this method in selecting areas of local similarity, even when limited. The methodology is applied to the maps in the dataset representing the Russian Compound and used as a tool to aid in the creation of the stemmata of the area's representation and to uncover new historical insight. These findings underscore the potential of computational cartometry in revealing hidden layers of cartographic knowledge and contributing to the digital genealogy of map collections.

Revealing the Structure of Land Ownership through the Automatic Vectorisation of Swiss Cadastral Plans

R. G. Petitpierre; I. Di Lenardo; L. A. A. Rappo

This paper explores how the dynamics and structure of land ownership can be studied by automating the vectorisation of historical cadastral sources. Our analysis focuses mainly on three cadastres of Lausanne (1722, 1831, 1883). The plans are first georeferenced, before being semantically segmented using a neural model, and vectorised. We study the dynamics of persistence using a spatial matching methodology, detecting parcel fusions and divisions through time. We also investigate the social structure of ownership by matching representative owners to their socioprofessional status, based on historical population censuses.

A fragment-based approach for computing the long-term visual evolution of historical maps

R. Petitpierre; J. H. Uhl; I. Di Lenardo; F. Kaplan

Cartography, as a strategic technology, is a historical marker. Maps are tightly connected to the cultural construction of the environment. The increasing availability of digital collections of historical map images provides an unprecedented opportunity to study large map corpora. Corpus linguistics has led to significant advances in understanding how languages change. Research on large map corpora could in turn significantly contribute to understanding cultural and historical changes. We develop a methodology for cartographic stylometry, with an approach inspired by structuralist linguistics, considering maps as visual language systems. As a case study, we focus on a corpus of 10,000 French and Swiss maps, published between 1600 and 1950. Our method is based on the fragmentation of the map image into elementary map units. A fully interpretable feature representation of these units is computed by contrasting maps from different, coherent cartographic series, based on a set of candidate visual features (texture, morphology, graphical load). The resulting representation effectively distinguishes between map series, enabling the elementary units to be grouped into types, whose distribution can be examined over 350 years. The results show that the analyzed maps underwent a steady abstraction process during the 17th and 18th centuries. The 19th century brought a lasting scission between small- and large-scale maps. Macroscopic trends are also highlighted, such as a surge in the production of fine lines, and an increase in map load, that reveal cultural fashion processes and shifts in mapping practices. This initial research demonstrates how cartographic stylometry can be used for exploratory research on visual languages and cultural evolution in large map corpora, opening an effective dialogue with the history of cartography. It also deepens the understanding of cartography by revealing macroscopic phenomena over the long term.

Artificial Intelligence for Digital Heritage Innovation: Setting up a R&D Agenda for Europe

S. Muenster; F. Maiwald; I. di Lenardo; J. Henriksson; A. Isaac  et al.

Artificial intelligence (AI) is a game changer in many fields, including cultural heritage. It supports the planning and preservation of heritage sites and cities, enables the creation of virtual experiences to enrich cultural tourism and engagement, supports research, and increases access and understanding of heritage objects. Despite some impressive examples, the full potential of AI for economic, social, and cultural change is not yet fully visible. Against this background, this article aims to (a) highlight the scope of AI in the field of cultural heritage and innovation, (b) highlight the state of the art of AI technologies for cultural heritage, (c) highlight challenges and opportunities, and (d) outline an agenda for AI, cultural heritage, and innovation.

Tracing Cartographic Errors in the Western Representation of 19th Century Jerusalem

B. Vaienti; I. Di Lenardo; F. Kaplan

SAOR : Scuola di Arte e Orizzonti culinari di Rialto

L. Ferretti

L’énoncé théorique, De la gloire à la crise. Perspectives futures pour le marché de Rialto, raconte l’histoire d’un marché qui a toujours été le cœur et le miroir de Venise. Au centre de l’économie de la République vénitienne, il grandissait avec la croissance de la ville et se réduisait avec ses difficultés. Ainsi, depuis la chute de la République en 1797 et davantage encore depuis les années 1950, le marché comme la ville sont dans une crise qui met en péril leur survie. Le projet souhaite redonner vie à la ville à partir de son cœur, l’île de Rialto, en réactivant un ensemble existant pour attirer une nouvelle population durable. Le projet s’implante dans les trois corps de bâtiment des anciens locaux du tribunal et s’articule autour de quatre places publiques existantes tout en offrant un visage direct sur le canal. SAOR, dont le nom évoque une préparation locale typique de conservation du poisson, propose une école de cuisine issue de la tradition vénitienne. L’objectif est de ramener des étudiant.e.s au cœur de la ville et de leur donner la possibilité d’entrer dans le marché du travail local, par le biais de la restauration, afin de s’y installer durablement. L’intégration d’un restaurant et d’un bar gérés par les étudiant.e.s dans le projet ainsi que l’utilisation des produits du marché permettent d’intégrer l’établissement dans le tissu social. Ceci doit favoriser l’installation d’une population pérenne, engendrer une hausse des services aux habitant.e.s et, enfin, rendre la ville plus attractive.

The Skin of Venice: Automatic Facade Extraction from Point Clouds

P. Guhennec; I. Di Lenardo

We propose a method to extract orthogonal views of facades from photogrammetric models of cities. This method was applied to extract all facades of the city of Venice. The result images open up new areas of research in architectural history.

1805-1898 Census Records of Lausanne : a Long Digital Dataset for Demographic History

R. Petitpierre; M. Kramer; L. Rappo; I. di Lenardo

This historical dataset stems from the project of automatic extraction of 72 census records of Lausanne, Switzerland. The complete dataset covers a century of historical demography in Lausanne (1805-1898), which corresponds to 18,831 pages, and nearly 6 million cells. Content. The data published in this repository correspond to a first release, i.e. a diachronic slice of one register every 8 to 9 years. Unfortunately, the remaining data are currently under embargo. Their publication will take place as soon as possible, and at the latest by the end of 2023. In the meantime, the data presented here correspond to a large subset of 2,844 pages, which already allows to investigate most research hypotheses. The population censuses, digitized by the Archives of the city of Lausanne, continuously cover the evolution of the population in Lausanne throughout the 19th century, starting in 1805, with only one long interruption from 1814 to 1831. Highly detailed, they are an invaluable source for studying migration, economic and social history, and traces of cultural exchanges not only with Bern, but also with France and Italy. Indeed, the system of tracing family origin, specific to Switzerland, allows to follow the migratory movements of families long before the censuses appeared. The bourgeoisie is also an essential economic tracer. In addition, censuses extensively describe the organization of the social fabric into family nuclei, around which gravitate various boarders, workers, servants or apprentices, often living in the same apartment with the family. Production. The structure and richness of censuses have also provided an opportunity to develop automatic methods for processing structured documents. The processing of censuses includes several steps, from the identification of text segments to the restructuring of information as digital tabular data, through Handwritten Text Recognition and the automatic segmentation of the structure using neural networks. Please note that the detailed extraction methodology, as well as the complete evaluation of performance and reliability is published in: Petitpierre R., Rappo L., Kramer M. (2023). An end-to-end pipeline for historical censuses processing. International Journal on Document Analysis and Recognition (IJDAR). doi: 10.1007/s10032-023-00428-9 Data structure. The data are structured in rows and columns, with each row corresponding to a household. Multiple entries in the same column for a single household are separated by vertical bars 〈|〉. The center point 〈·〉 indicates an empty entry. For some columns (e.g., street name, house number, owner name), an empty entry indicates that the last non-empty value should be carried over. The page number is in the last column. Liability. The data presented here are not curated nor verified. They are the raw results of the extraction, the reliability of which was thoroughly assessed in the above-mentioned publication. We insist on the fact that for any reuse of this data for research purposes, the implementation of an appropriate methodology is necessary. This may typically include string distance heuristics, or statistical methodologies to deal with noise and uncertainty. References: ["Petitpierre R., Rappo L., Kramer M. (2023). An end-to-end pipeline for historical censuses processing. International Journal on Document Analysis and Recognition (IJDAR). doi: 10.1007/s10032-023-00428-9"]

Recartographier l'espace napoléonien : Une lecture computationnelle du cadastre historique de Lausanne

R. Petitpierre; L. Rappo; I. di Lenardo

Le cadastre napoléonien est une source historique relativement homogène et largement répandue. Cela rend une approche computationnelle particulièrement pertinente. Dans cette étude, nous proposons une méthode de reconnaissance et de vectorisation automatique des cartes cadastrales à la pointe de la technologie. Nous démontrons son efficacité sur le cas lausannois et proposons des méthodes d'analyse congruentes.

Ce que les machines ont vu et que nous ne savons pas encore

I. Di Lenardo; F. Kaplan

Cet article conceptualise l’idée qu’il existe une « matière noire » composée des structurations latentes identifiées par le regard machinique sur de grandes collections photographiques patrimoniales. Les campagnes photographiques de l’histoire de l’art, au xxe siècle, avaient pour ambition implicite de transformer toutes les œuvres d’art en documents plus facilement étudiables. Au fil du temps, la création de ces collections visuelles a permis de produire un corpus d’informations potentiellement plus dense et plus riche que ce que ses créateurs avaient initialement imaginé. En effet, la conversion numérique de ces immenses corpus visuels permet aujourd’hui de réanalyser ces images avec des techniques de vision par ordinateur, l’intelligence artificielle ouvrant ainsi la voie à des perspectives d’études bien différentes de celles envisageables au siècle dernier. Nous pourrions ainsi dire qu’il y a dans ces images un immense potentiel latent de connaissance, un réseau dense de relations qui n’a pas encore été mis en lumière. Qu’est-ce que les machines ont vu ou vont pouvoir voir dans ces collections d’images que l’homme n’a pas encore identifié ? Quelle étendue la connaissance visuelle de l’homme couvre-t-elle par rapport à ce que la machine a pu analyser ? Les nouvelles techniques d’indexation des images et des motifs qui les constituent nous rapprochent d’une révolution copernicienne du visuel dans laquelle l’homme peut, grâce à la machine-prothèse, analyser beaucoup plus d’images qu’il ne pouvait le faire par une simple activité mnémonique et sélectionner des perspectives spécifiques en comparant des ensembles de motifs les uns par rapport aux autres. Cette vision augmentée est fondée sur une pré-analyse conduite par la machine sur l’ensemble de ces corpus visuels, un entraînement qui permet de retrouver la structure sous-jacente du système d’images. La vision humaine est ainsi étendue par le regard artificiel préalable de la machine. Pour comprendre les enjeux de cette nouvelle alliance, il faut étudier la nature de ce regard artificiel, comprendre son potentiel pour découvrir des structures jusqu’à présent inconnues et anticiper les nouvelles formes de connaissances humaines auxquelles il pourra donner naissance. L’enjeu sera donc, pour les prochaines années, de comprendre ce que les machines ont vu et que nous ne savons pas encore.

Machine-Learning-Enhanced Procedural Modeling for 4D Historical Cities Reconstruction

B. Vaienti; R. G. Petitpierre; I. Di Lenardo; F. Kaplan

The generation of 3D models depicting cities in the past holds great potential for documentation and educational purposes. However, it is often hindered by incomplete historical data and the specialized expertise required. To address these challenges, we propose a framework for historical city reconstruction. By integrating procedural modeling techniques and machine learning models within a Geographic Information System (GIS) framework, our pipeline allows for effective management of spatial data and the generation of detailed 3D models. We developed an open-source Python module that fills gaps in 2D GIS datasets and directly generates 3D models up to LOD 2.1 from GIS files. The use of the CityJSON format ensures interoperability and accommodates the specific needs of historical models. A practical case study using footprints of the Old City of Jerusalem between 1840 and 1940 demonstrates the creation, completion, and 3D representation of the dataset, highlighting the versatility and effectiveness of our approach. This research contributes to the accessibility and accuracy of historical city models, providing tools for the generation of informative 3D models. By incorporating machine learning models and maintaining the dynamic nature of the models, we ensure the possibility of supporting ongoing updates and refinement based on newly acquired data. Our procedural modeling methodology offers a streamlined and open-source solution for historical city reconstruction, eliminating the need for additional software and increasing the usability and practicality of the process.

A data structure for scientific models of historical cities: extending the CityJSON format

B. Vaienti; P. Guhennec; I. di Lenardo

In the field of the 3D reconstruction of cities in the past there is a raising interest in the creation of models that are not just geometrical, but also informative, semantic and georeferenced. Despite the advancements that were done in the historical reconstruction of architecture and archaeology, the solutions designed for larger scale models are still very limited. On the other hand, research on the digitisation of current-day cities provides useful instruments. In particular, CityJSON - a JSON encoding of CityGML - represents an easy-to-use and lightweight solution for storing 3D models of cities that are geolocated, semantic and that contain additional information in the form of attributes. This contribution proposes (1) to extend the schema to the needs of a historical representation; and (2) to incorporate the newly created model in a continuous flow pipeline, in which the geometry is dynamically updated each time an attribute is changed, as a means to foster collaboration.

Introducing the HIPE 2022 Shared Task: Named Entity Recognition and Linking in Multilingual Historical Documents

M. Ehrmann; M. Romanello; A. Doucet; S. Clematide

We present the HIPE-2022 shared task on named entity processing in multilingual historical documents. Following the success of the first CLEF-HIPE-2020 evaluation lab, this edition confronts systems with the challenges of dealing with more languages, learning domain-specific entities, and adapting to diverse annotation tag sets. HIPE-2022 is part of the ongoing efforts of the natural language processing and digital humanities communities to adapt and develop appropriate technologies to efficiently retrieve and explore information from historical texts. On such material, however, named entity processing techniques face the challenges of domain heterogeneity, input noisiness, dynamics of language, and lack of resources. In this context, the main objective of the evaluation lab is to gain new insights into the transferability of named entity processing approaches across languages, time periods, document types, and annotation tag sets.

The Replica Project: Co-Designing a Discovery Engine for Digital Art History

I. Di Lenardo

This article explains how the Replica project is a particular case of different professionals coming together to achieve the digitization of a historical photographic archive, intersecting complementary knowledge specific to normally unconnected communities. In particular the community of Art History researchers, brought together here in relation to their common methodologies in the practice of visual pattern research, became protagonists in the construction of a specific tool, the Morphograph, to navigate through the archive’s photos. A specific research problem, the recognition of visual patterns migrating from one work to another, became the key to developing a new technology initially intended for a specific community of users, but with such a generic character in its approach that it could easily be made available to other uninformed users as learning by doing tools. The Morphograph tool also made it possible to demonstrate how, within a community, the partial expertise of individuality needs to be related to each other and benefits enormously from the knowledge densification mechanism made possible by the sharing. The digital context easily makes it possible to create tools that are specific in terms of content but generic in form that can be communicated and shared with even diverse and uninformed communities.

Generic Semantic Segmentation of Historical Maps

R. G. Petitpierre; F. Kaplan; I. Di Lenardo

Research in automatic map processing is largely focused on homogeneous corpora or even individual maps, leading to inflexible models. Based on two new corpora, the first one centered on maps of Paris and the second one gathering maps of cities from all over the world, we present a method for computing the figurative diversity of cartographic collections. In a second step, we discuss the actual opportunities for CNN-based semantic segmentation of historical city maps. Through several experiments, we analyze the impact of figurative and cultural diversity on the segmentation performance. Finally, we highlight the potential for large-scale and generic algorithms. Training data and code of the described algorithms are made open-source and published with this article.

Aux portes du monde miroir

I. Di Lenardo; F. Kaplan

The Mirror World is no longer an imaginary device, a mirage in a distant future, it is a reality under construction. In Europe, Asia and on the American continent, large companies and the best universities are working to build the infrastructures, to define their functionalities, to specify their logistics. The Mirror World, in its asymptotic form, presents a quasi-continuous representation of the world in motion, integrating, virtually, all photographic perspectives. It is a new giant computational object, opening the way to new research methods or even probably to a new type of science. The economic and cultural stakes of this third platform are immense. If the Mirror World transforms access to knowledge for new generations, as the Web and Social Networks did in their time, it is our responsibility to understand and, if need be, bend its technological trajectory to make this new platform an environment for the critical knowledge of the past and the creative imagination of the future.

Une approche computationnelle du cadastre napoléonien de Venise

I. Di Lenardo; R. Barman; F. Pardini; F. Kaplan

At the beginning of the 19th century, the Napoleonic administration introduced a new standardised description system to give an objective account of the form and functions of the city of Venice. The cadastre, deployed on a European scale, was offering for the first time an articulated and precise view of the structure of the city and its activities, through a methodical approach and standardised categories. With the use of digital techniques, based in particular on deep learning, it is now possible to extract from these documents an accurate and dense representation of the city and its inhabitants. By systematically checking the consistency of the extracted information, these techniques also evaluate the precision and systematicity of the surveyors’ work and therefore indirectly qualify the trust to be placed in the extracted information. This article reviews the history of this computational protosystem and describes how digital techniques offer not only systematic documentation, but also extraction perspectives for latent information, as yet uncharted, but implicitly present in this information system of the past.

I sistemi di immagini nell’archivio digitale di Vico Magistretti

F. Kaplan; I. di Lenardo

La messa a disposizione in linea dell’archivio digitalizzato di Vico Magistretti che raggruppa decine di migliaia di disegni preparatori, disegni tecnici e fotografie prodotte tra 1946 e il 2006, apre la strada a un grande rinnovamento delle ricerche sul designer e architetto italiano. L’apertura di questo archivio così speciale ci invita a immaginare diverse prospettive che possono essere considerate per esplorare, visualizzare e studiare un tale insieme di documenti.

The Advent of the 4D Mirror World

F. Kaplan; I. di Lenardo

The 4D Mirror World is considered to be the next planetary-scale information platform. This commentary gives an overview of the history of the converging trends that have progressively shaped this concept. It retraces how large-scale photographic surveys served to build the first 3D models of buildings, cities, and territories, how these models got shaped into physical and virtual globes, and how eventually the temporal dimension was introduced as an additional way for navigating not only through space but also through time. The underlying assumption of the early large-scale photographic campaign was that image archives had deeper depths of latent knowledge still to be mined. The technology that currently permits the advent of the 4D World through new articulations of dense photographic material combining aerial imagery, historic photo archives, huge video libraries, and crowd-sourced photo documentation precisely exploits this latent potential. Through the automatic recognition of “homologous points,” the photographic material gets connected in time and space, enabling the geometrical computation of hypothetical reconstructions accounting for a perpetually evolving reality. The 4D world emerges as a series of sparse spatiotemporal zones that are progressively connected, forming a denser fabric of representations. On this 4D skeleton, information of cadastral maps, BIM data, or any other specific layers of a geographical information system can be easily articulated. Most of our future planning activities will use it as a way not only to have smooth access to the past but also to plan collectively shared scenarios for the future.

Territories in Time: Mapping Palimpsest Horizons

Reconstruire la densité historique du « Quartier Richelieu »

I. Di Lenardo

Dans le projet « Richelieu. Histoire du quartier », de jeunes ingénieurs en humanités numériques et des historiens ont commencé à analyser les adresses du Bottin du commerce sur la période 1839-1922. La technique a permis d’identifier les informations correspondant à environ 200 000 adresses du quartier Richelieu. Une base géographique a été développée pour visualiser, aujourd'hui en 2D, toutes les occupations et les adresses sur une carte, ainsi que toutes les évolutions historiques du réseau routier et des bâtiments. C’est la première fois que toutes les adresses des Annuaires du Commerce sont massivement extraites grâce à des méthodologies de pointe dans les documents anciens.

Introducing the CLEF 2020 HIPE Shared Task: Named Entity Recognition and Linking on Historical Newspapers

M. Ehrmann; M. Romanello; S. Bircher; S. Clematide

Since its introduction some twenty years ago, named entity (NE) processing has become an essential component of virtually any text mining application and has undergone major changes. Recently, two main trends characterise its developments: the adoption of deep learning architectures and the consideration of textual material originating from historical and cultural heritage collections. While the former opens up new opportunities, the latter introduces new challenges with heterogeneous, historical and noisy inputs. If NE processing tools are increasingly being used in the context of historical documents, performance values are below the ones on contemporary data and are hardly comparable. In this context, this paper introduces the CLEF 2020 Evaluation Lab HIPE (Identifying Historical People, Places and other Entities) on named entity recognition and linking on diachronic historical newspaper material in French, German and English. Our objective is threefold: strengthening the robustness of existing approaches on non-standard inputs, enabling performance comparison of NE processing on historical texts, and, in the long run, fostering efficient semantic indexing of historical documents in order to support scholarship on digital cultural heritage collections.

Building a Mirror World for Venice

F. Kaplan; I. di Lenardo

Between 2012 and 2019, ‘TheVeniceTime Machine Project’ developed a new methodology for modelling the past, present, and future of a city. This methodology is based on two pillars: (a) the vast digitisation and processing of the selected city’s historical records, (b) the digitisation of the city itself, another vast undertaking. The combination of these two processes has the potential to create a new kind of historical information system organised around a diachronic digital twin of a city.

A deep learning approach to Cadastral Computing

S. Ares Oliveira; I. di Lenardo; B. Tourenc; F. Kaplan

This article presents a fully automatic pipeline to transform the Napoleonic Cadastres into an information system. The cadastres established during the first years of the 19th century cover a large part of Europe. For many cities they give one of the first geometrical surveys, linking precise parcels with identification numbers. These identification numbers points to registers where the names of the proprietary. As the Napoleonic cadastres include millions of parcels , it therefore offers a detailed snapshot of large part of Europe’s population at the beginning of the 19th century. As many kinds of computation can be done on such a large object, we use the neologism “cadastral computing” to refer to the operations performed on such datasets. This approach is the first fully automatic pipeline to transform the Napoleonic Cadastres into an information system.

Repopulating Paris: massive extraction of 4 Million addresses from city directories between 1839 and 1922

I. di Lenardo; R. Barman; A. B. Descombes; F. Kaplan

In 1839, in Paris, the Maison Didot bought the Bottin company. Sébastien Bottin trained as a statistician was the initiator of a high impact yearly publication, called “Almanachs" containing the listing of residents, businesses and institutions, arranged geographically, alphabetically and by activity typologies (Fig. 1). These regular publications encountered a great success. In 1820, the Parisian Bottin Almanach contained more than 50 000 addresses and until the end of the 20th century the word “Bottin” was the colloquial term to designate a city directory in France. The publication of the “Didot-Bottin” continued at an annual rhythm, mapping the evolution of the active population of Paris and other cities in France.The relevance of automatically mining city directories for historical reconstruction has already been argued by several authors (e.g Osborne, N., Hamilton, G. and Macdonald, S. 2014 or Berenbaum, D. et al. (2016). This article reports on the extraction and analysis of the data contained in “Didot-Bottin” covering the period 1839-1922 for Paris, digitized by the Bibliotheque nationale de France. We process more than 27 500 pages to create a database of 4,2 Million entries linking addresses, person mention and activities.

Can Venice be saved?

S. Settis; I. Di Lenardo; F. Kaplan

Will Venice be inhabitable in 2100? What kinds of policies can we develop to navigate the best scenarios for this floating city? In 2012, the École Polytechnique Fédérale de Lausanne (EPFL) and the University Ca’Foscari launched a programme called the Venice Time Machine to create a large-scale digitisation project transforming Venice’s heritage into ‘big data’. Thanks to the support of the Lombard Odier Foundation, millions of pages and photographs have been scanned at the state archive in Venice and at the Fondazione Giorgio Cini. While commercial robotic scanners were used at the archives, a new typology of robotised circular table was developed by Adam Lowe and his team at Factum Arte to process the million photographs of Fondazione Giorgio Cini. The documents were analysed using deep-learning artificial-intelligence methods to extract their textual and iconographic content and to make the data accessible via a search engine. Also during this time, thousands of primary and secondary sources were compiled to create the first 4D model (3D + time) of the city, showing the evolution of its urban fabric. This model and the other data compiled by the Venice Time Machine were part of an exhibition at the Venice Pavilion of the Biennale of Architecture in 2018, shown side-by-side with potential projects for Venice’s future. Having reached an important milestone in convincing not only the Venetian stakeholders but also a growing number of partners around the world that care about Venice’s future, the Venice Time Machine is now raising funds for the most ambitious simulation of the city that has ever been developed. Its planned activities include a high-resolution digitisation campaign of the entire city at centimetre scale, a crucial step on which to base a future simulation of the city’s evolution, while also creating a digital model that can be used for preservation regardless of what occurs in the coming decades. On the island of San Giorgio Maggiore, a digitisation centre called ARCHiVe (Analysis and Recording of Cultural Heritage in Venice) opened in 2018 to process a large variety of Venetian artefacts. This is a joint effort of Factum Foundation, the École Polytechnique Fédérale de Lausanne and the Fondazione Giorgio Cini, along with philanthropic support from the Helen Hamlyn Trust. The centre aims to become a training centre for future cultural heritage professionals who would like to learn how they can use artificial intelligence and robotics to preserve documents, objects and sites. These operations will work together to create a multiscale digital model of Venice, combining the most precise 4D information on the evolution of the city and its population with all the available documentation of its past. The project aims to demonstrate how this ‘digital double’ can be achieved by using robotic technology to scan the city and its archives on a massive scale, using artificial intelligence techniques to process documents and collecting the efforts of thousands of enthusiastic Venetians. In a project called ‘Venice 2100’, the Venice Time Machine team’s ambition is to show how a collectively built information system can be used to build realistic future scenarios, blending ecological and social data into large-scale simulations. The Venice Time Machine’s ‘hypermodel’ will also create economic opportunities. If its hypotheses are valid, Venice could host the first incubators for start-ups using big data of the past to develop services for smart cities, creative industries, education, academic scholarship and policy making. This could be the beginning of a renewal of Venice’s economic life, encouraging younger generations to pursue activities in the historic city, at the heart of what may become one of the first AI-monitored cities of the world. Venice can reinvent itself as the city that put the most advanced information technology and cultural heritage at the core of its survival and its strategy for development. Artificial intelligence can not only save Venice, but Venice can be the place to invent a new form of artificial intelligence.

Frederic Kaplan Isabella di Lenardo

F. Kaplan; I. di Lenardo

The Daily Design of the Quantified Self

D. Rodighiero; A. Rigal

This article argues how the digital traces that are collected day by day can be used to reshape and improve our personal self. If in the past, we were used to collecting data through diaries, today the task of producing inscriptions is delegated to technological devices such as mobile phones. The article, then, discusses how technology can shape athletes using the Sky Team as an example of personal and collective design.

Digital Cultural Heritage meets Digital Humanities

S. Muenster; F. I. Apollonio; P. Bell; P. Kuroczynski; I. Di Lenardo  et al.

Digital Cultural Heritage and Digital Humanities are, historically seen, in focus of different communities as well as approaching different research topics and - from an organizational point of view - departments. However, are they that different? The idea of this joint article involving digital humanists and heritage researchers is to examine communities, concepts and research applications as well as shared challenges. Beyond a collection of problem-centred essays this is intended to initiate a fruitful discussion about commonalities and differences between both scholarly fields as well as to assess to which extent they are two sides of the same medal.

Automatic information extraction from historical collections: the case of the 1808 venetian cadaster

S. Ares Oliveira

The presentation reports on the on-going work to automatically process heterogeneous historical documents. After a quick overview of the general processing pipeline, a few examples are more comprehensively described. The recent progress in making large collections of digitised documents searchable is also be presented through the results of the automatic transcription of named entities in 18th century venetian fiscal documents. Finally, the case of the 1808 venetian cadaster is used to illustrate the general approach and the results of the processing of the whole 1808 cadaster are presented.

New Techniques for the Digitization of Art Historical Photographic Archives - the Case of the Cini Foundation in Venice

B. L. A. Seguin; L. Costiner; I. Di Lenardo; F. Kaplan

Numerous libraries and museums hold large art historical photographic collections, numbering millions of images. Because of their non-standard format, these collections pose special challenges for digitization. This paper address these difficulties by proposing new techniques developed for the digitization of the photographic archive of the Cini Foundation. This included the creation of a custom-built circular, rotating scanner. The resulting digital images were then automatically indexed, while artificial intelligence techniques were employed in information extraction. Combined, these tools vastly sped processes which were traditionally undertaken manually, paving the way for new ways of exploring the collections.

Dürer tra Norimberga e Venezia, 1506-1507

I. Di Lenardo

Making large art historical photo archives searchable

B. L. A. Seguin

In recent years, museums, archives and other cultural institutions have initiated important programs to digitize their collections. Millions of artefacts (paintings, engravings, drawings, ancient photographs) are now represented in digital photographic format. Furthermore, through progress in standardization, a growing portion of these images are now available online, in an easily accessible manner. This thesis studies how such large-scale art history collection can be made searchable using new deep learning approaches for processing and comparing images. It takes as a case study the processing of the photo archive of the Foundation Giorgio Cini, where more than 300'000 images have been digitized. We demonstrate how a generic processing pipeline can reliably extract the visual and textual content of scanned images, opening up ways to efficiently digitize large photo-collections. Then, by leveraging an annotated graph of visual connections, a metric is learnt that allows clustering and searching through artwork reproductions independently of their medium, effectively solving a difficult problem of cross-domain image search. Finally, the thesis studies how a complex Web Interface allows users to perform different searches based on this metric. We also evaluate the process by which users can annotate elements of interest during their navigation to be added to the database, allowing the system to be trained further and give better results. By documenting a complete approach on how to go from a physical photo-archive to a state-of-the-art navigation system, this thesis paves the way for a global search engine across the world's photo archives.

Mapping affinities: visualizing academic practice through collaboration

D. Rodighiero

Academic affinities are one of the most fundamental hidden dynamics that drive scientific development. Some affinities are actual, and consequently can be measured through classical academic metrics such as co-authoring. Other affinities are potential, and therefore do not have visible traces in information systems; for instance, some peers may share scientific interests without actually knowing it. This thesis illustrates the development of a map of affinities for scientific collectives, which is intended to be relevant to three audiences: the management, the scholars themselves, and the external public. Our case study involves the School of Architecture, Civil and Environmental Engineering of EPFL, which consists of three institutes, seventy laboratories, and around one thousand employees. The actual affinities are modeled using the data available from the academic systems reporting publications, teaching, and advising, whereas the potential affinities are addressed through text mining of the documents registered in the information system. The major challenge for designing such a map is to represent the multi-dimensionality and multi-scale nature of the information. The affinities are not limited to the computation of heterogeneous sources of information, they also apply at different scales. Therefore, the map shows local affinities inside a given laboratory, as well as global affinities among laboratories. The thesis presents a graphical grammar to represent affinities. This graphical system is actualized in several embodiments, among which a large-scale carpet of 250 square meters and an interactive online system in which the map can be parameterized. In both cases, we discuss how the actualization influences the representation of data, in particular the way key questions could be appropriately addressed considering the three target audiences: the insights gained by the management and the relative decisions, the understanding of the researchersâ own positioning in the academic collective that might reveal opportunities for new synergies, and eventually the interpretation of the structure from an external standpoint that suggesting the relevance of the tool for communication.

Extracting And Aligning Artist Names in Digitized Art Historical Archives

B. Seguin; L. Costiner; I. di Lenardo; F. Kaplan

The largest collections of art historical images are not found online but are safeguarded by museums and other cultural institutions in photographic libraries. These collections can encompass millions of reproductions of paintings, drawings, engravings and sculptures. The 14 largest institutions hold together an estimated 31 million images (Pharos). Manual digitization and extraction of image metadata undertaken over the years has succeeded in placing less than 100,000 of these items for search online. Given the sheer size of the corpus, it is pressing to devise new ways for the automatic digitization of these art historical archives and the extraction of their descriptive information (metadata which can contain artist names, image titles, and holding collection). This paper focuses on the crucial pre-processing steps that permit the extraction of information directly from scans of a digitized photo collection. Taking the photographic library of the Giorgio Cini Foundation in Venice as a case study, this paper presents a technical pipeline which can be employed in the automatic digitization and information extraction of large collections of art historical images. In particular, it details the automatic extraction and alignment of artist names to known databases, which opens a window into a collection whose contents are unknown. Numbering nearing one million images, the art history library of the Cini Foundation was established in the mid-twentieth century to collect and record the history of Venetian art. The current study examines the corpus of the 330’000+ digitized images.

"Een Italische Keucken van Dirick de Vriese" The Commercialisation of the Artistic Identity between Venice and the 'North'

I. di Lenardo

In the second half of the sixteenth century the artistic exchanges between Venice and the Low Countries intensified. Although no Venetian painters settled in Antwerp or in the cities of the Low Countries, several painters of Flemish origin, in particular Dirck de Vries and Ludovico Pozzoserrato, moved to Venice. These two personalities fostered the circulation in Venice of paintings produced in Flanders and, in the meantime, produced paintings featuring some subjects characterized by a marked Venetian identity.

Machine Vision algorithms on cadaster plans

S. Ares Oliveira

Cadaster plans are cornerstones for reconstructing dense representations of the history of the city. However, as some of these handwritten documents are more than 200 years old, the establishment of processing pipeline for interpreting them remains extremely challenging. The talk will present the implementation of an automated process capable of segmenting and interpreting Napoleonic Cadaster Maps of the Veneto Region dating from the beginning of the 19th century. Using a deep learning approach, the system extracts the geometry of each of the drawn parcels, reads and interprets the handwritten labels. This efficient and automated process opens new perspectives to reconstitute the past.

Tracking Transmission of Details in Paintings

B. L. A. Seguin; I. Di Lenardo; F. Kaplan

In previous articles (di Lenardo et al, 2016; Seguin et al, 2016), we explored how efficient visual search engines operating not on the basis of textual metadata but directly through visual queries, could fundamen- tally change the navigation in large databases of work of arts. In the present work, we extended our search engine in order to be able to search not only for global similarity between paintings, but also for matching de- tails. This feature is of crucial importance for retriev- ing the visual genealogy of a painting, as it is often the case that one composition simply reuses a few ele- ments of other works. For instance, some workshops of the 16th century had repertoires of specific charac- ters (a peasant smoking a pipe, a couple of dancing, etc.) and anatomical parts (head poses, hands, etc.) ,that they reused in many compositions (van den Brink, 2001; Tagliaferro et al, 2009). In some cases it is possible to track the circulation of these visual pat- terns over long spatial and temporal migrations, as they are progressively copied by several generations of painters. Identifying these links permits to recon- struct the production context of a painting, and the connections between workshops and artists. In addi- tion, it permits a fine-grained study of taste evolution in the history of collections, following specific motives successfully reused in a large number of paintings. Tracking these graphical replicators is challenging as they can vary in texture and medium. For instance, a particular character or a head pose of a painting may have been copied from a drawing, an engraving or a tapestry. It is therefore important that the search for matching details still detects visual reuse even across such different media and styles. In the rest of the pa- per, we describe the matching method and discuss some results obtained using this approach.

Big Data of the Past

F. Kaplan; I. di Lenardo

Big Data is not a new phenomenon. History is punctuated by regimes of data acceleration, characterized by feelings of information overload accompanied by periods of social transformation and the invention of new technologies. During these moments, private organizations, administrative powers, and sometimes isolated individuals have produced important datasets, organized following a logic that is often subsequently superseded but was at the time, nevertheless, coherent. To be translated into relevant sources of information about our past, these document series need to be redocumented using contemporary paradigms. The intellectual, methodological, and technological challenges linked to this translation process are the central subject of this article.

Optimized scripting in Massive Open Online Courses

F. Kaplan; I. di Lenardo

The Time Machine MOOC, currently under preparation, is designed to provide the necessary knowledge for students to use the editing tool of the Time Machine platform. The first test case of the platform in centered on our current work on the City of Venice and its archives. Small Teaching modules focus on specific skills of increasing difficulty: segmenting a word on a page, transcribing a word from a document series, georeferencing ancient maps using homologous points, disambiguating named entities, redrawing urban structures, finding matching details between paintings and writing scripts that perform automatically some of these tasks. Other skills include actions in the physical world, like scanning pages, books, maps or performing a photogrammetric reconstruction of a sculpture taking a large number of pictures. Eventually, some other modules are dedicated to general historic, linguistic, technical or archival knowledge that constitute prerequisites for mastering specific tasks. A general dependency graph has been designed, specifying in which order the skills can be acquired. The performance of most tasks can be tested using some pre-defined exercises and evaluation metrics, which allows for a precise evaluation of the level of mastery of each student. When the student successfully passes the test related to a skill, he or she gets the credentials to use that specific tool in the platform and starts contributing. However, the teaching options can vary greatly for each skill. Building upon the script concept developed by Dillenbourg and colleagues, we designed each tutorial as a parameterized sequence. A simple gradient descent method is used to progressively optimize the parameters in order to maximize the success rate of the students at the skill tests and therefore seek a form of optimality among the various design choices for the teaching methods. Thus, the more students use the platform, the more efficient teaching scripts become.

Machine Vision Algorithms on Cadaster Plans

S. Ares Oliveira; I. di Lenardo; F. Kaplan

Cadaster plans are cornerstones for reconstructing dense representations of the history of the city. They provide information about the city urban shape, enabling to reconstruct footprints of most important urban components as well as information about the urban population and city functions. However, as some of these handwritten documents are more than 200 years old, the establishment of processing pipeline for interpreting them remains extremely challenging. We present the first implementation of a fully automated process capable of segmenting and interpreting Napoleonic Cadaster Maps of the Veneto Region dating from the beginning of the 19th century. Our system extracts the geometry of each of the drawn parcels, classifies, reads and interprets the handwritten labels.

Machine Vision Algorithms on Cadaster Plans

S. Ares Oliveira; I. di Lenardo

Cadaster plans are cornerstones for reconstructing dense representations of the history of the city. They provide information about the city urban shape, enabling to reconstruct footprints of most important urban components as well as information about the urban population and city functions. However, as some of these handwritten documents are more than 200 years old, the establishment of processing pipeline for interpreting them remains extremely challenging. We present the first implementation of a fully automated process capable of segmenting and interpreting Napoleonic Cadaster Maps of the Veneto Region dating from the beginning of the 19th century. Our system extracts the geometry of each of the drawn parcels, classifies, reads and interprets the handwritten labels.

Visual Link Retrieval in a Database of Paintings

B. L. A. Seguin; C. Striolo; I. di Lenardo; F. Kaplan

This paper examines how far state-of-the-art machine vision algorithms can be used to retrieve common visual patterns shared by series of paintings. The research of such visual patterns, central to Art History Research, is challenging because of the diversity of similarity criteria that could relevantly demonstrate genealogical links. We design a methodology and a tool to annotate efficiently clusters of similar paintings and test various algorithms in a retrieval task. We show that pretrained convolutional neural network can perform better for this task than other machine vision methods aimed at photograph analysis. We also show that retrieval performance can be significantly improved by fine-tuning a network specifically for this task.

Pose and Pathosformel in Aby Warburg’s Bilderatlas

L. L. Impett; S. Süsstrunk

look at Aby Warburg’s concept of Pathosformel, the repeatable formula for the expression of emotion, through the depiction of human pose in art. Using crowdsourcing, we annotate 2D human pose in one-third of the panels of Warburg’s atlas of art, and perform some exploratory data analysis. Concentrating only on the relative angles of limbs, we find meaningful clusters of related poses, explore the structure using a hierarchical model, and describe a novel method for visualising salient characteristics of the cluster. We find characteristic pose-clusters which correspond to Pathosformeln, and investigate their historical distribution; at the same time, we find morphologically similar poses can represent wildly different emotions. We hypothesise that this ambiguity comes from the static nature of our encoding, and conclude with some remarks about static and dynamic representations of human pose in art.

Visual Patterns Discovery in Large Databases of Paintings

I. di Lenardo; B. L. A. Seguin; F. Kaplan

The digitization of large databases of works of arts photographs opens new avenue for research in art history. For instance, collecting and analyzing painting representations beyond the relatively small number of commonly accessible works was previously extremely challenging. In the coming years,researchers are likely to have an easier access not only to representations of paintings from museums archives but also from private collections, fine arts auction houses, art historian However, the access to large online database is in itself not sufficient. There is a need for efficient search engines, capable of searching painting representations not only on the basis of textual metadata but also directly through visual queries. In this paper we explore how convolutional neural network descriptors can be used in combination with algebraic queries to express powerful search queries in the context of art history research.

Venice Time Machine : Recreating the density of the past

I. di Lenardo; F. Kaplan

This article discusses the methodology used in the Venice Time Machine project (http://vtm.epfl.ch) to reconstruct a historical geographical information system covering the social and urban evolution of Venice over a period of 1,000 years. Given the time span considered, the project used a combination of sources and a specific approach to align heterogeneous historical evidence into a single geographic database. The project is based on a mass digitization project of one of the largest archives in Venice, the Archivio di Stato. One goal of the project is to build a kind of ‘Google map’ of the past, presenting a hypothetical reconstruction of Venice in 2D and 3D for any year starting from the origins of the city to present-day Venice.

Venezia e l’invenzione del paesaggio urbano tra laguna e città

I. di Lenardo

Carlo Helman : merchant, patron and collector in the Antwerp – Venice migrant network

I. di Lenardo

This contribution is part of the monographic number of the Nederlands Yearbook for History of Art dedicated to a large overview on the “Art and Migration. Nethelandish Artists on the Move, 1400-1750”. In the dynamics of migration, circulation, establishing trough Europe in the Modern Era, the network’s analysis play a fundamental role. The essay explores the prominent role played by Antwerp merchants in Venice in forging contacts between artists, patrons and agent of art in promoting the exchange of goods and ideas within their adopted home. In the course of the 16th century, and more particularly towards the end of that period, the complex network of Netherlandish merchant families, operating on a European level, played a crucial role in the circulation of artists, paintings and other artworks in Italy and beyond. The article proposed here deals with Carlo Helman, a Venetian resident of Antwerp origins, a major figure whose importance in this context has been insufficiently studied. Helman’s family firm traded in practically every kind of commodity, ranging from wool and spices to pearls and diamonds, and, indeed, artworks, “in omnibus mundis regnis”, as we read in the commemorative inscription on his monumental tomb in the Venetian church of Santa Maria Formosa. A high-class international trader in Venice, Helman was consul of the “Nattione Fiamenga”. Helman had a conspicuous collection of art, including classics of the “Venetian maniera” like Titian, Veronese and Bassano, but also important pictures by Northern masters. Moreover, his collection contained a remarkable cartographic section. In Venice, Helman had contacts with the Bassano dynasty, Paolo Fiammingo, Dirck de Vries, Lodewijck Toeput (Pozzoserrato) and the Sadeler brothers, artists who, in one way or another, introduced novel themes and typologies on the Italian, and, indeed, European market. The dedication to Helman on a print by Raphael Sadeler, reproducing Bassano’s Parable of the Sower, photographs the merchant’s role in the diffusion of Bassanesque themes in the North. Helman’s connections with the Zanfort brothers, dealers in tapestries and commercial agents of Hieronymus Cock are further indications of the merchant’s exemplary role of collector, merchant and agent of artists in a European network of “art” commerce.

Character Networks and Centrality

Y. Rochat

A character network represents relations between characters from a text; the relations are based on text proximity, shared scenes/events, quoted speech, etc. Our project sketches a theoretical framework for character network analysis, bringing together narratology, both close and distant reading approaches, and social network analysis. It is in line with recent attempts to automatise the extraction of literary social networks (Elson, 2012; Sack, 2013) and other studies stressing the importance of character- systems (Woloch, 2003; Moretti, 2011). The method we use to build the network is direct and simple. First, we extract co-occurrences from a book index, without the need for text analysis. We then describe the narrative roles of the characters, which we deduce from their respective positions in the network, i.e. the discourse. As a case study, we use the autobiographical novel Les Confessions by Jean-Jacques Rousseau. We start by identifying co-occurrences of characters in the book index of our edition (Slatkine, 2012). Subsequently, we compute four types of centrality: degree, closeness, betweenness, eigenvector. We then use these measures to propose a typology of narrative roles for the characters. We show that the two parts of Les Confessions, written years apart, are structured around mirroring central figures that bear similar centrality scores. The first part revolves around the mentor of Rousseau; a figure of openness. The second part centres on a group of schemers, depicting a period of deep paranoia. We also highlight characters with intermediary roles: they provide narrative links between the societies in the life of the author. The method we detail in this complete case study of character network analysis can be applied to any work documented by an index.

“Cities of Fire”. Iconography, Fortune and the Circulation of Fire Paintings in Flanders and Italy in the XVI Century.

I. di Lenardo

From calle to insula: the case of Santa Maria della Carità in Venice

I. di Lenardo

The histories of the monastery of Santa Maria della Carità and of the Scuola Grande della Carità are interwoven with the urban physiognomy of the extreme offshoot of the sestiere of Dorsoduro, the very tip of which overlooks St. Mark’s basin, the island of San Giorgio Maggiore (where the Benedictine monastery lies) and the island of the Giudecca.The history of the original founding of the monastery of Santa Maria della Carità has its beginnings on land owned by the Zulian family, on which Marco Zulian had decided to establish a place of worship surrounded by other properties owned by the family1. The land was located along the San Vidal canal, which would eventually become the Grand Canal. The monastery had been affiliated with Santa Maria in Porto outside Ravenna since1134, and the decision to relocate seems to have been imposed from aboveby Pope Innocent II, who urged the canons either to establish themselves inthe assigned seat or to give it up. A few years later the new coenoby cameinto its own, cutting loose from the founder’s family and following the Ruleof St. Augustine. The monastery’s next two settlements in the lagoon were San Salvador and San Clemente in Isola, the religious founding of which was promoted by Enrico Dandolo. Both were crucial parts of Venice’s early urban fabric: the church of San Salvador was built upon divine revelation in the central commercial area of Rialto while the monastery of San Clemente was a resting-place for pilgrims on the island of the same name, located on the route connecting the area of St. Mark’s, the Lido and the mouths of the lagoon. The complex of the Trinità, located near the abbey of San Gregorio and thus connected to the monastery of Santa Maria della Carità, was another transit point for pilgrims en route to the Holy Land2.

The Ghetto inside out

S. Bassi; I. di Lenardo