A data structure for scientific models of historical cities: extending the CityJSON format

B. Vaienti; P. Guhennec; I. di Lenardo

In the field of the 3D reconstruction of cities in the past there is a raising interest in the creation of models that are not just geometrical, but also informative, semantic and georeferenced. Despite the advancements that were done in the historical reconstruction of architecture and archaeology, the solutions designed for larger scale models are still very limited. On the other hand, research on the digitisation of current-day cities provides useful instruments. In particular, CityJSON - a JSON encoding of CityGML - represents an easy-to-use and lightweight solution for storing 3D models of cities that are geolocated, semantic and that contain additional information in the form of attributes. This contribution proposes (1) to extend the schema to the needs of a historical representation; and (2) to incorporate the newly created model in a continuous flow pipeline, in which the geometry is dynamically updated each time an attribute is changed, as a means to foster collaboration.

Generic Semantic Segmentation of Historical Maps

R. G. Petitpierre; F. Kaplan; I. Di Lenardo

Research in automatic map processing is largely focused on homogeneous corpora or even individual maps, leading to inflexible models. Based on two new corpora, the first one centered on maps of Paris and the second one gathering maps of cities from all over the world, we present a method for computing the figurative diversity of cartographic collections. In a second step, we discuss the actual opportunities for CNN-based semantic segmentation of historical city maps. Through several experiments, we analyze the impact of figurative and cultural diversity on the segmentation performance. Finally, we highlight the potential for large-scale and generic algorithms. Training data and code of the described algorithms are made open-source and published with this article.

Aux portes du monde miroir

I. Di Lenardo; F. Kaplan

The Mirror World is no longer an imaginary device, a mirage in a distant future, it is a reality under construction. In Europe, Asia and on the American continent, large companies and the best universities are working to build the infrastructures, to define their functionalities, to specify their logistics. The Mirror World, in its asymptotic form, presents a quasi-continuous representation of the world in motion, integrating, virtually, all photographic perspectives. It is a new giant computational object, opening the way to new research methods or even probably to a new type of science. The economic and cultural stakes of this third platform are immense. If the Mirror World transforms access to knowledge for new generations, as the Web and Social Networks did in their time, it is our responsibility to understand and, if need be, bend its technological trajectory to make this new platform an environment for the critical knowledge of the past and the creative imagination of the future.

I sistemi di immagini nell’archivio digitale di Vico Magistretti

F. Kaplan; I. di Lenardo

La messa a disposizione in linea dell’archivio digitalizzato di Vico Magistretti che raggruppa decine di migliaia di disegni preparatori, disegni tecnici e fotografie prodotte tra 1946 e il 2006, apre la strada a un grande rinnovamento delle ricerche sul designer e architetto italiano. L’apertura di questo archivio così speciale ci invita a immaginare diverse prospettive che possono essere considerate per esplorare, visualizzare e studiare un tale insieme di documenti.

The Advent of the 4D Mirror World

F. Kaplan; I. di Lenardo

The 4D Mirror World is considered to be the next planetary-scale information platform. This commentary gives an overview of the history of the converging trends that have progressively shaped this concept. It retraces how large-scale photographic surveys served to build the first 3D models of buildings, cities, and territories, how these models got shaped into physical and virtual globes, and how eventually the temporal dimension was introduced as an additional way for navigating not only through space but also through time. The underlying assumption of the early large-scale photographic campaign was that image archives had deeper depths of latent knowledge still to be mined. The technology that currently permits the advent of the 4D World through new articulations of dense photographic material combining aerial imagery, historic photo archives, huge video libraries, and crowd-sourced photo documentation precisely exploits this latent potential. Through the automatic recognition of “homologous points,” the photographic material gets connected in time and space, enabling the geometrical computation of hypothetical reconstructions accounting for a perpetually evolving reality. The 4D world emerges as a series of sparse spatiotemporal zones that are progressively connected, forming a denser fabric of representations. On this 4D skeleton, information of cadastral maps, BIM data, or any other specific layers of a geographical information system can be easily articulated. Most of our future planning activities will use it as a way not only to have smooth access to the past but also to plan collectively shared scenarios for the future.

Building a Mirror World for Venice

F. Kaplan; I. di Lenardo

Between 2012 and 2019, ‘TheVeniceTime Machine Project’ developed a new methodology for modelling the past, present, and future of a city. This methodology is based on two pillars: (a) the vast digitisation and processing of the selected city’s historical records, (b) the digitisation of the city itself, another vast undertaking. The combination of these two processes has the potential to create a new kind of historical information system organised around a diachronic digital twin of a city.

A deep learning approach to Cadastral Computing

S. Ares Oliveira; I. di Lenardo; B. Tourenc; F. Kaplan

This article presents a fully automatic pipeline to transform the Napoleonic Cadastres into an information system. The cadastres established during the first years of the 19th century cover a large part of Europe. For many cities they give one of the first geometrical surveys, linking precise parcels with identification numbers. These identification numbers points to registers where the names of the proprietary. As the Napoleonic cadastres include millions of parcels , it therefore offers a detailed snapshot of large part of Europe’s population at the beginning of the 19th century. As many kinds of computation can be done on such a large object, we use the neologism “cadastral computing” to refer to the operations performed on such datasets. This approach is the first fully automatic pipeline to transform the Napoleonic Cadastres into an information system.

Repopulating Paris: massive extraction of 4 Million addresses from city directories between 1839 and 1922

I. di Lenardo; R. Barman; A. B. Descombes; F. Kaplan

In 1839, in Paris, the Maison Didot bought the Bottin company. Sébastien Bottin trained as a statistician was the initiator of a high impact yearly publication, called “Almanachs" containing the listing of residents, businesses and institutions, arranged geographically, alphabetically and by activity typologies (Fig. 1). These regular publications encountered a great success. In 1820, the Parisian Bottin Almanach contained more than 50 000 addresses and until the end of the 20th century the word “Bottin” was the colloquial term to designate a city directory in France. The publication of the “Didot-Bottin” continued at an annual rhythm, mapping the evolution of the active population of Paris and other cities in France.The relevance of automatically mining city directories for historical reconstruction has already been argued by several authors (e.g Osborne, N., Hamilton, G. and Macdonald, S. 2014 or Berenbaum, D. et al. (2016). This article reports on the extraction and analysis of the data contained in “Didot-Bottin” covering the period 1839-1922 for Paris, digitized by the Bibliotheque nationale de France. We process more than 27 500 pages to create a database of 4,2 Million entries linking addresses, person mention and activities.

Digital Cultural Heritage meets Digital Humanities

S. Muenster; F. I. Apollonio; P. Bell; P. Kuroczynski; I. Di Lenardo  et al.

Digital Cultural Heritage and Digital Humanities are, historically seen, in focus of different communities as well as approaching different research topics and - from an organizational point of view - departments. However, are they that different? The idea of this joint article involving digital humanists and heritage researchers is to examine communities, concepts and research applications as well as shared challenges. Beyond a collection of problem-centred essays this is intended to initiate a fruitful discussion about commonalities and differences between both scholarly fields as well as to assess to which extent they are two sides of the same medal.

Frederic Kaplan Isabella di Lenardo

F. Kaplan; I. di Lenardo

New Techniques for the Digitization of Art Historical Photographic Archives - the Case of the Cini Foundation in Venice

B. L. A. Seguin; L. Costiner; I. Di Lenardo; F. Kaplan

Numerous libraries and museums hold large art historical photographic collections, numbering millions of images. Because of their non-standard format, these collections pose special challenges for digitization. This paper address these difficulties by proposing new techniques developed for the digitization of the photographic archive of the Cini Foundation. This included the creation of a custom-built circular, rotating scanner. The resulting digital images were then automatically indexed, while artificial intelligence techniques were employed in information extraction. Combined, these tools vastly sped processes which were traditionally undertaken manually, paving the way for new ways of exploring the collections.

Dürer tra Norimberga e Venezia, 1506-1507

I. Di Lenardo

Extracting And Aligning Artist Names in Digitized Art Historical Archives

B. Seguin; L. Costiner; I. di Lenardo; F. Kaplan

The largest collections of art historical images are not found online but are safeguarded by museums and other cultural institutions in photographic libraries. These collections can encompass millions of reproductions of paintings, drawings, engravings and sculptures. The 14 largest institutions hold together an estimated 31 million images (Pharos). Manual digitization and extraction of image metadata undertaken over the years has succeeded in placing less than 100,000 of these items for search online. Given the sheer size of the corpus, it is pressing to devise new ways for the automatic digitization of these art historical archives and the extraction of their descriptive information (metadata which can contain artist names, image titles, and holding collection). This paper focuses on the crucial pre-processing steps that permit the extraction of information directly from scans of a digitized photo collection. Taking the photographic library of the Giorgio Cini Foundation in Venice as a case study, this paper presents a technical pipeline which can be employed in the automatic digitization and information extraction of large collections of art historical images. In particular, it details the automatic extraction and alignment of artist names to known databases, which opens a window into a collection whose contents are unknown. Numbering nearing one million images, the art history library of the Cini Foundation was established in the mid-twentieth century to collect and record the history of Venetian art. The current study examines the corpus of the 330’000+ digitized images.

"Een Italische Keucken van Dirick de Vriese" The Commercialisation of the Artistic Identity between Venice and the 'North'

I. di Lenardo

In the second half of the sixteenth century the artistic exchanges between Venice and the Low Countries intensified. Although no Venetian painters settled in Antwerp or in the cities of the Low Countries, several painters of Flemish origin, in particular Dirck de Vries and Ludovico Pozzoserrato, moved to Venice. These two personalities fostered the circulation in Venice of paintings produced in Flanders and, in the meantime, produced paintings featuring some subjects characterized by a marked Venetian identity.

Making large art historical photo archives searchable

B. L. A. Seguin

In recent years, museums, archives and other cultural institutions have initiated important programs to digitize their collections. Millions of artefacts (paintings, engravings, drawings, ancient photographs) are now represented in digital photographic format. Furthermore, through progress in standardization, a growing portion of these images are now available online, in an easily accessible manner. This thesis studies how such large-scale art history collection can be made searchable using new deep learning approaches for processing and comparing images. It takes as a case study the processing of the photo archive of the Foundation Giorgio Cini, where more than 300'000 images have been digitized. We demonstrate how a generic processing pipeline can reliably extract the visual and textual content of scanned images, opening up ways to efficiently digitize large photo-collections. Then, by leveraging an annotated graph of visual connections, a metric is learnt that allows clustering and searching through artwork reproductions independently of their medium, effectively solving a difficult problem of cross-domain image search. Finally, the thesis studies how a complex Web Interface allows users to perform different searches based on this metric. We also evaluate the process by which users can annotate elements of interest during their navigation to be added to the database, allowing the system to be trained further and give better results. By documenting a complete approach on how to go from a physical photo-archive to a state-of-the-art navigation system, this thesis paves the way for a global search engine across the world's photo archives.

Machine Vision Algorithms on Cadaster Plans

S. Ares Oliveira; I. di Lenardo; F. Kaplan

Cadaster plans are cornerstones for reconstructing dense representations of the history of the city. They provide information about the city urban shape, enabling to reconstruct footprints of most important urban components as well as information about the urban population and city functions. However, as some of these handwritten documents are more than 200 years old, the establishment of processing pipeline for interpreting them remains extremely challenging. We present the first implementation of a fully automated process capable of segmenting and interpreting Napoleonic Cadaster Maps of the Veneto Region dating from the beginning of the 19th century. Our system extracts the geometry of each of the drawn parcels, classifies, reads and interprets the handwritten labels.

Machine Vision Algorithms on Cadaster Plans

S. Ares Oliveira; I. di Lenardo

Cadaster plans are cornerstones for reconstructing dense representations of the history of the city. They provide information about the city urban shape, enabling to reconstruct footprints of most important urban components as well as information about the urban population and city functions. However, as some of these handwritten documents are more than 200 years old, the establishment of processing pipeline for interpreting them remains extremely challenging. We present the first implementation of a fully automated process capable of segmenting and interpreting Napoleonic Cadaster Maps of the Veneto Region dating from the beginning of the 19th century. Our system extracts the geometry of each of the drawn parcels, classifies, reads and interprets the handwritten labels.

Big Data of the Past

F. Kaplan; I. di Lenardo

Big Data is not a new phenomenon. History is punctuated by regimes of data acceleration, characterized by feelings of information overload accompanied by periods of social transformation and the invention of new technologies. During these moments, private organizations, administrative powers, and sometimes isolated individuals have produced important datasets, organized following a logic that is often subsequently superseded but was at the time, nevertheless, coherent. To be translated into relevant sources of information about our past, these document series need to be redocumented using contemporary paradigms. The intellectual, methodological, and technological challenges linked to this translation process are the central subject of this article.

Optimized scripting in Massive Open Online Courses

F. Kaplan; I. di Lenardo

The Time Machine MOOC, currently under preparation, is designed to provide the necessary knowledge for students to use the editing tool of the Time Machine platform. The first test case of the platform in centered on our current work on the City of Venice and its archives. Small Teaching modules focus on specific skills of increasing difficulty: segmenting a word on a page, transcribing a word from a document series, georeferencing ancient maps using homologous points, disambiguating named entities, redrawing urban structures, finding matching details between paintings and writing scripts that perform automatically some of these tasks. Other skills include actions in the physical world, like scanning pages, books, maps or performing a photogrammetric reconstruction of a sculpture taking a large number of pictures. Eventually, some other modules are dedicated to general historic, linguistic, technical or archival knowledge that constitute prerequisites for mastering specific tasks. A general dependency graph has been designed, specifying in which order the skills can be acquired. The performance of most tasks can be tested using some pre-defined exercises and evaluation metrics, which allows for a precise evaluation of the level of mastery of each student. When the student successfully passes the test related to a skill, he or she gets the credentials to use that specific tool in the platform and starts contributing. However, the teaching options can vary greatly for each skill. Building upon the script concept developed by Dillenbourg and colleagues, we designed each tutorial as a parameterized sequence. A simple gradient descent method is used to progressively optimize the parameters in order to maximize the success rate of the students at the skill tests and therefore seek a form of optimality among the various design choices for the teaching methods. Thus, the more students use the platform, the more efficient teaching scripts become.

Visual Link Retrieval in a Database of Paintings

B. L. A. Seguin; C. Striolo; I. di Lenardo; F. Kaplan

This paper examines how far state-of-the-art machine vision algorithms can be used to retrieve common visual patterns shared by series of paintings. The research of such visual patterns, central to Art History Research, is challenging because of the diversity of similarity criteria that could relevantly demonstrate genealogical links. We design a methodology and a tool to annotate efficiently clusters of similar paintings and test various algorithms in a retrieval task. We show that pretrained convolutional neural network can perform better for this task than other machine vision methods aimed at photograph analysis. We also show that retrieval performance can be significantly improved by fine-tuning a network specifically for this task.

Visual Patterns Discovery in Large Databases of Paintings

I. di Lenardo; B. L. A. Seguin; F. Kaplan

The digitization of large databases of works of arts photographs opens new avenue for research in art history. For instance, collecting and analyzing painting representations beyond the relatively small number of commonly accessible works was previously extremely challenging. In the coming years,researchers are likely to have an easier access not only to representations of paintings from museums archives but also from private collections, fine arts auction houses, art historian However, the access to large online database is in itself not sufficient. There is a need for efficient search engines, capable of searching painting representations not only on the basis of textual metadata but also directly through visual queries. In this paper we explore how convolutional neural network descriptors can be used in combination with algebraic queries to express powerful search queries in the context of art history research.

Venezia e l’invenzione del paesaggio urbano tra laguna e città

I. di Lenardo

Venice Time Machine : Recreating the density of the past

I. di Lenardo; F. Kaplan

This article discusses the methodology used in the Venice Time Machine project (http://vtm.epfl.ch) to reconstruct a historical geographical information system covering the social and urban evolution of Venice over a period of 1,000 years. Given the time span considered, the project used a combination of sources and a specific approach to align heterogeneous historical evidence into a single geographic database. The project is based on a mass digitization project of one of the largest archives in Venice, the Archivio di Stato. One goal of the project is to build a kind of ‘Google map’ of the past, presenting a hypothetical reconstruction of Venice in 2D and 3D for any year starting from the origins of the city to present-day Venice.

Carlo Helman : merchant, patron and collector in the Antwerp – Venice migrant network

I. di Lenardo

This contribution is part of the monographic number of the Nederlands Yearbook for History of Art dedicated to a large overview on the “Art and Migration. Nethelandish Artists on the Move, 1400-1750”. In the dynamics of migration, circulation, establishing trough Europe in the Modern Era, the network’s analysis play a fundamental role. The essay explores the prominent role played by Antwerp merchants in Venice in forging contacts between artists, patrons and agent of art in promoting the exchange of goods and ideas within their adopted home. In the course of the 16th century, and more particularly towards the end of that period, the complex network of Netherlandish merchant families, operating on a European level, played a crucial role in the circulation of artists, paintings and other artworks in Italy and beyond. The article proposed here deals with Carlo Helman, a Venetian resident of Antwerp origins, a major figure whose importance in this context has been insufficiently studied. Helman’s family firm traded in practically every kind of commodity, ranging from wool and spices to pearls and diamonds, and, indeed, artworks, “in omnibus mundis regnis”, as we read in the commemorative inscription on his monumental tomb in the Venetian church of Santa Maria Formosa. A high-class international trader in Venice, Helman was consul of the “Nattione Fiamenga”. Helman had a conspicuous collection of art, including classics of the “Venetian maniera” like Titian, Veronese and Bassano, but also important pictures by Northern masters. Moreover, his collection contained a remarkable cartographic section. In Venice, Helman had contacts with the Bassano dynasty, Paolo Fiammingo, Dirck de Vries, Lodewijck Toeput (Pozzoserrato) and the Sadeler brothers, artists who, in one way or another, introduced novel themes and typologies on the Italian, and, indeed, European market. The dedication to Helman on a print by Raphael Sadeler, reproducing Bassano’s Parable of the Sower, photographs the merchant’s role in the diffusion of Bassanesque themes in the North. Helman’s connections with the Zanfort brothers, dealers in tapestries and commercial agents of Hieronymus Cock are further indications of the merchant’s exemplary role of collector, merchant and agent of artists in a European network of “art” commerce.

“Cities of Fire”. Iconography, Fortune and the Circulation of Fire Paintings in Flanders and Italy in the XVI Century.

I. di Lenardo

The oltramontani Network in Venice: Hans von Aachen in Context

I. di Lenardo

Thanks to recent archival and historical researches it is now possible to specify the identity of some personalities told in the Lives of Van Mander relating and in close contact with Hans von Aachen. The reconstruction of Venice and Treviso context, in which the artist moves, shows a thick network of relationships woven by Flemish and German communities. The presence of a portrait by Hans von Aachen in the collection of paintings of Francesco Vrients is information very valuable: firstly outlines the painter as an intimate friend of the family Vrients, and in the same time the discovery of the inscription on the drawings of Cephalus and Procri (presented for this exhbition) it is an important pointer for profiling the Vrients-circle and its relationships with the flemish jewellers lobby. Indeed is him the collector of Maastricht mentioned by Van Mander and one of the most eminent flemish personality in the lagoon, around whom, probably, gravitated intellectuals and artists: is a fact that in his house, in Campo Santa Maria Formosa, found hospitality the literate Pieter Cornelisz de Hooft on the occasion of his trip in Italy in 1599. Additional documents shall also specify the role of Gaspar Rem in a venetian and international context: his strong tie to the circle of the “Sadelers” who, especially with a shrewd art dealer like Giusto, play a crucial role promoting artists “Oltramontani” weaving friendship with Dirck de Vries, Rottenhammer, Joannes Koenig to name a few.