Kick off SPARK Project: Dynamic Data Ingestion

On February 1 2020, this SPARK project officially started. Together with Geert Kessels and Pim van Bree (LAB1100) the kick off meeting took place in Montreux on Lake Geneva and the milestones for the project were set. Core tasks are the development of a new module for the virtual open source research environment nodegoat (github) for central data harmonisation and the creation of a common ontology with selected database projects. The kick off it made clear that we have to do the module development and the harmonization of the database structures in small steps and in very close coordination, which is why we decided for an agile software development and project management. The progress of the project can be followed in this blog. The project will also be presented at several events and conferences, such as the Research Day of the University of Bern (March 2020), in a panel at the Annual Congress of Italian Medievalists (Bertinoro, June 2020), in a presentation at the Annual Congress of Atelier Heloise, a European network on digital academic history (Bologna, September 2020) and at a conference on the use of virtual research environments (VRE) with ancient manuscripts (Dorigny, September 2020) as well as at a conference of the the Swiss Academy of Humanities and Social Scienceson on ‘knowledge locations’ (Bern, October 2020), which i will co-organise. The public SPARK workshop with data testing and results is planned for november 2020 in Bern, later a public lecture on the results will be held at the University of Bern.

Cite this article as: Kaspar Gubler: Kick off SPARK Project: Dynamic Data Ingestion, in HistData, 03/02/2020, https://histdata.hypotheses.org/605.

Database Migration Case Study: Repertorium Academicum Germanicum

The goal of the Repertorium Academicum Germanicum (RAG) is to develop the history of the cultural reach of a pre-modern intellectual leadership and impulse group to gain a comprehensive insight into the medieval origins of the modern knowledge society with around 60.000 scholars with 360.000 observations on their life and career paths, within the framework of an analysis of contextualized prosopography. The RAG uses nodegoat as their primary data storage application and research environment. nodegoat is also used to create and publish diachronic geographical and social visualisations (networks).

Work on the RAG began in 2001 under the direction of Rainer Schwinges and Peter Moraw, financed by the Swiss National Science Foundation (SNSF), the German research foundation and the Fritz Thyssen Foundation. From 2007 to 2019 the project was funded by the Union of the German Academies of Sciences and Humanities and from 2008 on as well by the Swiss Academy of Humanities and Social Sciences. The project will be run from 2020 at the University of Bern as part of the larger project Repertorium Academicum (REPAC), which is led by Christian Hesse and Kaspar Gubler and advised by Rainer Schwinges.

In 2017, the RAG research project found itself in a difficult situation. The database that the RAG had been working with for years was technically so outdated that an update of this software was no longer possible. The frontend was running on MS Accesss 2003 (sic!) and the backend on a MS SQL Server 2005. A serious omission was that data entry could be completely free without technical restrictions, for example dependent combination lists. Although there was an internal Wiki that documented the rules for data entry, but not a few of these rules were no longer up-to-date. Accordingly, the large data set was heterogeneous and inconsistent when I started my work at the RAG. Furthermore, with the outdated software constellation it had become more and more difficult for the RAG to publish the research data on the internet, because the data first had to be exported from the SQL-Server and imported into a web application, which was very time-consuming and error-prone. Therefore a new software for the RAG had to be evaluated in a timely manner. A colleague at the University of Bern, who had attended a nodegoat workshop, drew my attention to nodegoat. It was immediately clear to me that nodegoat fulfilled exactly the functions we urgently needed for the RAG’s research data, namely data management and data visualization in one and the same software.

We invited LAB1100 to our office in Bern to discuss the data import in nodegoat. Before the meeting, LAB1100 had analysed the RAG data model and made some decisive suggestions at the meeting on how we could simplify the data model and make it easier to understand. An important change was that institutions were introduced as location types (e.g. university, school, town, court, princely court, church, monastery) and, in addition, these were linked hierarchically. In the previous database, such types existed only sporadically. The main part of the data was only assigned to the main type ‘Location’, which was not conducive to systematic evaluation.

The database migration was started in September 2017. First, the data was exported from the Microsoft SQL database, then cleaned, sorted and partially reordered in various programming steps. The final import into nodegoat was done with Python via the nodegoat JSON-API. By the end of 2017 the database migration was completed and the RAG teams at the universities of Bern and Giessen were able to enter and visualize research data in nodegoat in January 2018.

The biggest advantage of nodegoat for us is the acceleration of the work process. We now need a fraction of the time compared to the previous database to enter, analyze and publish research data. In addition, with the automated data checks of nodegoat we have data consistency under control, which is very important given the large dataset of the RAG.

Cite this article as: Kaspar Gubler: Database Migration Case Study: Repertorium Academicum Germanicum, in HistData, 03/02/2020, https://histdata.hypotheses.org/545.

Project Website: rag-online.org
Public User Interface: database.rag-online.org/viewer