Nodegoat – 10 years of research infrastructure for the digital humanities (2011 – 2021)
Let’s take a look at the basics when starting a project.
The ‘holy’ three sections of Nodegoat are: Data – Management – Model. When you log into your new Nodegoat research environment, you must first create a project in ‘Management‘. Then you create the data model, i.e. the database fields, in ‘Model‘. These database fields are then available in the ‘Data‘ section. This means that each database field of ‘Model’ also appears in the ‘Data’ section and we can immediately start filling in our research data there. Note: We must activate the database fields of ‘Model’ in our project in ‘Management’ by selecting them. This way we can control which database fields from ‘Model’ we want to display in our ‘Data’ section.
In the section of ‘Data’ we mainly work with Objects and Categories. We use Objects for the main types of our data model, for example for persons. We use Categories like attributes of these main types, for example the occupation of a person. Attention: the Objects and Categories are named differently in the ‘Data’ section than in the ‘Model’ section: Data: Objects – Categories / Model: Object Types – Classifications. See here the ‘Model’ section:
And what are the ‘Reversals’? With the ‘Reversals’ you can tag your data automatically. But look at it later when you have filled in your data (by hand or by data import (CSV) or by data ingestion).
Data modeling in Nodegoat
This is the ultra-short version on data modeling in Nodegoat. Basically, you build your model in Nodegoat with:
2) Sub-Objects where you can store Dates and Locations (for visualisations in time and space).
Difference between Objects and Categories? Objects are classified with Categories. Example: An Object can be a person, and a Category the person’s profession. So easy? Yes. Objects and Sub-Objects can further have descriptions (Text, Images, Links, Relations). Sounds a bit like Excell to you? Yes. Objects are similar to columns in Excell, descriptions similar to rows. And Sub-Objects? This is special in Nodegoat, like you store in Excell Locations and Dates for a column. So if you have data in Excell, it must be quite easy to import it into Nodegoat? Yes, if your data has a clean structure and is consistent.
Are there any mandatory requirements or restrictions for data modelling in Nodegoat?
No, you can define your model individually, adapted to the research questions.
Should you adapt my model to existing standards, such as the CIDOC reference model?
What you can do. Nodegoat can generate and provide standardized data and thus improve the interoperability of research data. However, it is advisable to first adapt the data model to the sources and the research questions and to consider what new insights can be gained with this data model (and the data collected and visualised in it). You should ask yourself whether your data model is consistent and uniformly structured and, most importantly, whether outsiders understand what you want to represent and say with your data model and the data. A secret tip: Think from the end. What new insights do you want to present with your data and how do you want to visualize them? What should a map, network or time series with your results look like so that outsiders can understand it?
But where should you start? Which objects and categories should be defined and how named?
That is the key question. Objects and Categories are on the same level at the beginning. Now, by creating relationships between the two (or between Objects), we form a structure (hierarchy) that represents our data model. Secret tip: First find about 5 Objects that are important for your data model. These are the archetypes of your model, the central types. If you are interested in people, you will create an Object ‘Person’. If you are researching books and their contents, then the object ‘Book’ is central to you. Pro tip: don’t make the Objects too small. For example, if you are interested in artists, it is better to create a ‘Person’ Object and a ‘Profession’ category where you can enter ‘Artist’. This way you can enter not only artists in the ‘Person’ Object, but also patrons or people related to the artists. This gives you an overview of all the people in your project in the ‘Person’ Object. And it prevents redundancies, for example if an artist is also a patron. Pro-tip: You can of course also use ‘Object’ as a name for the type Object and use it to record a wide variety of things (archaeological finds, manuscripts, books, pictures, maps, etc.).
For those in a hurry, skip tutorials no. 1-9 and start right away with tutorial no. 10: In this tutorial a short video (no sound, no comments) shows from scratch how to create a simple project with a data model and how to import and visualize data. For the tutorial you need a nodegoat account and the test data set I provide on this website. If you don’t have an account yet, ask your friend where to get one. Or your institution (university), if they provide nodegoat as a digital tool. If you are studying or working at the Faculty of Philosophy and History at the University of Bern, you can get a nodegoat account here: https://forms.gle/Gjm4682EJLsq5TCR7. Or get a student account directly at nodegoat: https://nodegoat.net
1. Getting started: create your first Project in ‘Management’ / add Object Types in ‘Model’ / add Object descriptions in Model / activate the Object Types in ‘Management’ / work with the Object Types in ‘Data’
If you have a new Nodegoat account, just follow the instructions in the video after logging in to create your first project with one object and two object descriptions (the videos have no sound, no comments, just watch exactly what is done). Additionally, after logging in, there are infotexts in Nodegoat that show you what you need to do to create a new project.
2. Your first visualization: Locations must be stored in the Sub-Object of an Object. Create a Sub-Object ‘Location’ for your Object in ‘Model’
In this video we will add a field to store locations in our “First Object” that we created in video 1. Locations are not stored in the object description (like the name of the Object), but in the Sub-Object, as you will see in the video. You’ll also see how to change a Location:
3. Entering dates: Dates must be stored in the Sub-Object of an Object
In this video we will store a date for our “first Object”. Then we go to “Model” and select “Period” in the Sub-Object so that we can store two dates (start date and end date) in our “first Object”. This means that for each Object in Nodegoat you can choose a point in time or a period of time. In addition, you can also save vague dates, as we will show later in video 4. Attention: the date format goes like this: 1-8-2020 (not 1.8.2020)
4. Entering vague dates: Vague dates must be stored in the Sub-Object in Chronology
In this video, a simple example shows how to work with vague dates in Nodegoat. There are many more ways to capture vague dates in Nodegoat, see: https://nodegoat.net/guides (working with temporal data). In the example we do not know the exact start date, but we estimate that it was 5 days after the start we entered earlier (1-8-2020). So we make a statement: ‘5 days after begin start date’. Such vague dates are entered in the Sub-Object. Not as ‘Point’, but as ‘Chronology’, as shown in the video.
5. Importing Locations with CSV data: Geo coordinates must be stored in Sub-Objects
In this video you see how to create a Object Type ‘Locations’ with Object Descriptions that match the column names in the CSV-File. After creating the Object, import the sample CSV data into your Object ‘Locations’. The sample data provides Locations (40k) with geonames.org-IDs and Geo Coordinates: Location1.csv
Hint: In Nodegoat are about 130k locations from geonames.org preinstalled (Type: City). These locations can be used and extended by all users as a collaborative work.
6. Create your first relation: Relations are created in the Model to be used in Data
In this video we establish a relationship between the ‘first object’ and locations, because we want to use the locations as a georeference for the ‘first object’. In Model we select ‘Locations’ as the georeference in the sub-object of the ‘new object’. In Data we enter a location and see that it is not visualised immediately because we have to set the Visual Settings correctly first (selecting Loaction as the reference for visualisation). We’ll change the Location in the ‘first Object’ and see that we have to activate the ‘Quicksearch’ field in Model (in the Object descriptions of Location), so that we can search for a new Location in the Quicksearch field.
7. Create your first Classification (Category): Classifications are created in the Model to be used in Data.
In this video we create a classification ‘Attribute’ in the Model. Then we go to the ‘first object’ in the Model to add an Object description ‘Attribute’ that is linked to the created classification.
8. Expand your Data Model: Person with ‘Event Birth’.
In this video we will change ‘first Object’ to ‘Person’, adding a Classification in Model ‘Event (kind of’, linking it to the Sub-Object description of Person. Then we add the Event ‘Birth’ to the Sub-Object. This gives us a simple model for biographies that we can extend with other events (such as death, activities, etc.). In the Sub-Object we can add to each of these events dates and a location.
9. Change the background map
In nodegoat you can integrate background maps as you like, if the maps are available on a tile server via link. So you need only this link to the map. But where can you find such links? Google is your friend. Or this short tutorial. Go in nodegoat to the Visualisation Settings.
Then go to the Visual Settings tab. You will automatically get to the Geographical Settings.
Standard map is the Google Map with its copyright. Remove the Google Map link in the Map field and insert your new link for your map. Change the copyright as desired.
Save your Map settings:
I have compiled some links here. Don’t be afraid of the length of the links, they are just like this, sometimes longer, sometimes shorter:
Google Satellite Map (without places, topography)
Background info on Google Maps: the standard link for Google Map tiles looks like:
If you want a different Google Map, just replace the letter in the URL at lyrs=
So for example lyrs=m
This will then display the standard street map.
The whole link will then look like this:
With these letters you can change the map:
h: shows only roads
m: is the standard roadmap
p: shows the terrain
r: is another roadmap
s: shows the satellit view
t: shows only the terrrain
y: is a hybrid map with terrain and roads
Grey map without places
Digital Atlas of the Roman Empire
See for this cool project: https://dh.gu.se/dare/
Mercator map from 1607
The example of the Mercator map refers to the Georeferencer, a service for online maps, where you can find a lot of links to historical maps. Many institutions have their own account at Georeferencer like the David Rumsey Collections with a useful overview of the referenced maps (world map):
Create an account at Georeferencer to get the link for a map provided (for example by the David Rumsey collection). Log in at Georeferencer, choose a map here:
Then go to: ‘This map’ and to ‘get Links’. Copy the link into the Map field of nodegoat.
As another example, the British Library also has an account at Georeferencer. You can find their maps here, on the interactive map:
10. Import your Excel data (CSV data), which have geo coordinates (longitude and latitude) into nodegoat and visualize the data on a map
In this tutorial I provide a set of test data with geocoordinates that you can easily import and visualize, like the map below. It shows positions of ships calculated from logbooks (18th – 19th century). The data are especially interesting for historical climate research.
Prerequisite is that you already have a nodegoat account. If you don’t have an account yet, ask your friend where to get one. Or your institution (university), if they provide nodegoat as a digital tool. If you are studying or working at the Faculty of Philosophy and History at the University of Bern, you can get a nodegoat account here: https://forms.gle/Gjm4682EJLsq5TCR7. Alternatively you can get a student account directly at nodegoat: https://nodegoat.net/
The following video (no sound, no comments) shows a step by step guide from scratch. The video starts with the login into your nodegoat account. The next steps are: Create a project, create an object type, download data sample from this website, import and visualize the data:
If you prefer written instructions, you can continue here. These instructions are identical to the video, but contain some background information.
Login into your nodegoat account. Import the CSV data into your already existing project or create a new one: ‘climate project’. We will import data (sample) from a cool project about logbooks of ships which are important for weather observations. Here is the website of the project where the data is available:
“The database consists of 287,114 logbooks written aboard Dutch, English, French, and Spanish sailing ships. The vast majority of these logbooks date from between 1750 and 1850, yet four ship logbooks were incorporated that predate 1750. These were centuries of European imperial expansion, and so the logbooks record the activities of sailors – both civilian and military – in oceans that span the entire globe.”
I have downloaded the following data: ‘Download as an Open Office Spreadsheet’
I opened the spreadsheet in Excel and first added a column on the far left to add to the records (rows) a unique identifier, because they don’t have any. I’ve added for this the following into field A2 in Excel which contains the first record:
Then double-click on the icon to the right of the cell and it fills the whole column with identifiers. The identifiers are very important. With these identifiers you can later update your data records in nodegoat (‘Update Existing Objects’). I always import the identifiers into nodegoat first and then update the records with additional information based on the identifiers. In the nodegoat Import web interface you can choose whether you want to create new records or update existing ones.
In nodegoat you can import 50k of data records (rows) at once. So if you want to import all the more than 200k data rows, you have to split them up. I’ve already done this by providing a test data sample here with 20k records that you can use for your import. I have prepared this data and just selected a few columns to get started: Identifier, ship name, year, longitude and latitude. You can download the test data sample here:
Your Data Model in your project for this data sample should look like this:
Just add one Sub-Object with ‘year + coordinates’
We will import the data into nodegoat via web interface (you can also import data via JSON interface, we will cover this in another tutorial).
Go to Model > Import > CSV Files, there you upload the downloaded CSV file (climate project test data).
Background: To import your data into nodegoat, the data must be available in a text file in UTF8 format. In Excel for example you can save your data as CSV data. Go to ‘Save as’ in your Excel sheet and choose CSV-UTF8 as data format. CSV means comma separated values (data). Open your CSV file with a text editor and you will see the many separators in the data.
Go to the Import Template. Map the fields of your CSV data to the fields in your data model. For the year (YR), select the Date Start field in your data model.
Now you can run your import template. You can first check a selection of the records to see if you have mapped the fields correctly. Click on Next to import the 20k data records.
Have a coffee now, you have already achieved a lot today.
After the import go to ‘Data’ and click on the Geografical Visualisation.
This is how your result should look like. Zoom in and don’t forget to play with the time slider. You will also discover ships in the desert of Africa, these are errors in the data that do not include either the longitude or latitude. So the visualization also helps you to detect such errors.
Reduce the points in the Visual Settings, if you like.
You can update now your data records based on the identifiers. Create a CSV file with the data you want to continue importing. When importing, select ‘Update Existing Objects’ and choose your identifier to map the CSV data to the corresponding record in your nodegoat database.
11. Advanced Tutorial – Import of Open Data into nodegoat: Benjamin Franklin’s Post Office Records
Benjamin West (1738-1820): Benjamin Franklin Drawing Electricity from the Sky, Philadelphia Museum of Art, image Source: Public domain, via Wikimedia Commons
In 1743 the American Philosophical Society (APS) – the oldest learned society in the US – was founded in Philadelphia by Benjamin Franklin, John Bartram, Francis Hopkinson and others for the purpose of “promoting useful knowledge”, as you can read on tha APS website. Today, APS follows an open data strategy that encourages researchers to use and reuse the open datasets provided under a Creative Commons Attribution 4.0 License. In this tutorial, we will work with such a dataset as it is a very useful resource to show what you can do with it in nodegoat, focusing on importing and mapping the data. Citation of the dataset: Heider, Cynthia, Bayard Miller and Scott Ziegler. Post Office Book, 1748-1752. BF85f6-8. Distributed by Philadelphia: American Philosophical Society Library & Museum, 2017. https://diglib.amphilsoc.org/islandora/object/compound:11.
Franklin was not only the founder of the society, he also became Postmaster of Philadelphia in 1737, appointed by the British Crown Post. For a later time when he was serving as a Postmaster, records of letters are still available, which are archived at the APS library and made available as Open Data, as we read on the APS website:
“Benjamin Franklin’s Post Office Records: Post Office Book, Philadelphia incoming and outgoing mail, 1748-1752. Created while Benjamin Franklin served as Postmaster of Philadelphia, these datasets reveal a wealth of previously untapped information about colonial correspondence.” (Source: APS website).
In the following tutorial, we will import into nodegoat only the outgoing letters from the post office in Philadelphia. For the tutorial there are three videos which show from scratch how the data for the outgoing letters are imported and visualised in nodegoat. The videos have no sound and comments, just observe what to do. Requirements: you need a nodegoat account and Google Tables (or Excell, but the step by step tutorial uses Google Tables).
The challenge in Video 1 is to reformat the dates for import into nodegoat and to convert the American date format into the European one. An additional location (Philadelphia Post Office) is added for visualisation. The tutorial starts with the data download from the APS website:
In video 2, a data model is first created in nodegoat, based on the column names of the data in Google Tables. Then the data is downloaded as CSV from Google Tables and uploaded to nodegoat, the data fields of the CSV file mapped with the data fields in nodegoat and the import process is started. During the import process nodegoat automatically makes assignments to the locations in the CSV data which can be accepted or rejected.
Video 3 shows how to work with the imported data. First a basic function is shown: if you click just on one object in nodegoat (a window with the object opens) only this object is visualised on the map (by clicking on the globe symbol). If you call up all objects, for example by selecting ‘all’ to the right of the filter symbol (screenshot below) or by scrolling between the pages (1,2,3 etc), all objects are visualised:
If you do this with our data, as in the video, by clicking on the globe symbol, the locations that we have previously assigned during the import process will be displayed on the map. Not all of these locations are correctly located, because during the import process we did not check if the location is really the right one, but simply chose a location to do the data cleansing afterwards in nodegoat. It would have been better if we had already cleaned the data in Google Tables or Excell BEFORE the data import, which is highly recommended! But no worries: you can also clean up the data within nodegoat, as an example shows in the video. The video also shows how to add and use a database field ‘Comment’ to the data model, for example if you want to indicate that you are unsure about the location of a place. And the video shows how to create a filter query to find locations that do not have geo-coordinates.