Session 3

12 May 2021

“Introduction by Kaspar Gubler

I would like to welcome in this session all participants who are already familiar with Nodegoat and have therefore skipped sessions 1 and 2 and will now attend sessions 3 and 4. In sessions 1 and 2 we created a data model for people and books and imported data, including geo-coordinates, into Nodegoat by uploading CSV files. Importing data into Nodegoat will also be the central topic of today’s session. We have three ways to import data into Nodegoat.

1) We can upload data into Nodegoat as a CSV file, as we did in session 2.

2) We can import data directly into Nodegoat using a graphical interface without having to upload it, we will look at this process of dynamic data ingestion today.

3) We can import data into Nodegoat via an application programming interface (API), which, unlike 1) and 2), requires programming knowledge.

Of course, we can also start a project in Nodegoat without importing data first, not even geodata. In every Nodegoat installation, the object ‘City’ is already present. In ‘City’ about 130k places are available, which have geo-coordinates as well as a GeoNames-ID. ‘City’ is a collaborative object: all projects of a Nodegoat installation can add and use ‘City’-locations and thus benefit from each other.”

Programme Session 3 (12-05-2021)

14:00 Welcome and recap of last week’s session

14:15 Create Linked Data Resource to query GND

14:30 Create Linked Data Resource to query VIAF

14:50 Break

15:00 Ingestion of VIAF IDs from Wikidata

15:50 Break

16:00 Ingestion of biographical data from Wikidata

16:35 Looking forward to next session

16:45 Q&A

• Where can I add an image caption? -> Create a separate Object for Images, store the caption as an Object Description
• This a very generic question from my experience with data importing this week: is there a way to mass-edit data in Nodegoat? An import that went wrong and created new undesired sub-objects. How to delete them? -> Filter on the specific data (e.g. the undesired sub-objects), and click the grey ‘multi’ button (top-right of the table view), click ‘c’ (Change) and set the value or whole sub-object to be removed.
• follow-up on the question above: if what I want to delete is all entities (for example, that miss a particular sub-object)? -> Filter sub-object on amount ‘no’ or 0, mass delete result using ‘multi’.
• also a question following my attempts this last week: is it possible to only include a set of relevant persons in a project? (I have a very large list of person records, and a large gazetteer, but particular projects only use small parts of them) → Create separate Projects in Management, create a Filter for each selection, apply this filter in each Project’s settings. – Ah! thanks.
• just as FYI for participants: some searchable catalogs of linked data resources are at http://bartoc.org and https://lov.linkeddata.es/dataset/lov/ …
• additional resources that could be considered:
  • Reconciliation protocol API (https://reconciliation-api.github.io/specs/latest/ ) , see list of services: https://reconciliation-api.github.io/testbench/ ;
  • Consortium of European Research Libraries (CERL): https://data.cerl.org/thesaurus/_search?query=Burckhard&format=json&style=jsonld
  • Orcid: https://info.orcid.org/documentation/api-tutorials/api-tutorial-searching-the-orcid-registry/
  • Rijksmuseum: https://www.rijksmuseum.nl/en/search?ii=0&p=1

• Can the mapping then only be done per 1 object? Or can you run it on a set (like reconciliation in open refine) and get unambiguous results automatically? → Yes, both.
• Can I add/concatenate more of the json fields to the “label”? Because just the preferred name may not be sufficient to identify which one is the correct entry …→ Yes, add/include more of the relevant Values in your response, then open a “filter” dialog if necessary (there the additional fields/values will be shown).
• Does nodegoat also support APIs that return XML instead of JSON? -> No
• Wikidata SPARQL Query to only get Gregorian Dates Example: https://w.wiki/3KFq ->thanks!
• Not really related to this session by I don’t see this referred to in any of the sessions: is there a nodegoat api from which one can draw the visualizations? or even simpler ‘embeds’? -> Public User Interfaces support embedding.
  • Does it expose JSON representations of our objects → next session
  • And is there a SPARQL endpoint (I know I would have to specify a kind of mapping)
•  I am trying to create a LD resource from my API. The query: https://data.geo-kima.org/api/Variants/PlaceVariants/8964/100/1 works outside nodegoat. How can I spit this to fill the uri and query in nodegoat? (I get an error message when I do https://data.geo-kima.org/api/ in the url and Variants/PlaceVariants/8964/100/1 in the query. → next session
• Is it possible to interact with database directly using SQL? → Not by design/purpose, API should be used

Links

https://nodegoat.net/usecases

Slides:

Preparation

If you are unfamiliar with the benefits of adding external identifiers to your dataset, please read this guide: https://nodegoat.net/guides/externalidentifiers.This example shows how to update 1 object at once. We can update multiple objects at once with the data ingestion (see below, data ingestion with Nodegoat).

Human Readable vs Machine Readable

Browse GND Data

Via Graphical User Interface (GUI):

https://d-nb.info/gnd/118637533 / https://lobid.org/gnd/118637533

Via Application Programming Interface (API): https://lobid.org/gnd/118637533.json

Query GND Data:

Via GUI: https://lobid.org/gnd/search?q=Zwingli

Via API: https://lobid.org/gnd/search?q=Zwingli&filter=type:Person&format=json

Data ingestion with Nodegoat

With the data ingestion in Nodegoat, you can enrich or update multiple objects with Linked Open Data (LOD) data from external data sources. This requires two steps. First, we configure a Linked Data Resource in Nodegoat, i.e. a query to the interface where the LOD data is available (data source). Secondly, we configure a data ingestion process, i.e. the mapping and storage of the LOD data in Nodegoat. Below are some examples of how to configure interfaces to Linked Data Resources.

Linked Data Resources

To start the ingestion process, we activate it for our project in ‘Management’. Then we define the process, the mapping and the storage of the LOD data, in the ‘Data’ section, like in the picture. We can add new objects (or values of objects = ‘object descriptions’ in Nodegoat), ad them if they not exist already or update objects, for example the object ‘Person’.

Session 2

05 May 2021

“Introduction by Kaspar Gubler

We are very pleased to have so many interesting projects and engaged participants in the workshop. And more participants have joined for session 2. For example, someone from Hamburg who used to do network analysis with the software Gephi and now wants to try out Nodegoat. A new participant is an archaeologist from the University of Bern who has documented sites in Excel and wants to import and visualise them in Nodegoat. Such data import is not difficult, especially if the data has been entered consistently in Excel. Another new participant plans to visualise cultural heritage with Nodegoat. A good example of how Nodegoat can be used for the presentation of digital, cultural heritage (thus also for art history) is the encyclopaedia on Romantic Nationalism: https://ernie.uva.nl/viewer.p/21/52/types/all/grid

Terminology

Before we start, I would like to remind you of the terminology of Nodegoat, in which we speak of Objects and Sub-Objects as well as Categories. We describe these Objects (column inExcel) in Nodegoat with Object descriptions (like rows in Excel). Object descriptions can be a text, a link, a picture or a link to another Object or a Category (= reference = relation). We can define in our data model the kind of description for each Object description. This gives us the possibility to describe an Object very precisely:

Find a common language

Important: if you want to communicate with another Nodegoat project it is very helpful if you use the terminology mentioned. So the first questions to another project would be: what Objects do you have? And how do you describe your Objects with what kind of Object descriptions? In which Sub-Object do you store your geo references? If you want to get in touch with other projects, you can organise your own zoom meetings on specific questions about Nodegoat. I see many projects that have a lot in common and could certainly benefit from an exchange. I would also like to draw your attention to the Nodegoat Day on 4 June, where you can present your project or your project idea.”

14:00 Welcome and recap of last week’s session

14:15 Object Type ‘Place’ Data Model + Data Entry

14:30 Object Type ‘Place’ Data Import

14:50 Break

15:00 Object Type ‘Person’ Data Import

15:30 Filter + Visualisation

15:50 Break

16:00 Scope & Visual Settings

16:15 Conditions & Export

16:35 Looking forward to next session

16:45 Q&A

• Difference between Gephi and Nodegoat? → Nodegoat departs from data management + visualization functionality
• Will there be a possibility to store Nodegoat Data in a data repository like Zenodo? → There are rumors about a Zenodo-Module in Nodegoat coming, currently it’s technically no problem to do it manually
• How to download a “dump”?  → Via API, export dump of the data + of the model in JSON
• Is it possible to export a complete project (instead of individual csv sheets)?  → Yes, via API you can export all of the data and the data model in JSON
• How can I “undo” an import from csv when I notice that some things did not work as intended? Can I mass delete objects? → Yes, cou can mass delete objects via graphical interface, choosing all objects deleting them with the grey multi button, or delete in ‘Model’ the whole of a Object Type with clicking on ‘empty’ or mass delete mass objects via API
• Can you import by just giving the URL of the Google Doc? → Yes, via API of Nodegoat, check what Google allows you to do via API
• Can visualisations be downloaded in any way to the desktop? → Yes, Screenshot, or for high resolution use the ‘Capture’ functionality in the visualisation settings
• Follow-up question to session 1:  can you create an itinerary of a person (object) with just knowing the sequence of the location but not the dates?→ Yes, with storing vague dates in Nodegoat, you make an statement in vague dates (‘Chronology’) like: ‘Studies came after Birth’. Or Yes, use as date: 1, 2, 3 etc. , or use the sequence identifier in a nodegoat date, so if you know a year use ‘1880 1’, ‘1818 2’, ‘1818 3’
• We can include both a geometry (polygon) AND a precise coordinate in a sub-object? Or as separate subobjects of the same object? → Yes, , yes both options are possible! One geometry can be polygon + point(s) + line(s). Or each in a separate sub-object to be able to add attributes.
• Are there any example projects that depicts more complex routes? → http://mnn.nodegoat.net/viewer.p/1/47/scenario/30/geo
• Can you add your own icons to be displayed on the map? → Yes, in SVG format.
• Nodegoat as Tool to visualise routes or itineraries? →  Yes
• Is there also a method to show place-specific meta-information on the map instead of the person’s? →  Yes
• In case of data model refactoring, how should we deal with the already inserted data? For example, if one wants to normalize repetitive data creating a new object type, how can he migrate the actual data to the new data model? Export + Transform + Import is the only way?  →  Yes, but because you now have nodegoat IDs, it’s a matter of a straightforward mapping. Or use an Ingestion process (session 4).
• is it possible to mark a node with multiple conditions (e.g. one condition for people born in the low countries (orange) + people died in Italy (blue), so objects that fall in both categories marked in two colours)? →  Yes

Links

http://mnn.nodegoat.net/viewer.p/1/47/scenario/30/geo

Slides:

Download Google Sheets as CSV files:

RAG Places small selection: https://docs.google.com/spreadsheets/d/1zvcVj66nr1tm7PAmNJSSf2BI_o5e2rrPCSE2l4PAsHQ/
RAG People small selection:https://docs.google.com/spreadsheets/d/1K2SGF0TkQTVnZ5WQqgMc0MbJdGps1kA_oWVL3Qir6rs/

Guides:https://nodegoat.net/guides/csvfilehttps://nodegoat.net/guides/gazetteer

Another sample data Import:
https://histdata.hypotheses.org/nodegoat-tutorials
Tutorial No 10, to create this map (positions of ships):

Session 1

28 April 2021

Introduction by Kaspar Gubler

“Welcome to the Nodegoat SPARK workshop ‘Dynamic Data ingestion’ We are very happy about the 140 participants from all over the world. On the map, which was of course created with Nodegoat, we can see the places of origin of the participants. They come from very different disciplines: history, literary history, German studies, English studies, legal history, historical geography, art history, musicology, theatre studies, film studies, African studies, Islamic studies, sociology, digital humanities and also from archives and libraries. This impressively shows how Nodegoat has established itself in the last ten years as an international research infrastructure for the humanities, an interdisciplinary research infrastructure that helps digital research to gain new insights and more visibility – and facilitates the collaboration of projects, especially beyond one’s own subject boundaries. Pim van Bree and Geert Kessels began developing Nodegoat as part of a project at the University of Amsterdam in 2011. Pim van Bree has a Master’s degree in Media Studies, Geert Kessels a Master’s degree in History. Both are also accomplished software developers. Their particular strength is that they know both worlds, the world of humanities and the world of programming. They combine these two worlds in their workshops, which they conduct at educational institutions worldwide. With their deep knowledge of methods, sources and questions in the humanities, they can create fitting and working data models for the different disciplines in Nodegoat to extract new scientific information and knowledge from the data.”

Programme

14:00 Welcome by Kaspar Gubler

14:15 General introduction to nodegoat

14:40 Login and set up your nodegoat project

14:50 Break

15:00 Object Type ‘Person’ Data Model

15:15 Object Type ‘Person’ Data Entry

15:35 Classification ‘Capacity’ Data Model + Data Entry

15:50 Break

16:00 Object Type ‘Book’ Data Model

16:20 Object Type ‘Book’ Data Entry

16:35 Looking forward to next session

16:45 Q&A

• I don’t have access to the ‘Model’ section in nodegoat → Check Page Clearence. Each nodegoat project has 1 administrator at the beginning, who can set up additional users. Management > Users > add User > add the user  and activate the ‘Model’ in the settings  for the page clearance (tab)
• How does the Scope work? For Visualizations? → In the Scope you define which of your databasefields you want to use for the visualization, so you activate the field that contains the georeference wiht the coordinates
• Are there facilities (planned?) helping to prepare a RDF rendition of the database? →Yes: on one’s own nodegoat installation, you can configure a translation module to translate the data model to some RDF vocabulary
• Is it possible to export, as static or dynamic representation, a computed spatio-temporal / network analysis ? →Yes
• Würden Sie sagen, dass sich Nodegoat grundsätzlich auch als Bilddatenbank (mit Zusatzbeschreibungen und Querverweisen) eignet? →Yes, absolutely, see the links below.
• Regarding custom gazetteers and prosopographies: Are there size limitations? → Not in general, size limit of CSV import is set to 60’000 rows at a time
• Is it better to store a region like Germany as JSON or via Reference >city > autofill option Germany? →It depends on what you want to show on your map, if you are more interested in areas it’s Geo JSON, if you are working in general with dots on your maps, it’s maybe better to store it as a point like your other data
• Can we import polygon data from an existing map with territorial circumscription so that we don’t have to draw them by hand? → Yes
• Can we specify a schema or other constraints to ensure consistency of the data (e.g. Birthdate < Deathdate, no overlapping residence periods etc.)? → You can use visualisation to do some error checking, but there can be no hard enforcement of such constraints at the moment. Or you can filter specifically on Birthdate < Deathdate, more advanced.
• Does database harmonise the different sources of location, e.g. if I put “Roma (IT)” from “City” and also point from “Geometry” which is actually Rome, will database understand it is the same? On the map it looks the same, but how is it in database? Is it linked as the same entity? → More on this next session!
• Is Arabic script supported or more generally, are scripts running from right to left supported? → Yes: Everything Unicode
• Can we geo-visualize more than one object type, say author’s places and book’s publishing houses’ places? → Yes: using the Scope (https://nodegoat.net/guides/visualisationsettings)
• Is there anything in the guides as regards Nodegoat and RDF? → Yes, it’s work in progress, see this sample nodegoat project working with a Subject-Predicate-Object (RDF Format) data model: https://www.manto-myth.org/blog/a-half-dozen-ways-to-die-mythically
• If I uncheck „Fixed field“ in Object „book“ it throws: The data Model does not have a configuration that can be used to generate Object names, please check your settings.“ Why? → Tick/check one or more of the Object Description ‘name’ (‘use object description for name’) checkboxes
• Can nodegoat handle localised object descriptions? E.g. book reviews in different languages? → Yes
• Can nodegoat handle uncertain dates / data? → Yes, see the following blogposts:

https://nodegoat.net/blog.s/45/how-to-store-uncertain-data-in-nodegoat-ambiguous-identities

https://nodegoat.net/blog.s/44/how-to-store-uncertain-data-in-nodegoat-conflicting-information

https://nodegoat.net/blog.s/43/how-to-store-uncertain-data-in-nodegoat-incomplete-source-material

https://nodegoat.net/blog.s/42/how-to-store-uncertain-data-in-nodegoat

      Slides

• Guides / Tutorialshttps://nodegoat.net/guides/guides

  https://histdata.hypotheses.org/nodegoat-tutorials

  Sample Projects / Visualizations

  https://ernie.uva.nl/

  https://devereeuwigdestad.nl/

  https://uspolitics.nodegoat.net/viewer.p/28/455/scenario/4/social/fullscreen

  https://uspolitics.nodegoat.net/viewer.p/28/455/scenario/3/geo/fullscreen

Workshop on the results of the SPARK project of the Swiss National Science Foundation (SNSF) “Dynamic Data Ingestion (DDI): Server-side data harmonization in historical research. A centralized approach to networking and providing interoperable research data to answer specific scientific questions” (http://p3.snf.ch/project-190161). The workshop will take place in four sessions via Zoom. In sessions 1 and 2, participants will be introduced to the functions of the virtual research environment Nodegoat (VRE) and create a data model and import a data sample, which they will use in sessions 3 and 4 for the exercises on data ingestion. At the end, each participant will have a working VRE that can be used for further research or also used in teaching. It is highly recommended to attend all 4 sessions. The workshop is primarily aimed at members of the Phil.-Hist. faculty of the University of Bern, but is generally open to other interested parties on planet earth. The zoom link to the workshop will be sent to participants after registration. The workshop will be led by Nodegoat developers Pim van Bree and Geert Kessels (LAB1100), together with Kaspar Gubler, Institute of History, University of Bern.

Members of the Phil.-Hist. faculty of the University of Bern can apply for an VRE free of charge at the following link: https://www.dh.unibe.ch/dienstleistungen/nodegoat_go/index_ger.html Other participants can obtain an VRE at nodegoat.net. Or get Nodegoat Open Soruce on GitHub: https://github.com/nodegoat/nodegoat

The workshops always take place on Wednesdays from 2 – 5 pm. The workshops are recorded and can therefore be re-watched if a session cannot be attended.

Dates: 28.04.2021 / 05.05.2021 / 12.5.2021 / 26.5.2021

Registration for the workshop until 25.04.2021 to: kaspar.gubler@hist.unibe.ch

Session 1: Data Modelling (people and books)

In session 1 we get to know the central functions of Nodegoat (NG). Since NG is managed via the web browser, no additional software needs to be installed on the computer and location-independent working is no problem. With NG, research projects can be created, research data managed, analyzed, visualized, published on the Internet and shared with other researchers without any special programming skills. We will create our first data model, which we will fill with data in the next sessions. As we will see, NG is not a rigid ”boutique solution” that only fits a specific question or data model. Students and researchers can use NG to create custom data models based on their specific questions.

Session 2: Importing Data (including a VIAF id for each person)

In Session 2, we will import our first data sample and an identifier (VIAF) for each person. So we will work with a prosopographically oriented data model, but we can easily extend it for other research questions.

Session 3: Ingesting Biographical Data (like other IDs, or birth/death)

In session 3, we will learn about the principles of “Dynamic Data Ingestion”. There are numerous data sources on the Internet, whether for research or for the interested public. What types of data sources are there? And what about data quality? We will first explore these questions before connecting our Nodegoat environment to a typical data source via an interface (API) and importing the first test data. These data can be further identifiers of persons or also information about the life data.

Session 4: Ingesting Related Data (like published books of people)

In Session 4, we will enrich the data on the persons and check what other data is available on the Internet. For example, publications that we can add and, if we have full texts, additionally analyze in nodegoat. Finally, we will also look at the data harmonization capabilities in nodegoat.