SNSF SPARK Workshop on Dynamic Data Ingestion

Session 3

12 May 2021

“Introduction by Kaspar Gubler

I would like to welcome in this session all participants who are already familiar with Nodegoat and have therefore skipped sessions 1 and 2 and will now attend sessions 3 and 4. In sessions 1 and 2 we created a data model for people and books and imported data, including geo-coordinates, into Nodegoat by uploading CSV files. Importing data into Nodegoat will also be the central topic of today’s session. We have three ways to import data into Nodegoat.

1) We can upload data into Nodegoat as a CSV file, as we did in session 2.

2) We can import data directly into Nodegoat using a graphical interface without having to upload it, we will look at this process of dynamic data ingestion today.

3) We can import data into Nodegoat via an application programming interface (API), which, unlike 1) and 2), requires programming knowledge.

Of course, we can also start a project in Nodegoat without importing data first, not even geodata. In every Nodegoat installation, the object ‘City’ is already present. In ‘City’ about 130k places are available, which have geo-coordinates as well as a GeoNames-ID. ‘City’ is a collaborative object: all projects of a Nodegoat installation can add and use ‘City’-locations and thus benefit from each other.”

Programme Session 3 (12-05-2021)

14:00 Welcome and recap of last week’s session

14:15 Create Linked Data Resource to query GND

14:30 Create Linked Data Resource to query VIAF

14:50 Break

15:00 Ingestion of VIAF IDs from Wikidata

15:50 Break

16:00 Ingestion of biographical data from Wikidata

16:35 Looking forward to next session

16:45 Q&A

  • Can the mapping then only be done per 1 object? Or can you run it on a set (like reconciliation in open refine) and get unambiguous results automatically? → Yes, both.
  • Can I add/concatenate more of the json fields to the “label”? Because just the preferred name may not be sufficient to identify which one is the correct entry …→ Yes, add/include more of the relevant Values in your response, then open a “filter” dialog if necessary (there the additional fields/values will be shown).
  • Does nodegoat also support APIs that return XML instead of JSON? -> No
  • Wikidata SPARQL Query to only get Gregorian Dates Example: https://w.wiki/3KFq ->thanks!
  • Not really related to this session by I don’t see this referred to in any of the sessions: is there a nodegoat api from which one can draw the visualizations? or even simpler ‘embeds’? -> Public User Interfaces support embedding.
    • Does it expose JSON representations of our objects → next session
    • And is there a SPARQL endpoint (I know I would have to specify a kind of mapping)
  •  I am trying to create a LD resource from my API. The query: https://data.geo-kima.org/api/Variants/PlaceVariants/8964/100/1 works outside nodegoat. How can I spit this to fill the uri and query in nodegoat? (I get an error message when I do https://data.geo-kima.org/api/ in the url and Variants/PlaceVariants/8964/100/1 in the query. → next session
  • Is it possible to interact with database directly using SQL? → Not by design/purpose, API should be used

Links

https://nodegoat.net/usecases

Slides:

 

Preparation

If you are unfamiliar with the benefits of adding external identifiers to your dataset, please read this guide: https://nodegoat.net/guides/externalidentifiers.This example shows how to update 1 object at once. We can update multiple objects at once with the data ingestion (see below, data ingestion with Nodegoat).

Human Readable vs Machine Readable

Browse GND Data

Via Graphical User Interface (GUI):

https://d-nb.info/gnd/118637533 / https://lobid.org/gnd/118637533

Via Application Programming Interface (API): https://lobid.org/gnd/118637533.json

Query GND Data:

Via GUI: https://lobid.org/gnd/search?q=Zwingli

Via API: https://lobid.org/gnd/search?q=Zwingli&filter=type:Person&format=json

 

Data ingestion with Nodegoat

With the data ingestion in Nodegoat, you can enrich or update multiple objects with Linked Open Data (LOD) data from external data sources. This requires two steps. First, we configure a Linked Data Resource in Nodegoat, i.e. a query to the interface where the LOD data is available (data source). Secondly, we configure a data ingestion process, i.e. the mapping and storage of the LOD data in Nodegoat. Below are some examples of how to configure interfaces to Linked Data Resources.

Linked Data Resources

Label Value
Name Search the GND API via lobid.org
Protocol API
URL https://lobid.org/gnd/search?q=
URL Options &filter=type:Person&format=json
Query [query=name][variable]zwingli[/variable][/query]&from=[[offset]]&size=[[limit]]

 

Label Value
Name Search the VIAF API
Protocol API
URL http://www.viaf.org/viaf/AutoSuggest?query=
URL Options
Query [query=name][variable]zwingli[/variable][/query]

 

Label Value
Name Query Wikidata for VIAF ID based on GND ID
Protocol SPARQL
URL https://query.wikidata.org/sparql?query=
URL Options &format=json
Query SELECT ?person ?viaf

WHERE {

[query=gnd]?person wdt:P227 “[variable]118637533[/variable]” .[/query]

?person wdt:P214 ?viaf .}

Link Click to open Query

 

Label Value
Name Query Wikidata for Religion based on GND ID
Protocol SPARQL
URL https://query.wikidata.org/sparql?query=
URL Options &format=json
Query SELECT ?person ?religion ?religion_label

WHERE {

[query=gnd]?person wdt:P227 “[variable]118637533[/variable]” .[/query]

?person wdt:P140 ?religion .

?religion rdfs:label ?religion_label .

FILTER(LANG(?religion_label) = “en”)

}

Link Click to open Query

 

Label Value
Name Query Wikidata for Date of Birth based on GND ID
Protocol SPARQL
URL https://query.wikidata.org/sparql?query=
URL Options &format=json
Query SELECT ?person ?date_of_birth

WHERE {

[query=gnd]?person wdt:P227 “[variable]118637533[/variable]” .[/query]

?person wdt:P569 ?date_of_birth .

}

Conversion INPUT

1484-01-10T00:00:00Z

Script:

const date = new Date(INPUT);

const day = date.getDate();

const month = date.getMonth() + 1;

const year = date.getFullYear();

var OUTPUT = {‘date’: day+’-‘+month+’-‘+year};

Link Click to open Query

Click to open Query with reference statement

 

To start the ingestion process, we activate it for our project in ‘Management’. Then we define the process, the mapping and the storage of the LOD data, in the ‘Data’ section, like in the picture. We can add new objects (or values of objects = ‘object descriptions’ in Nodegoat), ad them if they not exist already or update objects, for example the object ‘Person’.


OpenEdition suggests that you cite this post as follows:
Kaspar Gubler (May 13, 2021). SNSF SPARK Workshop on Dynamic Data Ingestion. HistData. Retrieved March 21, 2025 from https://doi.org/10.58079/pldy