Session 3
12 May 2021
“Introduction by Kaspar Gubler
I would like to welcome in this session all participants who are already familiar with Nodegoat and have therefore skipped sessions 1 and 2 and will now attend sessions 3 and 4. In sessions 1 and 2 we created a data model for people and books and imported data, including geo-coordinates, into Nodegoat by uploading CSV files. Importing data into Nodegoat will also be the central topic of today’s session. We have three ways to import data into Nodegoat.
1) We can upload data into Nodegoat as a CSV file, as we did in session 2.
2) We can import data directly into Nodegoat using a graphical interface without having to upload it, we will look at this process of dynamic data ingestion today.
3) We can import data into Nodegoat via an application programming interface (API), which, unlike 1) and 2), requires programming knowledge.
Of course, we can also start a project in Nodegoat without importing data first, not even geodata. In every Nodegoat installation, the object ‘City’ is already present. In ‘City’ about 130k places are available, which have geo-coordinates as well as a GeoNames-ID. ‘City’ is a collaborative object: all projects of a Nodegoat installation can add and use ‘City’-locations and thus benefit from each other.”
Programme Session 3 (12-05-2021)
14:00 Welcome and recap of last week’s session
14:15 Create Linked Data Resource to query GND
14:30 Create Linked Data Resource to query VIAF
14:50 Break
15:00 Ingestion of VIAF IDs from Wikidata
15:50 Break
16:00 Ingestion of biographical data from Wikidata
16:35 Looking forward to next session
16:45 Q&A
- Where can I add an image caption? -> Create a separate Object for Images, store the caption as an Object Description
- This a very generic question from my experience with data importing this week: is there a way to mass-edit data in Nodegoat? An import that went wrong and created new undesired sub-objects. How to delete them? -> Filter on the specific data (e.g. the undesired sub-objects), and click the grey ‘multi’ button (top-right of the table view), click ‘c’ (Change) and set the value or whole sub-object to be removed.
- follow-up on the question above: if what I want to delete is all entities (for example, that miss a particular sub-object)? -> Filter sub-object on amount ‘no’ or 0, mass delete result using ‘multi’.
- also a question following my attempts this last week: is it possible to only include a set of relevant persons in a project? (I have a very large list of person records, and a large gazetteer, but particular projects only use small parts of them) → Create separate Projects in Management, create a Filter for each selection, apply this filter in each Project’s settings. – Ah! thanks.
- just as FYI for participants: some searchable catalogs of linked data resources are at http://bartoc.org and https://lov.linkeddata.es/dataset/lov/ …
- additional resources that could be considered:
- Reconciliation protocol API (https://reconciliation-api.github.io/specs/latest/ ) , see list of services: https://reconciliation-api.github.io/testbench/ ;
- Consortium of European Research Libraries (CERL): https://data.cerl.org/thesaurus/_search?query=Burckhard&format=json&style=jsonld
- Orcid: https://info.orcid.org/documentation/api-tutorials/api-tutorial-searching-the-orcid-registry/
- Rijksmuseum: https://www.rijksmuseum.nl/en/search?ii=0&p=1
- Can the mapping then only be done per 1 object? Or can you run it on a set (like reconciliation in open refine) and get unambiguous results automatically? → Yes, both.
- Can I add/concatenate more of the json fields to the “label”? Because just the preferred name may not be sufficient to identify which one is the correct entry …→ Yes, add/include more of the relevant Values in your response, then open a “filter” dialog if necessary (there the additional fields/values will be shown).
- Does nodegoat also support APIs that return XML instead of JSON? -> No
- Wikidata SPARQL Query to only get Gregorian Dates Example: https://w.wiki/3KFq ->thanks!
- Not really related to this session by I don’t see this referred to in any of the sessions: is there a nodegoat api from which one can draw the visualizations? or even simpler ‘embeds’? -> Public User Interfaces support embedding.
- Does it expose JSON representations of our objects → next session
- And is there a SPARQL endpoint (I know I would have to specify a kind of mapping)
- I am trying to create a LD resource from my API. The query: https://data.geo-kima.org/api/Variants/PlaceVariants/8964/100/1 works outside nodegoat. How can I spit this to fill the uri and query in nodegoat? (I get an error message when I do https://data.geo-kima.org/api/ in the url and Variants/PlaceVariants/8964/100/1 in the query. → next session
- Is it possible to interact with database directly using SQL? → Not by design/purpose, API should be used
Links
Slides:
Preparation
If you are unfamiliar with the benefits of adding external identifiers to your dataset, please read this guide: https://nodegoat.net/guides/externalidentifiers.This example shows how to update 1 object at once. We can update multiple objects at once with the data ingestion (see below, data ingestion with Nodegoat).
Human Readable vs Machine Readable
Browse GND Data
Via Graphical User Interface (GUI):
https://d-nb.info/gnd/118637533 / https://lobid.org/gnd/118637533
Via Application Programming Interface (API): https://lobid.org/gnd/118637533.json
Query GND Data:
Via GUI: https://lobid.org/gnd/search?q=Zwingli
Via API: https://lobid.org/gnd/search?q=Zwingli&filter=type:Person&format=json
Data ingestion with Nodegoat
With the data ingestion in Nodegoat, you can enrich or update multiple objects with Linked Open Data (LOD) data from external data sources. This requires two steps. First, we configure a Linked Data Resource in Nodegoat, i.e. a query to the interface where the LOD data is available (data source). Secondly, we configure a data ingestion process, i.e. the mapping and storage of the LOD data in Nodegoat. Below are some examples of how to configure interfaces to Linked Data Resources.
Linked Data Resources
Label | Value |
Name | Search the GND API via lobid.org |
Protocol | API |
URL | https://lobid.org/gnd/search?q= |
URL Options | &filter=type:Person&format=json |
Query | [query=name][variable]zwingli[/variable][/query]&from=[[offset]]&size=[[limit]] |
Label | Value |
Name | Search the VIAF API |
Protocol | API |
URL | http://www.viaf.org/viaf/AutoSuggest?query= |
URL Options | |
Query | [query=name][variable]zwingli[/variable][/query] |
Label | Value |
Name | Query Wikidata for VIAF ID based on GND ID |
Protocol | SPARQL |
URL | https://query.wikidata.org/sparql?query= |
URL Options | &format=json |
Query | SELECT ?person ?viaf
WHERE { [query=gnd]?person wdt:P227 “[variable]118637533[/variable]” .[/query] ?person wdt:P214 ?viaf .} |
Link | Click to open Query |
Label | Value |
Name | Query Wikidata for Religion based on GND ID |
Protocol | SPARQL |
URL | https://query.wikidata.org/sparql?query= |
URL Options | &format=json |
Query | SELECT ?person ?religion ?religion_label
WHERE { [query=gnd]?person wdt:P227 “[variable]118637533[/variable]” .[/query] ?person wdt:P140 ?religion . ?religion rdfs:label ?religion_label . FILTER(LANG(?religion_label) = “en”) } |
Link | Click to open Query |
Label | Value |
Name | Query Wikidata for Date of Birth based on GND ID |
Protocol | SPARQL |
URL | https://query.wikidata.org/sparql?query= |
URL Options | &format=json |
Query | SELECT ?person ?date_of_birth
WHERE { [query=gnd]?person wdt:P227 “[variable]118637533[/variable]” .[/query] ?person wdt:P569 ?date_of_birth . } |
Conversion | INPUT
1484-01-10T00:00:00Z Script: const date = new Date(INPUT); const day = date.getDate(); const month = date.getMonth() + 1; const year = date.getFullYear(); var OUTPUT = {‘date’: day+’-‘+month+’-‘+year}; |
Link | Click to open Query |
To start the ingestion process, we activate it for our project in ‘Management’. Then we define the process, the mapping and the storage of the LOD data, in the ‘Data’ section, like in the picture. We can add new objects (or values of objects = ‘object descriptions’ in Nodegoat), ad them if they not exist already or update objects, for example the object ‘Person’.
OpenEdition suggests that you cite this post as follows:
Kaspar Gubler (May 13, 2021). SNSF SPARK Workshop on Dynamic Data Ingestion. HistData. Retrieved March 21, 2025 from https://doi.org/10.58079/pldy