Skip to content

PID/DOI Guide

This guide describes how to generate PIDs (DOIs) for resources in the Språkbanken Metadata repository. A Resources are assigned a PID with the gen_pids/gen_pids.py script, which is run periodically on the server.

All types of resources, including collections should have a PID. The PID is stored in the metadata YAML file for the resource and is also shown on the Språkbanken Text website. The PID/DOI is generated by the DataCite service.

Terminology

  • data set: (previously called resource) a resource which is not an analysis or utility, (i.e. a corpus, lexicon or model).
  • PID/DOI: a unique identifier for a resource, generated by DataCite. A PID is a persistent identifier, and a DOI is a type of PID. In this project, we use the terms PID and DOI interchangeably.
  • DMS (record): DataCite Metadata Schema (record): what we construct and send to DataCite to get (or update) a DOI
  • Metadata API: Språkbanken Text's API for getting information out of our repo
  • SND: Swedish National Data Service, which is the organization that manages the DOI service for Swedish research data. SND is a member of DataCite and has an account with them. SND has provided us with a test account and a production account for generating DOIs.

Method

The gen_pids/gen_pids.py script checks every resource in the Språkbanken Metadata repository for a DOI:

  • If it does not exist, create a DMS record and add DOI to Metadata YAML file.
  • (Add existing Handle if possible: We don't do that - Markus' decision.)
  • Update DMS record and fix relation between collection and members.

In more technical detail:

  • gen_pids.py is called periodically by cron on k2 which should:
    1. Read all data sets (including collections)
    2. Iterated over all data sets (including collections):
      • If data set has no AlternateIdentifier, look up if it already has a Handle and set it.
      • If data set has no PID, generate a DOI (via DataCite API) and add it to the metadata YAML file.
    3. Iterate over all collections:
      • Set DMS-12-RelatedIdentifier to HasPart for all members of each collection (found in YAML resources field).
      • Set DMS-12-RelatedIdentifier to IsPartOf for each member of each collection.
      • Update DMS record (via DataCite API).

Notes on gen_pids.py

  • Resources with unlisted=true are skipped (no PID/DOI is generated).
  • Script will only update an existing DataCite record if it is older than the YAML record (updated field) or if run with the '--force_update' flag.
  • The login credentials to DataCite saved in /home/fksbwww/.netrc on k2 (credentials were received from SND and can be found in Språkbanken Text's safe).
  • When testing locally with one of the test repositories, you can enter the credentials in gen_pids/settings.py (DMS_URL, DMS_REPOID and DMS_AUTH_PASSWORD) instead of using the netrc file. Remember to remove them before committing the code.
  • The identifiers tag in JSON is called alternateIdentifiers in XML format (https://support.datacite.org/docs/what-is-the-identifiers-attribute-in-the-rest-api)

DataCite Metadata Schema (DMS)

This is what WE have to generate to get a DOI. Shouldn't be a problem as the Metadata in our repository is generously populated. (Source: DataCite Metadata Schema Documentation)

M - Mandatory. R - recommended. 1 - 1 value allowed. n - multiple values allowed.

  1. M1. Identifier. (10.21384/foo; "DOI"). Let DataCite autogen a DOI (4-4 chars) (https://support.datacite.org/docs/api-create-dois)
  2. Mn. Creator. May be a corporate/institutional or personal name. (+ROR/ORCID) Comment from SND: även där blir det nog Språkbanken Text
  3. Mn. Title. Title of dataset in multiple languages.
  4. M1. Publisher. "Språkbanken Text"? Eller "Språkbanken"? (+ROR) Comment from SND: organisationen ansvarig för att tillhandlahålla resursen så i ert fall borde det vara Språkbanken Text då resurserna ligger i erat repositorie.
  5. M1. PublicationYear. Not in YAML, but add it. Update later. Possibly weClarin repos can provide some dates.
  6. Rn. Subject. språkteknologi; nyckelord (see below). Choose the 1st ("Svenska ...")
  7. Rn. Contributor. The institution and/or person responsible for... (+ROR/ORCID). Wait.
  8. Rn. Date. Several types. Wait.
  9. O1. Language. Primary language. https://en.wikipedia.org/wiki/List_of_ISO_639_language_codes. Use 39.3!
  10. M1. ResourceType. +subtype. Dataset/Collection; "Dataset"/"lexicon"
  11. On. AlternateIdentifier: ex Handle. From SweClarin repos.
  12. Rn. RelatedIdentifier. Id or related resources. Ex: IsVersionOf, HasPart/IsPartOf (collection - and vice versa), Obsoletes/IsObsoletedBy (successor!)
  13. On. Size. Free format. State bytes and tokens.
  14. On. Format. File ext or MIME type.
  15. O1. Version. (leave out for now) Suggested practice: track major_version.minor_version. Register a new identifier for a major version change. Use with 11 and 12.
  16. On. Rights.
  17. Rn. Description. Take short description. Filter HTML.
  18. Rn. GeoLocation. Named place or coord. Comment from SND: skulle nog skippa att fylla i något där.
  19. On. Funding reference. VR?
  20. On. RelatedItem. Where resource does not have an id.

Also see the SND guide.

More DataCite info

Viewing a DMS record

Insert a resource ID in the following URL to view the DMS record for that resource:

https://api.datacite.org/dois?client-id=SND.SPRKB&query=identifiers.identifier:[RESOURCE-ID]%20AND%20identifiers.identifierType:slug&detail=true%22

e.g: https://api.datacite.org/dois?client-id=SND.SPRKB&query=identifiers.identifier:attasidor%20AND%20identifiers.identifierType:slug&detail=true%22

REST API usage

DataCite references

DataCite rate limit

Changing a resource id (slug) from A to B

In the metadata repository, rename the metadata file from A.yaml to B.yaml (A and B are resource IDs).

On DataCite:

  • Go to https://datacite.org and hit "Member login" to sign in to Fabrica.
  • Find the metadata record/DOI.
  • Click "Update DOI (form)".
  • Update the following fields (change A to B):
    1. URL
    2. Alternate identifier
  • Check that the update was successful by querying the API:
    https://api.datacite.org/dois?client-id=SND.SPRKB&query=identifiers.identifier:<A>%20AND%20identifiers.identifierType:slug&detail=true%22

Further info

SND

Contact persons at SND for questions about DataCite and DOIs:

Python packages

Usage examples for gen_pids.py

# Test gen_pids, don't write yaml or update DataCite
uv run -m python gen_pids.py --debug --dry-run

# Create DOIs for resources without them (don't update)
uv run -m python gen_pids.py --debug --no-update

# Test one resource
uv run -m python gen_pids.py --debug --dry-run --file corpus/attasidor.yaml

# Force update of one
uv run -m gen_pids.gen_pids --debug --force-update --file corpus/attasidor.yaml

# Force update all resources
python gen_pids.py --debug --force-update

Other useful commands

  • Finding successors in YAML directory: grep -lzrP "successors:\n - .+\n" .
  • Removing "doi:" line from all YAML files: grep -rl 'doi:' yaml | xargs sed -i '/^doi:/d'

Geting DataCite data for a certain year

To check how many posts have been updated or created within a certain year use the following calls (and update the year param in the end of the URL):

This is used for "återrapportering" within Språkbanken.