Skip to content

Developer documentation

Project structure

This project contains the following main components:

  • metadata_api - A REST API that serves metadata for Språkbanken Text's corpora, lexicons, models, analyses, and utilities (mainly used by our site at spraakbanken.gu.se). For documentation, see below.
  • /metadata_api/parse_yaml.py - A script that prepares data for the REST API. This component is called automatically upon cache renewal but can also be run as a script locally (although this functionality is deprecated).
  • /metadata_api/tasks.py - Celery background tasks for the REST API, e.g. for cache renewal.
  • /gen_pids/gen_pids.py - A Python script that generates new PIDs (Datacite DOIs) by reading our metadata YAML files and registering resources at Datacite. It also handles log files, git add/commit/push, and log rotation when run as a script. For documentation, see the code comments and the PID guide.

Furthermore there are some scripts in the batch_jobs/ directory that are used for batch processing of resources, e.g. for adding new fields to all metadata YAML files or for extracting information.

Logging

Logging is configured via environment variables in metadata_api/settings.py. Important variables:

  • LOG_LEVEL (default: INFO)
  • LOG_TO_FILE (default: True): Logs always go to stdout; if True, they are also saved to logs/metadata_api_<DATE>.log.

When running the development server (python run.py), logs are always sent to the console and the log level is set to DEBUG. The log level can be changed with the --log-level argument.

Caching

Caching can be used in the API to improve response times. Memcached is used for this purpose and can be configured in .env with the following settings:

MEMCACHED_SERVER="localhost:11211"  # Set to "" to disable caching

What is being cached?

  • All data for each resource type as a dictionary (example key: corpus.json)
  • Data for each resource as a dictionary (example key: attasidor)
  • Resource descriptions for each resource that has one (example key: res_descr_attasidor)

Cache renewal

Cache renewal can be triggered by calling the /renew-cache route. This will trigger a background task with celery that will do the following:

  • Changes from the metadata repository will be pulled from GitHub to update the metadata YAML files.
  • Metadata YAML files will be reprocessed (either all of them or just the ones specified by the resource-paths parameter, or the files that were changed in the last push event, specified by the GitHub webhook call) and the static JSON files used by the API will be regenerated.
  • If memcached caching is activated, the cache is flushed and repopulated with data from the updated JSON files.

The /renew-cache route can be called manually (e.g. via curl) but it is usually also set up as a webhook in the metadata repository to be triggered automatically upon each push event. Configure the GitHub webhook to send a POST request with the application/json content type to the deployment's /renew-cache endpoint, for example:

https://ws.spraakbanken.gu.se/ws/metadata/dev/renew-cache

The endpoint returns 202 Accepted after queueing the Celery task. Pushes to main use the webhook payload to identify the changed metadata YAML files; pushes to other branches are ignored. If GitHub reports more changed files than the configured limit, all metadata is renewed because the payload may not contain a complete file list.

The response from the /renew-cache route will not contain the results of the cache renewal itself, since this is done in the background. If SLACK_WEBHOOK_URL is set in the configuration and any errors or warnings occur, a message with the results of the cache renewal will be sent to the specified Slack channel when the task is finished. Messages from the task will also be logged to the celery worker log.

Bumping the version number

If you want to bump the app version number, update the version field in pyproject.toml. If you change the major version, run ./set_version.sh to automatically update all version references in the URLs in the README file.

Deployment (SBX-specific)

Set up the metadata-api app by following the installation instructions in the README file to install the dependencies. Don't forget to add your own configuration to the app as described under the Configuration section in the README file.

Setting up the metadata repository

git clone git@github.com-metadata:spraakbanken/metadata.git

Setting up the services

  • Install Redis (used as broker for Celery background tasks) and Memcached (for optional caching).

  • Create a directory where Redis can store data and add it to the Redis configuration file, e.g.:

mkdir /home/fksbwww/redisdata
echo "dir /home/fksbwww/redisdata" >> /home/fksbwww/redis-install/redis.conf
  • Add entries in supervisord config for the metadata-api, the celery worker, redis and memcached, e.g:
[program:metadata]
directory=%(ENV_HOME)s/metadata-api/dev/
command=%(ENV_HOME)s/metadata-api/dev/.venv/bin/uvicorn --app-dir %(ENV_HOME)s/metadata-api/dev --root-path /ws/metadata/dev --host "localhost" --port 1337 --workers 4  --proxy-headers --forwarded-allow-ips=* metadata_api.main:app
redirect_stderr=true
stopasgroup=true

[program:metadata-celery]
command=%(ENV_HOME)s/metadata-api/dev/.venv/bin/celery -A metadata_api.tasks worker -P solo --loglevel=INFO
directory=%(ENV_HOME)s/metadata-api/dev/
stopwaitsecs=600
redirect_stderr=true
stopasgroup=true
killasgroup=true

[program:redis-metadata]
command=%(ENV_HOME)s/redis-install/src/redis-server %(ENV_HOME)s/redis-install/redis.conf
redirect_stderr=true

[program:memcached-metadata]
command=%(ENV_HOME)s/memcached-jox/memcached-install/bin/memcached -v
redirect_stderr=true
  • Update supervisord and start the services with supervisorctl update.

Final setup steps

  • When the app is up and running, call the /renew-cache route in order to create the necessary JSON files and populate the cache.

  • Store Datacite login credentials in /home/fksbwww/.netrc (check the PID guide for more info).

  • Set up cron jobs that periodically run gen_pids.py to add DOIs to resources, update Datacite, and push metadata changes. The following cron jobs are run on fksbwww@k2 (but /dev and localhost:1337 should be replaced with the actual deployment path and host/port):

# Generate pids every night
5 1 * * * cd /home/fksbwww/metadata-api/dev && uv run -m gen_pids.gen_pids --no-update > /dev/null
# Update Datacite metadata once per week
15 23 * * 0 cd /home/fksbwww/metadata-api/dev && uv run -m gen_pids.gen_pids > /dev/null
# Call /renew-cache every night (mainly for updating downloadables file sizes)
5 5 * * * curl --silent "localhost:1337/renew-cache" > /dev/null