Introduction

DSpace is a popular choice to run digital repositories of various sizes. The software consists, at a bare minimum, of at least four parts: The DSpace backend, the DSpace frontend, a PostgreSQL database and an Apache Solr search index. If you follow the official installation guide, you will arrive at a single node setup with all parts (except for maybe the database) ending up on a single server system. In larger production grade deployments one might encounter optimized variations of this; e.g. each component might have a dedicated physical or virtual machine on its own, there might be a (caching) proxy server in front of DSpace etc. However, at the UBFFM we chose a different approach: Deploying DSpace in containers on an on-premises Kubernetes Cluster.

Introduction

User management is a fundamental aspect of web application development. While a local login is the simplest method to grant users access to resources, it comes with certain limitations. If users want to access multiple services using the same credentials, a centralized authentication system is necessary.

For a diverse user base (spanning different contexts such as industry specialists, academia, and various organizations or locations) a Single Sign-On (SSO) solution is essential. Well-known identity providers (IdPs) like GitHub¹ in the tech world, ORCID² for the scientific community, and the lesser-known DNF-AAI³ in German academia enable users to reuse their credentials across services.

Introduction

At the UB JCS, we make extensive usage of the Python luigi framework for data engineering. The framework is capable of handling thousands of tasks, calculating non-circular task dependencies, and run over days. Additionally, it provides a convenient web control panel to see, e.g. the task dependencies in a tree diagram or start specific tasks.

Although luigi itself supports the user already by enforcing a very specific structure, there are still some things to consider when designing a data pipeline with luigi (for a general introduction, see in a previous post). In this post, I present ideas, that I learned while using luigi. Since luigi is a heavily object-oriented framework, some approaches in this post rely naturally on Software architecture patterns.

Introduction

For many automated data processing tasks within the context of the Specialised Information Services (FID) at the University Library Frankfurt, we use the Python package luigi. This package proves especially useful when a task (e.g. the loading of data into a database) depends on the work of other tasks that have to run successfully, before the next task starts (e.g. first you need to download the data). luigi orchestrates all required tasks and their respective required task(s) and then processes everything for you. This approach makes the maintenance of tasks very easy, since you only have to add or remove required tasks from any task and luigi handles the rest for you, while you don’t have to worry about the computer science behind it too much. But although luigi takes a lot of mental load off of you, it also requires strategies to handle common situations that you may find yourself in.

Introduction

In the first post of this series, we covered the necessary steps to populate VuFind’s Solr cores with title and authority records. This second post describes changes to configuration files, as well as modifications that are necessary to interact, display and export records. All our customizations are based on existing VuFind code and, to ensure maintainability, stored in the local/ folder and the custom module Fiddk. We want to remind the reader, that we assume basic VuFind knowledge, which can be acquired from the documentation.