Lin on UBLabs

Common engineering strategies in luigi

labs@ub.uni-frankfurt.de (UBLabs) — Thu, 29 Jun 2023 14:00:00 +0200

Introduction

For many automated data processing tasks within the context of the Specialised Information Services (FID) at the University Library Frankfurt, we use the Python package luigi. This package proves especially useful when a task (e.g. the loading of data into a database) depends on the work of other tasks that have to run successfully, before the next task starts (e.g. first you need to download the data). luigi orchestrates all required tasks and their respective required task(s) and then processes everything for you. This approach makes the maintenance of tasks very easy, since you only have to add or remove required tasks from any task and luigi handles the rest for you, while you don’t have to worry about the computer science behind it too much. But although luigi takes a lot of mental load off of you, it also requires strategies to handle common situations that you may find yourself in.

Upgrading OJS with the ojs_updater

labs@ub.uni-frankfurt.de (UBLabs) — Wed, 21 Dec 2022 12:07:00 +0000

Introduction

At the University Library Frankfurt, we currently host 21 OJS journals, with more to come. Since we apply a strategy that runs only a single journal within an OJS instance, we have to maintain 21 different OJS instances. In order to maintain and manage this multiplicity, we found it important to come up with structures on the server and helper tools.

Especially the process of updating a journal instance can be quite tedious, since it involves multiple manual steps and can cause problems when forgetting something in the process. Due to these manual steps, the upgrade of a single OJS instance can take up to 20 minutes – and longer, if something doesn’t go according to plan.

Selecting subject specific records from the Bielefeld Academic Search Engine (Part 1)

labs@ub.uni-frankfurt.de (UBLabs) — Fri, 10 Jun 2022 00:00:00 +0100

Introduction

Since 2004, the Bielefeld Academic Search Engine (BASE) offers an aggregated metadata search over scientific publications. By indexing well over 300 million records (60% of which are open access¹) from almost 10,000 international repositories, it provides researchers with an invaluable access to publications. In addition to offering a VuFind based discovery system, the aggregated data is also available via both a live search API and an OAI-PMH interface to enable re-use by 3rd party services.² As a general search engine for scientific documents, the indexed publications are not restricted to any academic discipline or field of research. And while BASE retains available subject terms and – to some extent – even tries to automatically assign Dewey Decimal Classification to a subset, it’s not always straight forward to find resources that pertain to a specific field. That’s where 3rd party services can come into play and try to cater to the needs of specific areas of research. To provide such a service, however, a method of selecting a subject specific subset is required.³