Ap on UBLabs

Data Engineering with luigi - Lessons learned

labs@ub.uni-frankfurt.de (UBLabs) — Thu, 17 Aug 2023 00:00:01 +0200

Introduction

At the UB JCS, we make extensive usage of the Python luigi framework for data engineering. The framework is capable of handling thousands of tasks, calculating non-circular task dependencies, and run over days. Additionally, it provides a convenient web control panel to see, e.g. the task dependencies in a tree diagram or start specific tasks.

Although luigi itself supports the user already by enforcing a very specific structure, there are still some things to consider when designing a data pipeline with luigi (for a general introduction, see in a previous post). In this post, I present ideas, that I learned while using luigi. Since luigi is a heavily object-oriented framework, some approaches in this post rely naturally on Software architecture patterns.

Common engineering strategies in luigi

labs@ub.uni-frankfurt.de (UBLabs) — Thu, 29 Jun 2023 14:00:00 +0200

Introduction

For many automated data processing tasks within the context of the Specialised Information Services (FID) at the University Library Frankfurt, we use the Python package luigi. This package proves especially useful when a task (e.g. the loading of data into a database) depends on the work of other tasks that have to run successfully, before the next task starts (e.g. first you need to download the data). luigi orchestrates all required tasks and their respective required task(s) and then processes everything for you. This approach makes the maintenance of tasks very easy, since you only have to add or remove required tasks from any task and luigi handles the rest for you, while you don’t have to worry about the computer science behind it too much. But although luigi takes a lot of mental load off of you, it also requires strategies to handle common situations that you may find yourself in.

Upgrading OJS with the ojs_updater

labs@ub.uni-frankfurt.de (UBLabs) — Wed, 21 Dec 2022 12:07:00 +0000

Introduction

At the University Library Frankfurt, we currently host 21 OJS journals, with more to come. Since we apply a strategy that runs only a single journal within an OJS instance, we have to maintain 21 different OJS instances. In order to maintain and manage this multiplicity, we found it important to come up with structures on the server and helper tools.

Especially the process of updating a journal instance can be quite tedious, since it involves multiple manual steps and can cause problems when forgetting something in the process. Due to these manual steps, the upgrade of a single OJS instance can take up to 20 minutes – and longer, if something doesn’t go according to plan.

Maintaining applications with external API dependencies with software tests

labs@ub.uni-frankfurt.de (UBLabs) — Sun, 31 Jul 2022 00:00:00 +0000

Introduction

When talking about software maintainability, we always imply that you can extend or change a given software (even by another person than the original maintainer) without breaking the code or introducing bugs. This premise assumes that there is a good test suite making sure that your software still works as intended after the change (or with a new version of an external dependency). For example, the complexity of the software in the BIOfid software framework, both in the backend and the frontend, can only be tamed by applying tests that make sure that the code works as intended - now and in the future.

Getting the (Semantic) Sense out of a User Query

labs@ub.uni-frankfurt.de (UBLabs) — Wed, 15 Sep 2021 08:00:00 +0000

The BIOfid Semantic Search

Within the BIOfid-project, we create a semantic search portal (hereafter “BIOfid portal”) to help our users to access legacy biodiversity literature more easily. Hence, since the BIOfid portal has a deeper “understanding” of both the texts and the included species, it allows the users to get more relevant documents. Moreover, the BIOfid portal interprets the user query and transform it ad hoc into a graph database query, to learn more about their intention.