Content

Introductory

One of the basic, regularly recurring tasks of maintaining our Debian or Ubuntu-based servers and systems is updating the package repositories and installed software. An apt update and apt upgrade (or apt full-upgrade) command usually runs quickly and smoothly, ensuring the security and up-to-dateness of our system. However, as the saying goes, sometimes the devil is in the details.

I was just doing the usual update round on my server these days, which has been a bit behind lately, so I expected that several packages would be updated. In such a case, especially if we are using external, third-party repositories (such as the popular Sury.org repository a PHP versions), we are more likely to encounter unexpected messages and warnings. And so it happened: the update process stopped with interesting messages that drew attention to an important change related to the management of the signing keys of the package repositories.

In this description, I will explain step by step what messages I encountered, what the underlying change is, and how we can fix the situation so that updates continue to run smoothly in the future.

The problem - signed-by warnings when updating

During package updates, we often encounter various warnings that provide useful information about immediate or later actions that may be required.

Among these, in this case, the following emerged:

Warning about apt-listchanges

During package updates, we often come across a tool called apt-listchanges. This is a useful utility that displays the changelog of packages waiting to be updated before the actual update takes place. This gives us an overview of what is new and what changes are coming in the new package versions. The messages from apt-listchanges are usually divided into "News", "Changes" and "TODO" sections.

The warning we experienced appeared in the "News" section, right at the beginning of the update process. Here is a screenshot showing it:

The warning message is clearly visible in the image, which debsuryorg-archive-keyring The most important parts of the message are as follows:

• global APT key will be removed in the next update of the package: This sentence indicates the essence of the change: what was previously the /etc/apt/trusted.gpg.d/debsuryorg-archive.gpg The globally trusted APT key stored in this location will disappear with the next package update.
• You need to manually add: [signed-by=/usr/share/keyrings/debsuryorg-archive-keyring.gpg]: This explains what to do: we need to manually add the [signed-by=/usr/share/keyrings/debsuryorg-archive-keyring.gpg] directive to the APT to our resource lists.

The message also refers to a To the Debian Wiki page also, where we can find more information on the subject. It is important to understand that this not an error message in the strict sense of the word, but rather a warning, which prepares us for an upcoming change and also communicates the necessary steps.

Debconf dialog warning

It is possible that during the upgrade process, you will encounter the warning not in the apt-listchanges tool, but directly while installing or upgrading packages. In this case, debconf The configuration system called debconf displays the message in a dialog box. debconf allows the user to specify options or confirm certain decisions during the installation of packages.

The warning about the signed-by directive appears in the debconf window when the debsuryorg-archive-keyring (or a similar keyring) is updated and the system detects that the source list entry for the repository does not yet contain the appropriate "signed-by=" directive. Here is an example of this that just happened to me:

The debconf message is essentially the same warning as the apt-listchanges message, just in a different form, during package installation. The action we need to take is the same: we need to modify the source list.

Why do we get these warnings? The background

Now that we've seen the warning messages and their content, it's time to examine why this change is necessary. In the following, we'll review the old and new ways of managing repository keys, and why the signed-by directive is a more secure solution than before.

The old, outdated method: /etc/apt/trusted.gpg.d

On Debian-based systems, GPG keys for third-party repositories are traditionally stored in /etc/apt/trusted.gpg.d/ This directory contains the key files that the system uses to verify the authenticity of packages during updates.

When a new repository was added to the system, the associated GPG key was usually downloaded and copied into this directory. This made the key globally trusted, meaning that the system accepted this key for all repositories to authenticate packages.

This method was simple and easy to use, but it is now considered obsolete and will be completely replaced by the signed-by directive in the future.

The new, more secure method: The signed-by directive

Instead of the old, global key management, APT has introduced a new, more secure method for authenticating repositories: the signed-by directive. This directive is used in source lists (the / Etc / apt / sources.list in file, or in /etc/apt/sources.list.d/ directory) as part of the container definition.

The signed-by directive explicitly associates a given repository with a given GPG key. This means that packages from the repository will only be accepted if they are signed with the specified key.

The syntax of the directive is as follows:

deb [signed-by=/elérési/út/a/kulcshoz.gpg] https://tároló.címe/ disztribúció komponens

As we saw in the previous examples (and warning messages), signed-by requires the path to the GPG key file. This path is usually located in the /usr/share/keyrings/ directory, where the key package (e.g. debsuryorg-archive-keyring) installs the key.

With this method, the system can verify the authenticity of packages much more accurately and avoid the security risks arising from global key management.

Why is the new method safer?

The introduction of the signed-by directive significantly increases the security of the system in terms of authenticating package repositories. The main reasons for this are as follows:

• Reduced attack surface: If a key is compromised for some reason (such as being stolen or obtained by unauthorized parties), it will only affect the specific repository to which the key was assigned using the signed-by directive. In contrast, with the old method, a compromised key could be used to forge a package from any trusted repository.
• Explicit trust: Using signed-by requires explicit trust: the administrator consciously and unambiguously trusts a given key for a given container. With the old method, it was easier to accidentally trust a key without being aware of its scope.
• Easier auditing: Using signed-by makes it much easier to control which keys are used for which repositories. The trust relationship is clearly defined in the source list files. With the old method, it was harder to keep track of which key belongs to which repositories.
• Principle of least privilege: The new method follows the principle of least privilege: a key has only the minimum necessary privileges (only to authenticate packages in a given repository).

Overall, the signed-by directive provides a much more precise, secure, and transparent way to manage package repository keys, which is essential for maintaining system integrity.

General APT change, not repository specific

This is a general trend Towards more secure management of package repositories in the APT package manager on Debian/Ubuntu systems. The goal is to make key management for third-party repositories as accurate and secure as possible.

This means that in the future, all third-party repositories will likely require the signed-by directive. Therefore, it is a good idea to review our source lists and check if the entries for third-party repositories include the "signed-by=" directive. If not, it is a good idea to add it as soon as possible to avoid future problems and warnings.

Solution: Add the signed-by directive to the source list

Now that we understand why we need to use the signed-by directive, it's time to get down to practical implementation. In the following, we'll look at how we can add the signed-by= option to our source lists to ensure that future updates go smoothly. The process is not complicated, and you can easily fix the problem by following these steps.

Finding the affected source list (.list) file

To add the signed-by directive, we first need to find the source list file that contains the definition of the repository we want to update. Source list files store the path and other settings for APT to use to access the package repositories.

The source lists can be found in two places:

• A In /etc/apt/sources.list file: This is the main source list file, which usually contains the default repositories for Debian systems.
• A In the /etc/apt/sources.list.d/ directory: This directory contains separate files with the extension .list that store definitions for third-party containers. This is the more common solution for external containers.

As we saw in the previous examples, the Sury.org repository definition is /etc/apt/sources.list.d/php.list In most cases, the correct file is likely to be in /etc/apt/sources.list.d/ It will be in a directory with a name similar to the name of the repository (for example, sury.list, php.list, nodesource.list, etc.). The file will definitely have a .list extension.

Although less common, it is also possible that the container definition is / Etc / apt / sources.list It all depends on how you added the repository to your own list.

Identifying the correct GPG key file path

To use the signed-by directive correctly, we need to specify the exact path to the GPG key file. The good news is that this path can usually be found in warning messages.

As shown in the images you saw earlier, the correct path for the debsuryorg-archive-keyring package is as follows:

/usr/share/keyrings/debsuryorg-archive-keyring.gpg

It is important that the path you provide points to a file with the .gpg extension.

The key file is usually installed by a package (such as debsuryorg-archive-keyring). These packages place the key file in the /usr/share/keyrings/ directory. This directory is the standard location for storing GPG keys.

Correctly modify the source list file

Once we have found the correct source list file and identified the path to the GPG key file, it is time to edit the file.

Open the file for editing: Use nano command to open the file:

sudo nano /etc/apt/sources.list.d/php.list

Incorrect approach (to be avoided!): One of the most common mistakes we can make is simply adding a new line to the file with the signed-by directive. Here is an example of how not to do it (shown in the image below):

This is not good because the repository database contains the same repository definition twice: once with the old method and once with the new method. This is unnecessary and can cause problems during updates.

Correct approach: Instead of adding a new line, we need to modify the existing line, or comment out the original and create a new one with the signed-by option. Here's how to do it correctly:

In the image, you can see that the original line has been commented out (inactivated), and below it is the new line with the signed-by directive. Instead of commenting out the original line, you can also delete it, whichever you prefer.

The signed-by directive must appear after the word deb, in square brackets:

deb [signed-by=/elérési/út/a/kulcshoz.gpg] https://tároló.címe/ disztribúció komponens

Save the file.

Checking for changes (apt update)

After saving the source list file with the signed-by directive, we need to verify that the change was successful. To do this, run the following command:

sudo apt-get update
sudo apt-get upgrade

We update our repositories with the commands. If we added the signed-by directive correctly, then the key warnings should not appear when running apt update.

If you still get warnings after adding the signed-by directive, you may have made an error (for example, you may have misspelled the path to the key file or there may be syntax errors in the source list). In this case, double-check your configuration and make sure you have entered everything correctly.

Additional tasks: Handling and cleaning of held packages

After adding the signed-by directive, we have solved this part of the upgrade problems. However, there may still be some work to be done after updating the repositories with the sudo apt-get upgrade command.

In this section, we will review the example after the update.  how to update held packagesand how to get rid of unnecessary packages.

Upgrading held packages (apt full-upgrade)

After adding the signed-by directive and running sudo apt-get update , the sudo apt-get upgrade command may still show that some packages are kept back. This means that the packages were not updated for some reason.

The most common reason is that upgrading a package would require installing new dependencies or removing existing packages. The apt-get upgrade command does not perform such operations by default, as this would involve major changes to the system.

In the previous tutorial linked above, we reviewed updating with the --with-new-pkgs switch, as well as apt-get install However, if there are many retained packages on the system, you should use the sudo apt full-upgrade (or the older sudo apt-get dist-upgrade) command to upgrade them all at once. This command intelligently handles dependency issues, installing new required packages and removing unnecessary ones.

It is very important to review the proposed changes before running the full-upgrade. The command will list which packages will be upgraded, which will be newly installed, and which will be removed. Make sure that the changes are acceptable to you before approving the installation.

If everything goes well, you should see output like this, with the APT progress bar at the bottom of the terminal:

Removing unnecessary packages (apt autoremove)

When you run sudo apt full-upgrade, the system may install new packages to satisfy dependencies. However, after the upgrade is complete, these packages may no longer be needed.

To remove unnecessary packages, you can use the apt autoremove command. This command automatically identifies packages that were installed as dependencies but are no longer needed:

sudo apt autoremove

Before running the command, it is worth reviewing the list of packages that autoremove is trying to remove. In some cases (for example, if you have installed a new kernel), old kernel packages may also be included in the list. If you are not sure whether a package is unnecessary, it is better not to remove it.

Final check

After running the sudo apt full-upgrade and sudo apt autoremove commands, it's a good idea to run the sudo apt upgrade command again to make sure the system is fully up to date and there are no held packages.

If the sudo apt upgrade command prints the following:

0 frissített, 0 újonnan telepített, 0 eltávolítandó és 0 nem frissített

then everything is fine, the system is fully up to date.

Finally, we check if the services running on our system (such as websites that use PHP) are working properly after the updates.

Troubleshooting

If for some reason a system update were to remove our GPG key, such as an apt autoremove command as part of a larger update, then a curl We can quickly obtain it manually and reinstall it using the command. Using the Sury.org repository as an example, use the following command:

curl -fsSL https://packages.sury.org/php/apt.gpg | gpg --dearmor -o /usr/share/keyrings/debsuryorg-archive-keyring.gpg

Where the parameters are:

• -F: fail silently: if the download encounters an error (e.g. 404), it does not display an HTML error and does not print unnecessary data.
• -s: silent: does not write anything to standard output (no status line, download info, progress bar, etc.).
• -S: show error: combined with -s switch, it still displays the error message if there is a problem. (It is useful to have only successful downloads be silent.)
• -L: follow redirects: if the server sends an HTTP 301/302 redirect, it will automatically follow it to its destination.

In summary: this combination will download the file reliably, silently but not silently, and will handle redirection as well.

The "| gpg --dearmor..." part converts the downloaded ASCII-armored GPG key (which is text, e.g. -----BEGIN PGP PUBLIC KEY BLOCK-----), into a binary .gpg format that apt can use for the "signed-by=" directive.

Switches:

• --dearmor: conversion from ASCII-armored (text) format to binary GPG format.
• -he : specifies the name of the output file (where the result will be placed).

Make sure the output file name matches the path previously set in the list file, so the next update can run smoothly.

Of course, we should download the GPG key from the source corresponding to our own package repository, this was just an example, continuing with the Sury.org package repository, in case our GPG key also suffers from cleaning up when removing outdated dependencies.

Here in the picture, this is exactly what happened to me: after a major update and a complete server restart,

apt autoremove --purge

command, my GPG key fell victim to an outdated dependency cleanup, which I checked with the ls command, so after a manual download, the package update was restored.

Conclusion

In this article, we looked at how to deal with warnings about the signed-by directive displayed by the APT package manager. We examined the background of the problem, understood why it is important to manage keys accurately, and went through the solution step by step.

Using the signed-by directive is crucial to the security of our system, as it explicitly associates a container with a given key, thus minimizing the potential attack surface.

Please note that using the signed-by directive will become the standard way to handle third-party repositories in the APT package manager in the future. Therefore, it is recommended to review your source lists and make the necessary changes as soon as possible.

I hope this description helped you understand the problem and find a solution!