Stiftung Warentest evaluates Password Managers

In April 2022, we learned that Stiftung Warentest would soon benchmark our Password Depot - a password manager “Made in Darmstadt,” which has been developed and sold in Germany for over 20 years – against other competitors in this field.

Our product was awarded 1st rank – twice - by the Fraunhofer Institute on behalf of a well-known computer magazine more than 10 years ago, has achieved countless other test victories and was most recently subjected to a rigorous pen test by SySS GmbH, which it passed with the rating "very secure".

In the 07/2022 issue of Stiftung Warentest, we have now taken note of the fact that Password Depot was only rated "Sufficient" (= D).

How could this happen?

In the case of the current benchmark test of Stiftung Warentest, it was decided that a product is devalued, i.e. cannot achieve more than "Sufficient" or D, if it permits a single character as master password when creating new databases - i.e. regardless of how well or poorly it performs in all other test criteria.

This was the only reason for the "Sufficient" or D rating. Password Depot indeed permitted single-character master passwords until version 16.0.3 - but only after explicit security warnings.

Without this feature, Password Depot would very likely have received a "Good" or B rating based on the remaining test criteria.

The imminent devaluation due to the 1-character master password was communicated to us by Stiftung Warentest in advance (April 2022). As a result, we implemented an update on 26.04.22 and since then, a minimum length of 15 characters is enforced when creating new databases. In addition, at least three out of four character types must be used (upper/lower case letters, numbers, special characters). (A note on this feature: This is certainly a sensible default setting for completely inexperienced users, to protect them. Even if it might be annoying to have to use such complex passwords in certain situations, for example when testing the application or creating temporary databases.)

Unfortunately, this update was not taken into account by the testers for the final evaluation, although we had published the update within 24 hours after being informed about it, i.e. about 2 months before the release of the 07/2022 issue (see update to version 16.0.3 from 26/04/22).

Our competitor Avira, on the other hand, had a "dangerous security vulnerability", but its correction was still very much considered in the published test. It is not entirely understandable why this could not happen in our case. Upon further inquiry, we were informed that: "A security vulnerability is to be evaluated differently than a deliberate design decision."

Now, one can argue whether a password manager must be devalued if it permits a single character as master password despite explicit warnings to the user. However, one of the most important criteria, from our point of view, was not at all considered during the test:

To evaluate the security of any password manager, it would at least be necessary to investigate how it operates with the user's passwords and whether it leaves any traces in the working memory. Let's assume the following scenario: A user works with a password manager at the office. Next, they lock their password manager and leave the workplace for a short time - trusting that everything is fine. However, if a malicious third party accesses the user's PC for just a few minutes, they can read the memory of the test winner, 1Password, and access all their passwords in plain text - without this unfortunate user ever noticing: that is a serious security flaw!

Many password managers are unable to encrypt the user's data in working memory. In fact, this presents a highly complex challenge since Windows' memory management can only be manipulated and controlled to a limited extent. Surprisingly, this behavior did not lead to a downgrade for the "test winner" 1Password.

Stiftung Warentest's reasoning for this was as follows:

"On a system that has already been compromised, many scenarios are conceivable that could endanger the user’s security. To assume this as a basis for testing makes virtually any further statement about the potential security of individual products impossible."

We must strongly disagree with this outlandish argumentation. Especially on compromised systems, we must try to protect every user in the best possible way. Otherwise, we might as well store our passwords in Excel spreadsheets again.

Even in the case of a car accident, "many scenarios are conceivable that could endanger the safety of users." That's precisely why we need an airbag and ABS and all the other safety measures in vehicles. Following the argumentation of the testers of Stiftung Warentest, these security mechanisms would not make sense in case of an accident, because “the horse has already bolted”.

Password Depot, on the other hand, not only encrypts data in the working memory. It also provides protection against malware on the user's PC - malware that tries to impersonate Password Depot's addon, for example - by enabling communication between Password Depot and its addon to be password-protected. Or, to pick another feature, it blocks access when trying to copy too many passwords to the clipboard in too short an amount of time.

We sincerely hope that we were able to communicate to you why, despite this - at first glance - bleak test result, you do not need to worry about the security of your data in Password Depot and can continue to trust Password Depot without any restrictions.

PS: The screenshots below demonstrate how dangerously easy it is to read your passwords from the test winner.

A sample entry in 1Password:

 

Although the program is locked, it is possible to read both the password and the username (conveniently even with the matching labels "Password" and "Username").