The most important category where the protective effect of products is concerned is the test against current online threats. This involves accessing known malicious websites or e-mails in order to test whether the protection product is able to ward off attacks.
In order to increase the statistical relevance of the tests, further analyses are carried out with regard to a large number of current threats. This involves decreasing the complexity of the test and in turn increasing the number of test cases many times over. This test refers to the static detection of files, which includes detection with signatures, heuristics and in-the-cloud queries. AV-TEST uses two different test sets to carry out these analyses:
Both test sets only use files that have been discovered and analysed by AV-TEST. In order to prevent the test sets from being influenced by the manufacturers in their favour, data from manufacturers are not incorporated into the tests. As a result, the independent analysis carried out by AV-TEST achieves a very high level of quality.
Typical actions are used to measure the influence on the system speed, for example:
Functions of the operating system, the protection program and other programs that may be disruptive are closed down prior to the beginning of the test. This includes both automatic updates and planned actions such as scans or backups. The tests are carried out on a limited number of computers that are identical in construction and have been verified as having the same system speeds in order to give all products the same chances.
Every individual action is carried out at least seven times so that a reliable average can be generated. If the standard deviation of the individual values exceeds a specific threshold, this indicates an error and the test is repeated in full. If the test achieves reliable values for all products and all test cases, these are compared with the reference system values and the difference is calculated. This difference then specifies the slowing down of the system in the case of the actions tested.
From 01/01/2016: Dennis Technology Labs is no longer operating.
In 2015, Kaspersky Lab participated in 94 independent tests and reviews. They were awarded 60 first places and achieved 77 top three finishes. The TOP 3 metric represents the aggregate scores achieved by almost 100 vendors in some of the security industry’s most vigorous independent tests and reviews. It’s much more meaningful than a ‘one hit wonder’ performance with a single product on a single test.