Q. Which measures reflect the effectiveness of a biometric authentication system?

False Acceptance Rate (FAR)
The FAR is the frequency that a non authorized person is accepted as authorized.  Because a false acceptance can often lead to damages, FAR is generally a security relevant measure. FAR is a non-stationary statistical quantity which does not only show a personal correlation, it can even be determined for each individual biometric characteristic (called personal FAR).
False Rejection Rate (FRR)
The FRR is the frequency that an authorized person is rejected access.  FRR is generally thought of as a comfort criteria, because a false rejection is most of all annoying. FRR is a non-stationary statistical quantity which does not only show a strong personal correlation, it can even be determined for each individual biometric characteristic (called personal FRR).
Failure To Enrol rate (FTE, also FER)
The FER is the proportion of people who fail to be enroled successfully. FER is a non-stationary statistical quantity which does not only show a strong personal correlation, it can even be determined for each individual biometric characteristic (called personal FER). Those who are enroled yet but are mistakenly rejected after many verification/identification attempts count for the Failure To Acquire (FTA) rate. FTA can originate through temporarily not measurable features ("bandage", non-sufficient sensor image quality, etc.). The FTA usually is considered within the FRR and need not be calculated separately, see also FNMR and FMR.
False Identification Rate (FIR)
The False Identification Rate is the probability in an identification that the biometric features are falsely assigned to a reference. The exact definition depends on the assignment strategy; namely, after feature comparison, often more than one reference will exceed the decision threshold.

Further Implicit Measures

False Match Rate (FMR). The FMR is the rate which non-authorized people are falsely recognized during the feature comparison. In contrast to the FAR, attempts previously rejected due to poor (image-) quality (Failure to Acquire, FTA) are not accounted for. Whether a falsely recognized biometric characteristic leads to increases in FAR or FRR depends upon the application. (There are applications, which define a successful recognition as a rejection, when, for example, double release of identification cards for a person with a false identity is prevented by comparing the actual reference features with the centrally stored reference features of all cards released so far.)

False Non-Match Rate (FNMR). The FNMR is the rate that authorized people are falsely not recognized during feature comparison. In contrast to the FRR, attempts previously rejected due to poor (image-) quality (Failure to Acquire, FTA) are not accounted for. Whether a falsely recognized biometric characteristic leads to increases in FAR or FRR depends upon the application.

Q. How is the Failure-to-Enrol Rate (FER/FTE) defined in detail?

Due to the statistical nature of the failure-to-enrol rate, a large number of enrolment attempts have to be undertaken to get statistical reliable results. The enrolment can be successful or unsuccessful. The probability for lack of success (FER(n)) for a certain person is measured:
FER(n) =

Number of unsuccessful enrolment attempts for a person (or feature) n


Number of all enrolment attempts for a person (or feature) n

These values are better with more independent attempts per person/feature. The overall FER for N participants is defined as the average of FER(n):
FER =

1


N

N

q

n=1

 FER(n)

The values are more accurate with higher numbers of participants (N). Alternatively, the median value may be calculated.

Finally, the result of an enrollment attempt has to be defined exactly:
  • An enrolment attempt is successful if the user interface of the application provides a "successful"- or "finished" message.
  • An enrollment attempt is unsuccessful if the user interface of the application provides an "unsuccessful" message.
  • In cases where no defined completion is available, a fixed enrollment time interval has to be given to ensure comparability. If the time interval has expired the enrollment attempt is counted unsuccessful.

Q. What needs to be considered in the definition of FRR?

Even though the false rejection rate, FRR, is intuitively easy to understand, there can be many problems when trying to fix an unequivocal or universal definition.  The following must be taken into account:
  • The FRR is a statistical value whose measurement accuracy depends on the number of measurements.  Now the FRR is not only dependent on the biometric system, but on the users as well.  There is thus a personal FRR.  If one wants to deal with large numbers of people, it is important that the end result is not negatively affected by an individual.  Such could occur when the number of attempts per person differs.  This problem can be avoided, if one first identifies each personal FRR curve and calculates the mean from those (or uses the median, but this provides different values!).
  • The exact meaning of rejection must be clarified.  Here for example, the total number of recognition attempts before the final assessment of a failed recognition play a role.  There are systems, which can continuously process a verification in real time.  Here a verification time slot is offered.
  • Many biometric systems reject a verification due to poor picture quality (e.g., dirty or worn down fingers in a fingerprint verification, noisy surroundings in a voice recognition, poor lighting in a facial recognition, or sensor problems).  When such problems are not due to a faulty operation, rejections due to picture quality problems are still false rejections.  The user is indifferent to the reason for false rejections.
  • Even the personal FRR can vary with time.  It sinks, for example, when one frequently uses the system, which can learn to avoid false rejections.  In such cases, it is only reasonable for comparisons to determine FRR during learning phases.
In the case that a liveness/fake recognition is also used, this needs to be considered when determining the FRR.

Q. How is FRR defined in detail?

Due to the statistical nature of the false rejection rate, a large number of verification attempts have to be undertaken to get statistical reliable results. The verification can be successful or unsuccessful. In determining the FRR, only fingerprints from successfully enroled users are considered. The probability for lack of success (FRR(n)) for a certain person is measured:
FRR(n) =

Number of rejected verification attempts for a qualified person (or feature) n


Number of all verification attempts for a qualified person (or feature) n

These values are better with more independent attempts per person/feature. The overall FRR for N participants is defined as the average of FRR(n):
FRR =

1


N

N

q

n=1

 FRR(n)

The values are more accurate with higher numbers of participants (N). Alternatively, the median value may be calculated.

Important: the determined FRR includes both poor picture quality and other rejection reasons such as finger position, rotation, etc. in the reasons for rejection.  In many systems, however, rejections due to bad quality are generally independent of the threshold.  The FRR after quality filtering is similarly defined:

Number of rejected "qualified" attempts

Total number of "qualified" attempts

An FRR defined as such, generally yields better data sheet values, but these lower numbers are not reflected in reality from a user's perspective.

Finally, the result of a verification attempt has to be defined exactly:
  • A verification attempt is successful if the user interface of the application provides a "successful" message or if the desired access is granted.
  • A verification attempt counts as rejected if the user interface of the application provides an "unsuccessful" message.
  • In cases of no reaction, a verification time interval has to be given to ensure comparability. If the time interval has expired the verification attempt is counted unsuccessful.

Q. What needs to be considered in the definition of FAR?

Similar to the FRR, the false acceptance rate can be defined differently.
  • The FAR is a statistical value, whose measurement accuracy depends on the number of measurements. The FAR depends not only on the biometric system, but on the user as well.  There is also a personal FAR. If one wants to deal with large numbers of people, it is important that one individual does not negatively affect the end result.  Such could occur when the number of attempts per person differs.  This problem can be avoided, if one first identifies each personal FAR curve and calculates the mean from those (or uses the median, but this provides different values!).  In determining FAR, it is generally easier to limit the number of recognition attempts to 1 per person.  Further attempts per person will smooth out the ROC graph, but add little to the statistical significance.
  • If the biometric system has picture quality management, which happens to reject a false user due to poor picture quality (click here for example) already before verification, this is of course a correct rejection, and leads to an improved FAR.
  • Strong behavioral biometric features (e.g., voice or signature) are often purposefully forged or copied. In investigating FAR, it needs to be determined whether tests simply recognize foreign features or also attempted forgeries.  This difference can be serious.

Q. How is FAR defined in detail?

Due to the statistical nature of the false acceptance rate, a large number of fraud attempts have to be undertaken to get statistical reliable results. The fraud trial can be successful or unsuccessful. The probability for success (FAR(n)) against a certain enroled person n is measured:
FAR(n) =

Number of successful independent fraud attempts against a person (or characteristic) n


Number of all independent fraud attempts against a person (or characteristic) n

These values are more reliable with more independent attempts per person/characteristic. In this context, independency means that all fraud attempts have to be performed with different persons or characteristics! The overall FAR for N participants is defined as the average of all FAR(n):
FAR =

1


N

N

q

n=1

 FAR(n)

The values are more accurate with higher numbers of different participants/characteristics (N). Alternatively, the median value may be calculated.

Whether a correct rejection is due to poor picture quality or really to a person's unauthorized status, remains (just like in practice) extraneous.

The crucial number for the determination of statistic significance is the number of independent attempts.  Obviously, two attempts in which alternately one person is the reference and another places the request, are not independent of each other. Likewise, multiple attempts from one unauthorized user are considered dependent and therefore have less meaning for statistical significance.

Finally, the following items have to be settled, or defined, respectively:
  • What is a fraud attempt?
  • How is the result of a fraud attempt defined exactly?
Usually, during FAR determination, a fraud attempt is an attack using the characteristics of non-authorized persons. This, however, pretends a high security which may not be present since there are a lot of further possibilities for promising attacks.
  • A fraud attempt is successful if the user interface of the application provides a "successful" message or if the desired access is granted.
  • A fraud attempt counts as rejected if the user interface of the application provides an "unsuccessful" message.
  • In cases where no "unsuccessful" message is available, a verification time interval has to be given to ensure comparability. If the verification time interval has expired the fraud attempt is counted unsuccessful.

Tags: , , , ,