ntqr.r2.evaluators ================== .. py:module:: ntqr.r2.evaluators .. autoapi-nested-parse:: Evaluators for binary classification tests (R=2). Classes: SupervisedEvaluation ErrorIndependentEvaluation MajorityVotingEvaluation Functions: Misc variables: Attributes ---------- .. autoapisummary:: ntqr.r2.evaluators.data_sketch Classes ------- .. autoapisummary:: ntqr.r2.evaluators.SupervisedEvaluation ntqr.r2.evaluators.ErrorIndependentEvaluation ntqr.r2.evaluators.MajorityVotingEvaluation Functions --------- .. autoapisummary:: ntqr.r2.evaluators.turn_numerical Module Contents --------------- .. py:function:: turn_numerical(val) Turns exact values into numerical ones. :param val: DESCRIPTION. :type val: TYPE :rtype: None. .. py:class:: SupervisedEvaluation(label_counts: ntqr.r2.datasketches.TrioLabelVoteCounts) Evaluation for experiments where the true labels are known. .. py:attribute:: vote_patterns .. py:attribute:: pairs :value: ((0, 1), (0, 2), (1, 2)) .. py:attribute:: label_counts .. py:attribute:: evaluation_exact .. py:attribute:: evaluation_float .. py:method:: prevalences() Calculate the prevalences of the two labels. :returns: Mapping from labels to percentage of appearance in the test. :rtype: Mapping[Label, Fraction] .. py:method:: classifier_label_accuracy(classifier: int, label: ntqr.r2.datasketches.Label) Compute classifier label accuracy. .. py:method:: other_label(label: ntqr.r2.datasketches.Label) Return the other binary classification label given label. .. py:method:: pair_label_error_correlation(pair, label) Calculate the label error correlation a classifier pair. .. py:method:: three_way_label_error_correlation(triplet, label) Calculate the label error correlation a classifier pair. .. py:class:: ErrorIndependentEvaluation(vote_counts: ntqr.r2.datasketches.TrioVoteCounts) Evaluate three binary classifiers assuming they are error independent. :returns: * *Absent labeled data, there are two logically consistent solutions* * *given only their decision voting frequencies. For binary classification,* * *this means that there are 2 possible points in evaluation space that* * *can possibly explain the test results. The ground truth evaluation is* * *one of these points -- if the assumption of error independence is true.* * *The exact algebraic results have a unique virtue that few alarm systems* * *have - it can warn about the failures of its own assumption of error* * *independence. If the two possible solutions for the 'a' label prevalence* * *return an unresolved integer square root - the classifiers are error* * *correlated in the evaluation.* * *In version 0.2 the math needed to take handle the almost certain detection* * *of error correlation will be added. It is already being built as* * *can be seen in ntqr.r2.postulates file where the postulates related to* * *computing the error correlation have been expressed using the SymPy* * *package.* * *Warnings* * *---------* * *A. The ntqr package uses a notion of 'error independence' that is* * *different than the one most familiar in the ML/AI community. There are* * *many notions of independence in mathematics. In the context of ML/AI* * *papers/discussions, the term 'error independence' is taken to be* * **A.1. Functional independence of distributions** (*P(x, y) = P(x)P(y)*) * *The one used in the ntqr package is sample defined since there is no* * *probability theory used in its logic. For that reason, you must define a* * *set of error correlation parameters. 'Error independence' in the ntqr* * *package means* * *A.2. pair_label_correlations = 0, trio_label_correlations = 0, ...* * *It is best to think of 'error independence' in the ntqr package as a* * *property that belongs to the classifiers AND the test they took.* * *B. This class currently assumes that the observed classifier* * *vote counts supplied by the user are not fake. The set of all valid* * *observations from a classification test is much smaller than the set* * *of all sets of eight positive integers. Future versions of the ntqr* * *package will implement the algebraic geometry needed to detect when* * *TrioVoteCounts objects are not explainable as observations from a* * *classification test.* * *The error independent solution can fail if, in fact, the classifiers* * *are highly correlated on the test being evaluated. Tests can fail.* * *Future versions will have implemented the exceptions.* -- 1. PrevalenceImaginaryException 2. NoSolutionException * *The PrevalenceImaginaryException is a iron-clad detection of highly* * *correlated classifiers. Its main utility will be in "warning light"* * *applications in AI safety.* * *The NoSolutionException means that NO independent system can possibly* * *explain the observations. There are two different reasons for this -* * *higher error correlations, or the data sketch is fake. Distinguishing* * *between the two comes down to the same computation of error correlation.* .. py:attribute:: vote_counts .. py:attribute:: vote_frequencies .. py:attribute:: evaluation_exact .. py:attribute:: evaluation_float .. py:method:: alpha_prevalence_quadratic_terms() Calculate the coefficients of the 'a' label prevalence quadratic. If the quadratic is represented as: a * (P_a)**2 + b * P_a + c then, a = terms[2], b = terms[1], c = terms[0]. The quadratic is written in the 'standard' way seen in algebra textbooks. Be careful to not mistake the 'a' or 'b' coefficients described here with the two labels being used for classification - currently implemented as ('a', 'b'). .. py:method:: alpha_prevalence_estimates() Calculate the prevalence of the alpha label. Since the quadratic equation has ordered solutions by the plus/minus operations, we arbitrarily return the 'a' label less than 50% solution first. .. py:method:: classifier_a_label_accuracy(classifier: int, a_prevalence) Calculate classifier 'a' label accuracies. :param classifier: One of (0, 1, 2). :type classifier: int :param a_prevalence: :type a_prevalence: Sympy expression :rtype: The a label accuracy given the a_prevalence value .. py:method:: classifier_b_label_accuracy(classifier: int, a_prevalence) Calculate classifier 'b' label accuracies. :param classifier: One of (0, 1, 2). :type classifier: int :returns: * *Two possible logically consistent estimates for P_{i,b} given the* * *test error independence assumption.* .. py:class:: MajorityVotingEvaluation(vote_counts: ntqr.r2.datasketches.TrioVoteCounts) Evaluate three binary classifiers using majority voting. Majority voting can be used to carry out evaluation algebraically. Typically, majority voting is used with the assumption that the crowd is always right. In the context of safety, however, that the crowd is always wrong is an equally valid a-priori assumption. Hence, this class returns TWO evaluations. The first assuming the crowd is always right and the second assuming they are always wrong. Its main virtue is that it is simple and rock solid - always returns logically consistent evaluations. .. py:attribute:: vote_patterns .. py:attribute:: vote_counts .. py:attribute:: vote_frequencies .. py:attribute:: labels :value: ('a', 'b') .. py:attribute:: majority_right_vote_patterns .. py:attribute:: majority_wrong_vote_patterns .. py:attribute:: evaluation_exact .. py:attribute:: evaluation_float .. py:method:: compute_vote_pattern_evaluation(vote_patterns, flip) .. py:method:: prevalences(vote_patterns) Compute label prevalences in the test. .. py:method:: classifier_label_accuracy(classifier, vote_patterns, label, flip) Compute the label accuracy for classifier. .. py:method:: to_float(sol) .. py:data:: data_sketch