ntqr.evaluations ================ .. py:module:: ntqr.evaluations .. autoapi-nested-parse:: Module for super classes of R-axioms based evaluations. Logically consistent evaluations are defined by the size of an ensemble and its associated axioms. Classes ------- .. autoapisummary:: ntqr.evaluations.AnswerKeyQSimplex ntqr.evaluations.PossibleSet ntqr.evaluations.ConsistentSet ntqr.evaluations.MLabelResponseSimplexes ntqr.evaluations.MVariety ntqr.evaluations.MVarietyTupleDict ntqr.evaluations.MAxiomsVarieties ntqr.evaluations.SingleClassifierEvaluations Functions --------- .. autoapisummary:: ntqr.evaluations.generate_simplex_points ntqr.evaluations.random_simplex_point ntqr.evaluations.generate_observable_cube_points ntqr.evaluations.random_bounded_simplex_point Module Contents --------------- .. py:function:: generate_simplex_points(target_sum: int, n_vars: int) -> Iterable[collections.abc.Sequence[int]] Generator of tuples of integers that sum to target_sum. :param target_sum: Sum of the variables. :type target_sum: int :param n_vars: Number of variables. :type n_vars: int :returns: The points on the simplex obeying the target_sum. :rtype: Iterable[Sequence[int]] .. py:function:: random_simplex_point(target_sum: int, n_vars: int) -> collections.abc.Sequence[int] Generates a random point on the simplex. :param target_sum: Required target sum for the variables. :type target_sum: int :param n_vars: Number of variables. :type n_vars: int :returns: A single point on the simplex obeynig target sum. :rtype: Sequence[int] .. py:function:: generate_observable_cube_points(target_sum: int, maxs: collections.abc.Sequence[int]) -> Iterable[collections.abc.Sequence[int]] Generator of points on a simple inside the observable hypercube. The sum of decision events must equal the assumed label count in the answer key, but, in addition, each event count cannot be larger than the min of the label count OR the observed count of the event. This generator produces the points that obey the target sum and the individual max possible value for an event. Note that this is not enough to produce the consistent set as the product of each label generator -- they must obey the same observable count ACROSS all labels. :param target_sum: Sum of the variables. :type target_sum: int :param maxs: The individual max for each variable. :type maxs: Sequence[int] :returns: DESCRIPTION. :rtype: Iterable[Sequence[int]] .. py:function:: random_bounded_simplex_point(total_sum: int, bounds: collections.abc.Sequence[int]) -> collections.abc.Sequence[int] Uniformly samples non-negative integers x_i such that: 0 <= x_i <= bounds[i] and sum(x_i) == total_sum :param total_sum: Sum required. :type total_sum: int :param bounds: Bounds for each variable, must be equal to or less than total_sum. :type bounds: Sequence[int] :raises ValueError: If no points satisfy the given bounds and sum to total_sum. :returns: The sampled point. :rtype: Sequence[int] .. py:class:: AnswerKeyQSimplex(Q: int, labels: ntqr.Labels) Class to generate all answer-key simplex points. Given the test size, Q, and the labels (R=r) of them. This is a set of tuples of dimension r, the number of labels. It exists in (R-1) space since, by construction, we must have their sum always equal to Q. The number of possible qs is 1/(R-1)!*(Q+1)*(Q+2)*...*(Q+R-1). To summarize: - The number of labels in the answer key is a tuple of size R. Its value is unknown in unsupervised settings. - The number of such tuples is 1/(R-1)!(Q+1)... - This set exists in a (R-1) dimensional space inside the R space for the tuples. .. py:attribute:: Q .. py:attribute:: N .. py:attribute:: labels .. py:attribute:: R .. py:method:: qs() -> Iterable[tuple[int]] Generate all possible answer-key simplex points. :returns: Generator of all possible answer-key simplex points. :rtype: Iterable[tuple[int]] .. py:class:: PossibleSet(labels: collections.abc.Sequence[str], classifiers: collections.abc.Sequence[str]) Class for the possible set of evaluations for N classifiers classifying Q items into R labels. Given a point in the answer key Q-simplex, we can count how many group evaluations are possible before we see any test results. For small tests, it is also possible to generate all of the points in the possible set but it can quickly blow up so caution must be exercised when calling generators of possible evaluations. .. py:attribute:: labels .. py:attribute:: classifiers .. py:method:: set_count_at_ql(ql: collections.abc.Sequence[int]) -> int Computes the size of the possible set at given answer key ql point. The size of the possible set at any given answer key Q-simplex, point ql can be quite large for small tests and labels so this function carries out the exact integer representation of the count using SymPy's symbolic functionality. :param ql: The answer key Q-simplex point, values must correspond to the number of labels used to initialize the class. :type ql: Sequence[int] :returns: The integer count of the possible set at the given ql value. :rtype: int .. py:method:: sum_count_at_ql(ql: collections.abc.Sequence[int]) -> int Computes the sum of the possible values over the labels. This is useful for considering if random sampling from the possible set is possible. To randomly draw one evaluation from the possbile set, you would have to create the sum of the evaluations over the labels, not the product. :param ql: The answer key Q-simplex point, values must correspond to the number of labels used to initialize the class. :type ql: Sequence[int] :returns: The number of values that need to be generated to sample one point from the possible set. :rtype: int .. py:method:: set_generator(ql: collections.abc.Sequence[int]) -> Iterable[collections.abc.Sequence[int]] Generator of the possible set at given answer key ql point. Points are returned as tuples where variables are sorted by label then event. This generator should be used with caution since even reasonably sized tests can be quite large, of the order of (R^N)! where R is the number of labels and N the number of classifiers. :param ql: The answer key Q-simplex point, values must correspond to the number of labels used to initialize the class. :type ql: Sequence[int] :returns: Points are represented as tuples of ints where the label response variables have been sorted by label, then event. :rtype: Iterable[Sequence[int]] .. py:method:: random_points(ql: collections.abc.Sequence[int], n: int) -> set[collections.abc.Sequence[collections.abc.Sequence[int]]] Generates n random points in possible set at given ql. :param ql: Point on the Q-simplex. :type ql: Sequence[int] :param n: Number of points wanted. :type n: int :returns: Sequence of length n of possible evaluation points. :rtype: Sequence[Sequence[int]] .. py:method:: __repr__() .. py:class:: ConsistentSet(labels: collections.abc.Sequence[str], classifiers: collections.abc.Sequence[str], counts: Mapping[collections.abc.Sequence[str], int]) Class for N classifier evaluations logically consistent with the observed counts of the R^N ways they can agree/disagree when using R labels. This is a subset of of the possible set, smaller both in the dimension of its geometry (the number of indepent variables) and its count in that space. This is due to the observable counts for an event setting a ceiling for the possible value of the count of the same event **given** true label. .. py:attribute:: labels .. py:attribute:: classifiers .. py:attribute:: counts .. py:method:: max_value_at_ql(ql: collections.abc.Sequence[int], vars: collections.abc.Sequence[sympy.Symbol]) -> collections.abc.Sequence[int] Computes maximum value for each variable at given ql Q-simplex point. The maximum value possible for the count of a decision event by the ensemble given true label is the minimum of the assumed count of that label in the answer key and event observed count. :param ql: DESCRIPTION. :type ql: Sequence[int] :param vars: DESCRIPTION. :type vars: Sequence[sympy.Symbol] :returns: DESCRIPTION. :rtype: Sequence[int] .. py:method:: set_generator(ql: collections.abc.Sequence[int]) -> Iterable[collections.abc.Sequence[int]] Generator of the possible set at given answer key ql point. Points are returned as tuples where variables are sorted by label then event. This generator should be used with caution since even reasonably sized tests can be quite large, of the order of (R^N)! where R is the number of labels and N the number of classifiers. :param ql: The answer key Q-simplex point, values must correspond to the number of labels used to initialize the class. :type ql: Sequence[int] :returns: Points are represented as tuples of ints where the label response variables have been sorted by label, then event. :rtype: Iterable[Sequence[int]] .. py:method:: random_points(ql: collections.abc.Sequence[int], n: int) -> set[collections.abc.Sequence[collections.abc.Sequence[int]]] Returns n random points in the consistent set at given ql. :param ql: Assumed value of labels in the answer key. :type ql: Sequence[int] :param n: Number of random points wanted. :type n: int :returns: A random set of n points from the consistent set of evaluations at given ql point. :rtype: set[Sequence[Sequence[int]]] .. py:method:: correct_cuboid_generator(ql: collections.abc.Sequence[int]) -> Iterable[collections.abc.Sequence[collections.abc.Sequence[int]]] Generates the correct cuboid for each label given Q-simplex point, ql. A correct cuboid point is defined by the number of correct label answers for each classifier. Strictly speaking, this set does not have the geometry of a cuboid, but can be confined within one. This set is the basis for no-knowledge alarms since they consist of points where classifiers may have enough disagreements that one of them cannot be working properly at the specified ql value. :param ql: Assumed point in the Q-simplex, the count of labels in the answer key. :type ql: Sequence[int] :returns: Points of the form (R_{l_i, l_j, ...; l_true},...). The number of label corrects for each classifier and label. :rtype: Iterable[Sequence[Sequence[int]]] .. py:method:: random_correct_cuboid_points(ql: collections.abc.Sequence[int], n: int) -> collections.abc.Sequence[collections.abc.Sequence[collections.abc.Sequence[int]]] Random list of points on the correct label cuboids. Points may not be unique since they are marginalized counts of unique points on the consistent set. :param ql: Assumed count of each label in the answer key. :type ql: Sequence[int] :param n: Number of points sampled in the consistent set. :type n: int :returns: Sequence, n long, of points in the correctness cuboid. :rtype: Sequence[Sequence[Sequence[int]]] .. py:method:: __repr__() .. py:class:: MLabelResponseSimplexes(labels: collections.abc.Sequence[str], classifiers: collections.abc.Sequence[str], responses: Mapping[tuple, int], qs, m) Class to manage the response simplexes associated with each label. Each subset of sized-M for N classifiers has its own set of label response simplexes, one for each of the R labels in the test. These are the set of all possible values for statistics of aligned decisions by the members of the m-subset GIVEN true label. The a-priori logic of a test is that the sum of all possible responses by a m-subset must exactly equal to the Q_label, the count of the label in the answer key. This defines a simplex for a given label. The posterior logic is that the simplexes for any given value of M, M=m, have axioms that depend on all simplexes of value less than m. Thus the M=2 simplexes involve variables from the two M=1 simplexes that come from the classifier pair responses. The M=3 simplexes involve all M=2 simplexes, and all M=1 simplexes. And so on. This class is meant to internally manage that logical complexity by keeping track of all the variables that are needed for each simplex as well as the enclosing 'shells' that provide the values for the response variables. A separate class, MAxiomsVarieties, combines the functionality of this class and the ntqr.raxioms.MAxiomsIdeal class to compute the subset of possible label test responses that are logically consistent with the observed test results. .. deprecated :: 0.8 Use :class:`.ConsistentSet` instead. .. py:attribute:: labels .. py:attribute:: qs .. py:attribute:: m .. py:attribute:: classifiers .. py:attribute:: qad .. py:method:: _initialize_response_dicts() .. py:method:: _initialize_sympy_response_dict() .. py:method:: m_responses(m: int) -> Mapping[tuple, Mapping[tuple, int]] Observed responses by all m-sized subsets of the classifiers. Observed responses set the ceilings for any possible value for the label responses - responses given true label. For example, if we observe that two classifiers agreed on the same label some number of times, no possible evaluation of that agreement given true label can exceed this number. Observed responses create the 'ratchet' of evaluation. No label response variable can have a value larger than that of any subset of the classifiers responding similarly. :param m: Size of the subsets of the classifiers that will be used. :type m: int :returns: This is semantically of the form, Mapping[m-subset-classifiers, Mapping[m-decisions, observed count] A tuple that identifies the m-sized subset of N classifiers points to a mapping of question aligned decisions by that subset to the observed integer count in the test. :rtype: Mapping[tuple, Mapping[tuple, int]] .. py:class:: MVariety Class for the points obeying all axiom orders up to M. The M=1 variety contains points that obey the M=1 axioms for a single classifier. The M=2 variety contains points that obey the M=2 axioms for a pair of classifiers, but also the M=1 axioms for each of them. This class follows the 'code smell' test, it appears because we need operations that can create logically consistent intersections of varieties of fixed order. These varieties, by construction, do not use any information about responses at higher order. So the union of m=1 varieties never uses, or can contain, information about pair responses or higher. Thus the intersection of m-varities is a containing variety for the variety that corresponds to all axioms up to m=N being obeyed. .. deprecated :: 0.8 Use :class:`.ConsistentSet` instead. .. py:attribute:: labels :type: tuple[str] .. py:attribute:: classifiers :type: tuple[str] .. py:attribute:: qs :type: Mapping[str, sympy.Symbol] .. py:attribute:: m :type: int .. py:attribute:: label_vars :type: tuple[tuple[sympy.Symbol]] .. py:attribute:: points :type: Mapping[tuple, Mapping[tuple, {}]] .. py:method:: __eq__(other_variety: Self) -> bool Test equality. The current implementation is a weaker check on equality. It just verifies that self.m and self.label_vars are equal. If so, it returns true. A strict check on equality would verify that all points are also equal. :param other_variety: Other variety. :type other_variety: Self :returns: Whether the varities are equal. :rtype: bool .. py:method:: __and__(other_variety: Self) -> Self Create logical intersection of self and other_variety. Constructing varieties of order M=m requires that we find the logically consistent intersection of m-1 varieties. Starting at M=3, two varieties of order m-1 share some of their variables. The 'and' operation returns the joined points that are logically consistent with each other: have the same value for the shared variables. :param other_variety: Variety to do logical intersection with. It must be of the same M=m order as this one. :type other_variety: MVariety :returns: The points whose intersection is logically consistent. :rtype: MVariety .. py:method:: compute_from_indices(other_variety: Self, var_order: collections.abc.Sequence[sympy.Symbol]) -> collections.abc.Sequence[int] Compute the indices for finding the values of the final variety. Whenever we do logical AND of varities, the final variety will contain values from both varieties. This function returns the indices in their concatenated variety point where we can find the variables specified in 'var_order.' :param var_order: Sequence that defines the order of the variables. :type var_order: Sequence[sympy.Symbol] :returns: Source index in a concatenated variety point. :rtype: Sequence[int] .. py:method:: intersection_label_vars(other_variety: Self) -> Set[sympy.Symbol] Find variables shared by both varieties. Beginning at m=2, varieties may share lower m variables. In addition, we want the __and__ operation to be idempotent when a previously joined variety is joined again - joining self with another variety gives the same result if we join it again. :param other_variety: The other variety. :type other_variety: Self :rtype: Set of shared variables between the two varieties. .. py:method:: union_classifiers(other_variety: Self) -> Set[str] Find union of the classifiers in self and 'other_variety'. :param other_variety: Variety to compare with. :type other_variety: Self :returns: The union of the classifiers in self and 'other_variety'. :rtype: Set[str] .. py:method:: var_order(other_variety: Self) -> tuple[sympy.Symbol] Construct var order of the intersection of self with 'other_variety'. :param other_variety: A variety of the same order as self. :type other_variety: Self :rtype: The var order for the points in the intersection of the varieties. .. py:method:: common_m1_vars(other_variety: Self, var_order: tuple[sympy.Symbol]) -> tuple[str] Find common m=1 vars following var order. :param other_variety: Variety to check. :type other_variety: Self :param var_order: Canonical var order. :type var_order: tuple[sympy.Symbol] :returns: m=1 label responses variables shared by self with other_variety. :rtype: tuple[str] .. py:method:: m1_var_indices(m1_vars: collections.abc.Sequence[sympy.Symbol], variety: Self) -> tuple[int] Find the position of m=1 vars in variety.label_vars. :param m1_vars: m=1 label response variables for which we want the index. :type m1_vars: Sequence[sympy.Symbol] :param variety: The variety whose m1 variables will be indexed. :type variety: Self :returns: Indices of the m=1 label response variables in a variety point. :rtype: tuple[int] .. py:method:: common_m2p_vars(other_variety: Self, var_order: tuple[sympy.Symbol]) -> tuple[str] Find common m=2 or higher vars following var order. :param other_variety: Other variety. :type other_variety: Self :param var_order: Canonical var order. :type var_order: tuple[sympy.Symbol] :returns: m>=2 label response variables shared by self and 'other_variety'. :rtype: tuple[str] .. py:method:: m2p_var_indices(m2p_vars: collections.abc.Sequence[sympy.Symbol], variety: Self) -> tuple[int] Find the indices for the m>=2 label responses variables. :param m2p_vars: m>=2 label responses variables. :type m2p_vars: Sequence[sympy.Symbol] :param other_variety: The other variety. :type other_variety: Self :returns: Indices of the m>=2 label response variables in a point from 'variety'. :rtype: tuple[int] .. py:method:: only_self_vars(other_variety: Self) -> tuple[sympy.Symbol] Find label response variables only self has. :param other_variety: The other variety. :type other_variety: Self :returns: * *tuple[sympy.Symbol]* * *Label response variables only found in self and not in 'other_variety'.* .. py:method:: only_other_vars(other_variety: Self) -> tuple[sympy.Symbol] Find label response variables only 'other_variety' has. :param other_variety: The variety we are comparing with. :type other_variety: Self :returns: * *tuple[sympy.Symbol]* * *Label response variables only found in 'other_variety'.* .. py:method:: common_vars(other_variety: Self) -> tuple[sympy.Symbol] Find label response variables in self and other_variety. :param other_variety: The other variety. :type other_variety: Self :returns: * *tuple[sympy.Symbol]* * *Label response variables found in self and other_variety.* .. py:method:: var_indices(vars: collections.abc.Sequence[sympy.Symbol], variety: Self) -> Iterable Get indices for vars in self.label_vars. :param vars: Variables for which we want indices in points from 'variety'. :type vars: Sequence[sympy.Symbol] :param variety: The variety where the indices are to be found. :type variety: Self :returns: The indices for 'vars' in a point from 'variety'. :rtype: Iterable .. py:method:: join_label_vars(var_order: collections.abc.Sequence[sympy.Symbol], other_variety: Self) -> tuple[tuple[sympy.Symbol]] Join label_vars by m-order. :param other_variety: The variety whose variables will be joined with self. :type other_variety: Self :rtype: The joined vars by m-order. .. py:class:: MVarietyTupleDict Bases: :py:obj:`MVariety` Concrete class for MVariety storing points as dict of tuples. Warning: this class is memory intensive. .. deprecated :: 0.8 Use :class:`.ConsistentSet` instead. .. py:method:: generate_points() -> Iterable Generate the points in this variety. :Yields: *Iterable* -- Points in the variety. .. py:method:: generate_consistent_point_pairs(var_order: collections.abc.Sequence[sympy.Symbol], other_variety: Self) -> Iterable[tuple[tuple[int], tuple[int]]] Generate pairs of points from each variety consisten with each other. Whenever we are joining varieties of order m >= 2, care must be taken to only combine points that agree on their common variables. For example, if we have the variety for classifiers 'i' and 'j' and want to join it with the variety for classifiers 'j' and 'k', we must make sure that we return point pairs, one from each variety, that agree in their values of 'j' responses. :param var_order: Sequence of vars that defines order of variables in the joined variety. :type var_order: Sequence[sympy.Symbol] :param other_variety: The variety to check consistency on common variables. :type other_variety: Self :Yields: *(Iterable[tuple[tuple[int], tuple[int]]])* -- Pairs of points, one from each variety, that are logically consistent with each other (agree on common variables). .. py:method:: generate_consistent_key_pairs(common_m1_vars: collections.abc.Sequence[sympy.Symbol], other_variety: Self) -> Iterable[Tuple[Tuple[int], Tuple[int]]] Generate key pairs for self and other_variety that are consistent. :param other_variety: The variety we are joining self with. :type other_variety: Self :rtype: Tuples of m1 keys for the points dict of each variety. .. py:method:: generate_consistent_tail_pairs(other_variety: Self, self_key: Tuple[int], other_key: Tuple[int], scm2pi: Tuple[int], ocm2pi: Tuple[int]) -> Iterable[Tuple[Tuple[int], Tuple[int]]] Generate key pairs for self and other_variety that are consistent. :param other_variety: The variety we are joining self with. :type other_variety: Self :rtype: Tuples of m1 keys for the points dict of each variety. .. py:method:: construct_intersection_variety(other_variety: MVariety) -> Self Construct the intersection. :param other_variety: The variety we are intersecting self with. :type other_variety: MVariety :returns: The intersection of the two varieties.. :rtype: Self .. py:class:: MAxiomsVarieties(labels: collections.abc.Sequence[str], classifiers: collections.abc.Sequence[str], responses: Mapping[tuple, int], qs, m) Class to compute the test evaluation variety. The test evaluation variety for M=m axioms is the set of evaluations that are logically consistent with how we observe the classifiers agreeing and disagreeing on the question responses. .. deprecated :: 0.8 Use :class:`.ConsistentSet` instead. .. py:attribute:: labels .. py:attribute:: classifiers .. py:attribute:: qs .. py:attribute:: m .. py:attribute:: r_simplexes .. py:attribute:: test_axioms .. py:attribute:: mm1_relevant_vars .. py:attribute:: var_max_responses .. py:method:: instantiate_axioms(m: int) Fill in all response variables in the test axioms. Any m-axiom contains three sets of variables: 1. The answer-key simplex variables. This function fills them in with the values given during class instantiation. 2. The observed response counts. This function fills them in. 3. All label m or less response variables. These need to be filled in as we move up the ladder of logical consistency. :param m: The order of the axioms, an integer of value 1 or greather. :type m: int :returns: **Mapping[m_subset** :rtype: instantiated_m_axioms] .. py:method:: _var_max_responses() -> Mapping[sympy.Symbol, int] Calculate the maximum possible value of all label response vars. :returns: * *Mapping[sympy.Symbol, int]* * *The observable value for the label response variable.* .. py:method:: _relevant_vars() .. py:method:: simplex_points_equal(total: int, maxs: collections.abc.Sequence[int], N: int) Generate all simplex points with values less than or equal to maxs. This is a recursive generator to handle arbitrary number of variables in a simplex. :param total: Total value required for the sum of the vars on the simplex. :type total: int :param maxs: The max integer value that a var can have on that simplex. :type maxs: Sequence[int] :param N: The number of vars defining the simplex. :type N: int :Yields: *tuple[int]* -- Simplex points, of dimension 'N', that sum to 'total' and whose var values do not exceed maxs values. .. py:method:: mvariety(classifiers: collections.abc.Sequence[str]) -> MVariety Construct the MVariety dataclass for these classifiers. :param classifiers: The classifiers. :type classifiers: Sequence[str] :returns: Dataclass containing the points in the variety. :rtype: MVariety .. py:method:: turn_axiom_exprs_to_vectors(classifiers: collections.abc.Sequence[str], labels_vars: collections.abc.Sequence[sympy.Symbol]) -> tuple[numpy.typing.NDArray[numpy.int16]] Turn label axioms into an array of coefficient vectors. :param classifiers: Sequence of the classifiers to consider. :type classifiers: Sequence[str] :param labels_vars: The order of the label vars in the returned tuple of ints. :type labels_vars: Sequence[sympy.Symbol] :returns: **var_coefficients** -- Sequence of integer coefficients for the axioms, one for each label. :rtype: tuple[npt.NDArray[np.int16]] .. py:method:: label_coefficients(labels_vars: collections.abc.Sequence[sympy.Symbol], axiom: sympy.UnevaluatedExpr) -> tuple[int] Compute the coefficients for labels_vars than appear in axiom. :param labels_vars: Variables for which we want coefficients in axiom. :type labels_vars: Sequence[sympy.Symbol] :param axiom: The algebraic expression of the axiom. :type axiom: sympy.UnevaluatedExpr :returns: The integer coefficients for label_vars in axiom. :rtype: tuple[int] .. py:method:: label_msimplex(label_mvars: collections.abc.Sequence[sympy.Symbol], var_indices: Mapping[sympy.Symbol, int], ql: int, mm1_point: numpy.typing.NDArray[numpy.uint16]) -> collections.abc.Sequence[numpy.typing.NDArray[numpy.uint16]] Generate all label m-response vars points. Each label has an m-response simplex that must be logically consistent with lower m response varieties. :param classifiers: The classifiers for this m-response simplex. :type classifiers: Sequence[str] :param label: The true label. :type label: str :param ql: Assumed count of the true label in the answer key. :type ql: int :param mm1_point: The m (m-1)-varieties that define the starting point for creating the m-variety of these classifiers. :type mm1_point: Iteratable[dict] :rtype: Generator of all points on the label m-response simplex. .. py:method:: vars_max_values(label_mvars: collections.abc.Sequence[sympy.Symbol], var_indices: Mapping[sympy.Symbol, int], ql: int, mm1_point: numpy.typing.NDArray[numpy.uint16]) -> collections.abc.Sequence[int] Calculate the 'ratchet of crowd evaluation'. Every label response var is a positive integer between zero and, at most, the Q_label assumed value. However, this max is most of the time larger than the logically consistent solutions. The max integer value is the minimum of: 1. Q_label 2. R_decisions 3. all R_decisions_label for the m (m-1)-sized subsets of the classifier decisions. Clearly the max of label response variables must be Q_label. But if we observe the classifiers collectively producing a lower count, then that is the max. For example, if we never observed a pair count for (l_1, l_2) then all label response vars for this tuple must be zero for all labels. The same applies to the (m-1)-subsets of the decisions tuple given the true label. No value of the response count for the m-sized decisions tuple can be higher than any response count for the (m-1)-sized subsets of the decisions. :param classifiers: The m classifiers. :type classifiers: Sequence[str] :param decisions: Their m-decisions. :type decisions: Sequence[str] :param label: The true label. :type label: str :param mm1_point: The values of the mm1 variables. :type mm1_point: dict :returns: Max integer value for these classifier decisions given true label and the m-1 varieties point. :rtype: int .. py:method:: make_consistent_if_possible(mm1_points: collections.abc.Sequence[numpy.typing.NDArray[numpy.uint16]]) -> numpy.typing.NDArray[numpy.uint16] Make a consistent m point or return an empty numpy array. :param mm1_points: Order m minus 1 points to be joined logically. :type mm1_points: Sequence[npt.NDArray[np.uint16]] :returns: An array of the joined points if possible, empty otherwise. :rtype: npt.NDArray[np.uint16] .. py:method:: satisfies_axioms(mpoint: tuple[tuple[int]], maxioms: collections.abc.Sequence[tuple[int, numpy.typing.NDArray[numpy.uint16]]]) -> bool Test if mpoint satisfies maxioms. The variety is defined by all those points in the label response simplexes that satisfy the m-axioms. :param classifiers: The classifiers being considered. :type classifiers: Sequence[str] :param mpoint: Iteratable of label response point dicts. :type mpoint: Iterable[dict] :param maxioms: The m-axioms reduced the the components for a linear operation. :type maxioms: Mapping[str, tuple] :returns: Does this order m point obey the m-order axioms? :rtype: bool .. py:class:: SingleClassifierEvaluations(Q: int, single_axioms: ntqr.r2.raxioms.SingleClassifierAxioms | ntqr.r3.raxioms.SingleClassifierAxioms) Class for the evaluations of a single classifier given test responses. The axioms of unsupervised evaluation are algebraic filters that narrow the set of possible evaluations for test takers. This filtering proceeds in a ladder-like manner. The single classifier axioms are the bottom rung of that filtering. For a test with R labels, the set of possible evaluations is a collection of has dimension R*(R-1) at each setting of the number of label correct in the answer key. This is the space of error responses. There being R-1 for each of the R labels. The single classifier axioms cut that dimension by R-1. So the set of evaluations for a single classifier consists of points that have dimension R*(R-1), but the set itself has dimension (R-1)*(R-1). .. deprecated :: 0.8 Use :class:`.ConsistentSet` instead. .. py:attribute:: Q .. py:attribute:: axioms .. py:method:: correct_at_qs(qs: collections.abc.Sequence[int], responses: collections.abc.Sequence[int]) -> set[collections.abc.Sequence[int]] Calculate all possible correct responses given qs. :param qs: Number of label questions. :type qs: Sequence[int] :param responses: Label responses by classifier. :type responses: Sequence[int] :returns: Set of possible label correct evaluations given qs and responses. :rtype: set[Sequence[int]] .. py:method:: _check_axiom_consistency_(eval_dict, wrong_vars, wrong_vals)