ntqr.evaluations¶

Module for super classes of R-axioms based evaluations.

Logically consistent evaluations are defined by the size of an ensemble and its associated axioms.

Classes¶

`AnswerKeyQSimplex`	Class to generate all answer-key simplex points.
`PossibleSet`	Class for the possible set of evaluations for N classifiers
`ConsistentSet`	Class for N classifier evaluations logically consistent with the
`MLabelResponseSimplexes`	Class to manage the response simplexes associated with each label.
`MVariety`	Class for the points obeying all axiom orders up to M.
`MVarietyTupleDict`	Concrete class for MVariety storing points as dict of tuples.
`MAxiomsVarieties`	Class to compute the test evaluation variety.
`SingleClassifierEvaluations`	Class for the evaluations of a single classifier given test responses.

Functions¶

`generate_simplex_points`(...)	Generator of tuples of integers that sum to target_sum.
`random_simplex_point`(→ collections.abc.Sequence[int])	Generates a random point on the simplex.
`generate_observable_cube_points`(...)	Generator of points on a simple inside the observable hypercube.
`random_bounded_simplex_point`(...)	Uniformly samples non-negative integers x_i such that:

Module Contents¶

ntqr.evaluations.generate_simplex_points(target_sum: int, n_vars: int) → Iterable[collections.abc.Sequence[int]]¶

Generator of tuples of integers that sum to target_sum.

Parameters:

target_sum (int) – Sum of the variables.
n_vars (int) – Number of variables.

Returns:

The points on the simplex obeying the target_sum.

Return type:

Iterable[Sequence[int]]

ntqr.evaluations.random_simplex_point(target_sum: int, n_vars: int) → collections.abc.Sequence[int]¶

Generates a random point on the simplex.

Parameters:

target_sum (int) – Required target sum for the variables.
n_vars (int) – Number of variables.

Returns:

A single point on the simplex obeynig target sum.

Return type:

Sequence[int]

ntqr.evaluations.generate_observable_cube_points(target_sum: int, maxs: collections.abc.Sequence[int]) → Iterable[collections.abc.Sequence[int]]¶

Generator of points on a simple inside the observable hypercube.

The sum of decision events must equal the assumed label count in the answer key, but, in addition, each event count cannot be larger than the min of the label count OR the observed count of the event.

This generator produces the points that obey the target sum and the individual max possible value for an event. Note that this is not enough to produce the consistent set as the product of each label generator – they must obey the same observable count ACROSS all labels.

Parameters:

target_sum (int) – Sum of the variables.
maxs (Sequence[int]) – The individual max for each variable.

Returns:

DESCRIPTION.

Return type:

Iterable[Sequence[int]]

ntqr.evaluations.random_bounded_simplex_point(total_sum: int, bounds: collections.abc.Sequence[int]) → collections.abc.Sequence[int]¶

Uniformly samples non-negative integers x_i such that: 0 <= x_i <= bounds[i] and sum(x_i) == total_sum

Parameters:

total_sum (int) – Sum required.
bounds (Sequence[int]) – Bounds for each variable, must be equal to or less than total_sum.

Raises:

ValueError – If no points satisfy the given bounds and sum to total_sum.

Returns:

The sampled point.

Return type:

Sequence[int]

class ntqr.evaluations.AnswerKeyQSimplex(Q: int, labels: ntqr.Labels)¶

Class to generate all answer-key simplex points.

Given the test size, Q, and the labels (R=r) of them. This is a set of tuples of dimension r, the number of labels.

It exists in (R-1) space since, by construction, we must have their sum always equal to Q. The number of possible qs is 1/(R-1)!*(Q+1)*(Q+2)*…*(Q+R-1).

To summarize:

The number of labels in the answer key is a tuple

of size R. Its value is unknown in unsupervised settings.

The number of such tuples is 1/(R-1)!(Q+1)…
This set exists in a (R-1) dimensional space inside the R space for the tuples.

Q¶

N¶

labels¶

R¶

qs() → Iterable[tuple[int]]¶

Generate all possible answer-key simplex points.

Returns:: Generator of all possible answer-key simplex points.
Return type:: Iterable[tuple[int]]

class ntqr.evaluations.PossibleSet(labels: collections.abc.Sequence[str], classifiers: collections.abc.Sequence[str])¶

Class for the possible set of evaluations for N classifiers classifying Q items into R labels.

Given a point in the answer key Q-simplex, we can count how many group evaluations are possible before we see any test results.

For small tests, it is also possible to generate all of the points in the possible set but it can quickly blow up so caution must be exercised when calling generators of possible evaluations.

labels¶

classifiers¶

set_count_at_ql(ql: collections.abc.Sequence[int]) → int¶

Computes the size of the possible set at given answer key ql point.

The size of the possible set at any given answer key Q-simplex, point ql can be quite large for small tests and labels so this function carries out the exact integer representation of the count using SymPy’s symbolic functionality.

Parameters:: ql (Sequence[int]) – The answer key Q-simplex point, values must correspond to the number of labels used to initialize the class.
Returns:: The integer count of the possible set at the given ql value.
Return type:: int

sum_count_at_ql(ql: collections.abc.Sequence[int]) → int¶

Computes the sum of the possible values over the labels.

This is useful for considering if random sampling from the possible set is possible. To randomly draw one evaluation from the possbile set, you would have to create the sum of the evaluations over the labels, not the product.

Parameters:: ql (Sequence[int]) – The answer key Q-simplex point, values must correspond to the number of labels used to initialize the class.
Returns:: The number of values that need to be generated to sample one point from the possible set.
Return type:: int

set_generator(ql: collections.abc.Sequence[int]) → Iterable[collections.abc.Sequence[int]]¶

Generator of the possible set at given answer key ql point.

Points are returned as tuples where variables are sorted by label then event.

This generator should be used with caution since even reasonably sized tests can be quite large, of the order of (R^N)! where R is the number of labels and N the number of classifiers.

Parameters:: ql (Sequence[int]) – The answer key Q-simplex point, values must correspond to the number of labels used to initialize the class.
Returns:: Points are represented as tuples of ints where the label response variables have been sorted by label, then event.
Return type:: Iterable[Sequence[int]]

random_points(ql: collections.abc.Sequence[int], n: int) → set[collections.abc.Sequence[collections.abc.Sequence[int]]]¶

Generates n random points in possible set at given ql.

Parameters:

ql (Sequence[int]) – Point on the Q-simplex.
n (int) – Number of points wanted.

Returns:

Sequence of length n of possible evaluation points.

Return type:

Sequence[Sequence[int]]

__repr__()¶

class ntqr.evaluations.ConsistentSet(labels: collections.abc.Sequence[str], classifiers: collections.abc.Sequence[str], counts: Mapping[collections.abc.Sequence[str], int])¶

Class for N classifier evaluations logically consistent with the observed counts of the R^N ways they can agree/disagree when using R labels.

This is a subset of of the possible set, smaller both in the dimension of its geometry (the number of indepent variables) and its count in that space.

This is due to the observable counts for an event setting a ceiling for the possible value of the count of the same event given true label.

labels¶

classifiers¶

counts¶

max_value_at_ql(ql: collections.abc.Sequence[int], vars: collections.abc.Sequence[sympy.Symbol]) → collections.abc.Sequence[int]¶

Computes maximum value for each variable at given ql Q-simplex point.

The maximum value possible for the count of a decision event by the ensemble given true label is the minimum of the assumed count of that label in the answer key and event observed count.

Parameters:

ql (Sequence[int]) – DESCRIPTION.
vars (Sequence[sympy.Symbol]) – DESCRIPTION.

Returns:

DESCRIPTION.

Return type:

Sequence[int]

set_generator(ql: collections.abc.Sequence[int]) → Iterable[collections.abc.Sequence[int]]¶

Generator of the possible set at given answer key ql point.

Points are returned as tuples where variables are sorted by label then event.

This generator should be used with caution since even reasonably sized tests can be quite large, of the order of (R^N)! where R is the number of labels and N the number of classifiers.

Parameters:: ql (Sequence[int]) – The answer key Q-simplex point, values must correspond to the number of labels used to initialize the class.
Returns:: Points are represented as tuples of ints where the label response variables have been sorted by label, then event.
Return type:: Iterable[Sequence[int]]

random_points(ql: collections.abc.Sequence[int], n: int) → set[collections.abc.Sequence[collections.abc.Sequence[int]]]¶

Returns n random points in the consistent set at given ql.

Parameters:

ql (Sequence[int]) – Assumed value of labels in the answer key.
n (int) – Number of random points wanted.

Returns:

A random set of n points from the consistent set of evaluations at given ql point.

Return type:

set[Sequence[Sequence[int]]]

correct_cuboid_generator(ql: collections.abc.Sequence[int]) → Iterable[collections.abc.Sequence[collections.abc.Sequence[int]]]¶

Generates the correct cuboid for each label given Q-simplex point, ql.

A correct cuboid point is defined by the number of correct label answers for each classifier. Strictly speaking, this set does not have the geometry of a cuboid, but can be confined within one.

This set is the basis for no-knowledge alarms since they consist of points where classifiers may have enough disagreements that one of them cannot be working properly at the specified ql value.

Parameters:: ql (Sequence[int]) – Assumed point in the Q-simplex, the count of labels in the answer key.
Returns:: Points of the form (R_{l_i, l_j, …; l_true},…). The number of label corrects for each classifier and label.
Return type:: Iterable[Sequence[Sequence[int]]]

random_correct_cuboid_points(ql: collections.abc.Sequence[int], n: int) → collections.abc.Sequence[collections.abc.Sequence[collections.abc.Sequence[int]]]¶

Random list of points on the correct label cuboids.

Points may not be unique since they are marginalized counts of unique points on the consistent set.

Parameters:

ql (Sequence[int]) – Assumed count of each label in the answer key.
n (int) – Number of points sampled in the consistent set.

Returns:

Sequence, n long, of points in the correctness cuboid.

Return type:

Sequence[Sequence[Sequence[int]]]

__repr__()¶

class ntqr.evaluations.MLabelResponseSimplexes(labels: collections.abc.Sequence[str], classifiers: collections.abc.Sequence[str], responses: Mapping[tuple, int], qs, m)¶

Class to manage the response simplexes associated with each label.

Each subset of sized-M for N classifiers has its own set of label response simplexes, one for each of the R labels in the test. These are the set of all possible values for statistics of aligned decisions by the members of the m-subset GIVEN true label.

The a-priori logic of a test is that the sum of all possible responses by a m-subset must exactly equal to the Q_label, the count of the label in the answer key. This defines a simplex for a given label.

The posterior logic is that the simplexes for any given value of M, M=m, have axioms that depend on all simplexes of value less than m. Thus the M=2 simplexes involve variables from the two M=1 simplexes that come from the classifier pair responses. The M=3 simplexes involve all M=2 simplexes, and all M=1 simplexes. And so on.

This class is meant to internally manage that logical complexity by keeping track of all the variables that are needed for each simplex as well as the enclosing ‘shells’ that provide the values for the response variables.

A separate class, MAxiomsVarieties, combines the functionality of this class and the ntqr.raxioms.MAxiomsIdeal class to compute the subset of possible label test responses that are logically consistent with the observed test results.

Deprecated since version 0.8: Use ConsistentSet instead.

labels¶

qs¶

m¶

classifiers¶

qad¶

_initialize_response_dicts()¶

_initialize_sympy_response_dict()¶

m_responses(m: int) → Mapping[tuple, Mapping[tuple, int]]¶

Observed responses by all m-sized subsets of the classifiers.

Observed responses set the ceilings for any possible value for the label responses - responses given true label. For example, if we observe that two classifiers agreed on the same label some number of times, no possible evaluation of that agreement given true label can exceed this number.

Observed responses create the ‘ratchet’ of evaluation. No label response variable can have a value larger than that of any subset of the classifiers responding similarly.

Parameters:

m (int) – Size of the subsets of the classifiers that will be used.

Returns:

This is semantically of the form,

Mapping[m-subset-classifiers,: Mapping[m-decisions, observed count]

A tuple that identifies the m-sized subset of N classifiers points to a mapping of question aligned decisions by that subset to the observed integer count in the test.

Return type:

Mapping[tuple, Mapping[tuple, int]]

class ntqr.evaluations.MVariety¶

Class for the points obeying all axiom orders up to M.

The M=1 variety contains points that obey the M=1 axioms for a single classifier. The M=2 variety contains points that obey the M=2 axioms for a pair of classifiers, but also the M=1 axioms for each of them.

This class follows the ‘code smell’ test, it appears because we need operations that can create logically consistent intersections of varieties of fixed order.

These varieties, by construction, do not use any information about responses at higher order. So the union of m=1 varieties never uses, or can contain, information about pair responses or higher.

Thus the intersection of m-varities is a containing variety for the variety that corresponds to all axioms up to m=N being obeyed.

Deprecated since version 0.8: Use ConsistentSet instead.

labels: tuple[str]¶

classifiers: tuple[str]¶

qs: Mapping[str, sympy.Symbol]¶

m: int¶

label_vars: tuple[tuple[sympy.Symbol]]¶

points: Mapping[tuple, Mapping[tuple, {}]]¶

__eq__(other_variety: Self) → bool¶

Test equality.

The current implementation is a weaker check on equality. It just verifies that self.m and self.label_vars are equal. If so, it returns true.

A strict check on equality would verify that all points are also equal.

Parameters:: other_variety (Self) – Other variety.
Returns:: Whether the varities are equal.
Return type:: bool

__and__(other_variety: Self) → Self¶

Create logical intersection of self and other_variety.

Constructing varieties of order M=m requires that we find the logically consistent intersection of m-1 varieties. Starting at M=3, two varieties of order m-1 share some of their variables. The ‘and’ operation returns the joined points that are logically consistent with each other: have the same value for the shared variables.

Parameters:: other_variety (MVariety) – Variety to do logical intersection with. It must be of the same M=m order as this one.
Returns:: The points whose intersection is logically consistent.
Return type:: MVariety

compute_from_indices(other_variety: Self, var_order: collections.abc.Sequence[sympy.Symbol]) → collections.abc.Sequence[int]¶

Compute the indices for finding the values of the final variety.

Whenever we do logical AND of varities, the final variety will contain values from both varieties. This function returns the indices in their concatenated variety point where we can find the variables specified in ‘var_order.’

Parameters:: var_order (Sequence[sympy.Symbol]) – Sequence that defines the order of the variables.
Returns:: Source index in a concatenated variety point.
Return type:: Sequence[int]

intersection_label_vars(other_variety: Self) → Set[sympy.Symbol]¶

Find variables shared by both varieties.

Beginning at m=2, varieties may share lower m variables. In addition, we want the __and__ operation to be idempotent when a previously joined variety is joined again - joining self with another variety gives the same result if we join it again.

Parameters:: other_variety (Self) – The other variety.
Return type:: Set of shared variables between the two varieties.

union_classifiers(other_variety: Self) → Set[str]¶

Find union of the classifiers in self and ‘other_variety’.

Parameters:: other_variety (Self) – Variety to compare with.
Returns:: The union of the classifiers in self and ‘other_variety’.
Return type:: Set[str]

var_order(other_variety: Self) → tuple[sympy.Symbol]¶

Construct var order of the intersection of self with ‘other_variety’.

Parameters:: other_variety (Self) – A variety of the same order as self.
Return type:: The var order for the points in the intersection of the varieties.

common_m1_vars(other_variety: Self, var_order: tuple[sympy.Symbol]) → tuple[str]¶

Find common m=1 vars following var order.

Parameters:

other_variety (Self) – Variety to check.
var_order (tuple[sympy.Symbol]) – Canonical var order.

Returns:

m=1 label responses variables shared by self with other_variety.

Return type:

tuple[str]

m1_var_indices(m1_vars: collections.abc.Sequence[sympy.Symbol], variety: Self) → tuple[int]¶

Find the position of m=1 vars in variety.label_vars.

Parameters:

m1_vars (Sequence[sympy.Symbol]) – m=1 label response variables for which we want the index.
variety (Self) – The variety whose m1 variables will be indexed.

Returns:

Indices of the m=1 label response variables in a variety point.

Return type:

tuple[int]

common_m2p_vars(other_variety: Self, var_order: tuple[sympy.Symbol]) → tuple[str]¶

Find common m=2 or higher vars following var order.

Parameters:

other_variety (Self) – Other variety.
var_order (tuple[sympy.Symbol]) – Canonical var order.

Returns:

m>=2 label response variables shared by self and ‘other_variety’.

Return type:

tuple[str]

m2p_var_indices(m2p_vars: collections.abc.Sequence[sympy.Symbol], variety: Self) → tuple[int]¶

Find the indices for the m>=2 label responses variables.

Parameters:

m2p_vars (Sequence[sympy.Symbol]) – m>=2 label responses variables.
other_variety (Self) – The other variety.

Returns:

Indices of the m>=2 label response variables in a point from ‘variety’.

Return type:

tuple[int]

only_self_vars(other_variety: Self) → tuple[sympy.Symbol]¶

Find label response variables only self has.

Parameters:

other_variety (Self) – The other variety.

Returns:

tuple[sympy.Symbol]
Label response variables only found in self and not in ‘other_variety’.

only_other_vars(other_variety: Self) → tuple[sympy.Symbol]¶

Find label response variables only ‘other_variety’ has.

Parameters:

other_variety (Self) – The variety we are comparing with.

Returns:

tuple[sympy.Symbol]
Label response variables only found in ‘other_variety’.

common_vars(other_variety: Self) → tuple[sympy.Symbol]¶

Find label response variables in self and other_variety.

Parameters:

other_variety (Self) – The other variety.

Returns:

tuple[sympy.Symbol]
Label response variables found in self and other_variety.

var_indices(vars: collections.abc.Sequence[sympy.Symbol], variety: Self) → Iterable¶

Get indices for vars in self.label_vars.

Parameters:

vars (Sequence[sympy.Symbol]) – Variables for which we want indices in points from ‘variety’.
variety (Self) – The variety where the indices are to be found.

Returns:

The indices for ‘vars’ in a point from ‘variety’.

Return type:

Iterable

join_label_vars(var_order: collections.abc.Sequence[sympy.Symbol], other_variety: Self) → tuple[tuple[sympy.Symbol]]¶

Join label_vars by m-order.

Parameters:: other_variety (Self) – The variety whose variables will be joined with self.
Return type:: The joined vars by m-order.

class ntqr.evaluations.MVarietyTupleDict¶

Bases: MVariety

Concrete class for MVariety storing points as dict of tuples.

Warning: this class is memory intensive.

Deprecated since version 0.8: Use ConsistentSet instead.

generate_points() → Iterable¶

Generate the points in this variety.

Yields:: Iterable – Points in the variety.

generate_consistent_point_pairs(var_order: collections.abc.Sequence[sympy.Symbol], other_variety: Self) → Iterable[tuple[tuple[int], tuple[int]]]¶

Generate pairs of points from each variety consisten with each other.

Whenever we are joining varieties of order m >= 2, care must be taken to only combine points that agree on their common variables. For example, if we have the variety for classifiers ‘i’ and ‘j’ and want to join it with the variety for classifiers ‘j’ and ‘k’, we must make sure that we return point pairs, one from each variety, that agree in their values of ‘j’ responses.

Parameters:

var_order (Sequence[sympy.Symbol]) – Sequence of vars that defines order of variables in the joined variety.
other_variety (Self) – The variety to check consistency on common variables.

Yields:

(Iterable[tuple[tuple[int], tuple[int]]]) – Pairs of points, one from each variety, that are logically consistent with each other (agree on common variables).

generate_consistent_key_pairs(common_m1_vars: collections.abc.Sequence[sympy.Symbol], other_variety: Self) → Iterable[Tuple[Tuple[int], Tuple[int]]]¶

Generate key pairs for self and other_variety that are consistent.

Parameters:: other_variety (Self) – The variety we are joining self with.
Return type:: Tuples of m1 keys for the points dict of each variety.

generate_consistent_tail_pairs(other_variety: Self, self_key: Tuple[int], other_key: Tuple[int], scm2pi: Tuple[int], ocm2pi: Tuple[int]) → Iterable[Tuple[Tuple[int], Tuple[int]]]¶

Generate key pairs for self and other_variety that are consistent.

Parameters:: other_variety (Self) – The variety we are joining self with.
Return type:: Tuples of m1 keys for the points dict of each variety.

construct_intersection_variety(other_variety: MVariety) → Self¶

Construct the intersection.

Parameters:: other_variety (MVariety) – The variety we are intersecting self with.
Returns:: The intersection of the two varieties..
Return type:: Self

class ntqr.evaluations.MAxiomsVarieties(labels: collections.abc.Sequence[str], classifiers: collections.abc.Sequence[str], responses: Mapping[tuple, int], qs, m)¶

Class to compute the test evaluation variety.

The test evaluation variety for M=m axioms is the set of evaluations that are logically consistent with how we observe the classifiers agreeing and disagreeing on the question responses.

Deprecated since version 0.8: Use ConsistentSet instead.

labels¶

classifiers¶

qs¶

m¶

r_simplexes¶

test_axioms¶

mm1_relevant_vars¶

var_max_responses¶

instantiate_axioms(m: int)¶

Fill in all response variables in the test axioms.

Any m-axiom contains three sets of variables:

The answer-key simplex variables. This function fills them in with the values given during class instantiation.
The observed response counts. This function fills them in.
All label m or less response variables. These need to be filled in as we move up the ladder of logical consistency.

Parameters:: m (int) – The order of the axioms, an integer of value 1 or greather.
Returns:: Mapping[m_subset
Return type:: instantiated_m_axioms]

_var_max_responses() → Mapping[sympy.Symbol, int]¶

Calculate the maximum possible value of all label response vars.

Returns:

Mapping[sympy.Symbol, int]
The observable value for the label response variable.

_relevant_vars()¶

simplex_points_equal(total: int, maxs: collections.abc.Sequence[int], N: int)¶

Generate all simplex points with values less than or equal to maxs.

This is a recursive generator to handle arbitrary number of variables in a simplex.

Parameters:

total (int) – Total value required for the sum of the vars on the simplex.
maxs (Sequence[int]) – The max integer value that a var can have on that simplex.
N (int) – The number of vars defining the simplex.

Yields:

tuple[int] – Simplex points, of dimension ‘N’, that sum to ‘total’ and whose var values do not exceed maxs values.

mvariety(classifiers: collections.abc.Sequence[str]) → MVariety¶

Construct the MVariety dataclass for these classifiers.

Parameters:: classifiers (Sequence[str]) – The classifiers.
Returns:: Dataclass containing the points in the variety.
Return type:: MVariety

turn_axiom_exprs_to_vectors(classifiers: collections.abc.Sequence[str], labels_vars: collections.abc.Sequence[sympy.Symbol]) → tuple[numpy.typing.NDArray[numpy.int16]]¶

Turn label axioms into an array of coefficient vectors.

Parameters:

classifiers (Sequence[str]) – Sequence of the classifiers to consider.
labels_vars (Sequence[sympy.Symbol]) – The order of the label vars in the returned tuple of ints.

Returns:

var_coefficients – Sequence of integer coefficients for the axioms, one for each label.

Return type:

tuple[npt.NDArray[np.int16]]

label_coefficients(labels_vars: collections.abc.Sequence[sympy.Symbol], axiom: sympy.UnevaluatedExpr) → tuple[int]¶

Compute the coefficients for labels_vars than appear in axiom.

Parameters:

labels_vars (Sequence[sympy.Symbol]) – Variables for which we want coefficients in axiom.
axiom (sympy.UnevaluatedExpr) – The algebraic expression of the axiom.

Returns:

The integer coefficients for label_vars in axiom.

Return type:

tuple[int]

label_msimplex(label_mvars: collections.abc.Sequence[sympy.Symbol], var_indices: Mapping[sympy.Symbol, int], ql: int, mm1_point: numpy.typing.NDArray[numpy.uint16]) → collections.abc.Sequence[numpy.typing.NDArray[numpy.uint16]]¶

Generate all label m-response vars points.

Each label has an m-response simplex that must be logically consistent with lower m response varieties.

Parameters:

classifiers (Sequence[str]) – The classifiers for this m-response simplex.
label (str) – The true label.
ql (int) – Assumed count of the true label in the answer key.
mm1_point (Iteratable[dict]) – The m (m-1)-varieties that define the starting point for creating the m-variety of these classifiers.

Return type:

Generator of all points on the label m-response simplex.

vars_max_values(label_mvars: collections.abc.Sequence[sympy.Symbol], var_indices: Mapping[sympy.Symbol, int], ql: int, mm1_point: numpy.typing.NDArray[numpy.uint16]) → collections.abc.Sequence[int]¶

Calculate the ‘ratchet of crowd evaluation’.

Every label response var is a positive integer between zero and, at most, the Q_label assumed value. However, this max is most of the time larger than the logically consistent solutions.

The max integer value is the minimum of:

Q_label
R_decisions
all R_decisions_label for the m (m-1)-sized subsets of the classifier decisions.

Clearly the max of label response variables must be Q_label. But if we observe the classifiers collectively producing a lower count, then that is the max. For example, if we never observed a pair count for (l_1, l_2) then all label response vars for this tuple must be zero for all labels.

The same applies to the (m-1)-subsets of the decisions tuple given the true label. No value of the response count for the m-sized decisions tuple can be higher than any response count for the (m-1)-sized subsets of the decisions.

Parameters:

classifiers (Sequence[str]) – The m classifiers.
decisions (Sequence[str]) – Their m-decisions.
label (str) – The true label.
mm1_point (dict) – The values of the mm1 variables.

Returns:

Max integer value for these classifier decisions given true label and the m-1 varieties point.

Return type:

int

make_consistent_if_possible(mm1_points: collections.abc.Sequence[numpy.typing.NDArray[numpy.uint16]]) → numpy.typing.NDArray[numpy.uint16]¶

Make a consistent m point or return an empty numpy array.

Parameters:: mm1_points (Sequence[npt.NDArray[np.uint16]]) – Order m minus 1 points to be joined logically.
Returns:: An array of the joined points if possible, empty otherwise.
Return type:: npt.NDArray[np.uint16]

satisfies_axioms(mpoint: tuple[tuple[int]], maxioms: collections.abc.Sequence[tuple[int, numpy.typing.NDArray[numpy.uint16]]]) → bool¶

Test if mpoint satisfies maxioms.

The variety is defined by all those points in the label response simplexes that satisfy the m-axioms.

Parameters:

classifiers (Sequence[str]) – The classifiers being considered.
mpoint (Iterable[dict]) – Iteratable of label response point dicts.
maxioms (Mapping[str, tuple]) – The m-axioms reduced the the components for a linear operation.

Returns:

Does this order m point obey the m-order axioms?

Return type:

bool

class ntqr.evaluations.SingleClassifierEvaluations(Q: int, single_axioms: ntqr.r2.raxioms.SingleClassifierAxioms | ntqr.r3.raxioms.SingleClassifierAxioms)¶

Class for the evaluations of a single classifier given test responses.

The axioms of unsupervised evaluation are algebraic filters that narrow the set of possible evaluations for test takers. This filtering proceeds in a ladder-like manner. The single classifier axioms are the bottom rung of that filtering.

For a test with R labels, the set of possible evaluations is a collection of has dimension R*(R-1) at each setting of the number of label correct in the answer key. This is the space of error responses. There being R-1 for each of the R labels. The single classifier axioms cut that dimension by R-1. So the set of evaluations for a single classifier consists of points that have dimension R*(R-1), but the set itself has dimension (R-1)*(R-1).

Deprecated since version 0.8: Use ConsistentSet instead.

Q¶

axioms¶

correct_at_qs(qs: collections.abc.Sequence[int], responses: collections.abc.Sequence[int]) → set[collections.abc.Sequence[int]]¶

Calculate all possible correct responses given qs.

Parameters:

qs (Sequence[int]) – Number of label questions.
responses (Sequence[int]) – Label responses by classifier.

Returns:

Set of possible label correct evaluations given qs and responses.

Return type:

set[Sequence[int]]

_check_axiom_consistency_(eval_dict, wrong_vars, wrong_vals)¶