*pj/|F%SrSSKrSSKrSSKrSSKrSSKrSSKJrJ r J r J r J r SSK Jr SSKJr \\\\4\\\\\\\\4444r0SS_SS_S S _S S _S S_SS_SS_SS_SS_SS_SS_SS_SS_SS _S!S"_S#S$_S%S&_S'S(S)S*S+S,S-S.S/S0S1S2S3S4. Er\\\4\S5'\R35VVs0sHupX_M snnr\\\4\S6'S7\S8\4S9jr\R:S8\4S:j5r0SS_SS_S S_S S_SS_SS_SS_SS_SS_SS_S S_S"S_S$S_S&S_S'S_S)S_S(S_S S S S S S SSSSSSSS;. Er\\\4\S<'S r Sr!S r"S=r#S r$\R:S>\S8\4S?j5r%S@\SA\SB\SC\S8\4 SDjr&S7\SE\SF\SG\\SH\\\\\44S8\\S-\\\44 SIjr'\"54S7\SJ\SK\SL\\S8\S-4 SMjjr(S7\SE\SF\SG\\SH\\\\\44S8\S-4 SNjr)\R:SG\\S8\\\44SOj5r*\r+\R:SP\S8\\4SQj5r,SE\SF\SL\\S8\\S-4SRjr-SS\ ST\\\.\\S-\\44SP\SL\\S-S8\/4 SUjr0\"54S7\SE\SF\SG\\SL\\S8\S-4 SVjjr1SW\SE\SF\S8\\\4S-4SXjr2SYr3SZr4S[r5S\r6S7\S]\.\S8\.\4S^jr7gs snnf)_aConfusion group resolution for similar single-byte encodings. At runtime, loads pre-computed distinguishing byte maps from confusion.bin and uses them to resolve statistical scoring ties between similar encodings. Build-time computation (``compute_confusion_groups``, ``compute_distinguishing_maps``, ``serialize_confusion_data``) lives in ``scripts/confusion_training.py``. Note: ``from __future__ import annotations`` is intentionally omitted because this module is compiled with mypyc, which does not support PEP 563 string annotations. N) ART_LANGUAGE BigramProfile get_enc_indexget_idf_weightsscore_with_profile)DetectionResult)lookup_encodingLuLlLtLmLoMnMcMeNd Nl No Pc Pd PsPePiPfPoSmScSkSoZsZlZpCcCfCsCoCn) _INT_TO_CATEGORY_CATEGORY_TO_INTdatareturnc0nSn[R"SX5unUS- n[U5GHn[R"SX5unUS- nXX%-RS5nX%- n[R"SX5unUS- nXX'-RS5nX'- n[R"SX5un US- n/n 0n [U 5Han[R"SX5upnUS- nU R U 5 [ R U S 5[ R US 54X'Mc [U 5U 4XU4'GM U$) zLoad confusion group data from raw bytes. :param data: The raw binary content of a confusion.bin file. :returns: A :data:`DistinguishingMaps` dictionary keyed by encoding pairs. rz!Hr z!Br zutf-8z!BBBrr7)struct unpack_fromrangedecodeappendrEget frozenset)rGresultoffset num_pairs_ name_a_lenname_a name_b_lenname_b num_diffsdiff_bytes_list categoriesbv cat_a_int cat_b_ints V/var/www/html/pdf-tiff/venv/lib/python3.13/site-packages/chardet/pipeline/confusion.py%deserialize_confusion_data_from_bytesr`Lsa "$F F%%dD9LY aKF 9 **4> ! v23::7C**4> ! v23::7C))$= ! %'13 y!A'-'9'9&$'O $B9 aKF  " "2 & $$Y5 $$Y5JN "%.o$> #K 14 Mc[RRS5RS5nUR 5nU(d[ R "S[SS9 0$[U5n0nUR5H5uupgn[U5=(d Un [U5=(d Un XX4'M7 U$![R[4anSU3n[U5UeSnAff=f)zLoad confusion group data from the bundled confusion.bin file. :returns: A :data:`DistinguishingMaps` dictionary keyed by encoding pairs. zchardet.modelsz confusion.binuZchardet confusion.bin is empty — confusion resolution disabled; reinstall chardet to fixr ) stacklevelzcorrupt confusion.bin: N) importlib resourcesfilesjoinpath read_byteswarningswarnRuntimeWarningr`rJerrorUnicodeDecodeError ValueErroritemsr ) refrawraw_mapsemsg normalizedabvaluenorm_anorm_bs r_load_confusion_datar{ts    # #$4 5 > > OC .. C   '    %8= &(J!)  #(q #(q',F#$*  LL, -%'s+o1$%s# B;;C+C&&C+) r.r/r0r1r2r4r3r6r5r7rrr_CATEGORY_PREFERENCEencodingc<[S5n[S5Hgn[U/5RU5n[ U5S:waM0[ R"U5nUS:XaSX'MRUSS:XdUS;dMcSX'Mi [U5$![a Mf=f)u{256-entry table: 0 = non-letter, 1 = uppercase letter, 2 = other letter. Combining marks count as letters: in decomposed text (Vietnamese under windows-1258) a base letter's neighbor is its diacritic, which is word-internal, not a word boundary. Whitespace deliberately counts as a plain non-letter: exempting space-adjacent letters from the isolated-letter demotion (to spare one-letter words like French ``à``) was tried and falsified by the accuracy suite — Irish/Finnish po files and the EBCDIC record set depend on space-adjacent demotions, so the cross-family decisive-override gate handles the ``à`` failure mode instead. r}r r rL)rrr ) bytearrayrLbytesrMrmlen unicodedatacategory)r~tablerwchcats r__letter_case_tablers cNE 3Z s""8,B r7a< ""2& $;EH Vs]c\1EH <"   sB  BBrleftright case_tablec[RUS5nUSS:waU$X1nX2nUS:Xa US:Xa[$US:Xa US:Xa[$U$)aPreference for reading a byte as *cat*, adjusted for word shape. A letter reading only deserves its high preference when its neighbors make it look like part of a word under the same encoding: a letter with no letter neighbors is quoted/isolated punctuation in disguise, and a lowercase letter immediately followed by an uppercase one is not a word shape any of the supported languages produce. rrr r )r|rO_IMPLAUSIBLE_LETTER_PREFERENCE)rrrrpref left_kind right_kinds r__context_preferencersa # #C +D 1v}  I"JA~*/-- d{zQ-- Kraenc_aenc_b diff_bytesr[c\[U5upV[URSU55nU(dg[U5n[U5n Sn Sn Sn Sn SnSn[ U5S- nUGH-nUUunn[ U45nUR U5nSnUS:dM3U[:dM?US:aUUS- OSnUU:aUUS-OSn[UUUU5n[UUUU 5nUU:a<USS:Xa USS:XaOjU UU- - n USS:XaU[:Xa U UU- - n US- nOAUU:a;USS:Xa USS:XaO(U UU- - n USS:XaU[:Xa U UU- - n US- nUS- nUR UUS-5nUS:dGM!U[:aMGM0 X:aXU - X4$X:aX+U - X4$g)uContext-aware category voting. Returns ``(winner, margin, demotion_margin, demotion_events)``. For each occurrence of a distinguishing byte, compare the two encodings' readings: Unicode category preference, adjusted for word shape (see :func:`_context_preference`). The reading that makes more linguistic sense of the byte *in its context* collects the vote; occurrences vote independently, so repeated evidence counts. ``demotion_margin`` counts only the winner's votes earned where the *losing* side's letter reading was word-shape-implausible — evidence against an impossible reading, which is far stronger than the naive letters-beat-symbols preference. ``demotion_events`` counts how many distinct occurrences contributed to it, so callers can tell repeated evidence from one loud byte. N)Nrrrrr rP) _pair_byte_tablesrP translaterrrfind_MAX_VOTE_OCCURRENCESrr)rGrrrr[non_diffrTrelevanttable_atable_bvotes_avotes_b demotion_a demotion_bevents_aevents_bendr\cat_acat_bneedleposexaminedrrpref_apref_bs r__vote_with_marginr s 8$J/KHh78H  'G 'GGGJJHH d)a-C!"~ uuiiQh8&;;$'!G4a=D%(3YDqMAE(eWEF(eWEF8s?uQx3v.GQx365S+S"fvo5  A &8s?uQx3v.GQx365S+S"fvo5  A  MH))FC!G,C;Qh8&;;; F'=='== raenc_xenc_y languagesc&[5n[XAU5nUcgXEupgUup[XXU5uppU b)U [:aU [:a[ U5[ :aU $[XXU5n[ U5[ :a UbX:XaU$gUbU$U $)uReturn the byte-evidence winner between two encodings, or ``None``. Mirrors the in-band pairwise rule of :func:`resolve_confusion_groups` (decisive demotion vote, else bigram rescore, else category vote) for callers outside the ranked-results scan — e.g. the classic-Mac line-ending promotion, whose platform prior must not override distinguishing-byte evidence. Returns ``None`` when the pair has no distinguishing map or the evidence is inconclusive. *languages* must carry what the two results being compared report, or the mirror breaks: the rescore would arbitrate the same pair under a different rule than the confusion stage just did, and a veto built on that answer can reverse a promotion the stage had settled. N)r{_find_pair_keyr_DECISIVE_VOTE_MARGIN_DECISIVE_MIN_EVENTSr_CROSS_FAMILY_MIN_DIFFSresolve_by_bigram_rescore)rGrrrmapspair_keyrr[rr cat_winner_margindemotion_margindemotion_events bigram_winners r_confusion_pair_winnerr^s(  Dd51H!^JLE# UHnTU(aMUv M g7fN).0rwmembers r_ $_pair_byte_tables..s< 1&)QQ s )rrrL)rrwrrs @r_rrsDs^F q <c <.s;q!;s  )rPrrO)rs r__modelled_languagesrs- ,223; racU(dg[U5n[U5nX4-nX%::dg[U;d [U;a U[1-$U$)u9Languages to score *enc_a* and *enc_b* under, or ``None`` for all. ``None`` means *unrestricted* — score every variant, the original max-over-models comparison. It is not an abstention. An encoding should not win on language coverage the other side lacks. On a Hungarian document the u-double-acute bytes score 0.014 against iso8859-2's *Czech* model (which reads them as a common r-hacek) and only 0.006 against iso8859-16's Hungarian one, so max-over-variants hands Hungarian text to the Czech reading. Dropping the languages only one side models removes that particular unfairness. This is deliberately a narrow rule, and two tempting generalisations were measured and rejected against the accuracy suite: * Restricting to *languages* themselves rather than the shared set costs 12 tests. As a consequence the restriction is a no-op for pairs whose language coverage already matches (67 of the 236 confusion pairs), which is accepted — those pairs have no coverage asymmetry to correct in the first place. * Restricting when *languages* is not wholly inside the shared set costs 6 tests: cp1125 models only Ukrainian, so a Belarusian cp866 document shares just ``uk`` with it, and scoring the *right* encoding under the *wrong* language loses to cp1125. Falling back to unrestricted is also what keeps a Vietnamese windows-1258 document, whose rival cp1252 models no Vietnamese, resolving correctly. The cost is that pairs modelling disjoint languages (koi8-r/koi8-u, mac-roman/mac-turkish) never restrict at all — about 20% of calls over the corpus. N)r _ART_LANGUAGE)rrrlangs_alangs_bshareds r__comparable_languagesrsSF !%(G!%(G  F  =G#;'' Mraprofileindexcb^^URU5nU(dg[UU4SjU5SS9$)arReturn the best bigram score for *enc*, restricted to *languages*. *languages* of ``None`` means every variant, the unrestricted max-over-models comparison. The ``default`` guards a caller-supplied set that names no variant of *enc*; note that scoring 0.0 hands the comparison to the rival rather than abstaining, so callers wanting an abstention must not rely on it. gc3X># UHupnTbUT;dM[TX#5v M! g7fr)r)rrmodel model_keyrrs r_r&_best_variant_score..s7 *2&Y DI$5 : w 9 9*2s**)default)rOmax)rrrrvariantss` ` r__best_variant_scorers7yy~H   *2   rac[U5S:ag[U5upV[URSU55nU(dg[XU5n[ 5n 0n [U5S- n U[ -[U5:a[ 5n [URSU55Hun [U 45nURU5nUS:dM(U(aU RUS- 5 X:aU RU5 URXS-5nUS:aMMMw U H/nUUS-UUS--nU RUS5U U-U U'M1 OT[U 5HEnUUnUUS-nUUUU-(dM"US-U-nU RUS5U U-U U'MG U (dg[R"U 5n[5n[!UUX5n[!UUX(5nUU:aU$UU:aU$g)uResolve between two encodings by re-scoring only distinguishing bigrams. Builds a focused bigram profile containing only bigrams where at least one byte is a distinguishing byte, then scores both encodings under the languages they can be compared in (see :func:`_comparable_languages`). There is no abstention path here: when the pair has no comparable language the comparison widens to every variant rather than declining to answer. That is why a Danish mac-roman/mac-turkish document, whose pair models disjoint languages, is still decided by the Turkish model the rescore has no business consulting — the caller's category vote is better placed on such evidence, and overriding this to abstain was measured as costing more accuracy than it recovers. :param data: The raw byte data to examine. :param enc_a: First encoding name. :param enc_b: Second encoding name. :param diff_bytes: Byte values where the two encodings differ. :param languages: Languages the ranked results report for this pair; empty when the caller has no ranking to draw on, which scores every variant. :returns: The winning encoding name, or ``None`` if tied or if no distinguishing byte occurs in *data*. r Nr rr)rrrrr_DENSE_HIT_DIVISORsetrPrraddrOrLrfrom_weighted_freqrr)rGrrrrris_diffhits comparableidffreqlimitstartsr\rriidxb1b2rrbest_abest_bs r_rr s> 4y1} **5H t~~dH- .D &uY?J  CD IME   3t9, 5DNN4:;BB5\F))F#C(JJsQw';JJsOiia0 (<A7a<4A;.Ca(3s83DIuAaBa!eBBK'"+-7b.Ca(3s83DI  ..t4G OE % CF % CF    rarc,X4U;aX4$X!4U;aX!4$g)zEFind the canonical key for a pair of encodings in the confusion maps.Nr)rrrs r_rrhs+ ~~ ~~ ra4g{Gzt?g?g?resultsc:[U5S:aU$USnURcU$URS:XaU$[5nURnU[ -nSnUnURn[ S[U55GHn Xn U RcMU S:H=(d XJR- [:*n U (dU[:dU RU:a GOX[X8U R5n U cMwX<upU unn[XUX5unnnnUb*U[:a U[:a[U 5[:aUnOf[SURU R455n[XUU U5nU (a[U 5[:aUbUOUnOUb UU:XaUnOSnUbUU R:waGM=U (a[[!U RUU RU R"5n[%U5VVs/sHunnUU :wdMUPM nnnU/UQs $U nU nU RnGM US:XaU$[!URUURUR"5n[%U5VVs/sHunnUU:wdMUPM nnnU/UQ$s snnfs snnf)aResolve confusion between similar encodings in the top results. Checks the top result against each candidate within a confidence band. Always checks position 1 (preserving original top-2 behavior); for positions 2+ only checks within the band. Uses bigram re-scoring with category voting as fallback. :param data: The raw byte data to examine. :param results: Detection results sorted by confidence descending. :returns: A reordered list of :class:`DetectionResult` with the winner first. r rNzxxr c30# UH nUcMUv M g7frr)rrs r_r+resolve_confusion_groups..sCDCs )rr~languager{ confidence_CONFUSION_FLOOR_RATIOrL_CONFUSION_BAND_STRICT_TIER_MAX_CONFrrrrrrPrr mime_type enumerate)rGrtoprtop_conffloor champion_idxchampion champion_encr candidatein_bandrrr[rrr _vote_marginrrrlangsrpromotedjrrests r_resolve_confusion_groupsrs 7|a !*C || ||u  D~~H - -ELH <Vy'9'99  '""""## H #,G"4?"4$!QQA"4D?$t$ $  )) y$|q H $G, B,$!Q\0AA,D B  t 5@2 Cs: J J2 JJ)8__doc__ functoolsimportlib.resourcesrdrJrrichardet.modelsrrrrrchardet.pipelinerchardet.registryr dicttuplestrrPintDistinguishingMapsrE__annotations__rorFrr`cacher{r|rrrrrrrrrrrrrrlistfloatrrrrrrrr)kvs00r_r$sN  -, #s(O )C.$sE#s(O34 457$t$t$t$t $ t $ t $t$t$t$t$$$$$$ !$"#$$             =$$sCx.D6F5K5K5M#N5MTQAD5M#N$sCx.N%%:L%P />("("( "( ! (  ! (  ! ( !( !( !( !( !( !( !( !( !(  !!(" !#($ =(d38nH"# @SCUs*P P P P# P S%S/)* P  3:sC $% Pn!* +F +F +F +F~ +F  4Z +F\   #  S%S/)*   4Z "  #)C. #U5%<5H # #&  SYs^2 2 2~2s^d 2j  T%d E3 6788 9 ~$   @!* Y Y Y Y# Y ~ Y  4Z Yx        38_t (W W / "W /WY$Os2J