Ƒi* %SrSSKrSSKrSSKrSSKrSSKrSSKJr Sr \ \ S'Sq \ \\4S-\ S'\R "5rSq\ \\\\S-\\44S-\ S'\R "5rSq\ \\4S-\ S'\R "5r0r\ \\4\ S '\R4"5H\r\"\R:5S :XdM\R:S\\R<'\R>Hr \R:S\\ 'M M^ S \ \\44S jr!S \ \\\\S-\\444S jr"S\S \S-4Sjr#S\S \$4Sjr%S \ \\44Sjr&"SS5r'SS\'S\S\S \4Sjjr(SS\)S\S\'S-S \\\S-44Sjjr*g)zModel loading and bigram scoring utilities. Note: ``from __future__ import annotations`` is intentionally omitted because this module is compiled with mypyc, which does not support PEP 563 string annotations. N)REGISTRYNON_ASCII_BIGRAM_WEIGHT _MODEL_CACHE _ENC_INDEX _MODEL_NORMS_SINGLE_LANG_MAPreturnc[b[$[ 0n[RR S5R S5nUR 5nU(d'[R"S[SS9 UqUsSSS5 $Sn[R"SX#5unUS - nUS :aS US 3n[U5e[U5Hn[R"SX#5unUS - nUS :aSUS3n[U5eX#X7-RS5nX7- n[R"SX#5un US - nU S:aSU S3n[U5e[S5n [U 5H,n[R"SX#5upn US- nXU S-U -'M. XU'M UqUsSSS5 $![R ["4anSU3n[U5UeSnAff=f!,(df  g=f)zLoad all bigram models from the bundled models.bin file. Each model is a bytearray of length 65536 (256*256). Index: (b1 << 8) | b2 -> weight (0-255). :returns: A dict mapping model key strings to 65536-byte lookup tables. Nzchardet.modelsz models.binuXchardet models.bin is empty — statistical detection disabled; reinstall chardet to fix) stacklevelrz!Ii'z"corrupt models.bin: num_encodings=z exceeds limitzcorrupt models.bin: name_len=z exceeds 256zutf-8z corrupt models.bin: num_entries=z exceeds 65536z!BBBrzcorrupt models.bin: )r_MODEL_CACHE_LOCK importlib resourcesfilesjoinpath read_byteswarningswarnRuntimeWarningstruct unpack_from ValueErrorrangedecode bytearrayerrorUnicodeDecodeError)modelsrefdataoffset num_encodingsmsg_name_lenname num_entriestableb1b2weightes W/var/www/html/banglarbhumi/venv/lib/python3.13/site-packages/chardet/models/__init__.py load_modelsr4&s (*!!''(89BB<P~~ MM+   "L!  $ )F%11$E ] aKFv%:=/X o%=)$00tD ! c>9(<PC$S/)V%67>>wG"!'!3!3D$!G! &<[MXC$S/)!%({+A%+%7%7%MNBFaKF,227b.), %t '*0 i  ^ 01 )(,CS/q ( )_  s1A%GDF'G'GGGG G(c[b[$[ [5n0nUR5H<up#UR SS5upEUR U/5R XCU45 M> 0n[R"5H$nURHnURXh'M M& UR5HupX;dM X;dMXX'M UqUsSSS5 $!,(df  g=f)zTReturn a pre-grouped index mapping encoding name -> [(lang, model, model_key), ...].N/r ) r_ENC_INDEX_LOCKr4itemssplit setdefaultappendrvaluesaliasesr,) r$indexkeymodellangencalias_to_primaryentryaliasprimarys r3 get_enc_indexrGis DF ,,.JC #q)ID   S" % , ,d3-? @) ,.__&E*/** '''/446NE~'"6!&7 ) sB4C* C* C** C8encodingc,[RU5$)zReturn the language for a single-language encoding, or None. :param encoding: The canonical encoding name. :returns: An ISO 639-1 language code, or ``None`` if the encoding is multi-language. )r getrHs r3infer_languagerLs    ))cU[5;$)zReturn True if the encoding has language variants in the model index. :param encoding: The canonical encoding name. :returns: ``True`` if bigram models exist for this encoding. )rGrKs r3has_model_variantsrOs } &&rMc0[b[$[ [5n0nUR5HEup#Sn[ S5HnX5nU(dMXFU-- nM [ R "U5X'MG UqUsSSS5 $!,(df  g=f)zAReturn cached L2 norms for all models, keyed by model key string.Nrr)r_MODEL_NORMS_LOCKr4r8rmathsqrt)r$normsr?r@sq_sumivs r3_get_model_normsrXs "$ ,,.JCF5\H1!eOF"6*EJ )    s;B*B BcT\rSrSrSrSrS\SS4Sjr\S\ \ \ 4SS4S j5r Sr g) BigramProfileaPre-computed bigram frequency distribution for a data sample. Computing this once and reusing it across all models reduces per-model scoring from O(n) to O(distinct_bigrams). Stores a single ``weighted_freq`` dict mapping bigram index to *count * weight* (weight is 8 for non-ASCII bigrams, 1 otherwise). This pre-multiplies the weight during construction so the scoring inner loop only needs a single dict traversal with no branching. ) input_norm weight_sum weighted_freqr&r Nc[U5S- nUS::a0UlSUlSUlg0nSn[nUR n[ U5HInXnXS-n US-U -n US:dU S:aU"U S5U-X:'XE- nM6U"U S5S-X:'US- nMK X0lX@l[R"[SUR5555Ulg)zbCompute the bigram frequency distribution for *data*. :param data: The raw byte data to profile. r rNrc3*# UH oU-v M g7fN.0rWs r3 )BigramProfile.__init__..s'E}!A}) lenr^r]r\rrJrrRrSsumr<) selfr& total_bigramsfreqw_sumhi_w_getrVr/r0idxs r3__init__BigramProfile.__init__s D A A 13D #$DO%(DO !&xx}%AB!eB7b.CDyBI aL4/   aL1,  &"))C'Et{{}'E$EFrMr^cU"S5nXl[UR55Ul[R "[SUR5555UlU$)a#Create a BigramProfile from pre-computed weighted frequencies. Computes ``weight_sum`` and ``input_norm`` from *weighted_freq* to ensure consistency between the three fields. :param weighted_freq: Mapping of bigram index to weighted count. :returns: A new :class:`BigramProfile` instance. rMc3*# UH oU-v M g7frcrdres r3rg3BigramProfile.from_weighted_freq..s*Q:PQq5:Pri)r^rkr<r]rRrSr\)clsr^profiles r3from_weighted_freq BigramProfile.from_weighted_freqsSc( - !5!5!78!YYs*Q-:N:N:P*Q'QRrM) __name__ __module__ __qualname____firstlineno____doc__ __slots__bytesrs classmethoddictintrz__static_attributes__rdrMr3rZrZsN >IGUGtG< tCH~ /  rMrZryr@ model_keyc|URS:Xag[5nU(aURU5OSnUc>Sn[S5HnXnU(dMXWU-- nM [R "U5nUS:XagSnUR R5HupXU U -- nM XUR-- $)zSScore a pre-computed bigram profile against a single model using cosine similarity.r`Nrr)r\rXrJrrRrSr^r8) ryr@rrT model_normrUrVrWdotrrwcounts r3score_with_profilersS   E)29%JuAAqa%YYv& S C,,224  SzF""5 w111 22rMr&cU(dUcg[5nURU5nUcgUc [U5nSnSnUHupxn [X(U 5n X:dMU nUnM XV4$)ahScore data against all language variants of an encoding. Returns (best_score, best_language). Uses a pre-grouped index for O(L) lookup where L is the number of language variants for the encoding. If *profile* is provided, it is reused instead of recomputing the bigram frequency distribution from *data*. :param data: The raw byte data to score. :param encoding: The canonical encoding name to match against. :param profile: Optional pre-computed :class:`BigramProfile` to reuse. :returns: A ``(score, language)`` tuple with the best cosine-similarity score and the corresponding language code (or ``None``). N)r`Nr`)rGrJrZr) r&rHryr>variants best_score best_langrAr@rss r3score_best_languagersz& GO OEyy"H%J I"*Y wy 9 >JI #+   rM)rc)+rimportlib.resourcesrrRr threadingrchardet.registryrrr__annotations__rrstrr!Lockrrlisttupler7rfloatrQr r<_encrj languagesr,r=_aliasr4rGrLboolrOrXrZrrrrdrMr3rs % ! ,0 d3 >"T)0NN$HL Dd5tY!;<== > EL.."(, d3:%,NN$$&$sCx.% OO D 4>>a&*nnQ&7#llF'+~~a'8 V $# @T#y.)@FtCeC$J 3,F&G!HHI8*S*S4Z*'''$sEz**::|@B3 3#,39<3 34%)&! &!&!T !&! 5#*  &!rM