ƑiZ%SrSSKJr SSKJr SSKJr SSKJrJ r J r J r SSK J r JrJrJr SSKJr SSKJr SS KJr SS KJr SS KJr SS KJr SS KJr SSKJ r J!r!J"r" SSK#J$r$ SSK%J&r& SSK'J(r( SSK)J*r*J+r+J,r, \"S\ SS9r-\"SSSS9r.\"SSSS9r/Sr0\1"1Sk5r2S\3S'\1"1Sk5r4S\3S'\1"1Sk5r5S\3S '\1"1S!k5r6S\3S"'\4\5\6S#.r7S$\3S%'\1"1S&k5r8S\3S''S8S(jr9S)r:S*r;S+rS9S.jr?S:S/jr@S;S0jrAS;S1jrBS2rCSu@Pipeline orchestrator — runs all detection stages in sequence.) annotations)DEFAULT_MAX_BYTES) EncodingEra) BigramProfilehas_model_variantsinfer_languagescore_best_language)DETERMINISTIC_CONFIDENCE HIGH_BYTESDetectionResultPipelineContext) detect_ascii) is_binary) detect_bom)resolve_confusion_groups)detect_escape_encoding)detect_markup_charset)score_candidates)compute_lead_byte_diversitycompute_multibyte_byte_coveragecompute_structural_score) detect_utf8)detect_utf1632_patterns)filter_by_validity)REGISTRY EncodingInfoget_candidatesNencoding confidencelanguageutf-8g? windows-1252g333333?> iso-8859-1 iso-8859-15r#zfrozenset[str]_COMMON_LATIN_ENCODINGS>.zfrozenset[int]_ISO_8859_10_DISTINGUISHING>r'r(r*r+r,r-r/r0r1r3r4r5r6r7r8r:r;r<r=r>r?r@rArIrR_ISO_8859_14_DISTINGUISHING>rXrYrZr[_WINDOWS_1254_DISTINGUISHING)z iso-8859-10z iso-8859-14z windows-1254zdict[str, frozenset[int]]_DEMOTION_CANDIDATES> r'r(r+r8_KOI8_T_DISTINGUISHINGcj^[RU5mTcg[U4SjU55(+$)aaReturn True if encoding is a demotion candidate with no distinguishing bytes. Checks whether any non-ASCII byte in *data* falls in the set of byte values that decode differently under the given encoding vs iso-8859-1. If none do, the data is equally valid under both encodings and there is no byte-level evidence for preferring the candidate encoding. Fc3<># UHoS:dM UT;v M g7fN).0bdistinguishings ]/var/www/html/banglarbhumi/venv/lib/python3.13/site-packages/chardet/pipeline/orchestrator.py !_should_demote..sA1D&1&s  )r`getany)rdatarqs @rr_should_demoterxs2*--h7NAAA AAg?gffffff?cD/nUGH nUR(a[XU5nXRRUR'U[:aMEUR c0[ U5[ URS[55- UlUR [:aM[XX"R S9nXbRUR'U[:aMUR [:a[XU5nU[:aMUR!U5 GM [#U5$)aEliminate CJK multi-byte candidates that lack genuine multi-byte structure. Four checks are applied in order to each multi-byte candidate: 1. **Structural pair ratio** (valid_pairs / lead_bytes) must be >= ``_CJK_MIN_MB_RATIO``. Catches files with many orphan lead bytes. 2. **Minimum non-ASCII byte count**: the data must contain at least ``_CJK_MIN_NON_ASCII`` bytes > 0x7F. Tiny files with 1-5 high bytes can accidentally form perfect pairs and score 1.0 structurally. 3. **Byte coverage** (non-ASCII bytes in valid multi-byte sequences / total non-ASCII bytes) must be >= ``_CJK_MIN_BYTE_COVERAGE``. Latin text has many high bytes that are NOT consumed by multi-byte pairs; genuine CJK text has nearly all high bytes accounted for. 4. **Lead byte diversity**: the number of distinct lead byte values in valid pairs must be >= ``_CJK_MIN_LEAD_DIVERSITY``. Genuine CJK text draws from a wide repertoire of lead bytes; European false positives cluster in a narrow band (e.g. 0xC0-0xDF for accented Latin). Returns the filtered candidate list. Structural scores are cached in ``ctx.mb_scores`` for reuse in Stage 2b. N)non_ascii_count) is_multibyter mb_scoresname_CJK_MIN_MB_RATIOr~len translater _CJK_MIN_NON_ASCIIr mb_coverage_CJK_MIN_BYTE_COVERAGE_CJK_DIVERSITY_MIN_NON_ASCIIr_CJK_MIN_LEAD_DIVERSITYappendtuple)rwvalid_candidatesctxgatedencmb_score byte_coveragelead_diversitys rr_gate_cjk_candidatesrs:!#E   /3?H&.MM#(( #++""*&)$i#dnnT:6V2W&W#""%77;30C0CM)6OOCHH %55""&BB!= 0.95) receive a confidence boost proportional to coverage. When nearly all non-ASCII bytes form valid multi-byte pairs, the structural evidence is strong and should increase the candidate's ranking relative to single-byte alternatives whose bigram models may score higher on small samples. Note: boosted confidence values may exceed 1.0 and are used only for relative ranking among candidates. ``run_pipeline`` clamps all confidence values to [0.0, 1.0] before returning to callers. c3B># UHupUT;dM TUv M g7fNrn)ror_sc enc_lookups rrrs/_score_structural_candidates..5s&*;YTtz?Q 4*;s  c3J# UHoR(aMUv M g7fr)r)roes rrrsr8sJ#3a>>#3s# #gffffff?rcUR$r)r xs rr._score_structural_candidates..Isq||ryTkeyreverse) rrrlistrrrrurr r r!sort) rwstructural_scoresrrrvalid_mb single_byteresultsboostedrcoveragers @rr_score_structural_candidatesrs .&6H%5)!&&!)%5HJ*;HJ#3JJK#D*CH*C{*CDEG&(G ;<::3??&&qzz373 t  NNZZ ||q8|<ZZ  NN1  LL+TL: N-Is D!D!c[U5S:aUSRb[USRU5(aUSRnUSSHlnUR[;dMUVs/sHoDRU:wdMXCLdMUPM nnUVs/sHoDRU:XdMUPM nnU/UQUQs $ U$s snfs snf)aDemote niche Latin encodings when no distinguishing bytes are present. Some bigram models (e.g. iso-8859-10, iso-8859-14, windows-1254) can win on data that contains only bytes shared with common Western Latin encodings. When there is no byte-level evidence for the winning encoding, promote the first common Western Latin candidate to the top and push the demoted encoding to last. rrN)rrrxr&)rwrdemoted_encodingrrothersdemoted_entriess rr_demote_niche_latinrMs G q AJ   + 71:.. 5 5"1:..Azz44&&!**8H*HAQZAw/6"XgGW9W1g"X5F5_55  N #Ys/CC CC /C c4U(aUSRS:waU$Sn[U5Hup4URS:XdMUn O UcU$[SU55(a/Xn[U5VVs/sHup4X2:wdM UPM nnnU/UQ$U$s snnf)agPromote KOI8-T over KOI8-R when Tajik-specific bytes are present. KOI8-T and KOI8-R share the entire 0xC0-0xFF Cyrillic letter block, making statistical discrimination difficult. However, KOI8-T maps 12 bytes in 0x80-0xBF to Tajik-specific Cyrillic letters where KOI8-R has box-drawing characters. If any of these bytes appear, KOI8-T is the better match. rzkoi8-rNzkoi8-tc3B# UHoS:dM U[;v M g7frl)ri)rorps rrrs!_promote_koi8t..s A1D &1& &s )r enumeraterv)rwr koi8t_idxir koi8t_resultrs rr_promote_koi8tris gaj))X5I'" :: !I # A AAA) )' 2E 2an! 2E&v&& NFs 5 BBic|US:XaU$URUSS9RSSS9$![[4a gf=f)a<Decode data from encoding and re-encode as UTF-8 for language scoring. Returns None if the encoding is unknown. For UTF-8, returns data as-is. Uses ``errors="ignore"`` because the data already passed byte-validity filtering for the detected encoding; any residual invalid bytes are irrelevant for language scoring. r"ignore)errors surrogatepassN)decodeencode LookupError TypeError)rwrs rr_to_utf8rsY7 {{8H{5<< O=    #s (;;c b/nSnSnUGH!nURcURb[UR5nUcEU(a>[UR5(a$Uc [ U5n[ XRUS9upvUc^U(aW[S5(aG[ XR5nU(a+UbURS:wa [ U5n[ USUS9upvUb1UR[URURUS95 GMURU5 GM$ U$)aFill in language for results missing it. Tier 1: single-language encodings via hardcoded map (instant). Tier 2: multi-language encodings via statistical bigram scoring (lazy). Tier 3: decode to UTF-8, score against UTF-8 language models (universal fallback). N)profiler"r) r!rrrrr rrr r ) rwrfilledr utf8_profileresultlang_ utf8_datas rr_fill_languagers %'F$(G)-L ?? "v'B!&//2D|);FOO)L)L?+D1G-dOOWU|);G)D)D$T??; #+v'/I'4Y'? 1!7LGA #!'#)#4#4!% f78 MrycD[X5n[X5n[X5$)zGApply confusion resolution, niche Latin demotion, and KOI8-T promotion.)rrr)rwrs rr_postprocess_resultsrs# 't5G!$0G $ ((ryc[5nUSUnU(d[/$[U5nUbU/$[U5nUbU/$[ U5nUbFUR b9[ R"UR 5nUbXR-(aU/$[U5nUc[XS9(a[/$[U5n U bU /$[U5n U bU /$UbU/$[U5n [X 5n U (d[ /$[#X U5n U (d[ /$/n U HonUR$(dMUR&RUR(5nUc [+XU5nUS:dMRU R-UR(U45 Mq U (a;U R/SSS9 U SunnU[0:a[3X X5n[5UU5$[7[9U[;U 555nU(d[ /$[5UU5$)zBCore pipeline logic. Returns list of results sorted by confidence.N) max_bytesrc US$)Nrrnrs rrr$_run_pipeline_core..0sQqTryTrr)r _EMPTY_RESULTrrrrrruerarr_BINARY_RESULTrrrr_FALLBACK_RESULTrrrrrrr _STRUCTURAL_CONFIDENCE_THRESHOLDrrrrr)rw encoding_erarr bom_resultutf1632_result escape_resultenc_info utf8_precheck markup_result ascii_result candidatesrrrscorer best_scorers rr_run_pipeline_corers,  C   D D!J| -T2N!+40M ]%;%;%G<< 6 67  |ll:!? " %M4!E *$/M  %L~  -J)$;  !!,DCH  !!24    MM%%chh/E}0C@s{!((#((E):;  >4@)!, : 9 92)9G(g6 6#D%0@*ABCG  !! g ..ryc "[XU5n[US[U5nU(d Sn[U5eUVs/sHJnURS:a5[ UR [URS5UR5OUPML sn$s snf)aRun the full detection pipeline. :param data: The raw byte data to analyze. :param encoding_era: Filter candidates to a specific era of encodings. :param max_bytes: Maximum number of bytes to process. :returns: A list of :class:`DetectionResult` sorted by confidence descending. Nz/pipeline must always return at least one resultg?) rr_LANG_SCORE_MAX_BYTES RuntimeErrorr r rminr!)rwrrrmsgrs rr run_pipeliner@s!Y?GT"8#897CG ?3  A <<#   C c$:AJJG     sAB )rstrrwbytesreturnbool)rwrrtuple[EncodingInfo, ...]rr rr) rwrrzlist[tuple[str, float]]rrrr rlist[DetectionResult])rwrrrrr)rwrrrrz bytes | None)rwrrrrintrr)I__doc__ __future__rchardet._utilsr chardet.enumsrchardet.modelsrrrr chardet.pipeliner r r r chardet.pipeline.asciirchardet.pipeline.binaryrchardet.pipeline.bomrchardet.pipeline.confusionrchardet.pipeline.escaperchardet.pipeline.markuprchardet.pipeline.statisticalrchardet.pipeline.structuralrrrchardet.pipeline.utf8rchardet.pipeline.utf1632rchardet.pipeline.validityrchardet.registryrrrrrrr frozensetr&__annotations__rUr\r_r`rirxrrrrrrrrrrrrrrrrnryrrrsF",%  0-+?:99 .<8CC 6  TDQ # t $( +4+/8/1/^1n/8 "/^"T09(0n/.03/*3L* B$   "3 3.3 3 3l- -.-/-  -  -`  "8  "B$( (/((V) ) "))'i/ i/i/i/ i/^'  ry