|-j,dZddlZddlZddlZddlmZddlmZddlm Z ddl m Z ddl m ZddlmZddlmZmZdd lmZdd lmZdd lmZdd lmZmZdd lmZm Z m!Z!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2ddl3m4Z4m5Z5m6Z6e6j7e8Z9dZ:dZ;dZe,dz Z,ee e!e"dZ?e:e=dZ@e5e,Gdde/ZAeAZBdS)z Tokenization classes for fast tokenizers (provided by HuggingFace's tokenizers library). For slow (python) tokenizers see tokenization_utils.py N) defaultdict)Iterable)copyfile)Any)is_offline_mode) AddedToken processors)Encoding) Tokenizer)Decoder)BPEUnigram) BpeTrainerUnigramTrainerWordLevelTrainerWordPieceTrainer cached_file) SpmConverter)convert_gguf_tokenizer)load_gguf_checkpoint)INIT_TOKENIZER_DOCSTRING BatchEncodingPreTokenizedInputPreTrainedTokenizerBase TextInputTruncationStrategygenerate_merges)PaddingStrategyadd_end_docstringsloggingztokenizer.jsonzspecial_tokens_map.jsonztokenizer_config.jsonztokenizer.modelzadded_tokens.jsonu tokenizer_object ([`tokenizers.Tokenizer`]): A [`tokenizers.Tokenizer`] object from 🤗 tokenizers to instantiate from. See [Using tokenizers from 🤗 tokenizers](../fast_tokenizers) for more information. tokenizer_file ([`str`]): A path to a local JSON file representing a previously serialized [`tokenizers.Tokenizer`] object from 🤗 tokenizers. )r r WordLevel WordPiece)tokenizer_file vocab_filec)eZdZdZeZdZdZedLdZ fdZ e de fdZ e de fdZdMd ed edzdeefd Zd Ze d Ze dZejdZejdZdZe defdZdeeeffdZe deeeffdZe deeeffdZe deeeffdZeZeZ deeeffdZ!de fdZ"defdZ#e de$fdZ%e de&fdZ' dNde(de dzde dzd e d!e d"e d#e d$e deeee)fe*e(ffd%Z+d&edefd'Z,d(ededzfd)Z-dLd*e*eezdefd+Z.dLd,e defd-Z/dLd.ee*ezd/e dee*ezfd0Z0dOd1ed,edzd2e de*efd3Z1d4e2d5e3d6ed7ed8edzd9edzf d:Z4dde2j5e3j6dd;ddddddddddddfd1e7e8ze*e7ze*e8zde dzde dzde dzd e d!e d"e d#e d$e d?e dzde9f(d@Z:dAe*edefdBZ; dPdCee*ezd/e dDe dzdefdEZ< dQd ee=j>zdFeedGfdHe dzd edzdeedGff dIZ? dRdJZ@e dSdKZAxZBS)TTokenizersBackendaQ Base class for all fast tokenizers (wrapping HuggingFace tokenizers library). Inherits from [`~tokenization_utils_base.PreTrainedTokenizerBase`]. Handles all the shared methods for tokenization and special tokens, as well as methods for downloading/caching/loading pretrained tokenizers, as well as adding tokens to the vocabulary. This class also contains the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). NFc $t|}|dd}|Ltj|r-|t us d|jvs|rtj||d<|S|ntj|rNt|d5}tj |}dddn #1swxYwY| di d}|d vret|}t|d} i| d <|d krg| d <| |d<g|d <tj tj|} ntj|} | j|d<| j|d<| j|d<| j | j|d<| j | j|d<| d} | r| dddkr | d} nt%| t&s| g} | D]C} | ddkr(d| vr$ddl} | | d|d<nD| di d d}|j8t%|t&r"t't/t0|}n|jjdkrFt%|t&r0|r.t%|dt&t0fr d|D}nx|jjdkrdt5|D}nN|jjd ks|jjdkr.t%|t&rdt5|D}||d <t7|dd}d | divr,|r*|jd kr|dd }d |D}||d <|S| d!}| d"}| d }| d }t%|t8rd|d#rOtj|r0d$d%lm}||& |\|d <|d <|St%|t8rtj|r|d'r d$d(lm!}||}|j"|jfi|} d$d)lm#}| |j}|tI|d*r |j%dCi|}n=#tL$r0}tN(d+|jd,|d-Yd}~nd}~wwxYwtI|d.r |j)dCi|}d|vr|t us d|jvr|d d}|d d}| d/pi}||rd0|*D}|*D]a\}}tW|}t9|}| |}|r'||kr!||vr||||<|||<btYj-|j.||1}|||d<|j.j/}|j0dkr|1d2|j2pd3|j3dkr|1d4|j4pd5|j5dkr|1d6|j6pd7nu#tL$rh}tN(d8|d9|d:d$d;lm7}||| d<=} | 8|d<Yd}~nd}~wwxYw|S|At%|t8r,tj|r ||d <|d }|At%|t8r,tj|r ||d <|d }||j|jjd krt%|trd>trttd?t&t8f$fd@ $gdA}!tw}"|!D]+}#|#|vr%|"<$||#g,t{||"B}||d <|S)Dz Build a `tokenizers.Tokenizer` backend from the available serialization files (tokenizer.json, sentencepiece models, tekken.json, vocab/merges). r%N__init__tokenizer_objectutf-8encodingmodeltype)Nrvocabr merges added_tokenspost_processortokenizer_paddingtokenizer_truncation_json_truncation _json_padding normalizerSequence normalizers Precompiledprecompiled_charsmapr_spm_precompiled_charsmaprc,g|]}t|S)tuple).0items j/var/www/html/banglarbhumi/venv/lib/python3.11/site-packages/transformers/tokenization_utils_tokenizers.py z>TokenizersBackend.convert_to_native_format..s;;;TU4[[;;;r#ci|]\}}|| Sr@r@rBitokens rD z>TokenizersBackend.convert_to_native_format..sCCChaCCCrFr$cTi|]%\}}t|tr|dn||&Sr) isinstancelistrHs rDrKz>TokenizersBackend.convert_to_native_format..s8pppS[STV[E4)@)@KU1XXeQppprFcg|]H}t|tr"t|dnt|IS) )rNstrrAsplit)rBmerges rDrEz>TokenizersBackend.convert_to_native_format..sKrrrbgZs5K5K]% C 0 0111QVW\Q]Q]rrrrFr& merges_filez tekken.jsonr)MistralConverter)r&.model)SentencePieceExtractor)SLOW_TO_FAST_CONVERTERSconvert_from_spmz,Could not reorder vocab using converter for z due to z/. Falling back to raw SentencePiece extraction.convert_from_spm_modeladded_tokens_decoderci|]\}}|| Sr@r@)rBrJtoken_ids rDrKz>TokenizersBackend.convert_to_native_format..s&\&\&\?5(x&\&\&\rF)protor1r2 bos_token eos_token unk_tokenzz+Could not extract SentencePiece model from z$ using sentencepiece library due to z%. Falling back to TikToken extractor.)TikTokenConverterextra_special_tokens)r&rfvaluesreturncg}|D]b}|t|ttfr||@|t |c|SN)rNrOrAextendappendrR)rg collectedval_iter_special_tokenss rDrozHTokenizersBackend.convert_to_native_format.._iter_special_tokens)s{') !33C{ !#e}553!(()=)=c)B)BCCCC!((S2222  rF) pad_tokenrdr`rb sep_token cls_token mask_tokenadditional_special_tokensrf) skip_tokensr@)>dictpopospathisfiler(__dict__ TokenizerFast from_fileopenjsonloadgetfrom_strdumpsr4padding truncationrNrObase64 b64decoder/maprA__name__ enumerategetattrrRendswithconvert_slow_tokenizerrVextract_vocab_merges_from_modelrXextractrYhasattrrZ Exceptionloggerwarningr[itemsintrbuild_tokenizer_from_spm_protor_ trainer_specbos_id setdefault bos_pieceeos_id eos_pieceunk_id unk_piecere convertedrrsetupdater)%clstrust_remote_codekwargs local_kwargsfast_tokenizer_filetokenizer_handletokenizer_json model_typeminimal_tokenizer_json minimal_model tok_from_filenormalizer_configr9rr1r2r&rUrVrX extractorrYconverter_classer\ id_to_tokenr^ new_token current_tokenr+ proto_specre converterspecial_tokens_keysrukeyros% @rDconvert_to_native_formatz*TokenizersBackend.convert_to_native_formates F|| *../?FF +233 ,)))Zs|-K-KO`-K/>CY]gCgCg% DJDTDT&'=>EE %@A"&&w3377FFEy eT**4 UE!2!233E#y00eT**--gr:::: :zObfkOkOk'0:rrkqrrr)/ X& !%%l33 "&&}55   ))!!(++ j# & & :+>+>}+M+M RTRYR`R`akRlRl @ @ @ @ @ @>*+E+EH *J]J]^fJgJgH F IJJJJJJ32:>> 0y0KKlKK  OOOOOO&=&A&A#,&O&OO&2wPb7c7c2'G'G'W'W,'W'W  NNPs|PP]^PPP3 899N#=3#=#M#M #M#ML &\99,,, #,0N0N(,,Wd;;E)--h==F,8+;+;jc::>rw~~k?Z?Z>%0L "!(+F >ci3 8Je8S8SXbchjnXoXo8S !Xc] !tCy ! ! ! ! ! ! # # # %(EEK* R R,&&&&';';\#=NNr7r8r>r+ gguf_filer%r\add_prefix_spaceFr&r1r2 name_or_pathconfigr tokenizertokenizer_configrc i|] \}\}}|| Sr@r@)rBrIw_s rDrKz.TokenizersBackend.__init__..qs%CkCkCkYQPVQRTUAqCkCkCkrFT)r1r2fuse_unkdropoutr)r1ra9Couldn't instantiate the backend tokenizer from one of: (1) a `tokenizers` library serialization file, (2) a slow tokenizer instance to convert or (3) an equivalent slow tokenizer class to instantiate and convert. You need to have sentencepiece or tiktoken installed to convert a slow tokenizer to a fast one.r`rarbrcz3The backend tokenizer is not correctly initialized.r6 max_lengthtruncation_side directionstridetruncation_strategystrategyr5rppad_token_type_id pad_type_id padding_sidelengthpad_to_multiple_ofbackend tokenizers add_bos_token add_eos_tokenr4cFh|]}tt|Sr@hashreprrBrJs rD z-TokenizersBackend.__init__..s&$^$^$^5T$u++%6%6$^$^$^rFcVg|]%\}}tt|v#|&Sr@r)rBindexrJadded_tokens_decoder_hashs rDrEz.TokenizersBackend.__init__..sA   uDKK  (AAA AAArFc|dSNrr@)xs rDz,TokenizersBackend.__init__..s STUVSWrFrc,g|]}t|Sr@rRrs rDrEz.TokenizersBackend.__init__..s;b;b;b5CJJ;b;b;brFc0g|]}|t|Sr@r)rBts rDrEz.TokenizersBackend.__init__..s$WWW1UVWAWWWrF)speciali pre_tokenizerfix_mistral_regex) init_kwargsrr@):rwrcopydeepcopyrxryrzr|r}rrrrlen _tokenizerrNrvrr rOrAr ValueErrorrrenable_truncation no_truncationrenable_padding_add_bos_token_add_eos_tokenr4_should_update_post_processorsuperr*r&rsplit_special_tokensencode_special_tokensr\sortedradded_tokens_encoderkeys_special_tokens_maprgrRrl_extra_special_tokensrr add_tokensget_vocab_sizeNotImplementedErrorr_patch_mistral_regexrupdate_post_processor)"selfargsrr7r8r+rrr\rr&r1r2fast_tokenizer gguf_path gguf_param architecturetokenizer_dictradditional_kwargs vocab_dict _truncation_paddingexplicit_bos_eos_in_kwargsr4 tokens_to_addencoderspecial_token_valuerJtokensall_named_tokens vocab_sizer __class__s" @rDr*zTokenizersBackend.__init__Hs"::&8$?? ?D99   .555!::&8$??JJ{D11 $jj)94@@%zz*@"EE!::&8%@@ZZ --  7##H%%  '!]+;<>p$/B\p`p  " -DO - < < < < <   lK ,E F F F   / J J J   h H(= > > >   3[5L M M M M O ) ) + + +::1488dDO ? ? ?   2H=Q4R S S S F " " ,F9 %4%>%[/U[B["$jj%@@$jj%@@#ZZ(8$?? ?> <-;DO *-G-q4?KimqKq*""6"""  !(DO 0040I-$^$^DD]$^$^$^!    &';'A'A'C'C X X X   t0557788;b;bTa;b;b;bb$(#;#B#B#D#D : : "*&''w66;NVc;c;c$$%8999/ , ,E5zz((U--G-G$$U+++ }   ! !FWW0H0O0O0Q0QWWW & % %eS))-&ud;;;EEz22- =-SZZ;K-K-K(,  e$$$$ (''' 7799JJ"   JJJ    74?OT#R#R#^ JJ{D ) ) )7d7 $$^T::!,"(**-@$"G"G   DO  . X$/2PTX2X *  - )  & & ( ( ( ( ( ) )s] ]/.]/rhcdS)NTr@rs rDis_fastzTokenizersBackend.is_faststrFcd|jvr]|jddr=t|dr+|jr$tj|jSdSdS)z `bool`: Whether or not the slow tokenizer can be saved. For a sentencepiece based slow tokenizer, this can only be `True` if the original `"sentencepiece.model"` was not deleted. r&rWFT)vocab_files_namesrrr&rxryrzrs rDcan_save_slow_tokenizerz)TokenizersBackend.can_save_slow_tokenizersi 41 1 1d6L\6Z6c6cdl6m6m 1t\** 7t 7w~~do66654rFsave_directoryfilename_prefixctj|s td|ddStj||r|dzndt dz}tj|jtj|krt|j||fS)NzVocabulary path (z) should be a directory-rr&) rxryisdirrerrorjoinVOCAB_FILES_NAMESabspathr&r)rrrout_vocab_files rDsave_vocabularyz!TokenizersBackend.save_vocabularysw}}^,,  LLT^TTT U U U F oM_s222QbcoQpp   7??4? + +rw~/N/N N N T_n 5 5 5  rFc|j}|j}||jrd|_|j}|j}||jrd|_|jr|dzndd|jrd|zdznd}||jrd|zdzndd |jrd|zdznd}g}|jr|||f|jr|||ftj||| |j _ dS) ze Updates the underlying post processor with the current `bos_token` and `eos_token`. NFz:0 rz$A:0rQz:0z:1z $B:1)singlepairspecial_tokens) r` bos_token_idrrb eos_token_idrrlr TemplateProcessingrr4)rbosr+eosr,r(r)r*s rDrz'TokenizersBackend.update_post_processor s]n( ;4-;!&D n( ;4-;!&D %)%7?S5[[Rww[_[mEucCiRVFVFVsuwwD0BJ39t++DDgkgyRBRUX[R[^bRbRb@BDD   7  ! !3 "5 6 6 6   7  ! !3 "5 6 6 6)3)F^* * * &&&rFc$t|ddS)NrFrrs rDrzTokenizersBackend.add_eos_token$t-u555rFc$t|ddS)NrFr1rs rDrzTokenizersBackend.add_bos_token(r2rFcft|d||dS)Nrobject __setattr__rrvalues rDrzTokenizersBackend.add_eos_token,34!15999 ""$$$$$rFcft|d||dS)Nrr5r8s rDrzTokenizersBackend.add_bos_token1r:rFcg}|jD]j}|t|tr||0t|t r%|t|ddk|jD]g}t|tr||-t|t r%|t|ddh|r||dt|dds |j j | dSdS)a[ Post-initialization hook that runs after the tokenizer is fully set up. This is called by from_pretrained() after loading the tokenizer, which allows us to add any special tokens that may have been passed as AddedToken objects. Child classes should call super()._post_init() if they override this method. NTF)r normalized)r*r) rrgrNrrlrRrrrrr4r)rr token_valuerJs rD _post_initzTokenizersBackend._post_init6sl 3::<< ^ ^K"+z22 ^$$[1111K-- ^$$Z TV[%\%\%\]]]/ X XE%,, X$$U++++E3'' X$$ZtPU%V%V%VWWW  @ OOM$O ? ? ? 48$ ? ? )4?CaCi  & & ( ( ( ( (DjCirFc8|jdS)zP `int`: Size of the base vocabulary (without the added tokens). Fwith_added_tokensrrrs rDrzTokenizersBackend.vocab_sizeVs ---FFFrFc8|jdS)NTrA)r get_vocabrs rDrEzTokenizersBackend.get_vocab]s((4(@@@rFc*|Srj)rErs rDr1zTokenizersBackend.vocab`s~~rFchdt|jdDS)z Returns the sorted mapping from string to index. The added tokens encoder is cached for performance optimisation in `self._added_tokens_encoder` for the slow tokenizers. c$i|] \}}|j|Sr@contentrBvks rDrKz:TokenizersBackend.added_tokens_encoder..j mmmA 1mmmrFc|dSrr@rCs rDrz8TokenizersBackend.added_tokens_encoder..j dhijdkrFrrr\rrs rDrz&TokenizersBackend.added_tokens_encoderds9 nm0I0O0O0Q0QWkWk)l)l)lmmmmrFc4|jS)z Returns the added tokens in the vocabulary as a dictionary of index to AddedToken. Returns: `dict[str, int]`: The added tokens. )rget_added_tokens_decoderrs rDr\z&TokenizersBackend.added_tokens_decoderls77999rFchdt|jdDS)z Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: `dict[str, int]`: The added tokens. c$i|] \}}|j|Sr@rIrKs rDrKz5TokenizersBackend.get_added_vocab..rNrFc|dSrr@rPs rDrz3TokenizersBackend.get_added_vocab..rQrFrrRrs rDget_added_vocabz!TokenizersBackend.get_added_vocab{s9nm0I0O0O0Q0QWkWk)l)l)lmmmmrFcdS)zN Returns True, to avoid expensive `assert tokenizer` gotchas. Tr@rs rD__bool__zTokenizersBackend.__bool__s trFc8|jdS)zD Size of the full vocabulary with the added tokens. TrArCrs rD__len__zTokenizersBackend.__len__s---EEErFc|jS)zc `tokenizers.implementations.BaseTokenizer`: The Rust tokenizer used as a backend. )rrs rDbackend_tokenizerz#TokenizersBackend.backend_tokenizers rFc|jjS)zU `tokenizers.decoders.Decoder`: The Rust decoder for this tokenizer. )rdecoderrs rDr`zTokenizersBackend.decoders &&rFTr.return_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbosec F| d|jv}| d|jv}|r|j |g|jz} n|g} tt} | D]} | d| j|r | d| j|r | d| j|r | d| j|r | d| j |r-| dt| j| | fS)a Convert the encoding representation (from low-level HuggingFace tokenizer output) to a python Dict and a list of encodings, take care of building a batch from overflowing tokens. Overflowing tokens are converted to additional examples (like batches) so the output values of the dict are lists (overflows) of lists (tokens). Output shape: (overflows, sequence length) Ntoken_type_idsattention_mask input_idsspecial_tokens_maskoffset_mappingr) model_input_names overflowingrrOrlidstype_idsrjrloffsetsr) rr.rarbrcrdrerfrg encodings encoding_dictrs rD_convert_encodingz#TokenizersBackend._convert_encodingsV( ! ($48N$N ! ($48N$N ! $ #)=)I! X%99II! I#D))  ; ;A + & - -ae 4 4 4$ C./66qzBBB$ I./66q7GHHH) S34;;A This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop. Args: pair (`bool`, *optional*, defaults to `False`): Whether the number of added tokens should be computed in the case of a sequence pair or a single sequence. Returns: `int`: Number of special tokens added to sequences. )rnum_special_tokens_to_add)rr)s rDrz+TokenizersBackend.num_special_tokens_to_adds&88>>>rFrpskip_special_tokensc<t|tr|j|Sg}|rt |jn t }|D]C}t|}||vr||j|D|S)a Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens. Args: ids (`int` or `list[int]`): The token id (or token ids) to convert to tokens. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Returns: `str` or `list[str]`: The decoded token(s). )rNrrrrall_special_idsrl)rrprr ids_to_skiprs rDconvert_ids_to_tokensz'TokenizersBackend.convert_ids_to_tokenss c3   4?..s33 33FQc$.///CEE  > >EJJE ## MM$/55e<< = = = = rFtextr~c H|jd|||d|S)N)r text_pairr~r@) _encode_plusr)rrr)r~rs rDtokenizezTokenizersBackend.tokenize s2 t lddOallekllssuuurFpadding_strategyrrrrrc |jj |jj}|tjkr |jn<|||j|jd} d} n fd|D} | |kr|jjdi||tj kr||j dSdS|tj kr|nd} | ||n|j |j|j|j|d}||kr|jjdi|dSdS)a Define the truncation and the padding strategies for fast tokenizers (provided by HuggingFace tokenizers library) and restore the tokenizer settings afterwards. The provided tokenizer has no padding / truncation strategy before the managed section. If your tokenizer set a padding / truncation strategy before, then it will be reset to no padding / truncation when exiting the managed section. Args: padding_strategy ([`~utils.PaddingStrategy`]): The kind of padding that will be applied to the input truncation_strategy ([`~tokenization_utils_base.TruncationStrategy`]): The kind of truncation that will be applied to the input max_length (`int`): The maximum size of a sequence. stride (`int`): The stride to use when handling overflow. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. N)rrrrc>i|]}||dSrjr)rBrMr s rDrKz@TokenizersBackend.set_truncation_and_padding..Ds)GGG11kooa66GGGrF)rrpad_idrprrr@)rrrrDO_NOT_TRUNCATErr9rrr DO_NOT_PAD no_padding MAX_LENGTHr pad_token_idrprr) rrrrrrrr targetcurrentrr s @rDset_truncation_and_paddingz,TokenizersBackend.set_truncation_and_paddingsiBo0 ?* "4"D D D&--///) /5!1 F"GGGGGGG&  11;;F;;; 9 9 9#**,,,,,$#$47Q#Q#QZZW[F -9-E\\4K\+!^#5&8 F6!!..8888888"!rFrris_split_into_wordsreturn_tensorsrc  #d}||std|||std|rAt|ttfo#|o!t|dttf}nt|ttf}|rt|trt d|Pt |t |kr0tdt |dt |d|tt||n|}n |r||fgn|g}t|ttfs t dt|d  ||||| | |j }j j |kr |j _ j ||| }  fd |D}i}|ddD]##fd |D}||#<d|D}r;g}t|D]$\}\}} ||gt |dzz }%||d<|dD]}!|!|t!||| }"|s5| 3s1t!d|"D|"j}"|"S)Ncrt|trdSt|ttfrt |dkrdSt|dtrdSt|dttfrt |ddks!t|ddtrdSt|ddttfrFt |dddkp&t|dddtSdSdSdS)NTrF)rNrRrOrAr)rs rD_is_valid_text_inputz._is_valid_text_inputqs!S!! tAe}-- q66Q;;4!c** !4!tUm44!1Q4yyA~~AaDGS)A)A~#t#AaDGdE];;%"1Q47||q0OJqtAwqz34O4OO$u 5urFztext input must be of type `str` (single example), `list[str]` (batch or single pretokenized example) or `list[list[str]]` (batch of pretokenized examples) or `list[tuple[list[str], list[str]]]` (batch of pretokenized sequence pairs).rzdwhen tokenizing batches of text, `text_pair` must be a list or tuple with the same length as `text`.zbatch length of `text`: z- does not match batch length of `text_pair`: .z:batch_text_or_text_pairs has to be a list or a tuple (got ))rrrrrr)r~is_pretokenizedc Jg|]}|  S))r.rarbrcrdrerfrg)ru) rBr.rbrfrercrdrarrgs rDrEz2TokenizersBackend._encode_plus..sX   " "!&;&;*C+E'=+ #   rFc0g|]\}}|D]}|Sr@r@)rBrCrrrs rDrEz2TokenizersBackend._encode_plus..s.NNN74DINNqQNNNNrFc"g|] \}}|D]}| Sr@r@)rBrrCrs rDrEz2TokenizersBackend._encode_plus..s)SSSWQdSSqSSSSrFrkoverflow_to_sample_mapping) tensor_typeci|]>\}}|t|dkr#t|dtr|dn|?SrM)rrNrO)rBrr9s rDrKz2TokenizersBackend._encode_plus..sW"Uc%jj1nnE!Hd9S9Sn%((Y^rF)rrNrOrArR TypeErrorrzipr0rrrr encode_batchr&_eventual_warn_about_too_long_sequencerrrs)$rrrr~rrrrrrrrrarbrcrdrerfrgrrr is_batchedbatch_text_or_text_pairsrstokens_and_encodingssanitized_tokensstacksanitized_encodingsrrItoksrrkbatched_outputrs$` ``````` @rDrzTokenizersBackend._encode_plusYs0   ($#D)) W   )=)=i)H)H W   9#D4-88hThjQUVWQX[_afZgFhFhJJ#D4-88J  T)S)) $Tc)nn)D)D *s4yy**I***FOEZtCi,@,@'A'A'A`d $ $?H'Sy(9':':dV $2UDMBB nTRjMkMknnn  ''- 3!1% (    '#'#< ? 04H H H4HDO 1O00 $1/1   & '*1- * *CNNNN&:NNNE$) S ! !SS0DSSS % X)+ & )*> ? ? K K 9D!*qcC[8I4J4J.JJ**=W 9 :)+6 X XI  7 7 :w W W W W&'79LZhiii n4=V4*&4&:&:&<&<( NrFrc|jj|jj|nd|S)NrQ)r^r`decoder")rrs rDconvert_tokens_to_stringz*TokenizersBackend.convert_tokens_to_strings@%-9  " * 1 1& 9 9 9&!! rF token_idsclean_up_tokenization_spacesc |ddt|tr|g}t|tr|d}|j||}||n|j}|rgt|jj j dkr0|j s)t d|jj dn||}|S)Nuse_source_tokenizerrk)rr z=Ignoring clean_up_tokenization_spaces=True for BPE tokenizer aE. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.)rwrNrrvrrrr0r^r/rGclean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_outputr warning_oncerclean_up_tokenization)rrrrrrs rD_decodezTokenizersBackend._decodes   )4000 i % % $" I i & & /!+.I%%iEX%YY,7 ) (2 % ( 8 T+122;uDDdE##-/---11$77 rF file_names. legacy_formatct|}tj||r|dzndtz}|j|||fz}|S)Nrr)rRrxryr"TOKENIZER_FILEr^save)rrrrrr%s rD_save_pretrainedz"TokenizersBackend._save_pretrained%sl^,, oM_s222Q__   ##N333>"33 rFc  tj|j}|d}|d} d} |dddkri|dd<g|dd<n|ddd kr^|dd O|dd } |dd| d }  | vr| } d |dd <| d gg|dd<n;|ddd vr i|dd<nt d|ddd7d|dvr-|ddvr|dd|dd<t jtj|g} |D]} | dd}| dd}|ddd kr|sC| dvr| d| d<| td*i| || ||dddkr#d|vr|dd|dd|d<|dddkr#d|vr|dd|dd|d<|ddd kr| | |d<|dt|dddksA|dddkrPd|dvrFtd|ddDr!tj|d<t |dd}|d*|| d|}|||| ,tj}d| vr| dD]}| d|d}fd |D}|| d|d<|D](}|}|t d!)fd"|D| d|d#<d$D]L}|| vrF| |\}} |vr|}|}|t d!||g| |<M| |d<t jtj||j}t*jD]}t/||t/||} |vr|}|j|d}t5|tr-t||j|j|j|jd%&||<|||<|jr|jng}|| |tA|d kr||d'<|d(< |j!d*i|S#tD$rH}d)tG|vr1|d(d|j!d*i|}|_|cYd}~Sd}~wwxYw)+uf Trains a tokenizer on a new corpus with the same defaults (in terms of special tokens or tokenization pipeline) as the current one. Args: text_iterator (generator of `list[str]`): The training corpus. Should be a generator of batches of texts, for instance a list of lists of texts if you have everything in memory. vocab_size (`int`): The size of the vocabulary you want for your tokenizer. length (`int`, *optional*): The total number of sequences in the iterator. This is used to provide meaningful progress tracking new_special_tokens (list of `str` or `AddedToken`, *optional*): A list of new special tokens to add to the tokenizer you are training. special_tokens_map (`dict[str, str]`, *optional*): If you want to rename some of the special tokens this tokenizer uses, pass along a mapping old special token name to new special token name in this argument. kwargs (`dict[str, Any]`, *optional*): Additional keyword arguments passed along to the trainer from the 🤗 Tokenizers library. Returns: [`PreTrainedTokenizerFast`]: A new tokenizer of the same type as the original one, trained on `text_iterator`. r3r4Nr/r0r r1r2rrrg)r#r$z;This method does not support this type of tokenizer (found z-) only BPE, Unigram, WordLevel and WordPiece.rdridrJcontinuing_subword_prefixend_of_word_suffixr ByteLevelr: pretokenizersc3.K|]}|ddkVdS)r0rNr@)rB pretokenizers rD z.s@$!(K7rFinitial_alphabet)rr*)rtrainerr*rc<g|]}||Sr@r)rBrJspecial_tokens_maps rDrEz=TokenizersBackend.train_new_from_iterator..s*![![![5"4"8"8"F"F![![![rFzQAttempted to set a token in the post processor that does not exist in the mappingc:g|]}|Sr@)rw)rBrJrs rDrEz=TokenizersBackend.train_new_from_iterator..s)CuCuCuejIDYDYZ_D`D`CuCuCurFrp)rsepT) single_wordlstriprstripr=rrfr+z7multiple values for keyword argument 'tokenizer_object'r@)$rloadsrto_strrwrr|rrrlrrkanypre_tokenizers_fastralphabetMODEL_TO_TRAINER_MAPPINGtrain_from_iteratorrwrrrSPECIAL_TOKENS_ATTRIBUTESrrrrNrrrr=rfrrrrR)r text_iteratorrrnew_special_tokensrrrr3r4rdrr* added_tokenrr trainer_classrtrained_tokenizer_jsonrrrJr^ special_tokenspecial_token_fullrfr new_tokenizerrs ` @rDtrain_new_from_iteratorz)TokenizersBackend.train_new_from_iterator6sDDO$:$:$<$<==%)).99 '++,<== ' "6 *e 3 3/1N7 #G ,02N7 #H - - G $V , 9 9g&x0<'0:*73G4D3Ew'0 G $V ,0J J J/1N7 #G , ,>n]dNeflNm>>>   *~g666w' 48JJJ3EnU\F]^iFj3kN7 #K 0!*4:n+E+EFF ' = =K!ooi66Gd++Ag&v.);;G;!-+i2HL^2^2^);K 11)*:; v vC+,<=cB8LF)5![![![![TZ![![![FLN#34S9(C!'#,#8#8#?#?#+", s##, DvCuCuCuntCuCuCuN#34S9%@@!/ F F  N22-m++F++ +   HCPQFFRR -t444 . 8 8 8 8 +4 ($$$$$$$ s$! V.. X8> Model type one of the mistral models (on older versions) 2) Remote models on the hub from official mistral models >> Tags including `base_model:.*mistralai` rN) lru_cache) model_info)versionr)maxsizemodel_idrhc |}n#t$rYdSwxYw|j0dd|jrdSdS)NFzbase_model:.*mistralairT)rtagssearchr")rr/rres rDis_base_mistralz?TokenizersBackend._patch_mistral_regex..is_base_mistralst " 8,,   uu z%995rwwuz7J7JKK 45s  Tz config.jsonF) cache_dirrJlocal_files_only%_raise_exceptions_for_missing_entries'_raise_exceptions_for_connection_errors _commit_hashr,r-transformers_versionrz5.0.0)mistralmistral3voxtral ministralpixtralrz$The tokenizer you are loading from 'a' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.z[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+|[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]+[\p{Ll}\p{Lm}\p{Lo}\p{M}]*|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n/]*|\s*[\r\n]+|\s+(?!\S)|\s+isolated)patternbehavior)r use_regex)r functoolsrhuggingface_hubr packagingrtransformers.utils.hubrrRboolrr~rrrparsesetattrrrrrpre_tokenizersSplitRegexrrNr: Metaspacer)rrpretrained_model_name_or_pathrJrrris_localrrrrrrr _config_filemistral_config_detectedf_configrtransformers_model_typersplit_pretokenizercurrent_pretokenizerrrs @@rDrz&TokenizersBackend._patch_mistral_regexs *  ''''''......%%%%%%666666 3    c d          00 H ( 4  5% 5*9/:W*X*X 5';-#!16;8=)   L', #',999+Q"illG+++++++++++++++'.{{3I'J'J$*1++l*C*C' (%GMM:N,O,ORYR_R_`gRhRh,h,h  )3?3  )()%gmmAAAA tCH~   X nd38nnnnXn:d3 ?&;:::X:10nc3hnnnn$ FFFFF =X ''''X'.2-1*/+0',#-(-(-( $d{-( $d{ -( $( -( %) -(!%-(-(-( tCH~tL11 2-(-(-(-(^ 7#7#*777766d3+;&<6WZ6666 ??d?s????*tCyt`cfjknfo`o4vvSvd vtvjnorjsvvvvI9)I90I9 I9  I9  $J I9DjI9I9I9I9\gk#',;,F2D2T!%$))-#'&*-1-1*/+0',#,0)XX++d9o=EV@WWX004 ?BTJ[E\\_ccX! X * X 0 X$JXX"X $JXDjXt X $d{X $d{X$(X %)!X"!%#X$%X&'X(#Tk)X, -XXXXt tCy S    %*48 ))c?)")'+Tk ) ))))^&*&* bk)#s(Od{  t  sCx * CCCCJ CCC[CCCCCrFr()Crrrrx collectionsrcollections.abcrshutilrtypingrtokenizers.pre_tokenizersr rrrrrr r rr r|tokenizers.decodersr rtokenizers.modelsr rtokenizers.trainersrrrrrrrrintegrations.ggmlrmodeling_gguf_pytorch_utilsrtokenization_utils_baserrrrrrrutilsr r!r" get_loggerrrrSPECIAL_TOKENS_MAP_FILETOKENIZER_CONFIG_FILETIKTOKEN_VOCAB_FILEADDED_TOKENS_FILErr#r(PreTrainedTokenizerFastr@rFrDr4s  ######$$$$$$777777++++++--------//////111111666666********^^^^^^^^^^^^......000000555555======@?????????  H % %"3/'( !! (6EXYY,--kkkkk/kk.-k^),rF