{-jjddlZddlZddlmZmZddlmZddlmZddl m Z m Z ddl m Z e drdd lmZe d rdd lmZe d r dd lmZddlmZe dd d Gddee ZdS)N)ABCabstractmethod)List)logging)class_requires_depsis_dep_available)AutoRegisterABCMetaClass langchainDocumentzlangchain-text-splitters)RecursiveCharacterTextSplitterzlangchain-community) vectorstores)FAISSc eZdZdZdZdZfdZedZedZ de fdZ d e de fd Z d ede fd Zd e defdZdgdfdee dedee ddfdZddde fdZde ddfdZ d!dee dddededede f d Z xZS)" BaseRetrieverzBase RetrieverTPADDLEX_VECTOR_STOREcdtd|_d|_dS)z*Initializes an instance of base retriever.N)super__init__ model_name embedding)self __class__s u/var/www/html/banglarbhumi/venv/lib/python3.11/site-packages/paddlex/inference/pipelines/components/retriever/base.pyrzBaseRetriever.__init__)s+ c td)z Declaration of an abstract method. Subclasses are expected to provide a concrete implementation of generate_vector_database. zCThe method `generate_vector_database` has not been implemented yet.NotImplementedErrorrs rgenerate_vector_databasez&BaseRetriever.generate_vector_database/s " Q   rc td)z Declaration of an abstract method. Subclasses are expected to provide a concrete implementation of similarity_retrieval. z?The method `similarity_retrieval` has not been implemented yet.rr s rsimilarity_retrievalz"BaseRetriever.similarity_retrieval9s " M   rreturnc|jS)zt Get the model name used for generating vectors. Returns: str: The model name. )rr s rget_model_namezBaseRetriever.get_model_nameCs rsc6||jS)z Check if the given string starts with the vector store prefix. Args: s (str): The input string to check. Returns: bool: True if the string starts with the vector store prefix, False otherwise. ) startswithVECTOR_STORE_PREFIX)rr's ris_vector_storezBaseRetriever.is_vector_storeLs||D4555rvector_store_bytesc`|jtj|dzS)z Encode the vector store bytes into a base64 string prefixed with a specific prefix. Args: vector_store_bytes (bytes): The bytes to encode. Returns: str: The encoded string with the prefix. ascii)r*base64 b64encodedecode)rr,s rencode_vector_storez!BaseRetriever.encode_vector_storeXs5'&*:;M*N*N*U*U + +   rvector_store_strc^tj|t|jdS)a Decodes the vector store string by removing the prefix and decoding the base64 encoded string. Args: vector_store_str (str): The vector store string with a prefix. Returns: bytes: The decoded vector store data. N)r/ b64decodelenr*)rr3s rdecode_vector_storez!BaseRetriever.decode_vector_storefs, 0T5M1N1N1P1P QRRRri,)  u。z text_list block_size separatorsrct|d|}|d|}d|D} tj||j}n#t $rd}YnwxYw|S)a Generates a vector database from a list of texts. Args: text_list (list[str]): A list of texts to generate the vector database from. block_size (int): The size of each chunk to split the text into. separators (list[str]): A list of separators to use when splitting the text. Returns: FAISS: The generated vector database. Raises: ValueError: If an unsupported API type is configured. ) chunk_size chunk_overlapr=r8c.g|]}t|S) page_contentr ).0texts r z:BaseRetriever.generate_vector_database..s#DDDdhD111DDDr) documentsrN)r split_textjoinrfrom_documentsr ValueError)rr;r<r= text_splittertexts all_splits vectorstores rr!z&BaseRetriever.generate_vector_databasers(7!    ((9)=)=>>DDeDDD  .$KK   KKK sA$$ A32A3rPch||j}n'||}|S)z Encode the vector store serialized to bytes. Args: vectorstore (FAISS): The vector store to be serialized and encoded. Returns: str: The encoded vector store. )r*r2serialize_to_bytes)rrPs rencode_vector_store_to_bytesz*BaseRetriever.encode_vector_store_to_bytess9  2KK22;3Q3Q3S3STTKrc||std||}|dkrtjddSt j||jd}|S)a Decode a vector store from bytes according to the specified API type. Args: vectorstore (str): The serialized vector store string. Returns: FAISS: Deserialized vector store object. Raises: ValueError: If the retrieved vector store is not for PaddleX or if an unsupported API type is specified. z-The retrieved vectorstore is not for PaddleX.rz5The retrieved vectorstore is empty,will empty vector.NT) embeddingsallow_dangerous_deserialization) r+rLr7rwarningrrdeserialize_from_bytesr)rrPvectors rdecode_vector_store_from_bytesz,BaseRetriever.decode_vector_store_from_bytess##K00 NLMM M..{;; #   OS T T T4#:: ~,0;    r? query_text_list sleep_timetopkmin_charactersc4d}||S|D]}|}tj||||} d| D} t| d} | dddD]5\} } | dkr*t |t | z|krn|| z }6|S) a Retrieve similar contexts based on a list of query texts. Args: query_text_list (list[str]): A list of query texts to search for similar contexts. vectorstore (FAISS): The vector store where to perform the similarity search. sleep_time (float): The time to sleep between each query, in seconds. Default is 0.5. topk (int): The number of results to retrieve per query. Default is 2. min_characters (int): The minimum number of characters required for text processing, defaults to 3500. Returns: str: A concatenated string of all unique contexts found. r:N)kc&g|]\}}|j|fSrC)rEdocumentscores rrGz6BaseRetriever.similarity_retrieval..s$RRR/(E-u5RRRrc|dS)Nre)xs rz4BaseRetriever.similarity_retrieval..s AaDr)keyg)timesleep'similarity_search_with_relevance_scoressortedr6) rr^rPr_r`raall_C query_textQUESTIONdocscontextrFrgs rr#z"BaseRetriever.similarity_retrievals(  L) " "J!H Jz " " "FFxSWFXXDRRTRRRGW..999G&ttt} " " eD==5zzCII->>TME r)r[r\r])__name__ __module__ __qualname____doc___BaseRetriever__is_baser*rrr!r#strr&boolr+bytesr2r7rintrSrZfloat __classcell__)rs@rrr!sI0   ^   ^  6 6 6 6 6 6  e        SC SE S S S S ? ? ? !!9!!I !  !!!!FC #'D " ""c"" "  "  " """"""""rr) metaclass)r/rnabcrrtypingr paddlex.utilsr utils.depsrr utils.subclass_registerr langchain_core.documentsr langchain_text_splittersrlangchain_communityr langchain_community.vectorstoresrrrerrrsi ########!!!!!!AAAAAAAAAAAAAAK  2111111.//HGGGGGG)**7000000666666["<>STTCCCCCC#;CCCUTCCCr