langchain 模子加载HuggingFaceEmbeddings、文本切割RecursiveCharacterTex ...

参考：
https://github.com/TommyTang930/LangChain_LLM_ChatBot
https://python.langchain.com/docs/integrations/vectorstores/faiss
1、文本切割RecursiveCharacterTextSplitter

这里对着类举行了改写，对中文切分更友好

1. import re
2. from typing import List, Optional, Any
3. from langchain.text_splitter import RecursiveCharacterTextSplitter
4. import logging
6. logger = logging.getLogger(__name__)
9. def _split_text_with_regex_from_end(
10.         text: str, separator: str, keep_separator: bool
11. ) -> List[str]:
12.     # Now that we have the separator, split the text
13.     if separator:
14.         if keep_separator:
15.             # The parentheses in the pattern keep the delimiters in the result.
16.             _splits = re.split(f"({separator})", text)
17.             splits = ["".join(i) for i in zip(_splits[0::2], _splits[1::2])]
18.             if len(_splits) % 2 == 1:
19.                 splits += _splits[-

复制代码

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。