first commit

.gradio/certificate.pem

@@ -0,0 +1,31 @@
+-----BEGIN CERTIFICATE-----
+MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw
+TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
+cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4
+WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu
+ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY
+MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc
+h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+
+0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U
+A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW
+T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH
+B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC
+B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv
+KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn
+OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn
+jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw
+qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI
+rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV
+HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq
+hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL
+ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ
+3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK
+NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5
+ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur
+TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC
+jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc
+oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq
+4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA
+mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d
+emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=
+-----END CERTIFICATE-----

README.md

@@ -0,0 +1,288 @@
+# 项目说明文档
+
+## 目录
+
+1.  [util\_old.py](#util_oldpy)
+2.  [utils.py](#utilspy)
+3.  [search\_ui.py](#search_uipy)
+4.  [main.py](#mainpy)
+5.  [search\_plus.py](#search_pluspy)
+6.  [search\_queries.py](#search_queriespy)
+7.  [search\_ui\_v2.py](#search_ui_v2py)
+
+---
+
+## 1. `util_old.py` <a name="util_oldpy"></a>
+
+该文件包含一些用于网络爬取、文件处理和数据保存的实用函数。
+
+### 主要功能
+
+1.  **`fetch_documents(ip_address, dataset_id, api_key)`**:
+    -   从指定的 API 端点获取文档数据。
+    -   使用 `requests` 库发送 HTTP GET 请求。
+    -   处理 HTTP 错误和其它异常。
+    -   返回 JSON 格式的响应数据。
+2.  **`is_file(url)`**:
+    -   检查给定的 URL 是否指向一个文件（基于文件扩展名）。
+    -   使用预定义的文件扩展名集合 `file_extensions` 进行判断。
+3.  **`is_webpage(url)`**:
+    -   检查给定的 URL 是否指向一个网页。
+    -   通过检查 `Content-Type` 头是否包含 `text/html` 来判断。
+4.  **`can_crawl_webpage(url)`**:
+    -   判断一个网页是否适合爬取。
+    -   使用 `BeautifulSoup` 解析网页内容。
+    -   检查否存在大量 JavaScript 代码、页面内容是否主要是乱码或图片、以及是否存在 `meta robots` 标签阻止爬取。
+    -   使用 `langdetect` 库检测页面语言是否为中文。
+5.  **`is_cosmetic_related(snippet)`**:
+    -   检查给定的文本片段是否与化妆品相关。
+    -   使用预定义的关键词列表进行判断。
+6.  **`parse_search(result)`**:
+    -   解析搜索结果，判断链接类型（文件、网页或未知类型）。
+    -   根据链接类型返回相应的描述字符串。
+7.  **`save_webpage_content(url, engine, search_id, output_dir, pool)`**:
+    -   保存网页内容到本地文件，并将相关信息记录到数据库。
+    -   使用 `BeautifulSoup` 提取网页文本内容。
+    -   检查网页是否已保存，避免重复保存。
+8.  **`download_file(url, engine, search_id, output_dir, pool)`**:
+    -   下载文件到本地，并将相关信息记录到数据库。
+    -   检查文件是否已下载，避免重复下载。
+9.  **`save_non_crawlable(url, engine, search_id, pool)`**:
+    -   将无法爬取的链接保存到数据库的黑名单中。
+    -   避免重复保存。
+10. **`get_blacklist(pool)`**:
+    -   从数据库中获取黑名单列表。
+11. **`libreoffice_to_pdf(file_path, output_dir)`**:
+    -   使用 LibreOffice 将文件转换为 PDF 格式。
+    -   处理文件不存在和转换失败的情况。
+12. **`pdf_to_images(pdf_path, output_dir)`**:
+    -   将 PDF 文件逐页转换为图像。
+    -   使用 `pdf2image` 库进行转换。
+    -   强制清理每页处理后的内存。
+13. **`process_files(base_dir)`**:
+    -   遍历指定目录下的所有文件，处理 PPT/PPTX、XLS/XLSX、DOC/DOCX 和 PDF 文件。
+    -   将这些文件转换为 PDF 格式，并将 PDF 文件转换为图像。
+
+### 依赖库
+
+-   `bs4` (BeautifulSoup4)
+-   `os`
+-   `gc`
+-   `requests`
+-   `langdetect`
+-   `langchain_core`
+-   `langchain.text_splitter`
+-   `pdf2image`
+-   `typing`
+
+---
+
+## 2. `utils.py` <a name="utilspy"></a>
+
+该文件包含一些用于判断 URL 类型和内容是否相关的实用函数。
+
+### 主要功能
+
+1.  **`is_file(url)`**:
+    -   检查给定的 URL 是否指向一个文件（基于文件扩展名）。
+    -   使用预定义的文件扩展名集合 `file_extensions` 进行判断。
+2.  **`is_webpage(url)`**:
+    -   检查给定的 URL 是否指向一个网页。
+    -   通过检查 `Content-Type` 头是否包含 `text/html` 来判断。
+3.  **`can_crawl_webpage(url)`**:
+    -   判断一个网页是否适合爬取。
+    -   使用 `BeautifulSoup` 解析网页内容。
+    -   检查是否存在 `meta robots` 标签阻止爬取。
+4.  **`is_cosmetic_related(snippet)`**:
+    -   检查给定的文本片段是否与化妆品相关。
+    -   使用预定义的关键词列表进行判断。
+5.  **`parse_search(result)`**:
+    -   解析搜索结果，判断链接类型（文件、网页或未知类型）。
+    -   根据链接类型返回相应的描述字符串。
+
+### 依赖库
+
+-   `bs4` (BeautifulSoup4)
+-   `requests`
+
+---
+
+## 3. `search_ui.py` <a name="search_uipy"></a>
+
+该文件使用 Gradio 库创建一个 Web UI，用于执行多搜索引擎聚合搜索。
+
+### 主要功能
+
+1.  **`SearchUI` 类**:
+    -   `__init__`: 初始化 API 密钥。
+    -   `parse_google_results(results)`: 解析 Google 搜索结果，提取标题、链接、摘要和位置信息。
+    -   `parse_bing_results(results)`: 解析 Bing 搜索结果，提取标题、链接、摘要和位置信息。
+    -   `parse_baidu_results(results)`: 解析百度搜索结果，提取标题、链接、摘要和位置信息，并处理百度特有的 `answer_box`。
+    -   `format_results(results)`: 将解析后的搜索结果格式化为 Markdown 文本。
+    -   `search_all_engines(query)`: 在 Google、Bing 和 Baidu 三个搜索引擎中执行搜索，并返回格式化后的结果。
+2.  **`create_ui()` 函数**:
+    -   创建 `SearchUI` 实例。
+    -   定义自义 CSS 样式，用于美化 UI。
+    -   使用 Gradio 创建 Web UI，包括搜索输入框、搜索按钮、加载动画和搜索结果展示区域。
+    -   定义搜索事件处理函数，用于在搜索时显示加载动画，并在搜索完成后更新搜索结果。
+
+### 依赖库
+
+-   `gradio`
+-   `json`
+-   `typing`
+-   `requests`
+
+---
+
+## 4. `main.py` <a name="mainpy"></a>
+
+该文件使用 FastAPI 创建一个 API，用于处理搜索请求和批量搜索请求。
+
+### 主要功能
+
+1.  **数据库连接配置**:
+    -   定义数据库连接参数。
+2.  **API 端点**:
+    -   `/search/`: 接收单个搜索请求，并返回搜索结果。
+    -   `/batch_search/`: 接收批量搜索请求，并返回搜索结果。
+3.  **数据模型**:
+    -   `SearchRequest`: 定义单个搜索请求的数据模型。
+    -   `SearchResult`: 定义单个搜索结果的数据模型。
+    -   `BatchSearchRequest`: 定义批量搜索请求的数据模型。
+    -   `SearchResultItem`: 定义批量搜索结果中单个搜索结果的数据模型。
+    -   `BatchSearchResponse`: 定义批量搜索响应的数据模型。
+4.  **`fetch_all_content(query)`**:
+    -   从数据库中获取与查询相关的已保存网页内容。
+5.  **`save_user_query(query, result_count)`**:
+    -   将用户查询记录保存到数据。
+6.  **`search(request)`**:
+    -   接收单个搜索请求，首先尝试从数据库获取结果，如果数据库中没有结果，则调用 `search_plus` 模块进行搜索。
+    -   保存用户查询记录。
+7.  **`batch_search(request)`**:
+    -   接收批量搜索请求，并使用 SerpAPI 在 Google、Bing 和 Baidu 中执行搜索。
+    -   返回包含所有搜索结果的响应。
+
+### 依赖库
+
+-   `fastapi`
+-   `pydantic`
+-   `aiomysql`
+-   `requests`
+-   `search_plus`
+-   `datetime`
+-   `json`
+
+---
+
+## 5. `search_plus.py` <a name="search_pluspy"></a>
+
+该文件包含用于执行搜索并将结果保存到数据库的函数。
+
+### 主要功能
+
+1.  **数据库连接配置**:
+    -   定义数据库连接参数。
+2.  **`search_and_save(query)`**:
+    -   在 Google、Bing 和 Baidu 三个搜索引擎中执行搜索。
+    -   将搜索结果以 JSON 字符串的形式保存到数据库的 `web_search_results` 表中。
+3.  **`main(queries)`**:
+    -   接收一个查询列表，并对每个查询调用 `search_and_save` 函数。
+
+### 依赖库
+
+-   `fastapi`
+-   `pydantic`
+-   `databases`
+-   `requests`
+-   `json`
+-   `datetime`
+
+---
+
+## 6. `search_queries.py` <a name="search_queriespy"></a>
+
+该文件用于定时执行搜索任务。
+
+### 主要功能
+
+-   **定时执行**:
+    -   该脚本配置为通过 Ubuntu 系统的定时任务 (crontab) 在每周日 0 点定时执行。它从数据库中读取预定义的查询列表，并使用 `search_plus.py` 中的 `main` 函数执行搜索，并将结果保存到数据库。  **设置方法见下文 "Crontab 设置说明" 部分。**
+
+### 依赖库
+
+-   `aiomysql`
+-   `search_plus`
+-   `datetime`
+
+---
+
+## Crontab 设置说明
+
+1. **打开 crontab 编辑器:** 使用命令 `crontab -e` 打开 crontab 编辑器。  如果这是你第一次使用 crontab，系统会询问你使用哪个编辑器 (例如 vim 或 nano)。
+
+2. **添加定时任务:**  在编辑器中添加以下一行，以在每周日 0 点执行 `search_queries.py` 脚本：
+   ```bash
+   0 0 * * 0 /usr/bin/python3 /path/to/your/search_queries.py  # 请替换 /path/to/your/ 为你的脚本实际路径   ```
+
+   *   `0 0 * * 0`:  表示每天的 0 分 0 秒，每周日执行 (0 代表星期日)。  你可以根据需要修改这个时间表达式。  更多关于 crontab 时间表达式的资料，请参考 `man 5 crontab`。
+   *   `/usr/bin/python3`:  指定 Python3 解释器。  如果你的 Python3 解释器不在这个路径，请修改为正确的路径。
+   *   `/path/to/your/search_queries.py`:  替换为 `search_queries.py` 脚本的完整路径。
+
+3. **保存并退出:** 保存 crontab 文件并退出编辑器。
+
+4. **查看 crontab 设置:** 使用命令 `crontab -l` 查看当前已设置的 crontab 任务。 你应该能够看到你刚刚添加的那一行。
+
+5. **验证定时任务:**  为了验证定时任务是否正确设置，你可以尝试在 `search_queries.py` 脚本中添加日志记录功能，例如使用 `logging` 模块记录执行时间和相关信息。  然后，你可以等待到下一个星期日 0 点，查看日志文件，确认脚本是否被正确执行。  如果脚本没有执行，请检查脚本路径、Python 解释器路径以及 crontab 时间表达式是否正确。  你也可以使用 `tail -f /path/to/your/logfile` 实时监控日志文件。
+
+---
+
+## 7. `search_ui_v2.py` <a name="search_ui_v2py"></a>
+
+该文件使用 Gradio 库创建一个 Web UI，用于执行多搜索引擎聚合搜索，并提供更高级的功能。
+
+### 主要功能
+
+1.  **`SearchUI` 类**:
+    -   `__init__`: 初始化 API 密钥。
+    -   `parse_google_results(results)`: 解析 Google 搜索结果，提取标题、链接、摘要和位置信息。
+    -   `parse_bing_results(results)`: 解析 Bing 搜索结果，提取标题、链接、摘要和位置信息。
+    -   `parse_baidu_results(results)`: 解析百度搜索结果，提取标题、链接、摘要和位置信息，并处理百度特有的 `answer_box`。
+    -   `format_results(results)`: 将解析后的搜索结果格式化为 Markdown 文本。
+    -   `search_all_engines(query)`: 在 Google、Bing 和 Baidu 三个搜索引擎中执行搜索，并返回格式化后的结果。
+    -   `save_search_results(query, results)`: 将搜索结果保存到数据库。
+2.  **`create_ui()` 函数**:
+    -   创建 `SearchUI` 实例。
+    -   定义自义 CSS 样式，用于美化 UI。
+    -   使用 Gradio 创建 Web UI，包括搜索输入框、搜索按钮、加载动画、搜索结果展示区域和保存搜索结果的选项。
+    -   定义搜索事件处理函数，用于在搜索时显示加载动画，并在搜索完成后更新搜索结果。
+    -   定义保存搜索结果的事件处理函数，用于将搜索结果保存到数据库。
+
+### 界面结构
+
+1. **标题:** 显示"九鑫多搜索引擎聚合搜索"。
+2. **搜索输入框:** 一个文本框，用于输入搜索关键词。
+3. **搜索按钮:** 一个按钮，点击后开始搜索。
+4. **加载动画:** 搜索过程中显示一个GIF动画，提示用户正在等待结果。搜索完成后自动隐藏。
+5. **搜索结果展示区域:** 分为三个部分，分别显示Google、Bing和百度搜索结果。每个部分都以HTML卡片的形式展示搜索结果，包含标题、链接和摘要。点击卡片中的链接会在新标签页打开搜索结果页面。每个搜索引擎的结果区域都有各自的边框颜色，方便区分。
+
+### 操作方法
+
+1. **输入关键词:** 在搜索输入框中输入你想要搜索的内容。
+2. **点击搜索:** 点击"搜索"按钮。此时加载动画会显示。
+3. **查看结果:** 搜索完成后，加载动画会消失，Google、Bing和百度的搜索结果会分别显示在各自的区域内。每个搜索结果以卡片形式呈现，点击卡片即可打开对应的链接。
+
+### 功能说明
+
+该UI使用SerpAPI接口在Google、Bing和百度三个搜索引擎上进行搜索。它会解析搜索结果，并以用户友好的方式展示。界面使用了自定义CSS样式，使界面更美观易用。程序还包含错误处理机制，如果搜索过程中出现错误，会显示相应的错误信息。
+
+总而言之，这是一个功能完善、界面友好的多搜索引擎聚合搜索工具。用户只需输入关键词并点击搜索按钮，即可快速获得来自三个不同搜索引擎的搜索结果。
+
+### 依赖库
+
+-   `gradio`
+-   `json`
+-   `typing`
+-   `requests`
+-   `aiomysql`
+ 

main.py

@@ -0,0 +1,167 @@
+import json
+import datetime
+from fastapi import FastAPI, HTTPException
+from pydantic import BaseModel
+import aiomysql
+import requests
+from search_plus import search_and_save
+
+# 数据库连接配置
+db_config = {
+    'user': 'root',
+    'password': 'zaq12wsx@9Xin',
+    'host': '183.11.229.79',
+    'port': 3316,
+    'db': 'gptDB',
+    'auth_plugin': 'mysql_native_password'
+}
+SERP_API_KEY = "8af097ae8b587bb0569425058e03e5ef33b4c7b8b1a505053764b62e7e4ab9d6"
+app = FastAPI()
+
+class SearchRequest(BaseModel):
+    query: str
+
+class SearchResult(BaseModel):
+    results: str 
+
+# 添加新的请求模型
+class BatchSearchRequest(BaseModel):
+    queries: list[str]
+
+class SearchResultItem(BaseModel):
+    query: str
+    engine: str
+    results: dict
+
+class BatchSearchResponse(BaseModel):
+    status: str
+    message: str
+    total_processed: int
+    search_results: list[SearchResultItem]
+
+async def fetch_all_content(query):
+    query_list = query.split('\n')
+    query_list = ["'"+item.split('.')[0]+"'" for item in query_list if item] 
+    placeholders = ', '.join(query_list)
+
+    sql = f"SELECT url, gpt_output FROM saved_webpages WHERE hash_code IN ({placeholders});"  
+    pool = await aiomysql.create_pool(**db_config)
+    contents = []
+    counts = 0
+    try:
+        async with pool.acquire() as conn:
+            async with conn.cursor(aiomysql.DictCursor) as cursor:
+                await cursor.execute(sql)
+                results = await cursor.fetchall() 
+                if not results:  # 如果没有找到结果
+                    return [], 0
+                    
+                urls = [result['url'] for result in results]
+                gpt_outputs = [result['gpt_output'] for result in results]
+                for url, gpt_output in zip(urls, gpt_outputs):
+                    final_result = gpt_output + '\n相关链接：' + url
+                    contents.append(final_result)
+                counts = len(results)
+    except Exception as e:
+        print(f"Error fetching summaries: {e}")
+    finally:
+        pool.close()
+        await pool.wait_closed()
+        return contents, counts
+
+async def save_user_query(query: str, result_count: int):
+    """保存用户查询到数据库"""
+    pool = await aiomysql.create_pool(**db_config)
+    try:
+        async with pool.acquire() as conn:
+            async with conn.cursor() as cursor:
+                sql = """
+                    INSERT INTO user_queries (query, query_datetime, result_count)
+                    VALUES (%s, %s, %s)
+                """
+                await cursor.execute(sql, (query, datetime.datetime.now(), result_count))
+                await conn.commit()
+    except Exception as e:
+        print(f"Error saving user query: {e}")
+    finally:
+        pool.close()
+        await pool.wait_closed()
+
+@app.post("/search/", response_model=list[SearchResult])
+async def search(request: SearchRequest): 
+    search_results = [] 
+    query = request.query
+    
+    # 首先尝试从saved_webpages获取结果
+    results, counts = await fetch_all_content(query)
+    
+    # 如果没有找到结果，使用search_plus进行搜索
+    # if counts == 0:
+    #     web_results = await search_and_save(query)
+    #     results = web_results
+    #     counts = len(web_results)
+    
+    # 无论是否找到结果，都保存用户的查询记录
+    try:
+        await save_user_query(query, counts)
+    except Exception as e:
+        print(f"Error saving user query: {e}")
+    
+    # 处理结果
+    for result in results:
+        search_results.append(SearchResult(results=result))
+    
+    if not search_results:
+        # 如果没有找到任何结果，返回空列表但仍然记录这次查询
+        await save_user_query(query, 0)
+        return []
+    
+    return search_results
+
+@app.post("/batch_search/", response_model=BatchSearchResponse)
+async def batch_search(request: BatchSearchRequest):
+    try:
+        processed_count = 0
+        search_results = [] 
+        try: 
+            # 处理每个查询
+            for query in request.queries:
+                for engine in ["google", "bing", "baidu"]:
+                    params = {
+                        "api_key": SERP_API_KEY,
+                        "engine": engine,
+                        "q": query
+                    }
+                    response = requests.get('https://serpapi.com/search', params=params)
+                    search = response.json()
+                    search_metadata = search.get('search_metadata', {})
+                    
+                    if search_metadata.get('status') == 'Success':
+                        json_endpoint = search_metadata.get('json_endpoint')
+                        response = requests.get(json_endpoint)
+                        
+                        if response.status_code == 200:
+                            data = response.json()
+                            json_record_str = json.dumps(data) 
+                            # 添加到搜索结果列表
+                            search_results.append(
+                                SearchResultItem(
+                                    query=query,
+                                    engine=engine,
+                                    results=data
+                                )
+                            )
+                            
+                                    
+        finally: 
+        
+        return BatchSearchResponse(
+            status="success",
+            message=f"成功处理了 {processed_count} 条搜索请求",
+            total_processed=processed_count,
+            search_results=search_results
+        )
+        
+    except Exception as e:
+        raise HTTPException(status_code=500, detail=f"批量搜索失败: {str(e)}")
+

models.py

@@ -0,0 +1,23 @@
+from sqlalchemy import Table, Column, Integer, String, MetaData, Text, DateTime, Boolean
+
+metadata = MetaData()
+
+search_results = Table(
+    "search_results",
+    metadata,
+    Column("id", Integer, primary_key=True),
+    Column("engine", String(50)),
+    Column("query", String(255)),
+    Column("results", Text),
+    Column("search_datetime", DateTime),
+    Column("is_processed", Boolean, default=False)
+)
+
+user_queries = Table(
+    "user_queries",
+    metadata,
+    Column("id", Integer, primary_key=True),
+    Column("query", Text),
+    Column("query_datetime", DateTime),
+    Column("result_count", Integer),
+)

nohup.out

@@ -0,0 +1,2 @@
+/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/gradio/components/base.py:201: UserWarning: 'scale' value should be an integer. Using 0.2 will cause issues.
+  warnings.warn(

out1/output.md

@@ -0,0 +1,321 @@
+蜚伟尔  
+控制膜系列 2 款  
+资料介绍  
+广东芒果 市场部 2207  
+
+
+# 声 明
+
+本文献仅基于我所了解的资料及掌握的资料进行的整理、归纳。条件不同结果就不同，可能某些字眼或表述与现行法律法规有出入，因此仅供参考。不会代表任何明确或暗示性的保证，最终均以相关法律法规表述执行，不承担任何损失责任。
+
+如有有效宣传或广告用途的应用，请做相关有效宣传评估，获取相关的有效检测报告结果及确认是否符合《广告法》等相关法律法规。
+
+注：本文献最终解释权归广东芒果生物科技股份有限公司所有。
+
+谢谢！
+
+
+# 男士的肌肤特点——成因
+
+1. **荷尔蒙作用**
+男性肌肤一般偏向油性，PH 值约为 4.5~6.0， debido a la actividad hormonal que estimula las glándulas sebáceas, así que el aceite y la sudoración son comparativamente más.
+
+2. **生活方式**
+抽烟、熬夜、酒肉应酬、饮食无规律，身体内分泌紊乱，再加上现代社会会工作的压力等，皮肤功能失调。
+
+3. **不恰于护肤管理**
+不少男性都认为护肤管理是女性的专利，男人最好 MAN 一点，不不着护肤理论。
+
+4. **户外运动**
+大部分男性喜欢户外运动，足球、篮球、泳泳等，皮肤长时间暴露在阳光下，孔伤接触而来。
+
+
+男性皮肤最大的特点——油 & 痤
+
+和女性肌肤相比，男性的皮脂腺更旺盛，分泌的皮脂要多出 40%~70%。而且，有 80% 的男性是油性或混合性偏油性皮肤。
+
+这意味着——男性的皮肤油性大、毛孔大、易受污染，尤其是脂溶性的有害物质和多种微生物积蓄，导致炎症和感染的机会增多，所以更容易出现肤色暗沉、毛孔粗大、粉刺、痤疮等皮肤问题。
+
+![](3_0.png) 
+- Epidermis
+- Plugged follicle
+- Sebaceous gland
+- Accumulation of epithelial cells and keratin
+- Propionibacterium acne proliferation
+- Mild inflammation
+- Marked inflammation
+- Scarring
+
+
+油性肌肤将会带来多种困扰
+
+清洁护理不当、护肤方法不对，造成肌肤出油越来越多，痘痘反复难祛，进而引发炎症痘痘带来的红、痒、痛等不适感，也令人苦不堪言。
+
+![](1.png) 炎症痘痘  
+![](2.png) 脸部毛孔  
+![](3.png) 月球脸  
+![](4.png) 油腻毛孔  
+![](5.png) 粉刺闭口  
+
+
+**皙威尔**
+
+品牌严选 科学护肤  
+BRAND SELECTION, SCIENTIFIC SKINCARE  
+
+致选原料·专利加持·安全高效
+
+
+新品方案
+
+2 款专研面膜  
+还原男士肌肤状态  
+
+睡眠型 ---- 男士控油滋润睡眠面膜  
+贴片型 ---- 男士控油净润面膜  
+
+![](6_0.png)
+
+
+从配方根源解决问题  
+不做 “表面功夫”  
+有效原料，精准改善肌肤问题
+
+
+男士控油净润面膜  
+解决过度出油 / 暗沉 / 易长痘 / 毛孔粗大等肌肤问题  
+高能净控油，为干燥肌肤源源不断补充水分  
+15分钟，即刻揭开净爽肌  
+
+产品基因  
+链路一：平衡肌肤脂质（CONTACTICEL水解红藻）  
+链路二：基础保湿，改善暗沉（烟酰胺，双重透明质酸）  
+链路三：科技专利认证，硬核注能（莹萃护肤）  
+
+
+# CONTACTICEL - 源头控油
+## 奇生藻的秘密
+调节污染环境下的油性肌肤
+
+### 关于 CONTACTICEL
+CONTACTICEL（顶丝石斑）是一种附生红藻类海藻，一种寄生在其他藻类上的红色藻类，它的颜色来自于过多的红色素（藻红蛋白）。它与许多其他微生物在一起，成为了宿主藻抵抗外部侵害（如紫外线、有害微生物、污染等）的屏障的一部分。
+
+### 工艺来源
+世界首创的藻类细胞培养技术 CELEBRITY™，其技术特点：细胞形态下收获和分离海藻；鉴定特殊稀有物种；以细胞形式在光生物反应器中增殖。
+
+### 作用原理
+源头调节肌肤脱油分泌，其独特作用是其他红藻不能比拟的；在污染条件下，可以保护角质形成细胞的新陈代谢及保护细胞 DNA。
+
+INCI 名称：水解红藻提取物
+
+
+链路二：烟酰胺 - 改善暗沉
+
+明星级成分 - 烟酰胺
+
+### 美白原理：
+能有效对抗黑色素和干扰细胞之间的信号传导成分，从而阻止出现过多的黑色素。同时对于已经产生的黑色素，它可以避免其它细胞之中进行转移。而且它还能增强细胞代谢，加上它的分子比较小，可以更好被吸收起来，使得肌肤保持平衡的状态，还能促进更多的胶原蛋白。
+
+#### 明亮肤色：
+被用于减少已经生成、沉淀的黑色素，阻隔其向表层细胞的转移，加速细胞更新代谢，加快黑色素角质细胞脱落。
+
+#### 舒缓泛红：
+淡化泛印、舒缓肌肤炎症、长痘等现象，帮助肌肤平衡油脂分泌，保持肌肤清爽。
+
+#### 补水保湿：
+促进肌肤角蛋白的合成，使皮肤中游离的神经酰胺水平提高，保持水分自然大大增加。
+
+#### 延缓衰老：
+减少自由基对皮肤的伤害，起到预防老化、修复皮肤和减少暗沉的作用。
+
+![](https://example.com/10_0.png)
+
+
+# 链路二：双重透明质酸钠 - 基础保湿
+
+## 2 种不同分子透明质酸
+双管齐下：表层锁水 + 深层补水
+
+玻尿酸广泛存在于人体各组织中，皮肤中玻尿酸含量的高低，直接决定了肤质的“视觉年龄”。
+
+### 大分子玻尿酸 表层锁水
+**INCI 名：透明质酸钠**
+- 快速在肌肤表层形成一层均匀的锁水膜
+- 有效减少干燥环境对肌肤的侵害
+- 同时减少皮肤内部水分的流失
+
+### 小分子玻尿酸 深层补水
+**INCI 名：水解透明质酸钠**
+- 促进皮肤营养吸收作用
+- 锁住细胞间质中的水分
+- 具有较强的消炎功能，增加皮肤弹性
+
+
+链路三：科技专利认证 - 粗粮护肤 
+
+从数十种植物种子中，精心甄选，最终选择粗粮、黄崎粟、野大豆、杆枣粟、黑麦粟、向日葵、栗粟，七种粗粮共同发挥抗炎舒缓、抗氧化及保温功能。
+
+1. 多糖类物质  
+   来源：粗粮淀粉、黄秋葵多糖等  
+
+2. 黄酮类  
+   来源：黑麦多酚、大豆黄酮  
+
+3. 蛋白质及氨基酸类  
+   来源：榨菜蛋白质含量在13%以上，最高可达17%含有人体必需的8种氨基酸，丰富的甲硫氨酸  
+
+4. 脂肪酸类  
+   来源：槐树粕油、黄麦粕不饱和脂肪酸含量丰富  
+
+![](12_0.png)
+
+
+链路三：科技专利认证 - 榨菜护肤
+
+含丰富的蛋白质、氨基酸、维生素、多糖等，含人体必需的8种氨基酸，且含量高于大米、玉米等，对人体有极好的亲和性。
+
+含黄葱榨油，对ABTS自由基具有良好的清除能力，其多糖具有良好的抗氧化与湿性能力。
+
+榨楂粽油含大量不饱和脂肪酸，可防止细胞衰老，提高细胞活性。同时，榨楂提取物还有较好的抗炎、平喘功效。
+
+野大豆粽含有丰富的黄酮类物质，能够促进细胞的生长，具有活跃、抗衰老的作用；同时能保持皮肤的水分，有保湿功效。
+
+含黑麦多酚天然抗氧化剂。此外含丰富的多糖、蛋白质，且黑麦中的天然色素良好的抗氧化能力。
+
+葵粟仁含量高，葵花籽油含大量不饱和脂肪酸，具有保温、增强皮肤屏障的功能。
+
+
+男士控油犹润睡眠面膜  
+解决过度出油 / 暗沉 / 易长痘 / 毛孔粗大等肌肤问题  
+独特冰沙型质地，冰凉感体验  
+涂上即眠，护肤一步到位  
+
+产品基因  
+链路一：平衡肌肤油脂（CONTACTICEL 水解红藻）  
+链路二：促进 ATP，补充肌肤能量（天冬氨酸盐 / 葡萄糖酸钠 / 葡萄糖）  
+链路三：科技专利认证，硬核注能（净肤土）  
+链路四：提升肌肤耐受力（乳酸杆菌 / 豆浆发酵产物滤液，舒缓组方）  
+
+
+# CONTACTICEL- 源头控油
+
+## 寄生藻的秘密
+调节污染环境下的油性肌肤
+
+### 关于 CONTACTICEL
+CONTACTICEL（顶丝石斛）是一种附生红藻类海藻，一种寄生在其他藻类上的红色藻类，它的颜色来自于过多的红色色素（藻红蛋白）。它与许多其他微生物在一起，成为了宿主藻抵抗外部侵害（如紫外线、有害微生物、污染等）的屏障的一部分。
+
+### 工艺来源
+世界首创的藻类细胞培育技术 CELEBRITY™，其技术特点：细胞形态下收获和分离海藻；鉴定特殊稀有物种；以细胞形式在光生物反应器中培养。
+
+### 作用原理
+源头调节肌肤脂肪分泌，其独特作用是其他红藻不能比拟的；在污染条件下，可以保护角质形成细胞的新陈代谢以及保护细胞 DNA。
+
+INCI 名称：水解红藻提取物
+
+
+链路二：促进 ATP，补充肌肤能量 - SEPTONIC®M3  
+含多元素矿质的能量补充剂，给你的细胞充电！  
+SEPTONIC®M3 是一种通过将矿物离子和有机离子进行结合的活性物。通过更好的细胞呼吸，SEPTONIC®M3 逐步地刺激细胞的能量和新陈代谢。  
+
+天冬氨酸 / 葡萄糖酸  
+« 智能型 » 有机离子  
+- 给矿物离子提供载体，使它们具有生物可用性  
+- 真实的生物功效 (ATP - ADN)  
+
+![](https://example.com/image.png)  
+
+INCI 名：天冬氨酸 // 葡萄糖酸钠 // 葡萄糖酸铜  
+
+
+链路三：科技专利认证 - 净肤卫生士
+
+MA-Whelker-净肤卫生士专利组方  
+由芦荟叶 / 梅果 / 药用层孔菌 / 薄荷叶 / 密蒙花组合而成  
+
+- 高效保湿√  
+- 控油收敛毛孔√  
+- 抗皱老√  
+- 抗污垢√  
+- 预防皮肤敏感√  
+
+
+# Link 4: Enhance Skin Resistance - Lactic Acid Bacteria / Soybean Fermented Beverage
+
+**Aomizō - Probiotic**
+
+Aomizō is a fermented product derived from 16 varieties of black soybean and 35 types of lactic acid bacteria. It employs "Human Simulation" fermentation technology, mimicking human digestive system operations, improving the quality of nutrients that better meet human nutritional needs, and enhancing absorption efficiency.
+
+![](image1.png) 
+
+1. **Microbial Compound**
+   - **16 varieties & 35 types of lactic acid bacteria** & bifidobacteria
+
+2. **Fermentation Process**
+   - **5 days, 37°C** 
+   - Human digestive system simulation
+
+Contains **500+ active compounds**, including **300+ structured molecules**.
+
+- Protein quality
+- Lactic acid, hyaluronic acid
+- Fatty acids
+- Polysaccharides
+- Saccharides
+- Glucans, and structural units of cellulose
+- Phospholipids
+- Vitamins 
+- Minerals
+- Yellow pigment, and yellowish pigment
+- Antioxidants
+
+
+链路四：提升肌肤耐受力 - 乳酸杆菌 / 豆浆发酵产物滤液  
+**奥敏婷 - 益生菌的代谢物**  
+作用于微生物群，明显可感知的功效  
+
+(1) 改善皮肤微生物质量和多样性  
+- 体内测试  
+
+(2) 保护皮肤酸膜  
+- 维持皮肤处在健康的弱酸性pH  
+- 体内测试  
+
+(3) 舒缓  
+- 防止过敏  
+- 体内测试  
+- 改善过敏症状  
+- 临床测试  
+
+(4) 重组皮肤  
+- 促进肽原白的合成  
+- 体外测试  
+- 加速透明质酸的合成  
+- 体外测试  
+
+(5) 皮肤水合作用  
+- 即时提升皮肤的含水量  
+- 体内测试  
+- 维持每层皮肤的保湿子，对肌肤有一个长效深层的保湿效果  
+- 体内测试  
+
+
+# 链路四：提升肌肤耐受力 - 植物组方
+
+## 舒敏组方
+缓解皮肤不适，修复皮肤干燥
+
+天然植物提取物的组合
+- 有抑制透明质酸酶活性，抑制组胺释放，降低肌肤敏感和瘙痒症状；
+- 可以增加皮肤耐受性，缓解和恢复潮红、红肿、疼痛等肌肤损伤。
+
+![攀刺仙人掌](https://example.com/image1.jpg)  
+![苦参](https://example.com/image2.jpg)  
+![麦冬](https://example.com/image3.jpg)  
+
+
+# 威尔  
+感谢观看  
+资料介绍  
+广东茂名 市场部 2207  
+![](21_0.png)

parse_results.py

@@ -0,0 +1,154 @@
+import json
+import datetime
+from databases import Database
+from typing import Dict, List, Any
+
+# 数据库连接配置
+db_config = {
+    'user': 'root',
+    'password': 'zaq12wsx@9Xin', 
+    'host': '183.11.229.79',
+    'port': 3316,
+    'database': 'gptDB',
+    'auth_plugin': 'mysql_native_password'
+}
+
+DATABASE_URL = f"mysql://{db_config['user']}:{db_config['password']}@{db_config['host']}:{db_config['port']}/{db_config['database']}?auth_plugin={db_config['auth_plugin']}"
+
+database = Database(DATABASE_URL)
+
+class SearchResultParser:
+    def __init__(self):
+        self.parsed_results = []
+    
+    async def parse_google_results(self, results: Dict) -> List[Dict]:
+        """解析Google搜索结果"""
+        parsed = []
+        
+        # 解析organic_results
+        if 'organic_results' in results:
+            for result in results['organic_results']:
+                parsed.append({
+                    'title': result.get('title', ''),
+                    'link': result.get('link', ''),
+                    'snippet': result.get('snippet', ''),
+                    'position': result.get('position', 0),
+                    'type': 'organic'
+                })
+                
+        # 解析answer_box如果存在
+        if 'answer_box' in results:
+            answer = results['answer_box']
+            parsed.append({
+                'title': answer.get('title', ''),
+                'answer': answer.get('answer', ''),
+                'type': 'answer_box'
+            })
+            
+        return parsed
+
+    async def parse_bing_results(self, results: Dict) -> List[Dict]:
+        """解析Bing搜索结果"""
+        parsed = []
+        
+        if 'organic_results' in results:
+            for result in results['organic_results']:
+                parsed.append({
+                    'title': result.get('title', ''),
+                    'link': result.get('link', ''),
+                    'snippet': result.get('snippet', ''),
+                    'position': result.get('position', 0),
+                    'type': 'organic'
+                })
+                
+        return parsed
+
+    async def parse_baidu_results(self, results: Dict) -> List[Dict]:
+        """解析百度搜索结果"""
+        parsed = []
+        
+        if 'organic_results' in results:
+            for result in results['organic_results']:
+                parsed.append({
+                    'title': result.get('title', ''),
+                    'link': result.get('link', ''),
+                    'snippet': result.get('snippet', ''),
+                    'position': result.get('position', 0),
+                    'type': 'organic'
+                })
+        
+        # 解析百度特有的answer_box
+        if 'answer_box' in results:
+            for answer in results['answer_box']:
+                parsed.append({
+                    'answer': answer.get('answer', ''),
+                    'snippet': answer.get('snippet', ''),
+                    'source': answer.get('source', ''),
+                    'type': 'answer_box'
+                })
+                
+        return parsed
+
+    async def process_search_results(self):
+        """处理数据库中未处理的搜索结果"""
+        await database.connect()
+        
+        # 获取未处理的搜索结果
+        query = """
+            SELECT id, engine, query, results 
+            FROM web_search_results 
+            WHERE is_processed = 0
+        """
+        unprocessed_results = await database.fetch_all(query)
+        
+        for record in unprocessed_results:
+            results = json.loads(record['results'])
+            engine = record['engine']
+            
+            # 根据不同搜索引擎选择相应的解析方法
+            if engine == 'google':
+                parsed_data = await self.parse_google_results(results)
+            elif engine == 'bing':
+                parsed_data = await self.parse_bing_results(results)
+            elif engine == 'baidu':
+                parsed_data = await self.parse_baidu_results(results)
+            
+            # 将解析后的结果存入新表
+            for item in parsed_data:
+                await database.execute(
+                    """
+                    INSERT INTO parsed_search_results 
+                    (original_search_id, engine, query, title, link, snippet, result_type, parsed_datetime)
+                    VALUES (:original_search_id, :engine, :query, :title, :link, :snippet, :result_type, :parsed_datetime)
+                    """,
+                    {
+                        'original_search_id': record['id'],
+                        'engine': engine,
+                        'query': record['query'],
+                        'title': item.get('title', ''),
+                        'link': item.get('link', ''),
+                        'snippet': item.get('snippet', ''),
+                        'result_type': item.get('type', ''),
+                        'parsed_datetime': datetime.datetime.now()
+                    }
+                )
+            
+            # 更新原记录为已处理
+            await database.execute(
+                """
+                UPDATE web_search_results 
+                SET is_processed = 1 
+                WHERE id = :id
+                """,
+                {'id': record['id']}
+            )
+            
+        await database.disconnect()
+
+async def main():
+    parser = SearchResultParser()
+    await parser.process_search_results()
+
+if __name__ == "__main__":
+    import asyncio
+    asyncio.run(main()) 

search_plus.py

@@ -0,0 +1,90 @@
+import json
+import datetime
+from fastapi import FastAPI, HTTPException
+from pydantic import BaseModel
+from databases import Database
+import requests
+import json
+
+# 数据库连接配置
+db_config = {
+    'user': 'root',
+    'password': 'zaq12wsx@9Xin',
+    'host': '183.11.229.79',
+    'port': 3316,
+    'database': 'gptDB',
+    'auth_plugin': 'mysql_native_password'
+}
+
+# 创建数据库 URL
+DATABASE_URL = f"mysql://{db_config['user']}:{db_config['password']}@{db_config['host']}:{db_config['port']}/{db_config['database']}?auth_plugin={db_config['auth_plugin']}"
+
+database = Database(DATABASE_URL)
+
+SERP_API_KEY = "8af097ae8b587bb0569425058e03e5ef33b4c7b8b1a505053764b62e7e4ab9d6"
+
+async def search_and_save(query):
+    """
+    执行单个查询的搜索并保存结果
+    
+    参数:
+    query (str): 要搜索的查询
+    
+    返回:
+    list: 搜索结果列表
+    """
+    search_results = []
+    await database.connect()
+    
+    for engine in ["google", "bing", "baidu"]:
+        params = {
+            "api_key": SERP_API_KEY,
+            "engine": engine,
+            "q": query
+        }
+        response = requests.get('https://serpapi.com/search', params=params)
+        search = response.json()
+        search_metadata = search.get('search_metadata', {})
+        if search_metadata.get('status') == 'Success':
+            json_endpoint = search_metadata.get('json_endpoint')
+            response = requests.get(json_endpoint)
+            if response.status_code == 200:
+                data = response.json()
+                json_record_str = json.dumps(data)
+                
+                # 保存到数据库
+                query_ = """
+                    INSERT INTO web_search_results (engine, query, results, search_datetime, is_processed)
+                    VALUES (:engine, :query, :results, :search_datetime, :is_processed)
+                """
+                values = {
+                    "engine": engine,
+                    "query": query,
+                    "results": json_record_str,
+                    "search_datetime": datetime.datetime.now(),
+                    "is_processed": 0
+                }
+                await database.execute(query_, values)
+                search_results.append(json_record_str)
+    
+    await database.disconnect()
+    return search_results
+
+async def main(queries):
+    """
+    执行批量搜索查询
+    
+    参数:
+    queries (list): 要搜索的查询列表
+    """
+    for query in queries:
+        await search_and_save(query)
+
+if __name__ == "__main__":
+    example_queries = [
+        "化妆品 面部护理",
+        "化妆品 眼部护理",
+        "化妆品 面部清洁"
+    ]
+    import asyncio
+    asyncio.run(main(example_queries)) 

search_queries.py

@@ -0,0 +1,81 @@
+import json
+import datetime
+from fastapi import FastAPI, HTTPException
+from pydantic import BaseModel
+from databases import Database
+import requests
+import json
+
+# 数据库连接配置
+db_config = {
+    'user': 'root',
+    'password': 'zaq12wsx@9Xin',
+    'host': '183.11.229.79',
+    'port': 3316,
+    'database': 'gptDB',
+    'auth_plugin': 'mysql_native_password'
+}
+
+# 创建数据库 URL
+DATABASE_URL = f"mysql://{db_config['user']}:{db_config['password']}@{db_config['host']}:{db_config['port']}/{db_config['database']}?auth_plugin={db_config['auth_plugin']}"
+
+database = Database(DATABASE_URL)
+
+SERP_API_KEY = "8af097ae8b587bb0569425058e03e5ef33b4c7b8b1a505053764b62e7e4ab9d6" 
+
+queries = [
+"小红书化妆品 面部护理",
+"小红书化妆品 眼部护理",
+"小红书化妆品 面部清洁",
+"小红书化妆品 防晒及晒后护理",
+"小红书化妆品 体毛处理",
+"小红书化妆品 洗发护发",
+"小红书化妆品 洗涤",
+"小红书化妆品 面部清洁",
+"小红书化妆品 精油",
+"小红书化妆品 身体护理",
+"小红书化妆品 头皮洗护",
+"小红书化妆品 洗发护发",
+"小红书化妆品 洗浴产品",
+"小红书化妆品 婴童洗护",
+"小红书化妆品 婴童护肤"
+]
+
+async def main():
+    await database.connect()
+    for query in queries:
+        for engine in ["google", "bing", "baidu"]:
+            params = {
+                "api_key": SERP_API_KEY,
+                "engine": engine,
+                "q": query
+            }
+            response = requests.get('https://serpapi.com/search', params=params)
+            search = response.json()
+            search_metadata = search.get('search_metadata', {})
+            if search_metadata.get('status') == 'Success':
+                json_endpoint = search_metadata.get('json_endpoint') 
+                # 检查请求是否成功
+                response = requests.get(json_endpoint)
+                if response.status_code == 200:
+                    # 解析JSON数据
+                    data = response.json()
+                    json_record_str = json.dumps(data)  # 将JSON数据转换为字符串
+               
+                    # 保存到数据库
+                    query_ = """
+                        INSERT INTO web_search_results (engine, query, results, search_datetime, is_processed)
+                        VALUES (:engine, :query, :results, :search_datetime, :is_processed)
+                    """
+                    values = {
+                        "engine": engine,
+                        "query": query,
+                        "results": json_record_str,
+                        "search_datetime": datetime.datetime.now(),
+                        "is_processed":0
+                    }
+                    await database.execute(query_, values)
+    await database.disconnect()
+
+import asyncio
+asyncio.run(main())

search_ui.py

@@ -0,0 +1,246 @@
+import gradio as gr 
+import json
+from typing import Dict, List, Any
+import requests
+
+class SearchUI:
+    def __init__(self):
+        self.api_key = "8af097ae8b587bb0569425058e03e5ef33b4c7b8b1a505053764b62e7e4ab9d6"
+    
+    def parse_google_results(self, results: Dict) -> List[Dict]:
+        """解析Google搜索结果"""
+        parsed = []
+        
+        # 解析organic_results
+        if 'organic_results' in results:
+            for result in results['organic_results']:
+                parsed.append({
+                    'title': result.get('title', ''),
+                    'link': result.get('link', ''),
+                    'snippet': result.get('snippet', ''),
+                    'position': result.get('position', 0)
+                })
+                
+        # 解析answer_box如果存在
+        if 'answer_box' in results:
+            answer = results['answer_box']
+            parsed.append({
+                'title': '直接回答:',
+                'link': answer.get('link', ''),
+                'snippet': answer.get('answer', ''),
+                'position': 0
+            })
+            
+        return parsed
+
+    def parse_bing_results(self, results: Dict) -> List[Dict]:
+        """解析Bing搜索结果"""
+        parsed = []
+        
+        if 'organic_results' in results:
+            for result in results['organic_results']:
+                parsed.append({
+                    'title': result.get('title', ''),
+                    'link': result.get('link', ''),
+                    'snippet': result.get('snippet', ''),
+                    'position': result.get('position', 0)
+                })
+                
+        return parsed
+
+    def parse_baidu_results(self, results: Dict) -> List[Dict]:
+        """解析百度搜索结果"""
+        parsed = []
+        
+        if 'organic_results' in results:
+            for result in results['organic_results']:
+                parsed.append({
+                    'title': result.get('title', ''),
+                    'link': result.get('link', ''),
+                    'snippet': result.get('snippet', ''),
+                    'position': result.get('position', 0)
+                })
+        
+        # 解析百度特有的answer_box
+        if 'answer_box' in results:
+            for answer in results['answer_box']:
+                parsed.append({
+                    'title': '直接回答:',
+                    'link': answer.get('link', ''),
+                    'snippet': f"{answer.get('answer', '')}\n来源: {answer.get('source', '')}",
+                    'position': 0
+                })
+                
+        return parsed
+
+    def format_results(self, results: List[Dict]) -> str:
+        """将解析后的结果格式化为markdown文本"""
+        formatted = ""
+        for result in sorted(results, key=lambda x: x['position']):
+            formatted += f"### {result['title']}\n\n"
+            formatted += f"🔗 [{result['link']}]({result['link']})\n\n"
+            formatted += f"{result['snippet']}\n\n"
+            formatted += "---\n\n"
+        return formatted
+
+    def search_all_engines(self, query: str) -> tuple:
+        """在所有搜索引擎中执行搜索并返回格式化结果"""
+        try:
+            results = {}
+            for engine in ["google", "bing", "baidu"]:
+                params = {
+                    "api_key": self.api_key,
+                    "engine": engine,
+                    "q": query
+                }
+                # 为Google添加特殊参数
+                if engine == "google":
+                    params.update({
+                        "gl": "cn",
+                        "hl": "zh-cn"
+                    })
+                
+                response = requests.get('https://serpapi.com/search', params=params)
+                if response.status_code == 200:
+                    results[engine] = response.json()
+                else:
+                    return f"{engine}搜索出错: HTTP {response.status_code}", "", ""
+
+            # 解析各引擎结果
+            google_parsed = self.parse_google_results(results['google'])
+            google_text = self.format_results(google_parsed)
+
+            bing_parsed = self.parse_bing_results(results['bing'])
+            bing_text = self.format_results(bing_parsed)
+
+            baidu_parsed = self.parse_baidu_results(results['baidu'])
+            baidu_text = self.format_results(baidu_parsed)
+
+            return google_text, bing_text, baidu_text
+
+        except Exception as e:
+            return f"搜索出错: {str(e)}", f"搜索出错: {str(e)}", f"搜索出错: {str(e)}"
+
+def create_ui():
+    search_ui = SearchUI()
+    
+    # 自定义CSS样式
+    custom_css = """
+    .container {padding: 20px;}
+    
+    /* 搜索引擎区域样式 */
+    .google-container {background: rgba(66, 133, 244, 0.05); border-radius: 15px; padding: 15px;}
+    .bing-container {background: rgba(0, 169, 157, 0.05); border-radius: 15px; padding: 15px;}
+    .baidu-container {background: rgba(45, 97, 255, 0.05); border-radius: 15px; padding: 15px;}
+    
+    /* 结果卡片样式 */
+    .result-card {
+        background: white;
+        border-radius: 10px;
+        padding: 15px;
+        margin: 10px 0;
+        box-shadow: 0 2px 4px rgba(0,0,0,0.1);
+        transition: all 0.3s ease;
+    }
+    
+    .result-card:hover {
+        transform: translateY(-3px);
+        box-shadow: 0 4px 8px rgba(0,0,0,0.15);
+    }
+    
+    /* 链接样式 */
+    .result-card a {
+        color: #1a0dab;
+        text-decoration: none;
+    }
+    
+    .result-card a:hover {
+        text-decoration: underline;
+    }
+    
+    /* 分隔线样式 */
+    .google-container hr {border-color: rgba(66, 133, 244, 0.2);}
+    .bing-container hr {border-color: rgba(0, 169, 157, 0.2);}
+    .baidu-container hr {border-color: rgba(45, 97, 255, 0.2);}
+    """
+    
+    def format_with_css(results: str, engine: str) -> str:
+        """为结果添加CSS类"""
+        # 将markdown结果分割成单独的卡片
+        cards = results.split("---")
+        formatted_cards = []
+        
+        for card in cards:
+            if card.strip():
+                formatted_cards.append(f'<div class="result-card">{card}</div>')
+                
+        return f'<div class="{engine}-container">{"".join(formatted_cards)}</div>'
+    
+    with gr.Blocks(title="九鑫多搜索引擎聚合搜索", theme=gr.themes.Soft(), css=custom_css) as demo:
+        gr.Markdown("## 🔍 九鑫多搜索引擎聚合搜索")
+        
+        with gr.Row():
+            query_input = gr.Textbox(
+                label="搜索关键词",
+                placeholder="请输入要搜索的内容...",
+                lines=1
+            )
+            search_button = gr.Button("搜索", variant="primary")
+        
+        # 修改加载动画实现
+        loading_indicator = gr.HTML(
+            """
+            <div style="text-align:center; margin:20px;">
+                <img src="https://media.giphy.com/media/3oEjI6SIIHBdRxXI40/giphy.gif" alt="loading..." />
+                <p>正在搜索中...</p>
+            </div>
+            """,
+            visible=False
+        )
+        
+        with gr.Row():
+            with gr.Column():
+                gr.Markdown("### 🔵 Google 搜索结果")
+                google_output = gr.HTML()
+            
+            with gr.Column():
+                gr.Markdown("### 🟢 Bing 搜索结果")
+                bing_output = gr.HTML()
+            
+            with gr.Column():
+                gr.Markdown("### 🔷 百度搜索结果")
+                baidu_output = gr.HTML()
+        
+        def on_search(query):
+            return [
+                True,  # loading indicator visible
+                "",    # clear google output
+                "",    # clear bing output
+                ""     # clear baidu output
+            ]
+        
+        def after_search(query):
+            google, bing, baidu = search_ui.search_all_engines(query)
+            return [
+                False,  # hide loading indicator
+                format_with_css(google, "google"),
+                format_with_css(bing, "bing"),
+                format_with_css(baidu, "baidu")
+            ]
+        
+        search_button.click(
+            fn=on_search,
+            inputs=[query_input],
+            outputs=[loading_indicator, google_output, bing_output, baidu_output],
+        ).then(
+            fn=after_search,
+            inputs=[query_input],
+            outputs=[loading_indicator, google_output, bing_output, baidu_output],
+            show_progress="minimal"
+        )
+    
+    return demo
+
+if __name__ == "__main__":
+    demo = create_ui()
+    demo.launch(share=True, server_name="0.0.0.0", server_port=10083) 

search_ui_v2.py

@@ -0,0 +1,253 @@
+import gradio as gr 
+import json
+from typing import Dict, List, Any
+import requests
+
+class SearchUI:
+    def __init__(self):
+        self.api_key = "8af097ae8b587bb0569425058e03e5ef33b4c7b8b1a505053764b62e7e4ab9d6"
+    
+    def parse_google_results(self, results: Dict) -> List[Dict]:
+        """解析Google搜索结果"""
+        parsed = []
+        if 'organic_results' in results:
+            for result in results['organic_results']:
+                parsed.append({
+                    'title': result.get('title', ''),
+                    'link': result.get('link', ''),
+                    'snippet': result.get('snippet', ''),
+                    'position': result.get('position', 0)
+                })
+                
+        if 'answer_box' in results:
+            answer = results['answer_box']
+            parsed.append({
+                'title': '直接回答:',
+                'link': answer.get('link', ''),
+                'snippet': answer.get('answer', ''),
+                'position': 0
+            })
+        return parsed
+
+    def parse_bing_results(self, results: Dict) -> List[Dict]:
+        """解析Bing搜索结果"""
+        parsed = []
+        if 'organic_results' in results:
+            for result in results['organic_results']:
+                parsed.append({
+                    'title': result.get('title', ''),
+                    'link': result.get('link', ''),
+                    'snippet': result.get('snippet', ''),
+                    'position': result.get('position', 0)
+                })
+        return parsed
+
+    def parse_baidu_results(self, results: Dict) -> List[Dict]:
+        """解析百度搜索结果"""
+        parsed = []
+        if 'organic_results' in results:
+            for result in results['organic_results']:
+                parsed.append({
+                    'title': result.get('title', ''),
+                    'link': result.get('link', ''),
+                    'snippet': result.get('snippet', ''),
+                    'position': result.get('position', 0)
+                })
+        
+        if 'answer_box' in results:
+            for answer in results['answer_box']:
+                parsed.append({
+                    'title': '直接回答:',
+                    'link': answer.get('link', ''),
+                    'snippet': f"{answer.get('answer', '')}\n来源: {answer.get('source', '')}",
+                    'position': 0
+                })
+        return parsed
+
+    def format_results(self, results: List[Dict], engine: str) -> str:
+        """将解析后的结果格式化为HTML卡片"""
+        formatted = ""
+        for result in sorted(results, key=lambda x: x['position']):
+            formatted += f"""
+                <div class="result-card" onclick="window.open('{result['link']}', '_blank')">
+                    <div class="card-title">{result['title']}</div>
+                    <div class="card-link">{result['link']}</div>
+                    <div class="card-snippet">{result['snippet']}</div>
+                </div>
+            """
+        return f'<div class="{engine}-container">{formatted}</div>'
+
+    def search_all_engines(self, query: str) -> tuple:
+        """在所有搜索引擎中执行搜索并返回格式化结果"""
+        try:
+            results = {}
+            for engine in ["google", "bing", "baidu"]:
+                params = {
+                    "api_key": self.api_key,
+                    "engine": engine,
+                    "q": query
+                }
+                if engine == "google":
+                    params.update({"gl": "cn", "hl": "zh-cn"})
+                
+                response = requests.get('https://serpapi.com/search', params=params)
+                if response.status_code == 200:
+                    results[engine] = response.json()
+                else:
+                    return f"{engine}搜索出错: HTTP {response.status_code}", "", ""
+
+            google_parsed = self.parse_google_results(results['google'])
+            google_text = self.format_results(google_parsed, "google")
+
+            bing_parsed = self.parse_bing_results(results['bing'])
+            bing_text = self.format_results(bing_parsed, "bing")
+
+            baidu_parsed = self.parse_baidu_results(results['baidu'])
+            baidu_text = self.format_results(baidu_parsed, "baidu")
+
+            return google_text, bing_text, baidu_text
+
+        except Exception as e:
+            return f"搜索出错: {str(e)}", f"搜索出错: {str(e)}", f"搜索出错: {str(e)}"
+
+def create_ui():
+    search_ui = SearchUI()
+    
+    custom_css = """
+    /* 全局样式 */
+    .container { max-width: 1400px; margin: 0 auto; }
+    
+    /* 搜索引擎区域样式 */
+    .google-container, .bing-container, .baidu-container {
+        background: white;
+        border-radius: 15px;
+        padding: 15px;
+        height: calc(100vh - 200px);
+        overflow-y: auto;
+        scrollbar-width: thin;
+    }
+    
+    .google-container { border: 2px solid rgba(66, 133, 244, 0.3); }
+    .bing-container { border: 2px solid rgba(0, 169, 157, 0.3); }
+    .baidu-container { border: 2px solid rgba(45, 97, 255, 0.3); }
+    
+    /* 结果卡片样式 */
+    .result-card {
+        background: white;
+        border-radius: 10px;
+        padding: 15px;
+        margin: 10px 0;
+        box-shadow: 0 2px 4px rgba(0,0,0,0.1);
+        transition: all 0.3s ease;
+        cursor: pointer;
+        overflow: hidden;
+        word-wrap: break-word;
+    }
+    
+    .result-card:hover {
+        transform: translateY(-3px);
+        box-shadow: 0 4px 8px rgba(0,0,0,0.15);
+    }
+    
+    .card-title {
+        font-size: 1.1em;
+        font-weight: bold;
+        color: #1a0dab;
+        margin-bottom: 5px;
+        line-height: 1.4;
+    }
+    
+    .card-link {
+        color: #006621;
+        font-size: 0.9em;
+        margin-bottom: 8px;
+        overflow: hidden;
+        text-overflow: ellipsis;
+        white-space: nowrap;
+    }
+    
+    .card-snippet {
+        color: #545454;
+        font-size: 0.95em;
+        line-height: 1.5;
+        overflow: hidden;
+        display: -webkit-box;
+        -webkit-line-clamp: 4;
+        -webkit-box-orient: vertical;
+    }
+    
+    /* 滚动条样式 */
+    .google-container::-webkit-scrollbar,
+    .bing-container::-webkit-scrollbar,
+    .baidu-container::-webkit-scrollbar {
+        width: 6px;
+    }
+    
+    .google-container::-webkit-scrollbar-thumb { background: rgba(66, 133, 244, 0.3); }
+    .bing-container::-webkit-scrollbar-thumb { background: rgba(0, 169, 157, 0.3); }
+    .baidu-container::-webkit-scrollbar-thumb { background: rgba(45, 97, 255, 0.3); }
+    
+    .google-container::-webkit-scrollbar-track,
+    .bing-container::-webkit-scrollbar-track,
+    .baidu-container::-webkit-scrollbar-track {
+        background: rgba(0, 0, 0, 0.05);
+    }
+    """
+    
+    with gr.Blocks(title="九鑫多搜索引擎聚合搜索", theme=gr.themes.Soft(), css=custom_css) as demo:
+        gr.Markdown("## 🔍 九鑫多搜索引擎聚合搜索")
+        
+        with gr.Row():
+            query_input = gr.Textbox(
+                label="搜索关键词",
+                placeholder="请输入要搜索的内容...",
+                lines=1
+            )
+            search_button = gr.Button("搜索", variant="primary", scale=0.2)
+        
+        loading_indicator = gr.HTML(
+            """
+            <div style="text-align:center; margin:20px;">
+                <img src="https://media.giphy.com/media/3oEjI6SIIHBdRxXI40/giphy.gif" alt="loading..." />
+                <p>正在搜索中...</p>
+            </div>
+            """,
+            visible=False
+        )
+        
+        with gr.Row():
+            with gr.Column(scale=1):
+                gr.Markdown("### 🔵 Google 搜索结果")
+                google_output = gr.HTML(container=False)
+            
+            with gr.Column(scale=1):
+                gr.Markdown("### 🟢 Bing 搜索结果")
+                bing_output = gr.HTML(container=False)
+            
+            with gr.Column(scale=1):
+                gr.Markdown("### 🔷 百度搜索结果")
+                baidu_output = gr.HTML(container=False)
+        
+        def on_search(query):
+            return [True, "", "", ""]
+        
+        def after_search(query):
+            google, bing, baidu = search_ui.search_all_engines(query)
+            return [False, google, bing, baidu]
+        
+        search_button.click(
+            fn=on_search,
+            inputs=[query_input],
+            outputs=[loading_indicator, google_output, bing_output, baidu_output],
+        ).then(
+            fn=after_search,
+            inputs=[query_input],
+            outputs=[loading_indicator, google_output, bing_output, baidu_output],
+            show_progress="minimal"
+        )
+    
+    return demo
+
+if __name__ == "__main__":
+    demo = create_ui()
+    demo.launch(share=True, server_name="0.0.0.0", server_port=10083) 

server.log

@@ -0,0 +1,653 @@
+INFO:     Will watch for changes in these directories: ['/home/ubuntu/apps/search_tools/app']
+INFO:     Uvicorn running on http://0.0.0.0:10083 (Press CTRL+C to quit)
+INFO:     Started reloader process [2715327] using WatchFiles
+INFO:     Started server process [2715330]
+INFO:     Waiting for application startup.
+INFO:     Application startup complete.
+INFO:     38.54.50.90:44248 - "GET /?query=%E9%AB%98%E7%88%B7%E5%AE%B6%E7%8C%AB%E7%B2%AE HTTP/1.1" 404 Not Found
+INFO:     38.54.50.90:51656 - "POST / HTTP/1.1" 404 Not Found
+Error fetching summaries: (1146, "Table '9xin.saved_webpages' doesn't exist")
+INFO:     38.54.50.90:38324 - "POST /search/ HTTP/1.1" 200 OK
+WARNING:  WatchFiles detected changes in 'search_queries.py'. Reloading...
+INFO:     Shutting down
+INFO:     Waiting for application shutdown.
+INFO:     Application shutdown complete.
+INFO:     Finished server process [2715330]
+INFO:     Started server process [2724749]
+INFO:     Waiting for application startup.
+INFO:     Application startup complete.
+WARNING:  WatchFiles detected changes in 'util_old.py'. Reloading...
+INFO:     Shutting down
+INFO:     Waiting for application shutdown.
+INFO:     Application shutdown complete.
+INFO:     Finished server process [2724749]
+INFO:     Started server process [2738595]
+INFO:     Waiting for application startup.
+INFO:     Application startup complete.
+WARNING:  WatchFiles detected changes in 'util_old.py'. Reloading...
+INFO:     Shutting down
+INFO:     Waiting for application shutdown.
+INFO:     Application shutdown complete.
+INFO:     Finished server process [2738595]
+INFO:     Started server process [2738602]
+INFO:     Waiting for application startup.
+INFO:     Application startup complete.
+INFO:     165.154.51.225:19230 - "GET / HTTP/1.1" 404 Not Found
+WARNING:  Invalid HTTP request received.
+WARNING:  Invalid HTTP request received.
+INFO:     165.154.118.9:36588 - "GET / HTTP/1.1" 404 Not Found
+INFO:     165.154.118.9:36776 - "GET /favicon.ico HTTP/1.1" 404 Not Found
+INFO:     165.154.118.9:36884 - "GET /sitemap.xml HTTP/1.1" 404 Not Found
+INFO:     165.154.118.9:36882 - "GET /robots.txt HTTP/1.1" 404 Not Found
+INFO:     118.193.59.142:58970 - "GET / HTTP/1.1" 404 Not Found
+WARNING:  Invalid HTTP request received.
+WARNING:  Invalid HTTP request received.
+INFO:     165.154.182.174:41872 - "GET / HTTP/1.1" 404 Not Found
+INFO:     165.154.182.174:41944 - "GET /favicon.ico HTTP/1.1" 404 Not Found
+INFO:     165.154.182.174:42398 - "GET /sitemap.xml HTTP/1.1" 404 Not Found
+INFO:     165.154.182.174:42396 - "GET /robots.txt HTTP/1.1" 404 Not Found
+INFO:     106.75.156.189:56564 - "POST /token HTTP/1.1" 404 Not Found
+INFO:     185.242.226.4:33756 - "GET / HTTP/1.1" 404 Not Found
+INFO:     165.154.182.53:55626 - "GET / HTTP/1.1" 404 Not Found
+INFO:     165.154.182.53:55660 - "GET /favicon.ico HTTP/1.1" 404 Not Found
+INFO:     165.154.182.53:55724 - "GET /sitemap.xml HTTP/1.1" 404 Not Found
+INFO:     165.154.182.53:55722 - "GET /robots.txt HTTP/1.1" 404 Not Found
+INFO:     152.32.134.89:52042 - "GET / HTTP/1.1" 404 Not Found
+INFO:     152.32.134.89:52102 - "GET /favicon.ico HTTP/1.1" 404 Not Found
+INFO:     152.32.134.89:52182 - "GET /sitemap.xml HTTP/1.1" 404 Not Found
+INFO:     152.32.134.89:52184 - "GET /robots.txt HTTP/1.1" 404 Not Found
+INFO:     106.75.186.101:41622 - "POST /token HTTP/1.1" 404 Not Found
+INFO:     144.126.213.177:55772 - "GET / HTTP/1.1" 404 Not Found
+INFO:     185.242.226.4:42024 - "GET / HTTP/1.1" 404 Not Found
+WARNING:  WatchFiles detected changes in 'main.py'. Reloading...
+INFO:     Shutting down
+INFO:     Waiting for application shutdown.
+INFO:     Application shutdown complete.
+INFO:     Finished server process [2738602]
+INFO:     Started server process [4073521]
+INFO:     Waiting for application startup.
+INFO:     Application startup complete.
+WARNING:  WatchFiles detected changes in 'main.py'. Reloading...
+INFO:     Shutting down
+INFO:     Waiting for application shutdown.
+INFO:     Application shutdown complete.
+INFO:     Finished server process [4073521]
+INFO:     Started server process [4075998]
+INFO:     Waiting for application startup.
+INFO:     Application startup complete.
+Error fetching summaries: (1146, "Table '9xin.saved_webpages' doesn't exist")
+INFO:     218.86.185.185:54433 - "POST /search/ HTTP/1.1" 500 Internal Server Error
+ERROR:    Exception in ASGI application
+Traceback (most recent call last):
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
+    result = await app(  # type: ignore[func-returns-value]
+             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
+    return await self.app(scope, receive, send)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
+    await super().__call__(scope, receive, send)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/applications.py", line 123, in __call__
+    await self.middleware_stack(scope, receive, send)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
+    raise exc
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
+    await self.app(scope, receive, _send)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
+    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
+    raise exc
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
+    await app(scope, receive, sender)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/routing.py", line 756, in __call__
+    await self.middleware_stack(scope, receive, send)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/routing.py", line 776, in app
+    await route.handle(scope, receive, send)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/routing.py", line 297, in handle
+    await self.app(scope, receive, send)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/routing.py", line 77, in app
+    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
+    raise exc
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
+    await app(scope, receive, sender)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/routing.py", line 72, in app
+    response = await func(request)
+               ^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/fastapi/routing.py", line 278, in app
+    raw_response = await run_endpoint_function(
+                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
+    return await dependant.call(**values)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/app/main.py", line 94, in search
+    web_results = await search_and_save(query)
+                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/app/search_plus.py", line 67, in search_and_save
+    await database.execute(query_, values)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/databases/core.py", line 199, in execute
+    return await connection.execute(query, values)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/databases/core.py", line 318, in execute
+    return await self._connection.execute(built_query)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/databases/backends/mysql.py", line 161, in execute
+    await cursor.execute(query_str, args)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/aiomysql/cursors.py", line 239, in execute
+    await self._query(query)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/aiomysql/cursors.py", line 457, in _query
+    await conn.query(q)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/aiomysql/connection.py", line 469, in query
+    await self._read_query_result(unbuffered=unbuffered)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/aiomysql/connection.py", line 683, in _read_query_result
+    await result.read()
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/aiomysql/connection.py", line 1164, in read
+    first_packet = await self.connection._read_packet()
+                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/aiomysql/connection.py", line 652, in _read_packet
+    packet.raise_for_error()
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/pymysql/protocol.py", line 219, in raise_for_error
+    err.raise_mysql_exception(self._data)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/pymysql/err.py", line 150, in raise_mysql_exception
+    raise errorclass(errno, errval)
+pymysql.err.DataError: (1406, "Data too long for column 'results' at row 1")
+INFO:     218.86.185.185:54483 - "POST /batch_search/ HTTP/1.1" 500 Internal Server Error
+WARNING:  WatchFiles detected changes in 'search_plus.py'. Reloading...
+INFO:     Shutting down
+INFO:     Waiting for application shutdown.
+INFO:     Application shutdown complete.
+INFO:     Finished server process [4075998]
+INFO:     Started server process [4086197]
+INFO:     Waiting for application startup.
+INFO:     Application startup complete.
+Error fetching summaries: (1146, "Table '9xin.saved_webpages' doesn't exist")
+INFO:     218.86.185.185:54959 - "POST /search/ HTTP/1.1" 500 Internal Server Error
+ERROR:    Exception in ASGI application
+Traceback (most recent call last):
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
+    result = await app(  # type: ignore[func-returns-value]
+             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
+    return await self.app(scope, receive, send)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
+    await super().__call__(scope, receive, send)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/applications.py", line 123, in __call__
+    await self.middleware_stack(scope, receive, send)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
+    raise exc
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
+    await self.app(scope, receive, _send)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
+    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
+    raise exc
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
+    await app(scope, receive, sender)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/routing.py", line 756, in __call__
+    await self.middleware_stack(scope, receive, send)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/routing.py", line 776, in app
+    await route.handle(scope, receive, send)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/routing.py", line 297, in handle
+    await self.app(scope, receive, send)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/routing.py", line 77, in app
+    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
+    raise exc
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
+    await app(scope, receive, sender)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/routing.py", line 72, in app
+    response = await func(request)
+               ^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/fastapi/routing.py", line 278, in app
+    raw_response = await run_endpoint_function(
+                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
+    return await dependant.call(**values)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/app/main.py", line 94, in search
+    web_results = await search_and_save(query)
+                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/app/search_plus.py", line 67, in search_and_save
+    await database.execute(query_, values)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/databases/core.py", line 199, in execute
+    return await connection.execute(query, values)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/databases/core.py", line 318, in execute
+    return await self._connection.execute(built_query)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/databases/backends/mysql.py", line 161, in execute
+    await cursor.execute(query_str, args)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/aiomysql/cursors.py", line 239, in execute
+    await self._query(query)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/aiomysql/cursors.py", line 457, in _query
+    await conn.query(q)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/aiomysql/connection.py", line 469, in query
+    await self._read_query_result(unbuffered=unbuffered)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/aiomysql/connection.py", line 683, in _read_query_result
+    await result.read()
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/aiomysql/connection.py", line 1164, in read
+    first_packet = await self.connection._read_packet()
+                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/aiomysql/connection.py", line 652, in _read_packet
+    packet.raise_for_error()
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/pymysql/protocol.py", line 219, in raise_for_error
+    err.raise_mysql_exception(self._data)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/pymysql/err.py", line 150, in raise_mysql_exception
+    raise errorclass(errno, errval)
+pymysql.err.ProgrammingError: (1146, "Table '9xin.web_search_results' doesn't exist")
+INFO:     218.86.185.185:54970 - "POST /batch_search/ HTTP/1.1" 500 Internal Server Error
+WARNING:  WatchFiles detected changes in 'main.py'. Reloading...
+INFO:     Shutting down
+INFO:     Waiting for application shutdown.
+INFO:     Application shutdown complete.
+INFO:     Finished server process [4086197]
+INFO:     Started server process [4087364]
+INFO:     Waiting for application startup.
+INFO:     Application startup complete.
+Error fetching summaries: (1146, "Table '9xin.saved_webpages' doesn't exist")
+INFO:     218.86.185.185:50392 - "POST /search/ HTTP/1.1" 500 Internal Server Error
+ERROR:    Exception in ASGI application
+Traceback (most recent call last):
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
+    result = await app(  # type: ignore[func-returns-value]
+             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
+    return await self.app(scope, receive, send)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
+    await super().__call__(scope, receive, send)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/applications.py", line 123, in __call__
+    await self.middleware_stack(scope, receive, send)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
+    raise exc
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
+    await self.app(scope, receive, _send)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
+    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
+    raise exc
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
+    await app(scope, receive, sender)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/routing.py", line 756, in __call__
+    await self.middleware_stack(scope, receive, send)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/routing.py", line 776, in app
+    await route.handle(scope, receive, send)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/routing.py", line 297, in handle
+    await self.app(scope, receive, send)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/routing.py", line 77, in app
+    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
+    raise exc
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
+    await app(scope, receive, sender)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/starlette/routing.py", line 72, in app
+    response = await func(request)
+               ^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/fastapi/routing.py", line 278, in app
+    raw_response = await run_endpoint_function(
+                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
+    return await dependant.call(**values)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/app/main.py", line 94, in search
+    web_results = await search_and_save(query)
+                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/app/search_plus.py", line 67, in search_and_save
+    await database.execute(query_, values)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/databases/core.py", line 199, in execute
+    return await connection.execute(query, values)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/databases/core.py", line 318, in execute
+    return await self._connection.execute(built_query)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/databases/backends/mysql.py", line 161, in execute
+    await cursor.execute(query_str, args)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/aiomysql/cursors.py", line 239, in execute
+    await self._query(query)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/aiomysql/cursors.py", line 457, in _query
+    await conn.query(q)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/aiomysql/connection.py", line 469, in query
+    await self._read_query_result(unbuffered=unbuffered)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/aiomysql/connection.py", line 683, in _read_query_result
+    await result.read()
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/aiomysql/connection.py", line 1164, in read
+    first_packet = await self.connection._read_packet()
+                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/aiomysql/connection.py", line 652, in _read_packet
+    packet.raise_for_error()
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/pymysql/protocol.py", line 219, in raise_for_error
+    err.raise_mysql_exception(self._data)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/pymysql/err.py", line 150, in raise_mysql_exception
+    raise errorclass(errno, errval)
+pymysql.err.ProgrammingError: (1146, "Table '9xin.web_search_results' doesn't exist")
+INFO:     218.86.185.185:50413 - "POST /batch_search/ HTTP/1.1" 500 Internal Server Error
+WARNING:  WatchFiles detected changes in 'search_plus.py'. Reloading...
+INFO:     Shutting down
+INFO:     Waiting for application shutdown.
+INFO:     Application shutdown complete.
+INFO:     Finished server process [4087364]
+INFO:     Started server process [4091530]
+INFO:     Waiting for application startup.
+INFO:     Application startup complete.
+WARNING:  WatchFiles detected changes in 'main.py'. Reloading...
+INFO:     Shutting down
+INFO:     Waiting for application shutdown.
+INFO:     Application shutdown complete.
+INFO:     Finished server process [4091530]
+INFO:     Started server process [4091703]
+INFO:     Waiting for application startup.
+INFO:     Application startup complete.
+Error saving user query: (1146, "Table 'gptDB.user_queries' doesn't exist")
+INFO:     218.86.185.185:50885 - "POST /search/ HTTP/1.1" 200 OK
+Error saving user query: (1146, "Table 'gptDB.user_queries' doesn't exist")
+Error saving user query: (1146, "Table 'gptDB.user_queries' doesn't exist")
+Error saving user query: (1146, "Table 'gptDB.user_queries' doesn't exist")
+Error saving user query: (1146, "Table 'gptDB.user_queries' doesn't exist")
+Error saving user query: (1146, "Table 'gptDB.user_queries' doesn't exist")
+Error saving user query: (1146, "Table 'gptDB.user_queries' doesn't exist")
+INFO:     218.86.185.185:50905 - "POST /batch_search/ HTTP/1.1" 200 OK
+WARNING:  WatchFiles detected changes in 'main.py'. Reloading...
+INFO:     Shutting down
+INFO:     Waiting for application shutdown.
+INFO:     Application shutdown complete.
+INFO:     Finished server process [4091703]
+INFO:     Started server process [4094713]
+INFO:     Waiting for application startup.
+INFO:     Application startup complete.
+Error saving user query: (1146, "Table 'gptDB.user_queries' doesn't exist")
+INFO:     218.86.185.185:51254 - "POST /search/ HTTP/1.1" 200 OK
+Error saving user query: (1146, "Table 'gptDB.user_queries' doesn't exist")
+Error saving user query: (1146, "Table 'gptDB.user_queries' doesn't exist")
+Error saving user query: (1146, "Table 'gptDB.user_queries' doesn't exist")
+Error saving user query: (1146, "Table 'gptDB.user_queries' doesn't exist")
+Error saving user query: (1146, "Table 'gptDB.user_queries' doesn't exist")
+Error saving user query: (1146, "Table 'gptDB.user_queries' doesn't exist")
+INFO:     218.86.185.185:51262 - "POST /batch_search/ HTTP/1.1" 200 OK
+Error saving user query: (1146, "Table 'gptDB.user_queries' doesn't exist")
+INFO:     218.86.185.185:51402 - "POST /search/ HTTP/1.1" 200 OK
+Error saving user query: (1146, "Table 'gptDB.user_queries' doesn't exist")
+Error saving user query: (1146, "Table 'gptDB.user_queries' doesn't exist")
+Error saving user query: (1146, "Table 'gptDB.user_queries' doesn't exist")
+INFO:     218.86.185.185:51406 - "POST /batch_search/ HTTP/1.1" 200 OK
+Error saving user query: (1146, "Table 'gptDB.user_queries' doesn't exist")
+INFO:     218.86.185.185:51493 - "POST /search/ HTTP/1.1" 200 OK
+WARNING:  WatchFiles detected changes in 'main.py'. Reloading...
+INFO:     Shutting down
+INFO:     Waiting for application shutdown.
+INFO:     Application shutdown complete.
+INFO:     Finished server process [4094713]
+INFO:     Started server process [4099227]
+INFO:     Waiting for application startup.
+INFO:     Application startup complete.
+Error saving user query: (1146, "Table 'gptDB.user_queries' doesn't exist")
+Error saving user query: (1146, "Table 'gptDB.user_queries' doesn't exist")
+INFO:     218.86.185.185:51558 - "POST /search/ HTTP/1.1" 200 OK
+WARNING:  WatchFiles detected changes in 'main.py'. Reloading...
+INFO:     Shutting down
+INFO:     Waiting for application shutdown.
+INFO:     Application shutdown complete.
+INFO:     Finished server process [4099227]
+Process SpawnProcess-13:
+Traceback (most recent call last):
+  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
+    self.run()
+  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
+    self._target(*self._args, **self._kwargs)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/_subprocess.py", line 80, in subprocess_started
+    target(sockets=sockets)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/server.py", line 65, in run
+    return asyncio.run(self.serve(sockets=sockets))
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
+    return runner.run(main)
+           ^^^^^^^^^^^^^^^^
+  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
+    return self._loop.run_until_complete(task)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/server.py", line 69, in serve
+    await self._serve(sockets)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/server.py", line 76, in _serve
+    config.load()
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/config.py", line 434, in load
+    self.loaded_app = import_from_string(self.app)
+                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/importer.py", line 19, in import_from_string
+    module = importlib.import_module(module_str)
+             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
+    return _bootstrap._gcd_import(name[level:], package, level)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
+  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
+  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
+  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
+  File "<frozen importlib._bootstrap_external>", line 991, in exec_module
+  File "<frozen importlib._bootstrap_external>", line 1129, in get_code
+  File "<frozen importlib._bootstrap_external>", line 1059, in source_to_code
+  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
+  File "/home/ubuntu/apps/search_tools/app/main.py", line 158
+    return BatchSearchResponse(
+    ^^^^^^
+IndentationError: expected an indented block after 'finally' statement on line 156
+WARNING:  WatchFiles detected changes in 'main.py'. Reloading...
+Process SpawnProcess-14:
+Traceback (most recent call last):
+  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
+    self.run()
+  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
+    self._target(*self._args, **self._kwargs)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/_subprocess.py", line 80, in subprocess_started
+    target(sockets=sockets)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/server.py", line 65, in run
+    return asyncio.run(self.serve(sockets=sockets))
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
+    return runner.run(main)
+           ^^^^^^^^^^^^^^^^
+  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
+    return self._loop.run_until_complete(task)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/server.py", line 69, in serve
+    await self._serve(sockets)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/server.py", line 76, in _serve
+    config.load()
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/config.py", line 434, in load
+    self.loaded_app = import_from_string(self.app)
+                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/importer.py", line 19, in import_from_string
+    module = importlib.import_module(module_str)
+             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
+    return _bootstrap._gcd_import(name[level:], package, level)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
+  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
+  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
+  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
+  File "<frozen importlib._bootstrap_external>", line 991, in exec_module
+  File "<frozen importlib._bootstrap_external>", line 1129, in get_code
+  File "<frozen importlib._bootstrap_external>", line 1059, in source_to_code
+  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
+  File "/home/ubuntu/apps/search_tools/app/main.py", line 158
+    return BatchSearchResponse(
+    ^^^^^^
+IndentationError: expected an indented block after 'finally' statement on line 156
+WARNING:  WatchFiles detected changes in 'parse_results.py'. Reloading...
+Process SpawnProcess-15:
+Traceback (most recent call last):
+  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
+    self.run()
+  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
+    self._target(*self._args, **self._kwargs)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/_subprocess.py", line 80, in subprocess_started
+    target(sockets=sockets)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/server.py", line 65, in run
+    return asyncio.run(self.serve(sockets=sockets))
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
+    return runner.run(main)
+           ^^^^^^^^^^^^^^^^
+  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
+    return self._loop.run_until_complete(task)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/server.py", line 69, in serve
+    await self._serve(sockets)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/server.py", line 76, in _serve
+    config.load()
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/config.py", line 434, in load
+    self.loaded_app = import_from_string(self.app)
+                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/importer.py", line 19, in import_from_string
+    module = importlib.import_module(module_str)
+             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
+    return _bootstrap._gcd_import(name[level:], package, level)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
+  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
+  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
+  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
+  File "<frozen importlib._bootstrap_external>", line 991, in exec_module
+  File "<frozen importlib._bootstrap_external>", line 1129, in get_code
+  File "<frozen importlib._bootstrap_external>", line 1059, in source_to_code
+  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
+  File "/home/ubuntu/apps/search_tools/app/main.py", line 158
+    return BatchSearchResponse(
+    ^^^^^^
+IndentationError: expected an indented block after 'finally' statement on line 156
+WARNING:  WatchFiles detected changes in 'parse_results.py'. Reloading...
+Process SpawnProcess-16:
+Traceback (most recent call last):
+  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
+    self.run()
+  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
+    self._target(*self._args, **self._kwargs)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/_subprocess.py", line 80, in subprocess_started
+    target(sockets=sockets)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/server.py", line 65, in run
+    return asyncio.run(self.serve(sockets=sockets))
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
+    return runner.run(main)
+           ^^^^^^^^^^^^^^^^
+  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
+    return self._loop.run_until_complete(task)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/server.py", line 69, in serve
+    await self._serve(sockets)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/server.py", line 76, in _serve
+    config.load()
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/config.py", line 434, in load
+    self.loaded_app = import_from_string(self.app)
+                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/importer.py", line 19, in import_from_string
+    module = importlib.import_module(module_str)
+             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
+    return _bootstrap._gcd_import(name[level:], package, level)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
+  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
+  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
+  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
+  File "<frozen importlib._bootstrap_external>", line 991, in exec_module
+  File "<frozen importlib._bootstrap_external>", line 1129, in get_code
+  File "<frozen importlib._bootstrap_external>", line 1059, in source_to_code
+  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
+  File "/home/ubuntu/apps/search_tools/app/main.py", line 158
+    return BatchSearchResponse(
+    ^^^^^^
+IndentationError: expected an indented block after 'finally' statement on line 156
+WARNING:  WatchFiles detected changes in 'search_ui.py'. Reloading...
+Process SpawnProcess-17:
+Traceback (most recent call last):
+  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
+    self.run()
+  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
+    self._target(*self._args, **self._kwargs)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/_subprocess.py", line 80, in subprocess_started
+    target(sockets=sockets)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/server.py", line 65, in run
+    return asyncio.run(self.serve(sockets=sockets))
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
+    return runner.run(main)
+           ^^^^^^^^^^^^^^^^
+  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
+    return self._loop.run_until_complete(task)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/server.py", line 69, in serve
+    await self._serve(sockets)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/server.py", line 76, in _serve
+    config.load()
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/config.py", line 434, in load
+    self.loaded_app = import_from_string(self.app)
+                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/importer.py", line 19, in import_from_string
+    module = importlib.import_module(module_str)
+             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
+    return _bootstrap._gcd_import(name[level:], package, level)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
+  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
+  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
+  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
+  File "<frozen importlib._bootstrap_external>", line 991, in exec_module
+  File "<frozen importlib._bootstrap_external>", line 1129, in get_code
+  File "<frozen importlib._bootstrap_external>", line 1059, in source_to_code
+  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
+  File "/home/ubuntu/apps/search_tools/app/main.py", line 158
+    return BatchSearchResponse(
+    ^^^^^^
+IndentationError: expected an indented block after 'finally' statement on line 156
+WARNING:  WatchFiles detected changes in 'search_ui.py'. Reloading...
+Process SpawnProcess-18:
+Traceback (most recent call last):
+  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
+    self.run()
+  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
+    self._target(*self._args, **self._kwargs)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/_subprocess.py", line 80, in subprocess_started
+    target(sockets=sockets)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/server.py", line 65, in run
+    return asyncio.run(self.serve(sockets=sockets))
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
+    return runner.run(main)
+           ^^^^^^^^^^^^^^^^
+  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
+    return self._loop.run_until_complete(task)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/server.py", line 69, in serve
+    await self._serve(sockets)
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/server.py", line 76, in _serve
+    config.load()
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/config.py", line 434, in load
+    self.loaded_app = import_from_string(self.app)
+                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/apps/search_tools/search_tools/lib/python3.12/site-packages/uvicorn/importer.py", line 19, in import_from_string
+    module = importlib.import_module(module_str)
+             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
+    return _bootstrap._gcd_import(name[level:], package, level)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
+  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
+  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
+  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
+  File "<frozen importlib._bootstrap_external>", line 991, in exec_module
+  File "<frozen importlib._bootstrap_external>", line 1129, in get_code
+  File "<frozen importlib._bootstrap_external>", line 1059, in source_to_code
+  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
+  File "/home/ubuntu/apps/search_tools/app/main.py", line 158
+    return BatchSearchResponse(
+    ^^^^^^
+IndentationError: expected an indented block after 'finally' statement on line 156

testapi.py

@@ -0,0 +1,22 @@
+#from serpapi import GoogleSearch
+import serpapi
+params = {
+  "engine": "google",
+  "q": "小红书化妆品市场趋势2024",
+#   "location": "Shanghai, Shanghai, China",
+  # "hl": "en",
+  # "gl": "us",
+  "api_key": "036f27edccf81fa40e05cd2175ad730a9c8a44fe9f3e136acd9c67199ce7d272"
+}
+
+search = serpapi.search(params)
+for key in search.keys():
+    print(key)
+search_metadata = search['search_metadata']
+print(search_metadata)
+if search_metadata['status'] == 'Success':
+  json_record = search_metadata['json_endpoint']
+
+# print(search.keys())
+# results = search.get_dict()
+# categories = results["categories"]

util_old.py

@@ -0,0 +1,249 @@
+from bs4 import BeautifulSoup  
+import os
+import gc
+import requests
+from langdetect import detect, LangDetectException
+from typing import AsyncIterator, Iterator
+ 
+from langchain_core.document_loaders import BaseLoader
+from langchain_core.documents import Document
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+import requests
+
+def fetch_documents(ip_address, dataset_id, api_key):
+    url = f'http://{ip_address}/v1/datasets/{dataset_id}/documents'
+    headers = {
+        'Authorization': f'Bearer {api_key}'
+    }
+    
+    try:
+        response = requests.get(url, headers=headers)
+        response.raise_for_status()  # Raise HTTPError for bad responses (4xx and 5xx)
+        
+        return response.json()  # Assuming the response is in JSON format
+    
+    except requests.exceptions.HTTPError as http_err:
+        print(f'HTTP error occurred: {http_err}')
+    except Exception as err:
+        print(f'Other error occurred: {err}')
+    return None
+# 文件扩展名集合
+file_extensions = {'.pdf', '.doc', '.docx', '.xls', '.xlsx', '.ppt', '.pptx'}
+
+def is_file(url):
+    return any(url.lower().endswith(ext) for ext in file_extensions)
+
+def is_webpage(url):
+    try:
+        response = requests.get(url, timeout=5)
+        content_type = response.headers.get('Content-Type', '')
+        return 'text/html' in content_type
+    except requests.RequestException:
+        return False
+
+def can_crawl_webpage(url):
+    try:
+        response = requests.get(url, timeout=5)
+        soup = BeautifulSoup(response.text, 'html.parser')
+        
+        # 检查是否有大量JavaScript代码
+        scripts = soup.find_all('script')
+        if len(scripts) > 10:
+            return False
+        
+        # 检查页面内容是否主要是乱码或图片
+        text_content = ' '.join([t.get_text() for t in soup.find_all('p')])
+        try:
+            detected_language = detect(text_content)
+            if detected_language != 'zh-cn':
+                return False
+        except LangDetectException:
+            return False
+        
+        images = soup.find_all('img')
+        if len(images) > len(soup.find_all('p')):
+            return False
+        
+        # Example criteria: Check if there is a meta robots tag with noindex or nofollow
+        meta_robots = soup.find('meta', attrs={'name': 'robots'})
+        if meta_robots and ('noindex' in meta_robots['content'] or 'nofollow' in meta_robots['content']):
+            return False
+        
+        return True
+    except requests.RequestException:
+        return False
+
+def is_cosmetic_related(snippet):
+    keywords = ['化妆品', '美妆', '彩妆', '护肤']
+    return any(keyword in snippet for keyword in keywords)
+
+def parse_search(result):
+    ret_str = ""
+    link = result.get('link')
+    #snippet = result.get('snippet')
+    if is_file(link):
+        ret_str = (f"File: {link}")
+    elif is_webpage(link):
+        if can_crawl_webpage(link):# and is_cosmetic_related(snippet):
+            ret_str = (f"Webpage crawlable: {link}")
+        else:
+            ret_str = (f"Webpage not crawlable: {link}")
+    else:
+        ret_str = (f"Unknown type: {link}")
+    return ret_str
+
+async def save_webpage_content(url, engine, search_id, output_dir, pool):
+    try:
+        # 检查网页是否已经保存
+        async with pool.acquire() as conn:
+            async with conn.cursor() as cursor:
+                sql = "SELECT COUNT(*) FROM saved_webpages WHERE url = %s"
+                await cursor.execute(sql, (url,))
+                count = await cursor.fetchone()
+                if count[0] > 0:
+                    print(f"Webpage already saved: {url}")
+                    return
+
+        response = requests.get(url, timeout=10)
+        response.raise_for_status()
+        
+        # 使用 BeautifulSoup 解析网页内容
+        soup = BeautifulSoup(response.text, 'html.parser')
+        
+        # 提取有用的文字内容
+        text_content = ' '.join([t.get_text() for t in soup.find_all('p')])
+
+        # 保存网页内容
+        filename = os.path.join(output_dir, f"{url.split('//')[-1].replace('/', '_')}.html")
+        with open(filename, 'w', encoding='utf-8') as file:
+            file.write(response.text)
+
+        # 保存提取的文字内容
+        text_filename = os.path.join(output_dir, f"{url.split('//')[-1].replace('/', '_')}_text.txt")
+        with open(text_filename, 'w', encoding='utf-8') as file:
+            file.write(text_content)
+
+        # 将保存的网页记录到数据库
+        async with pool.acquire() as conn:
+            async with conn.cursor() as cursor:
+                sql = "INSERT INTO saved_webpages (url, engine, search_id, content) VALUES (%s, %s, %s, %s)"
+                await cursor.execute(sql, (url, engine, search_id,text_content,))
+                await conn.commit()
+
+        print(f"Saved webpage: {filename}")
+    except Exception as e:
+        print(f"Failed to save webpage {url}: {e}")
+
+async def download_file(url, engine, search_id, output_dir, pool):
+    try:
+        # 检查文件是否已经下载
+        async with pool.acquire() as conn:
+            async with conn.cursor() as cursor:
+                sql = "SELECT COUNT(*) FROM downloaded_files WHERE url = %s"
+                await cursor.execute(sql, (url,))
+                count = await cursor.fetchone()
+                if count[0] > 0:
+                    print(f"File already downloaded: {url}")
+                    return
+
+        response = requests.get(url, timeout=10, stream=True)
+        response.raise_for_status()
+        local_filename = url.split('/')[-1]
+        local_filepath = os.path.join(output_dir, local_filename)
+        with open(local_filepath, 'wb') as file:
+            for chunk in response.iter_content(chunk_size=8192):
+                file.write(chunk)
+
+        # 将下载的文件记录到数据库
+        async with pool.acquire() as conn:
+            async with conn.cursor() as cursor:
+                sql = "INSERT INTO downloaded_files (url, engine, search_id ) VALUES (%s,%s,%s)"
+                await cursor.execute(sql, (url,engine, search_id,))
+                await conn.commit()
+
+        print(f"Downloaded file: {local_filepath}")
+    except Exception as e:
+        print(f"Failed to download file {url}: {e}")
+
+async def save_non_crawlable(url,engine, search_id, pool):
+    async with pool.acquire() as conn:
+        async with conn.cursor() as cursor:
+            # 检查黑名单是否已经存在该URL
+            sql = "SELECT COUNT(*) FROM non_crawlable_links WHERE url = %s"
+            await cursor.execute(sql, (url,))
+            count = await cursor.fetchone()
+            if count[0] == 0:
+                sql = "INSERT INTO non_crawlable_links (url,engine, search_id) VALUES (%s,%s,%s)"
+                await cursor.execute(sql, (url,engine, search_id,))
+                await conn.commit()
+                print(f"Saved non-crawlable link to database: {url}")
+            else:
+                print(f"Non-crawlable link already exists in database: {url}")
+
+async def get_blacklist(pool):
+    blacklist = set()
+    async with pool.acquire() as conn:
+        async with conn.cursor() as cursor:
+            sql = "SELECT url FROM non_crawlable_links"
+            await cursor.execute(sql)
+            results = await cursor.fetchall()
+            blacklist = {result[0] for result in results}
+    return blacklist
+
+def libreoffice_to_pdf(file_path, output_dir):
+    if not os.path.exists(file_path):
+        print(f"Error: The file '{file_path}' does not exist.")
+        return False
+
+    # 创建输出目录
+    if not os.path.exists(output_dir):
+        os.makedirs(output_dir)
+
+    # 使用LibreOffice将文件转换为PDF
+    pdf_path = os.path.join(output_dir, os.path.splitext(os.path.basename(file_path))[0] + '.pdf')
+    command = f'soffice --headless --convert-to pdf --outdir "{output_dir}" "{file_path}"'
+    os.system(command)
+
+    if not os.path.exists(pdf_path):
+        print(f"Error: Failed to convert '{file_path}' to PDF.")
+        return False
+
+    return pdf_path
+
+def pdf_to_images(pdf_path, output_dir):
+    # 将PDF文件逐页转换为图像
+    page_num = 1
+    for page in convert_from_path(pdf_path, fmt='png', single_file=False):
+        output_slide_path = os.path.join(output_dir, f'slide_{page_num}.png')
+        page.save(output_slide_path, 'PNG')
+        print(f'Saved slide {page_num} as image at {output_slide_path}')
+        page_num += 1
+
+        # 强制清理每页处理后的内存
+        del page
+        gc.collect()
+
+def process_files(base_dir):
+    for root, dirs, files in os.walk(base_dir):
+        for file in files:
+            if file.endswith((".ppt", ".pptx", ".xls", ".xlsx", ".doc", ".docx")):
+                file_path = os.path.join(root, file)
+                output_dir = os.path.join(root, os.path.splitext(file)[0])
+                if not os.path.exists(output_dir):
+                    os.makedirs(output_dir)
+                
+                pdf_path = libreoffice_to_pdf(file_path, output_dir)
+                if pdf_path:
+                    pdf_to_images(pdf_path, output_dir) 
+                    gc.collect()
+            elif file.endswith(".pdf"):
+                pdf_path = os.path.join(root, file)
+                output_dir = os.path.join(root, os.path.splitext(file)[0])
+                if not os.path.exists(output_dir):
+                    os.makedirs(output_dir)
+                
+                pdf_to_images(pdf_path, output_dir)
+
+                # 清理内存
+                gc.collect()
+

utils.py

@@ -0,0 +1,48 @@
+from bs4 import BeautifulSoup  
+
+# 文件扩展名集合
+file_extensions = {'.pdf', '.doc', '.docx', '.xls', '.xlsx', '.ppt', '.pptx'}
+
+def is_file(url):
+    return any(url.lower().endswith(ext) for ext in file_extensions)
+
+def is_webpage(url):
+    try:
+        response = requests.get(url, timeout=5)
+        content_type = response.headers.get('Content-Type', '')
+        return 'text/html' in content_type
+    except requests.RequestException:
+        return False
+
+def can_crawl_webpage(url):
+    try:
+        response = requests.get(url, timeout=5)
+        soup = BeautifulSoup(response.text, 'html.parser')
+        # Example criteria: Check if there is a meta robots tag with noindex or nofollow
+        meta_robots = soup.find('meta', attrs={'name': 'robots'})
+        if meta_robots and ('noindex' in meta_robots['content'] or 'nofollow' in meta_robots['content']):
+            return False
+        return True
+    except requests.RequestException:
+        return False
+
+def is_cosmetic_related(snippet): 
+    if snippet is None:
+        return False
+    keywords = ['化妆品', '美妆', '彩妆', '护肤']
+    return any(keyword in snippet for keyword in keywords)
+
+
+def parse_search(result):
+    ret_str = ""
+    link = result.get('link') 
+    if is_file(link):
+        ret_str = (f"File: {link}")
+    elif is_webpage(link):
+        if can_crawl_webpage(link):
+            ret_str = (f"Webpage crawlable: {link}")
+        else:
+            ret_str = (f"Webpage not crawlable: {link}")
+    else:
+        ret_str = (f"Unknown type: {link}")
+    return ret_str