0.6.33 (#18118)

* feat: improve ollama model management experience This commit introduces several improvements to the Ollama model management modal: - Adds a cancel button to the model pulling operation, using the existing 'x' button pattern. - Adds a cancel button to the "Update All" models operation, allowing the user to cancel the update for the currently processing model. - Cleans up toast notifications when updating all models. A single toast is now shown at the beginning and a summary toast at the end, preventing notification spam. - Refactors the `ManageOllama.svelte` component to support these new cancellation features. - Adds tooltips to all buttons in the modal to improve clarity. - Disables buttons when their corresponding input fields are empty to prevent accidental clicks. * fix * i18n: improve Chinese translation * fix: handle non‑UTF8 chars in third‑party responses without error * German translation of new strings in i18n * log web search queries only with level 'debug' instead of 'info' * Tool calls now only include text and dont inlcude other content like image b64 * fix onedrive * fix: discovery url * fix: default permissions not being loaded * fix: ai hallucination * fix: non rich text input copy * refac: rm print statements * refac: disable direct models from model editors * refac/fix: do not process xlsx files with azure doc intelligence * Update pull_request_template.md * Update generated image translation in DE-de * added missing danish translations * feat(onedrive): Enable search and "My Organization" pivot * style(onedrive): Formatting fix * feat: Implement toggling for vertical and horizontal flow layouts This commit introduces the necessary logic and UI controls to allow users to switch the Flow component layout between vertical and horizontal orientations. * **`Flow.svelte` Refactoring:** * Updates logic for calculating level offsets and node positions to consistently respect the current flow orientation. * Adds a control panel using `<Controls>` and `<SwitchButton>` components. * Provides user interface elements to easily switch the flow layout between horizontal and vertical orientations. * build(deps): bump pydantic from 2.11.7 to 2.11.9 in /backend Bumps [pydantic](https://github.com/pydantic/pydantic) from 2.11.7 to 2.11.9. - [Release notes](https://github.com/pydantic/pydantic/releases) - [Changelog](https://github.com/pydantic/pydantic/blob/v2.11.9/HISTORY.md) - [Commits](https://github.com/pydantic/pydantic/compare/v2.11.7...v2.11.9) --- updated-dependencies: - dependency-name: pydantic dependency-version: 2.11.9 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): bump black from 25.1.0 to 25.9.0 in /backend Bumps [black](https://github.com/psf/black) from 25.1.0 to 25.9.0. - [Release notes](https://github.com/psf/black/releases) - [Changelog](https://github.com/psf/black/blob/main/CHANGES.md) - [Commits](https://github.com/psf/black/compare/25.1.0...25.9.0) --- updated-dependencies: - dependency-name: black dependency-version: 25.9.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): bump markdown from 3.8.2 to 3.9 in /backend Bumps [markdown](https://github.com/Python-Markdown/markdown) from 3.8.2 to 3.9. - [Release notes](https://github.com/Python-Markdown/markdown/releases) - [Changelog](https://github.com/Python-Markdown/markdown/blob/master/docs/changelog.md) - [Commits](https://github.com/Python-Markdown/markdown/compare/3.8.2...3.9.0) --- updated-dependencies: - dependency-name: markdown dependency-version: '3.9' dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): bump chromadb from 1.0.20 to 1.1.0 in /backend Bumps [chromadb](https://github.com/chroma-core/chroma) from 1.0.20 to 1.1.0. - [Release notes](https://github.com/chroma-core/chroma/releases) - [Changelog](https://github.com/chroma-core/chroma/blob/main/RELEASE_PROCESS.md) - [Commits](https://github.com/chroma-core/chroma/compare/1.0.20...1.1.0) --- updated-dependencies: - dependency-name: chromadb dependency-version: 1.1.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): bump opentelemetry-api from 1.36.0 to 1.37.0 Bumps [opentelemetry-api](https://github.com/open-telemetry/opentelemetry-python) from 1.36.0 to 1.37.0. - [Release notes](https://github.com/open-telemetry/opentelemetry-python/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-python/blob/main/CHANGELOG.md) - [Commits](https://github.com/open-telemetry/opentelemetry-python/compare/v1.36.0...v1.37.0) --- updated-dependencies: - dependency-name: opentelemetry-api dependency-version: 1.37.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * refac: ollama embed form data * fix: non rich text handling * fix: oauth client registration * refac * chore: dep bump * chore: fastapi bump * chore/refac: bump bcrypt and remove passlib * Improving Korean Translation * refac * Improving Korean Translation * feat: PWA share_target implementation Co-Authored-By: gjveld <19951982+gjveld@users.noreply.github.com> * refac: message input mobile detection behaviour * feat: model_ids per folder * Update translation.json (pt-BR) inclusion of new translations of items that have been added * refac * refac * refac * refac * refac/fix: temp chat * refac * refac: stop task * refac/fix: azure audio escape * refac: external tool validation * refac/enh: start.sh additional args support * refac * refac: styling * refac/fix: direct connection floating action buttons * refac/fix: system prompt duplication * refac/enh: openai tts additional params support * refac * feat: load data in parallel to accelerate page loading speed * i18n: improve Chinese translation * refac * refac: model selector * UPD: i18n es-ES Translation v0.6.33 UPD: i18n es-ES Translation v0.6.33 Updated new strings. * refac * improved query pref by querying only relevant columns * refac/enh: docling params * refac * refac: openai additional headers support * refac * FEAT: Add Vega Char Visualizer Renderer ### FEAT: Add Vega Char Visualizer Renderer Feature required in https://github.com/open-webui/open-webui/discussions/18022 Added npm vega lib to package.json Added function for visualization renderer to src/libs/utils/index.ts Added logic to src/lib/components/chat/Messages/CodeBlock.svelte The treatment is similar as for mermaid diagrams. Reference: https://vega.github.io/vega/ * refac * chore * refac * FEAT: Add Vega-Lite Char Visualizer Renderer ### FEAT: Add Vega Char Visualizer Renderer Add suport for Vega-Lite Specifications. Vega-Lite is a "compiled" version of Vega Char Visualizer. For be rendered with Vega it have to be compiled. This PR add the check and compile if necessary, is a complement of recent Vega Renderer Feature added. * refac * refac/fix: switch * enh/refac: url input handling * refac * refac: styling * UPD: Add Validators & Error Toast for Mermaid & Vega diagrams ### UPD: Feat: Add Validators & Error Toast for Mermaid & Vega diagrams Description: As many time the diagrams generated or entered have syntax errors the diagrams are not rendered due to that errors, but as there isn't any notification is difficult to know what happend. This PR add validator and toast notification when error on Mermaid and Vega/Vega-Lite diagrams, helping the user to fix its. * removed redundant knowledge API call * Fix Code Format * refac: model workspace view * refac * refac: knowledge * refac: prompts * refac: tools * refac * feat: attach folder * refac: make tencentcloud-sdk-python optional * refac/fix: oauth * enh: ENABLE_OAUTH_EMAIL_FALLBACK * refac/fix: folders * Update requirements.txt * Update pyproject.toml * UPD: Add Validators & Error Toast for Mermaid & Vega diagrams ### UPD: Feat: Add Validators & Error Toast for Mermaid & Vega diagrams Description: As many time the diagrams generated or entered have syntax errors the diagrams are not rendered due to that errors, but as there isn't any notification is difficult to know what happend. This PR add validator and toast notification when error on Mermaid and Vega/Vega-Lite diagrams, helping the user to fix its. Note: Another possibility of integrating this Graph Visualizer is through its svelte component: https://github.com/vega/svelte-vega/tree/main/packages/svelte-vega * Removed unused toast import & Code Format * refac * refac: external tool server view * refac * refac: overview * refac: styling * refac * Update bug_report.yaml * refac * refac * refac * refac * refac: oauth client fallback * Fixed: Cannot handle batch sizes > 1 if no padding token is defined Fixes Cannot handle batch sizes > 1 if no padding token is defined For reranker models that do not have this defined in their config by using the eos_token_id if present as pad_token_id. * refac: fallback to reasoning content * fix(i18n): corrected typo in Spanish translation for "Reasoning Tags" Typo fixed in Spanish translation file at line 1240 of `open-webui/src/lib/i18n/locales/es-ES/translation.json`: - Incorrect: "Eriquetas de Razonamiento" - Correct: "Etiquetas de Razonamiento" This improves clarity and consistency in the UI. * refac/fix: ENABLE_STAR_SESSIONS_MIDDLEWARE * refac/fix: redirect * refac * refac * refac * refac: web search error handling * refac: source parsing * refac: functions * refac * refac/enh: note pdf export * refac/fix: mcp oauth2.1 * chore: format * chore: Changelog (#17995) * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * refac * chore: dep bump --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: silentoplayz <jacwoo21@outlook.com> Co-authored-by: Shirasawa <764798966@qq.com> Co-authored-by: Jan Kessler <jakessle@uni-mainz.de> Co-authored-by: Jacob Leksan <jacob.leksan@expedient.com> Co-authored-by: Classic298 <27028174+Classic298@users.noreply.github.com> Co-authored-by: sinejespersen <sinejespersen@protonmail.com> Co-authored-by: Selene Blok <selene.blok@rws.nl> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Cyp <cypher9715@naver.com> Co-authored-by: gjveld <19951982+gjveld@users.noreply.github.com> Co-authored-by: joaoback <156559121+joaoback@users.noreply.github.com> Co-authored-by: _00_ <131402327+rgaricano@users.noreply.github.com> Co-authored-by: expruc <eygabi01@gmail.com> Co-authored-by: YetheSamartaka <55753928+YetheSamartaka@users.noreply.github.com> Co-authored-by: Akutangulo <akutangulo@gmail.com>
2025-10-07 16:20:27 -05:00
parent 4d7fddaf7e
commit 8d7d79d54b
156 changed files with 6119 additions and 3198 deletions
@@ -3,6 +3,7 @@ import json
 import logging
 import os
 import uuid
+import html
 from functools import lru_cache
 from pydub import AudioSegment
 from pydub.silence import split_on_silence
@@ -153,6 +154,7 @@ def set_faster_whisper_model(model: str, auto_update: bool = False):
 class TTSConfigForm(BaseModel):
    OPENAI_API_BASE_URL: str
    OPENAI_API_KEY: str
+    OPENAI_PARAMS: Optional[dict] = None
    API_KEY: str
    ENGINE: str
    MODEL: str
@@ -189,6 +191,7 @@ async def get_audio_config(request: Request, user=Depends(get_admin_user)):
        "tts": {
            "OPENAI_API_BASE_URL": request.app.state.config.TTS_OPENAI_API_BASE_URL,
            "OPENAI_API_KEY": request.app.state.config.TTS_OPENAI_API_KEY,
+            "OPENAI_PARAMS": request.app.state.config.TTS_OPENAI_PARAMS,
            "API_KEY": request.app.state.config.TTS_API_KEY,
            "ENGINE": request.app.state.config.TTS_ENGINE,
            "MODEL": request.app.state.config.TTS_MODEL,
@@ -221,6 +224,7 @@ async def update_audio_config(
 ):
    request.app.state.config.TTS_OPENAI_API_BASE_URL = form_data.tts.OPENAI_API_BASE_URL
    request.app.state.config.TTS_OPENAI_API_KEY = form_data.tts.OPENAI_API_KEY
+    request.app.state.config.TTS_OPENAI_PARAMS = form_data.tts.OPENAI_PARAMS
    request.app.state.config.TTS_API_KEY = form_data.tts.API_KEY
    request.app.state.config.TTS_ENGINE = form_data.tts.ENGINE
    request.app.state.config.TTS_MODEL = form_data.tts.MODEL
@@ -261,12 +265,13 @@ async def update_audio_config(

    return {
        "tts": {
-            "OPENAI_API_BASE_URL": request.app.state.config.TTS_OPENAI_API_BASE_URL,
-            "OPENAI_API_KEY": request.app.state.config.TTS_OPENAI_API_KEY,
-            "API_KEY": request.app.state.config.TTS_API_KEY,
            "ENGINE": request.app.state.config.TTS_ENGINE,
            "MODEL": request.app.state.config.TTS_MODEL,
            "VOICE": request.app.state.config.TTS_VOICE,
+            "OPENAI_API_BASE_URL": request.app.state.config.TTS_OPENAI_API_BASE_URL,
+            "OPENAI_API_KEY": request.app.state.config.TTS_OPENAI_API_KEY,
+            "OPENAI_PARAMS": request.app.state.config.TTS_OPENAI_PARAMS,
+            "API_KEY": request.app.state.config.TTS_API_KEY,
            "SPLIT_ON": request.app.state.config.TTS_SPLIT_ON,
            "AZURE_SPEECH_REGION": request.app.state.config.TTS_AZURE_SPEECH_REGION,
            "AZURE_SPEECH_BASE_URL": request.app.state.config.TTS_AZURE_SPEECH_BASE_URL,
@@ -336,6 +341,11 @@ async def speech(request: Request, user=Depends(get_verified_user)):
            async with aiohttp.ClientSession(
                timeout=timeout, trust_env=True
            ) as session:
+                payload = {
+                    **payload,
+                    **(request.app.state.config.TTS_OPENAI_PARAMS or {}),
+                }
+
                r = await session.post(
                    url=f"{request.app.state.config.TTS_OPENAI_API_BASE_URL}/audio/speech",
                    json=payload,
@@ -458,7 +468,7 @@ async def speech(request: Request, user=Depends(get_verified_user)):

        try:
            data = f"""<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis" xml:lang="{locale}">
-                <voice name="{language}">{payload["input"]}</voice>
+                <voice name="{language}">{html.escape(payload["input"])}</voice>
            </speak>"""
            timeout = aiohttp.ClientTimeout(total=AIOHTTP_CLIENT_TIMEOUT)
            async with aiohttp.ClientSession(
@@ -340,11 +340,12 @@ async def model_response_handler(request, channel, message, user):
                        if file.get("type", "") == "image":
                            images.append(file.get("url", ""))

+                thread_history_string = "\n\n".join(thread_history)
                system_message = {
                    "role": "system",
                    "content": f"You are {model.get('name', model_id)}, participating in a threaded conversation. Be concise and conversational."
                    + (
-                        f"Here's the thread history:\n\n{''.join([f'{msg}' for msg in thread_history])}\n\nContinue the conversation naturally as {model.get('name', model_id)}, addressing the most recent message while being aware of the full context."
+                        f"Here's the thread history:\n\n\n{thread_history_string}\n\n\nContinue the conversation naturally as {model.get('name', model_id)}, addressing the most recent message while being aware of the full context."
                        if thread_history
                        else ""
                    ),
@@ -384,19 +385,34 @@ async def model_response_handler(request, channel, message, user):
                )

                if res:
-                    await update_message_by_id(
-                        channel.id,
-                        response_message.id,
-                        MessageForm(
-                            **{
-                                "content": res["choices"][0]["message"]["content"],
-                                "meta": {
-                                    "done": True,
-                                },
-                            }
-                        ),
-                        user,
-                    )
+                    if res.get("choices", []) and len(res["choices"]) > 0:
+                        await update_message_by_id(
+                            channel.id,
+                            response_message.id,
+                            MessageForm(
+                                **{
+                                    "content": res["choices"][0]["message"]["content"],
+                                    "meta": {
+                                        "done": True,
+                                    },
+                                }
+                            ),
+                            user,
+                        )
+                    elif res.get("error", None):
+                        await update_message_by_id(
+                            channel.id,
+                            response_message.id,
+                            MessageForm(
+                                **{
+                                    "content": f"Error: {res['error']}",
+                                    "meta": {
+                                        "done": True,
+                                    },
+                                }
+                            ),
+                            user,
+                        )
            except Exception as e:
                log.info(e)
                pass
@@ -436,7 +452,7 @@ async def new_message_handler(
            }

            await sio.emit(
-                "channel-events",
+                "events:channel",
                event_data,
                to=f"channel:{channel.id}",
            )
@@ -447,7 +463,7 @@ async def new_message_handler(

                if parent_message:
                    await sio.emit(
-                        "channel-events",
+                        "events:channel",
                        {
                            "channel_id": channel.id,
                            "message_id": parent_message.id,
@@ -644,7 +660,7 @@ async def update_message_by_id(

        if message:
            await sio.emit(
-                "channel-events",
+                "events:channel",
                {
                    "channel_id": channel.id,
                    "message_id": message.id,
@@ -708,7 +724,7 @@ async def add_reaction_to_message(
        message = Messages.get_message_by_id(message_id)

        await sio.emit(
-            "channel-events",
+            "events:channel",
            {
                "channel_id": channel.id,
                "message_id": message.id,
@@ -774,7 +790,7 @@ async def remove_reaction_by_id_and_user_id_and_name(
        message = Messages.get_message_by_id(message_id)

        await sio.emit(
-            "channel-events",
+            "events:channel",
            {
                "channel_id": channel.id,
                "message_id": message.id,
@@ -839,7 +855,7 @@ async def delete_message_by_id(
    try:
        Messages.delete_message_by_id(message_id)
        await sio.emit(
-            "channel-events",
+            "events:channel",
            {
                "channel_id": channel.id,
                "message_id": message.id,
@@ -862,7 +878,7 @@ async def delete_message_by_id(

            if parent_message:
                await sio.emit(
-                    "channel-events",
+                    "events:channel",
                    {
                        "channel_id": channel.id,
                        "message_id": parent_message.id,
@@ -213,7 +213,7 @@ async def verify_tool_servers_config(
                    )
                    async with aiohttp.ClientSession() as session:
                        async with session.get(
-                            discovery_urls[0]
+                            discovery_url
                        ) as oauth_server_metadata_response:
                            if oauth_server_metadata_response.status == 200:
                                try:
@@ -234,7 +234,7 @@ async def verify_tool_servers_config(
                                    )
                                    raise HTTPException(
                                        status_code=400,
-                                        detail=f"Failed to parse OAuth 2.1 discovery document from {discovery_urls[0]}",
+                                        detail=f"Failed to parse OAuth 2.1 discovery document from {discovery_url}",
                                    )

                raise HTTPException(
@@ -50,7 +50,15 @@ async def get_folders(user=Depends(get_verified_user)):
    folders = Folders.get_folders_by_user_id(user.id)

    # Verify folder data integrity
+    folder_list = []
    for folder in folders:
+        if folder.parent_id and not Folders.get_folder_by_id_and_user_id(
+            folder.parent_id, user.id
+        ):
+            folder = Folders.update_folder_parent_id_by_id_and_user_id(
+                folder.id, user.id, None
+            )
+
        if folder.data:
            if "files" in folder.data:
                valid_files = []
@@ -74,12 +82,9 @@ async def get_folders(user=Depends(get_verified_user)):
                    folder.id, user.id, FolderUpdateForm(data=folder.data)
                )

-    return [
-        {
-            **folder.model_dump(),
-        }
-        for folder in folders
-    ]
+        folder_list.append(FolderNameIdResponse(**folder.model_dump()))
+
+    return folder_list


 ############################
@@ -265,21 +270,31 @@ async def delete_folder_by_id(
                detail=ERROR_MESSAGES.ACCESS_PROHIBITED,
            )

-    folder = Folders.get_folder_by_id_and_user_id(id, user.id)
-    if folder:
-        try:
-            folder_ids = Folders.delete_folder_by_id_and_user_id(id, user.id)
-            for folder_id in folder_ids:
-                Chats.delete_chats_by_user_id_and_folder_id(user.id, folder_id)
+    folders = []
+    folders.append(Folders.get_folder_by_id_and_user_id(id, user.id))
+    while folders:
+        folder = folders.pop()
+        if folder:
+            try:
+                folder_ids = Folders.delete_folder_by_id_and_user_id(id, user.id)
+                for folder_id in folder_ids:
+                    Chats.delete_chats_by_user_id_and_folder_id(user.id, folder_id)
+
+                return True
+            except Exception as e:
+                log.exception(e)
+                log.error(f"Error deleting folder: {id}")
+                raise HTTPException(
+                    status_code=status.HTTP_400_BAD_REQUEST,
+                    detail=ERROR_MESSAGES.DEFAULT("Error deleting folder"),
+                )
+            finally:
+                # Get all subfolders
+                subfolders = Folders.get_folders_by_parent_id_and_user_id(
+                    folder.id, user.id
+                )
+                folders.extend(subfolders)

-            return True
-        except Exception as e:
-            log.exception(e)
-            log.error(f"Error deleting folder: {id}")
-            raise HTTPException(
-                status_code=status.HTTP_400_BAD_REQUEST,
-                detail=ERROR_MESSAGES.DEFAULT("Error deleting folder"),
-            )
    else:
        raise HTTPException(
            status_code=status.HTTP_404_NOT_FOUND,
@@ -10,6 +10,7 @@ from open_webui.models.functions import (
    FunctionForm,
    FunctionModel,
    FunctionResponse,
+    FunctionUserResponse,
    FunctionWithValvesModel,
    Functions,
 )
@@ -42,6 +43,11 @@ async def get_functions(user=Depends(get_verified_user)):
    return Functions.get_functions()


+@router.get("/list", response_model=list[FunctionUserResponse])
+async def get_function_list(user=Depends(get_admin_user)):
+    return Functions.get_function_list()
+
+
 ############################
 # ExportFunctions
 ############################
@@ -1020,6 +1020,10 @@ class GenerateEmbedForm(BaseModel):
    options: Optional[dict] = None
    keep_alive: Optional[Union[int, str]] = None

+    model_config = ConfigDict(
+        extra="allow",
+    )
+

@router.post("/api/embed")
@router.post("/api/embed/{url_idx}")
@@ -190,6 +190,9 @@ async def get_headers_and_cookies(
    if token:
        headers["Authorization"] = f"Bearer {token}"

+    if config.get("headers") and isinstance(config.get("headers"), dict):
+        headers = {**headers, **config.get("headers")}
+
    return headers, cookies


@@ -5,6 +5,7 @@ import os
 import shutil
 import asyncio

+import re
 import uuid
 from datetime import datetime
 from pathlib import Path
@@ -70,6 +71,7 @@ from open_webui.retrieval.web.firecrawl import search_firecrawl
 from open_webui.retrieval.web.external import search_external

 from open_webui.retrieval.utils import (
+    get_content_from_url,
    get_embedding_function,
    get_reranking_function,
    get_model_path,
@@ -189,6 +191,26 @@ def get_rf(
                    log.error(f"CrossEncoder: {e}")
                    raise Exception(ERROR_MESSAGES.DEFAULT("CrossEncoder error"))

+                # Safely adjust pad_token_id if missing as some models do not have this in config
+                try:
+                    model_cfg = getattr(rf, "model", None)
+                    if model_cfg and hasattr(model_cfg, "config"):
+                        cfg = model_cfg.config
+                        if getattr(cfg, "pad_token_id", None) is None:
+                            # Fallback to eos_token_id when available
+                            eos = getattr(cfg, "eos_token_id", None)
+                            if eos is not None:
+                                cfg.pad_token_id = eos
+                                log.debug(
+                                    f"Missing pad_token_id detected; set to eos_token_id={eos}"
+                                )
+                            else:
+                                log.warning(
+                                    "Neither pad_token_id nor eos_token_id present in model config"
+                                )
+                except Exception as e2:
+                    log.warning(f"Failed to adjust pad_token_id on CrossEncoder: {e2}")
+
    return rf


@@ -429,6 +451,7 @@ async def get_rag_config(request: Request, user=Depends(get_admin_user)):
        "EXTERNAL_DOCUMENT_LOADER_API_KEY": request.app.state.config.EXTERNAL_DOCUMENT_LOADER_API_KEY,
        "TIKA_SERVER_URL": request.app.state.config.TIKA_SERVER_URL,
        "DOCLING_SERVER_URL": request.app.state.config.DOCLING_SERVER_URL,
+        "DOCLING_PARAMS": request.app.state.config.DOCLING_PARAMS,
        "DOCLING_DO_OCR": request.app.state.config.DOCLING_DO_OCR,
        "DOCLING_FORCE_OCR": request.app.state.config.DOCLING_FORCE_OCR,
        "DOCLING_OCR_ENGINE": request.app.state.config.DOCLING_OCR_ENGINE,
@@ -590,6 +613,7 @@ class ConfigForm(BaseModel):
    # Content extraction settings
    CONTENT_EXTRACTION_ENGINE: Optional[str] = None
    PDF_EXTRACT_IMAGES: Optional[bool] = None
+
    DATALAB_MARKER_API_KEY: Optional[str] = None
    DATALAB_MARKER_API_BASE_URL: Optional[str] = None
    DATALAB_MARKER_ADDITIONAL_CONFIG: Optional[str] = None
@@ -601,11 +625,13 @@ class ConfigForm(BaseModel):
    DATALAB_MARKER_FORMAT_LINES: Optional[bool] = None
    DATALAB_MARKER_USE_LLM: Optional[bool] = None
    DATALAB_MARKER_OUTPUT_FORMAT: Optional[str] = None
+
    EXTERNAL_DOCUMENT_LOADER_URL: Optional[str] = None
    EXTERNAL_DOCUMENT_LOADER_API_KEY: Optional[str] = None

    TIKA_SERVER_URL: Optional[str] = None
    DOCLING_SERVER_URL: Optional[str] = None
+    DOCLING_PARAMS: Optional[dict] = None
    DOCLING_DO_OCR: Optional[bool] = None
    DOCLING_FORCE_OCR: Optional[bool] = None
    DOCLING_OCR_ENGINE: Optional[str] = None
@@ -782,6 +808,11 @@ async def update_rag_config(
        if form_data.DOCLING_SERVER_URL is not None
        else request.app.state.config.DOCLING_SERVER_URL
    )
+    request.app.state.config.DOCLING_PARAMS = (
+        form_data.DOCLING_PARAMS
+        if form_data.DOCLING_PARAMS is not None
+        else request.app.state.config.DOCLING_PARAMS
+    )
    request.app.state.config.DOCLING_DO_OCR = (
        form_data.DOCLING_DO_OCR
        if form_data.DOCLING_DO_OCR is not None
@@ -1104,6 +1135,7 @@ async def update_rag_config(
        "EXTERNAL_DOCUMENT_LOADER_API_KEY": request.app.state.config.EXTERNAL_DOCUMENT_LOADER_API_KEY,
        "TIKA_SERVER_URL": request.app.state.config.TIKA_SERVER_URL,
        "DOCLING_SERVER_URL": request.app.state.config.DOCLING_SERVER_URL,
+        "DOCLING_PARAMS": request.app.state.config.DOCLING_PARAMS,
        "DOCLING_DO_OCR": request.app.state.config.DOCLING_DO_OCR,
        "DOCLING_FORCE_OCR": request.app.state.config.DOCLING_FORCE_OCR,
        "DOCLING_OCR_ENGINE": request.app.state.config.DOCLING_OCR_ENGINE,
@@ -1522,6 +1554,7 @@ def process_file(
                            "picture_description_mode": request.app.state.config.DOCLING_PICTURE_DESCRIPTION_MODE,
                            "picture_description_local": request.app.state.config.DOCLING_PICTURE_DESCRIPTION_LOCAL,
                            "picture_description_api": request.app.state.config.DOCLING_PICTURE_DESCRIPTION_API,
+                            **request.app.state.config.DOCLING_PARAMS,
                        },
                        PDF_EXTRACT_IMAGES=request.app.state.config.PDF_EXTRACT_IMAGES,
                        DOCUMENT_INTELLIGENCE_ENDPOINT=request.app.state.config.DOCUMENT_INTELLIGENCE_ENDPOINT,
@@ -1680,49 +1713,6 @@ def process_text(


@router.post("/process/youtube")
-def process_youtube_video(
-    request: Request, form_data: ProcessUrlForm, user=Depends(get_verified_user)
-):
-    try:
-        collection_name = form_data.collection_name
-        if not collection_name:
-            collection_name = calculate_sha256_string(form_data.url)[:63]
-
-        loader = YoutubeLoader(
-            form_data.url,
-            language=request.app.state.config.YOUTUBE_LOADER_LANGUAGE,
-            proxy_url=request.app.state.config.YOUTUBE_LOADER_PROXY_URL,
-        )
-
-        docs = loader.load()
-        content = " ".join([doc.page_content for doc in docs])
-        log.debug(f"text_content: {content}")
-
-        save_docs_to_vector_db(
-            request, docs, collection_name, overwrite=True, user=user
-        )
-
-        return {
-            "status": True,
-            "collection_name": collection_name,
-            "filename": form_data.url,
-            "file": {
-                "data": {
-                    "content": content,
-                },
-                "meta": {
-                    "name": form_data.url,
-                },
-            },
-        }
-    except Exception as e:
-        log.exception(e)
-        raise HTTPException(
-            status_code=status.HTTP_400_BAD_REQUEST,
-            detail=ERROR_MESSAGES.DEFAULT(e),
-        )
-
-
@router.post("/process/web")
 def process_web(
    request: Request, form_data: ProcessUrlForm, user=Depends(get_verified_user)
@@ -1732,19 +1722,16 @@ def process_web(
        if not collection_name:
            collection_name = calculate_sha256_string(form_data.url)[:63]

-        loader = get_web_loader(
-            form_data.url,
-            verify_ssl=request.app.state.config.ENABLE_WEB_LOADER_SSL_VERIFICATION,
-            requests_per_second=request.app.state.config.WEB_LOADER_CONCURRENT_REQUESTS,
-        )
-        docs = loader.load()
-        content = " ".join([doc.page_content for doc in docs])
-
+        content, docs = get_content_from_url(request, form_data.url)
        log.debug(f"text_content: {content}")

        if not request.app.state.config.BYPASS_WEB_SEARCH_EMBEDDING_AND_RETRIEVAL:
            save_docs_to_vector_db(
-                request, docs, collection_name, overwrite=True, user=user
+                request,
+                docs,
+                collection_name,
+                overwrite=True,
+                user=user,
            )
        else:
            collection_name = None
@@ -2047,7 +2034,7 @@ async def process_web_search(
    result_items = []

    try:
-        logging.info(
+        logging.debug(
            f"trying to web search with {request.app.state.config.WEB_SEARCH_ENGINE, form_data.queries}"
        )

@@ -2081,6 +2068,12 @@ async def process_web_search(
            detail=ERROR_MESSAGES.WEB_SEARCH_ERROR(e),
        )

+    if len(urls) == 0:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail=ERROR_MESSAGES.DEFAULT("No results found from web search"),
+        )
+
    try:
        if request.app.state.config.BYPASS_WEB_SEARCH_WEB_LOADER:
            search_results = [