1. Mcp
  2. Building Chatgpt Apps With Gradio

Building ChatGPT Apps with Gradio and Apps SDK

Apps in ChatGPT are a great way to let users try your machine learning models or other kinds of apps entirely by chatting in familiar chat application. OpenAI has released the Apps SDK for developers to build complete applications, but you can use Gradio to build ChatGPT apps very quickly, based off of your Gradio MCP server. We will also see how Gradio's built-in share links make it especially easy to iterate on your ChatGPT app!

Introduction

Building a ChatGPT app requires doing two things:

  • Building a Gradio MCP server with at least one tool exposed. If you're not already familiar with building a Gradio MCP server, we recommend reading this guide first.

  • Building a custom UI with HTML, JavaScript, and CSS that will be displayed when your tool is called, an exposing that as an MCP resource.

We will walk through the steps in more detail below.

Prerequisites

  • You will need to enable "developer mode" in ChatGPT under Settings → Apps & Connectors → Advanced settings in ChatGPT. This currently requires a paid ChatGPT account.
  • You need to have gradio>=6.0 installed with the mcp add-on:
pip install --upgrade gradio[mcp]

Now, let's walk through two examples of how you can build build ChatGPT apps with Gradio.

Example 1: Letter Counter App

The first example is an ChatGPT app that counts the occurrence of letters in a word and displays a card with the word and specified letters highlighted, like this:

So how do we build this? You can find the complete code for the letter counter app in a single file here, or follow the steps below:

  1. Start by writing your Python function. In our case, the function is simply a letter counter:
def letter_counter(word: str, letter: str) -> int:
    """
    Count the number of letters in a word or phrase.

    Parameters:
        word (str): The word or phrase to count the letters of.
        letter (str): The letter to count the occurrences of.
    """
    return word.count(letter)
  1. Then, wrap your Python function with a Gradio UI, something along these lines:
with gr.Blocks() as demo:
    with gr.Row():
        with gr.Column():
            word = gr.Textbox(label="Word")
            letter = gr.Textbox(label="Letter")
            btn = gr.Button("Count Letters")
        with gr.Column():
            count = gr.Number(label="Count")

    btn.click(letter_counter, inputs=[word, letter], outputs=count)
  1. Now, launch your Gradio app with the MCP server enabled, i.e. with mcp_server=True
    demo.launch(mcp_server=True)

As covered in earlier guides, you will now be able to test the tool using any MCP Client, such as the MCP Inspector tool. Test it and confirm that it behaves as you expect.

  1. Create a UI for your ChatGPT app and expose it as a resource. This part requires writing some frontend code and may be unfamiliar at first, but a few examples will help you create an app that works well for your use case. In our case, we'll create a card with HTML, Javascript, and CSS. Inside the card, we'll display the word presented by the user, highlighting each occurrence of the specified letter. Note that we access the user's tool input using window.openai?.toolInput?.word and window.openai?.toolInput?.letter. The window.openai object is automatically inserted by ChatGPT with the data from the user's tool call. This is what the complete function looks like:
@gr.mcp.resource("ui://widget/app.html", mime_type="text/html+skybridge")
def app_html():
    visual = """
    <div id="letter-card-container"></div>
    <script>
        const container = document.getElementById('letter-card-container');

        function render() {
            const word = window.openai?.toolInput?.word || "strawberry";
            const letter = window.openai?.toolInput?.letter || "r";

            let letterHTML = '';
            for (let i = 0; i < word.length; i++) {
                const char = word[i];
                const color = char.toLowerCase() === letter.toLowerCase() ? '#b8860b' : '#000000';
                letterHTML += `<span style="color: ${color};">${char}</span>`;
            }

            container.innerHTML = `
                <div style="
                    background: linear-gradient(135deg, #f5f5dc 0%, #e8e4d0 100%);
                    background-image:
                        repeating-linear-gradient(45deg, transparent, transparent 2px, rgba(139, 121, 94, 0.03) 2px, rgba(139, 121, 94, 0.03) 4px),
                        linear-gradient(135deg, #f5f5dc 0%, #e8e4d0 100%);
                    border-radius: 16px;
                    padding: 40px;
                    box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1), 0 1px 3px rgba(0, 0, 0, 0.08);
                    max-width: 600px;
                    margin: 20px auto;
                    font-family: 'Georgia', serif;
                    text-align: center;
                ">
                    <div style="
                        font-size: 48px;
                        font-weight: bold;
                        letter-spacing: 8px;
                        line-height: 1.5;
                    ">
                        ${letterHTML}
                    </div>
                </div>
            `;
        }
        render();
        window.addEventListener("openai:set_globals", (event) => {
            if (event.detail?.globals?.toolInput) {
                render();
            }
        }, { passive: true });
    </script>
    """
    return visual

Note that we've provided a URI for the gr.mcp.resource at ui://widget/app.html. This is arbitrary, but we'll need to use the same URI later on. We also need to specify the mimetype of the resource to be mime_type="text/html+skybridge". Finally, note that we attached an event listener in the JavaScript for "openai:set_globals", which is generally a good practice as it allows the widget to update whenever a new tool call is triggered.

  1. Create an event in your Gradio app corresponding to the resource function. This is necessary because your Gradio app only picks up MCP tools, resources, prompts, etc. if they are associated with a Gradio event. Typically, the convention is to simply display the code for your MCP resource in a gr.Code component, e.g. like this:
    html = gr.Code(language="html", max_lines=20)
    
    # ... the rest of your Gradio app

    btn.click(app_html, outputs=html)
  1. Add _meta attributes to your MCP tool. We need to connect the MCP tool that we created to the UI that we created for our app. We can do this by adding this decorator to our MCP tool function:
@gr.mcp.tool(
    _meta={
        "openai/outputTemplate": "ui://widget/app.html",
        "openai/resultCanProduceWidget": True,
        "openai/widgetAccessible": True,
    }
)

The key thing to observe is that the "openai/outputTemplate" must match the URI of the MCP resource that we created earlier.

  1. Relaunch your Gradio app with share=True. This will make it very easy to test within ChatGPT. Note the MCP server URL that is printed to your terminal, e.g. https://2e879c6066d729b11b.gradio.live/gradio_api/mcp/.
    demo.launch(share=True, mcp_server=True)

This will print a public URL that your Gradio app will be running on.

  1. Now, navigate to ChatGPT (https://chat.com/). As mentioned earlier, you need to enable "developer mode" in ChatGPT under Settings → Apps & Connectors → Advanced settings in ChatGPT. Then, navigate to Settings → Apps & Connectors and click the "Create" button. Give your connector a name, a description (optional), and paste in the MCP server URL that was printed to your terminal. Choose "No authentication" and create.

And that's it! Once the Connector has been created, you can start prompting it by saying something like, "Use @letter-counter to count the number of r's in Gradio."

Example 2: An Image Brightener

Next, let's see a more complex ChatGPT app for image enhancement. The ChatGPT app includes a "Brighten" button that lets the user call the tool directly from the app UI.

Here's the complete code for this app:

import gradio as gr
import tempfile
from PIL import Image
import numpy as np


@gr.mcp.tool(
    _meta={
        "openai/outputTemplate": "ui://widget/app.html",
        "openai/resultCanProduceWidget": True,
        "openai/widgetAccessible": True,
    }
)

def power_law_image(input_path: str, gamma: float = 0.5) -> str:
    """
    Applies a power-law (gamma) transformation to an image file and saves
    the result to a temporary file.

    Args:
        input_path (str): Path to the input image.
        gamma (float): Power-law exponent. <1 brightens, >1 darkens.

    Returns:
        str: Path to the saved temporary output image.
    """
    img = Image.open(input_path).convert("RGB")
    arr = np.array(img, dtype=np.float32) / 255.0
    arr = np.power(arr, gamma)
    arr = np.clip(arr * 255, 0, 255).astype(np.uint8)
    out_img = Image.fromarray(arr)

    tmp_file = tempfile.NamedTemporaryFile(delete=False, suffix=".png")
    out_img.save(tmp_file.name)
    tmp_file.close()

    return tmp_file.name


@gr.mcp.resource("ui://widget/app.html", mime_type="text/html+skybridge")
def app_html():
    visual = """
    <style>
        #image-container {
            position: relative;
            display: inline-block;
            max-width: 100%;
        }
        #image-display {
            max-width: 100%;
            height: auto;
            display: block;
            border-radius: 8px;
        }
        #brighten-btn {
            position: absolute;
            bottom: 16px;
            right: 26px;
            padding: 12px 24px;
            background: #1a1a1a;
            color: white;
            border: none;
            border-radius: 8px;
            font-weight: 600;
            cursor: pointer;
            box-shadow: 0 2px 8px rgba(0, 0, 0, 0.15);
        }
        #brighten-btn:hover {
            background: #000000;
        }
    </style>
    <div id="image-container">
        <img id="image-display" alt="Processed image" />
        <button id="brighten-btn">Brighten</button>
    </div>
    <script>
        const imageEl = document.getElementById('image-display');
        const btnEl = document.getElementById('brighten-btn');

        function extractImageUrl(data) {
            if (data?.text?.startsWith('Image URL: ')) {
                return data.text.substring('Image URL: '.length).trim();
            }
            if (data?.content) {
                for (const item of data.content) {
                    if (item.type === 'text' && item.text?.startsWith('Image URL: ')) {
                        return item.text.substring('Image URL: '.length).trim();
                    }
                }
            }
        }

        function render() {
            const url = extractImageUrl(window.openai?.toolOutput);
            if (url) imageEl.src = url;
        }

        async function brightenImage() {
            btnEl.disabled = true;
            btnEl.textContent = 'Brightening...';
            const result = await window.openai.callTool('power_law_image', {
                input_path: imageEl.src
            });
            const newUrl = extractImageUrl(result);
            if (newUrl) imageEl.src = newUrl;
            btnEl.disabled = false;
            btnEl.textContent = 'Brighten';
        }

        btnEl.addEventListener('click', brightenImage);
        window.addEventListener("openai:set_globals", (event) => {
            if (event.detail?.globals?.toolOutput) render();
        }, { passive: true });

        render();
    </script>
    """
    return visual


with gr.Blocks() as demo:
    with gr.Row():
        with gr.Column():
            original_image = gr.Image(label="Original Image", type="filepath")
            btn = gr.Button("Brighten Image")
        with gr.Column():
            output_image = gr.Image(label="Output Image", type="filepath")
            html = gr.Code(language="html", max_lines=20)

    btn.click(power_law_image, inputs=original_image, outputs=original_image)
    btn.click(app_html, outputs=html)

if __name__ == "__main__":
    demo.launch(mcp_server=True, share=True)

We won't break down the code in as much detail since many of the pieces are the same. But note the following differences from the earlier example:

  • Calling tools from the widget: The app uses window.openai.callTool() to invoke the MCP tool directly from a button click, without requiring ChatGPT to call it:
const result = await window.openai.callTool('power_law_image', {
    input_path: imageEl.src
});
  • Parsing tool call results: The result from callTool() contains a content array that needs to be parsed to extract data:
function extractImageUrl(data) {
    if (data?.content) {
        for (const item of data.content) {
            if (item.type === 'text' && item.text?.startsWith('Image URL: ')) {
                return item.text.substring('Image URL: '.length).trim();
            }
        }
    }
}
  • Updating UI based on tool results: After calling the tool, the app immediately updates the displayed image with the new result:
const newUrl = extractImageUrl(result);
if (newUrl) imageEl.src = newUrl;

With these examples, you've seen how to build both simple reactive widgets and more advanced interactive apps that can call tools directly from the UI. By combining Gradio's MCP server capabilities with the OpenAI Apps SDK, it's time to start create richer ChatGPT integrations that enhance the conversational experience with custom visualizations and user interactions!

←

Using Docs Mcp

Using Hugging Face Integrations

→