When you demo a machine learning model, you might want to collect data from users who try the model, particularly data points in which the model is not behaving as expected. Capturing these "hard" data points is valuable because it allows you to improve your machine learning model and make it more reliable and robust.
Gradio simplifies the collection of this data by including a Flag button with every
Interface. This allows a user or tester to easily send data back to the machine where the demo is running. In this Guide, we discuss more about how to use the flagging feature, both with
gradio.Interface as well as with
Flagging with Gradio's
Interface is especially easy. By default, underneath the output components, there is a button marked Flag. When a user testing your model sees input with interesting output, they can click the flag button to send the input and output data back to the machine where the demo is running. The sample is saved to a CSV log file (by default). If the demo involves images, audio, video, or other types of files, these are saved separately in a parallel directory and the paths to these files are saved in the CSV file.
There are four parameters in
gradio.Interface that control how flagging works. We will go over them in greater detail.
allow_flagging: this parameter can be set to either
manual: users will see a button to flag, and samples are only flagged when the button is clicked.
auto: users will not see a button to flag, but every sample will be flagged automatically.
never: users will not see a button to flag, and no sample will be flagged.
flagging_options: this parameter can be either
None(default) or a list of strings.
None, then the user simply clicks on the Flag button and no additional options are shown.
["Incorrect", "Ambiguous"], then buttons labeled Flag as Incorrect and Flag as Ambiguous appear. This only applies if
flagging_dir: this parameter takes a string.
flagging_callback: this parameter takes an instance of a subclass of the
gr.HuggingFaceDatasetSaverwhich can allow you to pipe any flagged data into a HuggingFace Dataset. (See more below.)
Within the directory provided by the
flagging_dir argument, a CSV file will log the flagged data.
Here's an example: The code below creates the calculator interface embedded below it:
import gradio as gr def calculator(num1, operation, num2): if operation == "add": return num1 + num2 elif operation == "subtract": return num1 - num2 elif operation == "multiply": return num1 * num2 elif operation == "divide": return num1 / num2 iface = gr.Interface( calculator, ["number", gr.Radio(["add", "subtract", "multiply", "divide"]), "number"], "number", allow_flagging="manual" ) iface.launch()
When you click the flag button above, the directory where the interface was launched will include a new flagged subfolder, with a csv file inside it. This csv file includes all the data that was flagged.
+-- flagged/ | +-- logs.csv
num1,operation,num2,Output,timestamp 5,add,7,12,2022-01-31 11:40:51.093412 6,subtract,1.5,4.5,2022-01-31 03:25:32.023542
If the interface involves file data, such as for Image and Audio components, folders will be created to store those flagged data as well. For example an
image input to
image output interface will create the following structure.
+-- flagged/ | +-- logs.csv | +-- image/ | | +-- 0.png | | +-- 1.png | +-- Output/ | | +-- 0.png | | +-- 1.png
im,Output timestamp im/0.png,Output/0.png,2022-02-04 19:49:58.026963 im/1.png,Output/1.png,2022-02-02 10:40:51.093412
If you wish for the user to provide a reason for flagging, you can pass a list of strings to the
flagging_options argument of Interface. Users will have to select one of these choices when flagging, and the option will be saved as an additional column to the CSV.
If we go back to the calculator example, the following code will create the interface embedded below it.
iface = gr.Interface( calculator, ["number", gr.Radio(["add", "subtract", "multiply", "divide"]), "number"], "number", allow_flagging="manual", flagging_options=["wrong sign", "off by one", "other"] ) iface.launch()
When users click the flag button, the csv file will now include a column indicating the selected option.
num1,operation,num2,Output,flag,timestamp 5,add,7,-12,wrong sign,2022-02-04 11:40:51.093412 6,subtract,1.5,3.5,off by one,2022-02-04 11:42:32.062512
Sometimes, saving the data to a local CSV file doesn't make sense. For example, on Hugging Face Spaces, developers typically don't have access to the underlying ephemeral machine hosting the Gradio demo. That's why, by default, flagging is turned off in Hugging Face Space. However, you may want to do something else with the flagged data.
We've made this super easy with the
For example, below we're going to pipe flagged data from our calculator example into a Hugging Face Dataset, e.g. so that we can build a "crowd-sourced" dataset:
import os HF_TOKEN = os.getenv('HF_TOKEN') hf_writer = gr.HuggingFaceDatasetSaver(HF_TOKEN, "crowdsourced-calculator-demo") iface = gr.Interface( calculator, ["number", gr.Radio(["add", "subtract", "multiply", "divide"]), "number"], "number", description="Check out the crowd-sourced dataset at: [https://huggingface.co/datasets/aliabd/crowdsourced-calculator-demo](https://huggingface.co/datasets/aliabd/crowdsourced-calculator-demo)", allow_flagging="manual", flagging_options=["wrong sign", "off by one", "other"], flagging_callback=hf_writer ) iface.launch()
Notice that we define our own
gradio.HuggingFaceDatasetSaver using our Hugging Face token and
the name of a dataset we'd like to save samples to. In addition, we also set
because on Hugging Face Spaces,
allow_flagging is set to
"never" by default. Here's our demo:
You can now see all the examples flagged above in this public Hugging Face dataset.
We created the
gradio.HuggingFaceDatasetSaver class, but you can pass your own custom class as long as it inherits from
FLaggingCallback defined in this file. If you create a cool callback, contribute it to the repo!
What about if you are using
gradio.Blocks? On one hand, you have even more flexibility
with Blocks -- you can write whatever Python code you want to run when a button is clicked,
and assign that using the built-in events in Blocks.
At the same time, you might want to use an existing
FlaggingCallback to avoid writing extra code.
This requires two steps:
.setup()somewhere in the code prior to the first time you flag data
.flag()method, making sure to collect the arguments correctly and disabling the typical preprocessing.
Here is an example with an image sepia filter Blocks demo that lets you flag
data using the default
import numpy as np import gradio as gr def sepia(input_img, strength): sepia_filter = strength * np.array( [[0.393, 0.769, 0.189], [0.349, 0.686, 0.168], [0.272, 0.534, 0.131]] ) + (1-strength) * np.identity(3) sepia_img = input_img.dot(sepia_filter.T) sepia_img /= sepia_img.max() return sepia_img callback = gr.CSVLogger() with gr.Blocks() as demo: with gr.Row(): with gr.Column(): img_input = gr.Image() strength = gr.Slider(0, 1, 0.5) img_output = gr.Image() with gr.Row(): btn = gr.Button("Flag") # This needs to be called at some point prior to the first call to callback.flag() callback.setup([img_input, strength, img_output], "flagged_data_points") img_input.change(sepia, [img_input, strength], img_output) strength.change(sepia, [img_input, strength], img_output) # We can choose which components to flag -- in this case, we'll flag all of them btn.click(lambda *args: callback.flag(args), [img_input, strength, img_output], None, preprocess=False) demo.launch()
Important Note: please make sure your users understand when the data they submit is being saved, and what you plan on doing with it. This is especially important when you use
allow_flagging=auto (when all of the data submitted through the demo is being flagged)