Create an AI Agent to Mint a MNIST NFT

Agents are entities designed to assist users in interacting with Smart Contracts by managing the proof verification of verifiable ML models and executing these contracts using Ape's framework.

Agents serve as intermediaries between users and Smart Contracts, facilitating seamless interaction with verifiable ML models and executing associated contracts. They handle the verification of proofs, ensuring the integrity and authenticity of data used in contract execution.

In this notebook, we will create an AI Agent to mint a MNIST NFT. Using an existing MNIST endpoint, we will perfomr a prediction on the MNIST dataset and mint an NFT based on the prediction.

This tutorial can be thought as the continuation of the previous tutorial Build a Verifiable Neural Network with Giza Actions where we created a verifiable MNIST model (notebook).

Before you begin

1. Python 3.11 or later must be installed on your machine

2. Giza CLI must be installed on your machine. You can install it by running pip install giza-cli

3. Actions-SDK should be installed with the extra agents. You can install it by running pip install giza-actions[agents]

4. You must have an active Giza account. If you don't have one, you can create one here.

5. You must have a MNIST model deployed on Giza. You can follow the tutorial Build a Verifiable Neural Network with Giza Actions to deploy a MNIST model on Giza.

6. During the tutorial, you will need to interact with the Giza CLI, Ape's framework, and provide multiple inputs, like model-id, version-id, account name, etc.

7. You have to be logged in to the Giza CLI or have an API KEY.

Installing the required libraries

Let's start by installing the required libraries.

pip install giza-actions[agents]

This will install the giza-actions library along with the agents extra, which contains the necessary tools to create an AI Agent.

Creating an account

Before we can create an AI Agent, we need to create an account using Ape's framework. We can do this by running the following command:

$ ape accounts generate <account name>
Enhance the security of your account by adding additional random input:
Show mnemonic? [Y/n]: n
Create Passphrase to encrypt account:
Repeat for confirmation:
SUCCESS: A new account '0x766867bB2E3E1A6E6245F4930b47E9aF54cEba0C' with HDPath m/44'/60'/0'/0/0 has been added with the id '<account name>'

This will create a new account under $HOME/.ape/accounts using the keyfile structure from the eth-keyfile library , for more information on the account management, you can refer to the Ape's framework documentation.

In this account generation we will be prompted to enter a passphrase to encrypt the account, this passphrase will be used to unlock the account when needed, so make sure to keep it safe.

We encourage the creation of a new account for each agent, as it will allow you to manage the agent's permissions and access control more effectively, but importing accounts is also possible.

Funding the account

Before we can create an AI Agent, we need to fund the account with some ETH. You can do this by sending some ETH to the account address generated in the previous step.

In this case we will use Sepolia testnet, you can get some testnet ETH from a faucet like Alchemy Sepolia Faucet or LearnWeb3 Faucet. This faucets will ask for security meassures to make sure that you are not a bot like having a specific amount of ETH in mainnet or a Github account. There are many faucets available, you can choose the one that suits you the best.

Once, we can recieve the testnet ETH and we have a funded account, we can proceed to create an AI Agent.

Funded account

Creating an AI Agent

Now that we have a funded account, we can create an AI Agent. We can do this by running the following command:

giza agents create --model-id <model-id> --version-id <version-id> --name <agent name> --description <agent description>

# or if you have the endpoint-id

giza agents create --endpoint-id <endpoint-id> --name <agent name> --description <agent description>

This command will prompt you to choose the account you want to use with the agent, once you select the account, the agent will be created and you will receive the agent id. The output will look like this:

[giza][2024-04-10 11:50:24.005] Creating agent ✅
[giza][2024-04-10 11:50:24.006] Using endpoint id to create agent, retrieving model id and version id
[giza][2024-04-10 11:50:53.480] Select an existing account to create the agent.
[giza][2024-04-10 11:50:53.480] Available accounts are:
┏━━━━━━━━━━━━━┓
┃  Accounts   ┃
┡━━━━━━━━━━━━━┩
│ my_account  │
└─────────────┘
Enter the account name: my_account
{
  "id": 1,
  "name": <agent_name>,
  "description": <agent_description>,
  "parameters": {
    "model_id": <model_id>,
    "version_id": <version_id>,
    "endpoint_id": <endpoint_id>,
    "alias": "my_account"
  },
  "created_date": "2024-04-10T09:51:04.226448",
  "last_update": "2024-04-10T09:51:04.226448"
}

This will create an AI Agent that can be used to interact with the deployed MNIST model.

How to use the AI Agent

Now that the agent is created we can start using it through giza-actions library. As we will be using the agent to mint a MNIST NFT, we will need to provide the MNIST image to the agent and preprocess it before sending it to the model.

import pprint
import numpy as np
from PIL import Image

from giza_actions.agent import GizaAgent, AgentResult
from giza_actions.action import action
from giza_actions.task import task

# Make sure to fill these in
MODEL_ID = <model-id>
VERSION_ID = <version-id>
# As we are executing in sepolia, we need to specify the chain
CHAIN = "ethereum:sepolia:geth"
# The address of the deployed contract
MNIST_CONTRACT = "0x17807a00bE76716B91d5ba1232dd1647c4414912"

Now we will start creating the functions that are necessary to create an Action that will:

• Load and preprocess the MNIST image

• Create an instance of an agent

• Predict the MNIST image using the agent

• Access the prediction result

• Mint an NFT based on the prediction result

To load and preprocess the image we can use the function that we already created in the previous tutorial Build a Verifiable Neural Network with Giza Actions.

# This function is for the previous MNIST tutorial
@task(name="Preprocess an image for the MNIST model.")
def preprocess_image(image_path: str):
    """
    Preprocess an image for the MNIST model.

    Args:
        image_path (str): Path to the image file.

    Returns:
        np.ndarray: Preprocessed image.
    """

    # Load image, convert to grayscale, resize and normalize
    image = Image.open(image_path).convert('L')
    # Resize to match the input size of the model
    image = image.resize((14, 14))
    image = np.array(image).astype('float32') / 255
    image = image.reshape(1, 196)  # Reshape to (1, 196) for model input
    return image

Now, we will create a function to create an instance of the agent. The agent is an exttension of the GizaModel class, so we can execute the predict as if we were using a model, but the agents needs more information:

• chain: The chain where the contract and account are deployed

• contracts: This is a dictionary in the form of {"contract_alias": "contract_address"} that contains the contract alias and address.

This contract_alias is the alias that we will use when executing the contract through code.

We create an instance so we can use it through the action.

@task(name="Create a Giza agent for the MNIST model")
def create_agent(model_id: int, version_id: int, chain: str, contract: str):
    """
    Create a Giza agent for the MNIST model with MNIST 
    """
    agent = GizaAgent(
        contracts={"mnist": contract},
        id=model_id,
        version_id=version_id,
        chain=chain,
        account="my_account1"
    )
    return agent

This new task will execute the predict method of the agent following the same format as a GizaModel instance. But in this case, the agent will return an AgentResult object that contains the prediction result, request id and multiple utilities to handle the verification of the prediction.

@task(name="Predict the digit in an image.")
def predict(agent: GizaAgent, image: np.ndarray):
    """
    Predict the digit in an image.

    Args:
        image (np.ndarray): Image to predict.

    Returns:
        int: Predicted digit.
    """
    prediction = agent.predict(
        input_feed={"image": image}, verifiable=True, dry_run=True
    )
    return prediction

Once we have the result, we need to access it. The AgentResult object contains the value attribute that contains the prediction result. In this case, the prediction result is a number from 0 to 9 that represents the digit that the model predicted.

When we access the value, the execution will be blocked until the proof of the prediction has been created and once created it is verified. If the proof is not valid, the execution will raise an exception.

This task is more for didactic purposes, to showcase in the workspaces the time that it can take to verify the proof. In a real scenario, you can use the AgentResult value directly in any other task.

@task(name="Get the digit from the prediction.")
def get_digit(prediction: AgentResult):
    """
    Get the digit from the prediction.

    Args:
        prediction (dict): Prediction from the model.

    Returns:
        int: Predicted digit.
    """
    # This will block the executon until the prediction has generated the proof and the proof has been verified
    return int(prediction.value[0].argmax())

Finally, we will create a task to mint an NFT based on the prediction result. This task will use the prediction result to mint an NFT with the image of the MNIST digit predicted.

To execute the contract we have the GizaAgent.execute context that will yield all the initialized contracts and the agent instance. This context will be used to execute the contract as it handles all the connections to the nodes, networks, and contracts.

In our instantiation of the agent we added an mnist alias to access the contract to ease its use. For example:

# A context is created that yields all the contracts used in the agent
with agent.execute() as contracts:
    # This is how we access the contract
    contracs.mnist
    # This is how we access the functions of the contract
    contracts.mnsit.mint(...)

@task(name="Execute the MNIST to mint a new NFT.")
def execute_contract(agent: GizaAgent, digit: int):
    """
    Execute the MNIST contract with the predicted digit to mint a new NFT.

    Args:
        agent (GizaAgent): Giza agent.
        digit (int): Predicted digit.

    Returns:
        str: Transaction hash.
    """
    with agent.execute() as contracts:
        contract_result = contracts.mnist.mint(digit)
    return contract_result

Now that we have all the steps defined, we can create the action to execute

@action(name="Mint an NFT with a prediction.", log_prints=True)
def mint_nft_with_prediction():
    # Preprocess image
    image = preprocess_image("seven.png")
    # Create Giza agent
    agent = create_agent(MODEL_ID, VERSION_ID, CHAIN, MNIST_CONTRACT)
    # Predict digit
    prediction = predict(agent, image)
    # Get digit
    digit = get_digit(prediction)
    # Execute contract
    result = execute_contract(agent, digit)
    pprint.pprint(result)

Remember that we should have kept the passphrase in a safe place? Now it is time to use it. For learning purposes, we will use the passphrase in the code, but in a real scenario, you should keep it safe and not hardcode it in the code.

# DO NOT COMMIT YOUR PASSPHRASE
import os
os.environ['MY_ACCOUNT1_PASSPHRASE'] = 'a'

Now let's execute the action.

Using Ape's default networks (chains) relies on public RPC nodes that could hit request limits and make the execution of the contract fail. For a better experience think about using a private RPC node with higher quotas

mint_nft_with_prediction()

What we have learned

In this tutorial, we learned how to create an AI Agent to mint an MNIST NFT. We created an AI Agent using Ape's framework and interacted with the agent to mint an NFT based on the prediction result of an MNIST image.

We learned how to load and preprocess the MNIST image, create an instance of the agent, predict the MNIST image using the agent, access the prediction result, and mint an NFT based on the prediction result.