Zero to AI Hero, Part 2: Understanding Plugins in Semantic Kernel, A Deep Dive With Examples

gowebing

I talked a little about semantic kernel in Part 1: Jumpstart Your Journey With Semantic Kernel, but I decided to leave out the juicy part. There are a few gears in SK. They spin and connect like the inside of the watch to make it do the magic. One of those gears is plugins. Plugins are the robotic arms of Semantic Kernel, capable of doing some work beyond chit-chatting.

Why Do We Need Plugins?

We all know GenAI is about — surprise, surprise: "Generative AI". What do they train on? Tokens. These tokens are then mapped onto vector databases and can be used to predict the next token, etc, and the story goes on and on. What are these models capable of? Predicting subsequent tokens, and that's about it (at least as of now). So, for those who think GAI is equal to AGI, no! Not yet!

Now, back to plugins. These token-predicting machines were not capable of handling real-world challenges in the beginning. What if we ask a simple question like, "What is the time now?" It couldn't tell us as it was not part of the training data. This is where plugins come into play. Plugins give Semantic Kernel these little code snippets, which it can run to get an answer to a specific query it wouldn't otherwise know the answer to.

SK Plugins are built using a technique called function/tool calling, which is baked into most Large Language Models and some Small Language Models nowadays (we will dig into SLMs later). In simple terms, function calling allows language models to create planning and invoking code snippets/APIs of your existing code. Once the language model requests a function, SK serves as a router to this and calls an existing code snippet, then serves the return value back into the language model. As you can see, the potential for this expands when a model and a bunch of plugins start talking to each other and exchanging information.

Let’s Charge an Electric Car With the Help of Plugins

AI overview of electric cars/PSE&G

Let's start with our former example from Part 1. The end goal of our plugin is to charge an electric vehicle (or hybrid) during off-peak hours. A bit of context: The electricity company where I live, PSE&G, rewards Electric/Hybrid vehicle owners who charge their vehicles during off-peak hours through an EV residential Charging Program. Let's start by taking advantage of this.

C#
  
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Connectors.OpenAI;
using System.ComponentModel;

var builder = Kernel.CreateBuilder();
builder.AddAzureOpenAIChatCompletion(
    deploymentName: "<YOUR_DEPLOYMENT_NAME>",
    endpoint: "<YOUR_ENDPOINT>",
    apiKey: "<YOUR_AZURE_OPENAI_API_KEY>"
);

builder.Plugins.AddFromType<TimeTeller>(); // <------------ Telling kernel about time plugins
builder.Plugins.AddFromType<ElectricCar>(); // <------------ Telling kernel about car plugins
var kernel = builder.Build();

OpenAIPromptExecutionSettings settings = new() { ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions };
while (true)
{
    Console.Write("User > ");
    string userMessage = Console.ReadLine();
    Console.WriteLine(await kernel.InvokePromptAsync(userMessage, new(settings)));
    Console.WriteLine("--------------------------------------------------------------");
}

public class TimeTeller // <------------ Time teller plugin. An expert on time, peak and off-peak periods
{
    [Description("This function retrieves the current time.")]
    [KernelFunction]
    public string GetCurrentTime() => DateTime.Now.ToString("F");

    [Description("This function checks if in off-peak period between 9pm and 7am")]
    [KernelFunction]
    public bool IsOffPeak() => DateTime.Now.Hour < 7 || DateTime.Now.Hour >= 21;
}

public class ElectricCar // <------------ Car plugin. Knows about states and conditions of the electric car.
{
    private bool isCarCharging = false;
    [Description("This function starts charging the electric car.")]
    [KernelFunction]
    public string StartCharging()
    {
        if (isCarCharging)
        {
            return "Car is already charging.";
        }
        isCarCharging = true; // This is where you would call httos://tesla/api/mycar/start. Kidding, you got the idea.
        return "Charging started.";
    }

    [Description("This function stops charging the electric car.")]
    [KernelFunction]
    public string StopCharging()
    {
        if (!isCarCharging)
        {
            return "Car is not charging.";
        }
        isCarCharging = false;
        return "Charging stopped.";
    }
}


We will dive into the code later. For now, let's play with our car charging plugins a bit. Please take a look at the conversation below. It is interesting, isn't it? To take explicit action, it needs:

  1. To know the current time
  2. If the current time is in the off-peak period
  3. To be able to stop or start the charging of the car

And interestingly enough, we have two plugins: one for time handling and one for car handling. Doesn't it feel like they are aware of each other and able to work together? That's the magic of the semantic kernel.

Prompt dialog

Semantic Kernel, the Orchestrator

The semantic kernel (SK) is this beautiful orchestrator that passes the ball between the model and available plugins, thus producing the desired output by getting a collaborative effort. In our example above, When we asked the question, "Can you start charging it if it is an off-peak period?" SK has to determine if I am on an off-peak period using the TimeTeller plugin and then call the functions of the ElectricCar plugin to charge the car if needed. Since I didn't ask during an off-peak period, SK decided not to charge the vehicle. Quite smart.

Tennis graphic depicting SK plugins/model

How Is This Possible? Let’s Dissect.

Let's look at the plugins TimeTeller and ElectricCar. They are just a C# class with a few public functions. Each function is decorated with a KernelFunction attribute and a Description attribute. This Description attribute, also called Semantic Description, tells the SK/model what this function actually does in simple language. When we ask a question, SK can determine if it needs to invoke any of these functions based on this description and function calling.

Plugins as Classes

If we were to ask, "What is the current time?" SK would use GetCurrentTime() to get an answer. If we were to ask, "Charge car if it is an off-peak period now." SK would call IsOffPeak() to determine if the current time is on an off-peak period; if it is, SK would then call the function StartCharging() to start charging the car, like chaining.

C#
  
public class TimeTeller // <------------ Time teller plugin. An expert on time, peak and off-peak periods
{
    [Description("This function retrieves the current time.")]
	[KernelFunction]
	public string GetCurrentTime() => DateTime.Now.ToString("F");
    
    [Description("This function checks if in off-peak period between 9pm and 7am")]
    [KernelFunction]
    public bool IsOffPeak() => DateTime.Now.Hour < 7 || DateTime.Now.Hour >= 21;
}

public class ElectricCar // <------------ Car plugin. Knows about states and conditions of the electric car.
{
    private bool isCarCharging = false;
    [Description("This function starts charging the electric car.")]
    [KernelFunction]
    public string StartCharging()
    {
        if (isCarCharging)
        {
            return "Car is already charging.";
        }
        isCarCharging = true; // This is where you would call httos://tesla/api/mycar/start. Kidding, you got the idea.
        return "Charging started.";
    }

    [Description("This function stops charging the electric car.")]
    [KernelFunction]
    public string StopCharging()
    {
        if (!isCarCharging)
        {
            return "Car is not charging.";
        }
        isCarCharging = false;
        return "Charging stopped.";
    }
}


Registering Plugins With Semantic Kernel

Now that we have plugins, we need to register them onto Semantic Kernel — without it, it won't be able to use them. We will register it like this.

C#
  
var builder = Kernel.CreateBuilder();

// Omitted for brevity

builder.Plugins.AddFromType<TimeTeller>(); <------ Tell SK Builder about the plugins we want to use
builder.Plugins.AddFromType<ElectricCar>(); <------ Tell SK Builder about the plugins we want to use
var kernel = builder.Build(); <--- Then build the kernel


Wire It All Up

We have plugins — we added them to the Semantic Kernel. Now, we need to tell the kernel that it is capable of using any of these plugins in the way it wants. That gives the kernel confidence to use them as necessary. This is done through OpenAIPromptExecutionSettings passed as a parameter to the InvokePrompt call.

C#
  
OpenAIPromptExecutionSettings settings = new() { ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions };
Console.WriteLine(await kernel.InvokePromptAsync("Can you start charging if it is past 10pm?", new(settings)));


That's it! We connected all the pieces. Now, we should be able to get the kernel to do some actual work for us beyond being super chatty all the time! This isn't much, but you can find all this code on GitHub. I promise that by Part 3, we will be rocking the dance floor.

Wrap Up

The code we wrote is "just fine." It doesn't utilize the best of Semantic Kernel yet. Plugins are cool pieces of the Agent puzzle. Plugins are powerful tools for building fully autonomous agents if needed by mixing with plans and persona. Now that we have a clear understanding of plugins, we are ready to step foot onto the actual realm of Semantic Kernel, where it shines and amazes oldie C# folks like me most — agents. We will explore agents in the next part of this series and build an actual agent to plan our day trip, including ensuring our electric car is charged enough before we start the journey.

What's Next?

Plugins are great. But they alone can't go far. In our case, Semantic Kernel may be able to choose the right plugin, but as the problem gets complex, it will be stretched thin. To achieve autonomous decision-making, we must jump to the next step: Agents. Agents combine the power of plugins with a persona, planner, and a few other ingredients to start behaving as if it has a mind. We need that to build absolute autonomy, at least to an extent. We will explore building agents in the next part of this series. We will use the plugins we built here and build an actual trip planner that is capable of understanding that our car battery does not have enough juice to go for a day trip.

 

 

 

 

Top