OpenAI Agent Toolkit

Project Omnibus

Posted by dave on February 20, 2024


The goal of Project Omnibus is to provide a simple, open-source Python toolkit that can help build and experiment with powerful, custom-crafted OpenAI Assistants based on ChatGPT-4. This toolkit can be modified and extended to enable GPT-4 to escape the cloud and interact with the real world through ubiquitous MQTT messaging. Build your own personal advisory sidekick.

OpenAI Assistants can write and run code, reference their own documents and call functions when responding to a prompt. This opens the door to a plethora of opportunities for leveraging the advanced reasoning capabilities of the leading large-language-model AI.

MQTT stands for "MQ Telemetry Transport" ( It is a standardized, lightweight publish/subscribe messaging protocol designed for machine to machine telemetry in low bandwidth environments.

Application Scenarios

Project Description

Python code, built on the OpenAI Beta Assistant API, to aid the integration of OpenAI LLMs (e.g. ChatGPT-4) into applications using MQTT messaging. Documentation for the API is available here.

MQTT is a light-weight, easy-to-use, publish-subscribe messaging bus widely used in internet of things (IoT) communications. It is multi-platform (e.g. Windows, Linux) and is supported by all major software languages. The site is and downloads are available at

Access to the GPT-4 Assistant API requires an OpenAI account and an OpenAI key from Set the OPENAI_API_KEY environment variable using the Windows Powershell "setx" command.



Code Summary

The code is 100% Python 3.12 developed on Windows 11 with Microsoft VSCode.

The code consists of a folder with a set of console apps that perform different functions associated with a specific GPT-4 Assistant that resides in the OpenAI cloud.

The GPT-4 Assistant is created by Use to verify creation. GPT-4 Assistant behavior is defined in this file.

The is the primary local proxy agent that communicates directly with the cloud Assistant. It sends messages to the Assistant and waits for responses. Other functions include:

  • Prompt messages can be entered on the command line or received in an MQTT message
  • Maintains a local cache of tag/data values that is refreshed from the agent. The agent collects whatever external data it desires and sends that data in JSON to the agent via an MQTT message. The data is then available to the GPT-4 Assistant through function call-backs
  • Implements a set of call-back functions that the GPT-4 Assistant can call whenever he/she/it desires. These represent external actions that the Assistant can perform based on the combined prompt data submitted. Typically these would be JSON encoded strings directed to a destination sink. Messages handled include: COMMAND, CONTROL, NOTICE, ALERT, ALARM
  • Command-line: python

Agent libraries include:

  • Handles Assistant communications using the Assistant API
  • Handles function call-backs from the Assistant API
  • Handles MQTT messaging
  • Handles tag/value caching
  • Handles data access to the MATLAB ThingSpeak data service (

Other agents are optional. They are:

    • Sends and receives prompts using MQTT to and from
    • Command-line: python
    • Receives MQTT messages from GPT-4 function call-backs in
    • Speaks the message received using OpenAI text-to-speech
    • Command-line: python
    • Sends JSON MQTT messages to using tag/value. This data is available to GPT-4 using function call-backs and is accessible using the "getSensorValueByName" function.
    • Sends hard-coded sample data
    • Command-line: python
    • Periodically sends fully-dressed prompt messages with both text and data to
    • Accesses and sends public weather data from MATLAB in Boston, MA to
    • Command-line: python
    • Deletes the Assistant
    • Command-line: python

These console apps have a simple structure that extends the OpenAI Assistant API in a straight forward if not elegant way. The purpose of this code is to help experiment and explore the capabilities of GPT-4 while providing a means to integrate GPT-4 with other applications, regardless of where the applications are running.

Each console app is multi-threaded. They create threads to handle keyboard and MQTT message dispatching based on topic. The topic tree is defined in to avoid conflict between agents. Incoming MQTT messages are pushed into topic queues and the topic worker threads get and service the messages received. Other worker threads are spawned as needed. Most of the code is plumbing and can be changed to do whatever.


  • Git clone the repository to a local folder
  • Obtain an OpenAI key from your API account and set the OPENAI_API_KEY using the Windows Powershell "setx" command
  • Create a new agent by copying and renaming the "OmniBus" folder
  • Rename agent_name in Agent version is encoded in name. (e.g. OmniBus-1-0-0) The agents should now run under the new name
  • Modify to create the assistant with the characteristics desired
  • Run and confirm creation on OpenAI API site
  • Modify and prompt data files to synch with assistant and change the MQTT hub IP address from "localhost" if an external hub is used
  • Test using keyboard text prompts
  • Run more agents as desired. Custom agents in any language interact with the using MQTT messages

Have Fun With Agents...

Dave Hardin