Skip to content

Open Source Models

While OpenAI is generally recommended, there are situations where you might prefer open-source models. The following projects offer alternatives by mimicking the Assistants API:

✅ Tested Projects

🔜 Other Projects

Astra Assistants API

To use agency-swarm with Astra Assistants API, follow these steps:

1. Create an account on Astra Assistants API and obtain an API key.

Astra Assistants API Example

2. Add Astra DB Token to your .env file: Copy token from the file that starts with "AstraCS:" and paste it into your .env file.

ASTRA_DB_APPLICATION_TOKEN=AstraCS:dsfkgn...

3. Add other model provider API keys to .env as well:

PERPLEXITYAI_API_KEY=your_perplexityai_api_key
ANTHROPIC_API_KEY=your_anthropic_api_key
TOGETHER_API_KEY=your_together_api_key
GROQ_API_KEY=your_groq_api_key

4. Install the Astra Assistants API and gradio:

pip install astra-assistants-api gradio

5. Patch the OpenAI client:

from openai import OpenAI
from astra_assistants import patch
from agency_swarm import set_openai_client
from dotenv import load_dotenv

load_dotenv()

client = patch(OpenAI())

set_openai_client(client)

6. Create an agent: Create an agent and replace the model parameter with the name of the model you want to use. With Astra Assistants you can upload files like usual using files_folder.

from agency_swarm import Agent

ceo = Agent(name="ceo",
            description="I am the CEO",
            model='ollama/llama3',
            # model = 'perplexity/llama-3-8b-instruct'
            # model = 'anthropic/claude-3-5-sonnet-20240620'
            # model = 'groq/mixtral-8x7b-32768'
            # model="gpt-4o",
            files_folder="path/to/your/files"
            )

7. Create an agency:

You can add more agents as needed, just make sure all manager agents support function calling.

from agency_swarm import Agency

agency = Agency([ceo])

8. Start gradio:

To utilize your agency in gradio, apply a specific non-streaming demo_gradio method from the agency-swarm-lab repository:

from agency_swarm import Agency
from .demo_gradio import demo_gradio

agency = Agency([ceo])

demo_gradio(agency)

For a complete example, see the notebook.

General Instructions

To use agency-swarm with any other projects that mimic the Assistants API, generally, you need to follow these steps:

1. Install the previous version of agency-swarm as most projects are not yet compatible with streaming and Assistants V2:

pip install agency-swarm==0.1.7

2. Switch out the OpenAI client:

import openai
from agency_swarm import set_openai_client

client = openai.OpenAI(api_key="whatever", base_url="http://127.0.0.1:8000/")

set_openai_client(client)

3. Set the model parameter:

from agency_swarm import Agent

ceo = Agent(name="ceo", description="I am the CEO", model='ollama/llama3')

4. Start Gradio:

To utilize your agency in gradio, apply a specific non-streaming demo_gradio method from the agency-swarm-lab repository:

from agency_swarm import Agency
from .demo_gradio import demo_gradio

agency = Agency([ceo])

demo_gradio(agency)

5. For backend integrations, simply use:

agency.get_completion("I am the CEO")

Limitations

  • Function calling is not supported by most open-source models: This limitation prevents the agent from communicating with other agents in the agency. So, it must be positioned at the end of the agency chart and cannot utilize any tools.
  • RAG is typically limited: Most open-source assistants API implementations have restricted Retrieval-Augmented Generation capabilities. It is recommended to develop a custom tool with your own vector database.
  • CodeInterpreter is not supported: The Code Interpreter feature is still under development for all open-source assistants API implementations.

Future Plans

Updates will be provided as new open-source assistant API implementations stabilize.

If you successfully integrate other projects with agency-swarm, please share your experience through an issue or pull request.