Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Implement TalkHier framework for hierarchical multi-agent content generation #783

Merged
merged 1 commit into from
Feb 20, 2025

Conversation

harshalmore31
Copy link
Collaborator

@harshalmore31 harshalmore31 commented Feb 20, 2025

This PR introduces the TalkHier framework, a novel hierarchical multi-agent system designed for structured content generation and refinement. TalkHier leverages a team of specialized agents, each with a distinct role, to collaboratively produce high-quality content through iterative communication and evaluation.

Key Features:

  • Hierarchical Multi-Agent System: TalkHier employs a hierarchical structure with agents specializing in different aspects of content creation:
    • Supervisor: Orchestrates the entire process, analyzes tasks, selects evaluation criteria, and guides revision.
    • Generator: Creates initial content based on instructions.
    • Evaluator (Multiple): Critically assesses content quality against dynamically generated criteria.
    • Revisor: Refines content based on evaluator feedback.
    • Criteria Generator: Dynamically generates task-specific evaluation criteria.
  • Structured Communication: Agents communicate using structured CommunicationEvent objects, ensuring clear and organized information exchange in JSON format. This includes messages, background context, intermediate outputs, sender/receiver information, and timestamps.
  • Iterative Refinement: The system iteratively refines content through evaluation and revision cycles, aiming to meet a defined quality threshold or a maximum number of iterations.
  • Dynamic Evaluation Criteria: TalkHier dynamically generates evaluation criteria tailored to each specific content generation task, ensuring relevant and targeted feedback.
  • Parallel Evaluation: Utilizes ThreadPoolExecutor to enable parallel content evaluation by multiple evaluator agents, speeding up the refinement process.
  • Robust JSON Handling: Implements robust JSON parsing with error handling and fallback mechanisms to ensure smooth communication between agents, even with variations in LLM output formats.
  • State Management: Includes functionality for saving and loading agent states, allowing for persistence and continuation of content generation processes.
  • Configurable Parameters: Offers configurable parameters such as max_iterations, quality_threshold, and model_name to customize the content generation process.
  • Conversation Tracking: Maintains a Conversation object to log agent interactions and the overall content generation history for debugging and analysis.

Benefits:

  • Improved Content Quality: The hierarchical and iterative approach, combined with structured evaluation and revision, leads to higher quality content compared to single-agent systems.
  • Task-Specific Content: Dynamic criteria generation ensures content is evaluated and refined based on the specific requirements of each task.
  • Enhanced Collaboration: The framework demonstrates effective collaboration between specialized agents, mimicking a team-based content creation workflow.
  • Transparency and Explainability: Structured communication and conversation logging provide insights into the content generation process and agent decision-making.
  • Extensibility: The modular design allows for easy extension and customization, such as adding new agent roles or modifying existing agent behaviors.

📚 Documentation preview 📚: https://swarms--783.org.readthedocs.build/en/783/

Copy link

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Summary

The TalkHier framework introduces a sophisticated hierarchical multi-agent system for content generation and refinement. Here are the key points from analyzing the implementation:

  • Implements a robust hierarchical structure in /swarms/structs/Talk_Hier.py with specialized agents (supervisor, generator, evaluator, revisor) communicating through structured CommunicationEvent objects
  • Utilizes ThreadPoolExecutor for parallel evaluation processing to improve performance when multiple evaluators assess content simultaneously
  • Adds dynamic criteria generation through a dedicated CriteriaGenerator agent that tailors evaluation metrics based on task requirements
  • Includes comprehensive error handling and fallback mechanisms for JSON parsing, though the current implementation may mask important errors by converting any text to a dict
  • The example code contains a potential typo using model name "gpt-4o" which should be addressed for proper functionality

💡 (1/5) You can manually trigger the bot by mentioning @greptileai in a comment!

1 file(s) reviewed, 3 comment(s)
Edit PR Review Bot Settings | Greptile

talkhier = TalkHier(
max_iterations=1,
quality_threshold=0.8,
model_name="gpt-4o",
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

syntax: model_name='gpt-4o' appears to be a typo and should be 'gpt-4'

Suggested change
model_name="gpt-4o",
model_name="gpt-4",

Comment on lines +401 to +405
with ThreadPoolExecutor() as executor:
evaluations = list(executor.map(
lambda x: run_evaluator(*x),
zip(self.evaluators, eval_inputs)
))
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style: ThreadPoolExecutor should specify max_workers to prevent resource exhaustion with many evaluators

Comment on lines +98 to +99
json_match = re.search(r"\{.*\}", json_str, re.DOTALL)
if (json_match):
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logic: regex pattern r"{.}" is greedy and may match too much - use r"{.?}" for non-greedy matching

Suggested change
json_match = re.search(r"\{.*\}", json_str, re.DOTALL)
if (json_match):
json_match = re.search(r"\{.*?\}", json_str, re.DOTALL)
if (json_match):

message=response.get("message", ""),
background=response.get("background", ""),
intermediate_output=response.get("intermediate_output", {}),
sender=sender.agent_name,

Check failure

Code scanning / Pyre

Incompatible parameter type Error

Incompatible parameter type [6]: In call CommunicationEvent.\_\_init\_\_, for argument sender, expected str but got Optional[str].
background=response.get("background", ""),
intermediate_output=response.get("intermediate_output", {}),
sender=sender.agent_name,
receiver=receiver.agent_name,

Check failure

Code scanning / Pyre

Incompatible parameter type Error

Incompatible parameter type [6]: In call CommunicationEvent.\_\_init\_\_, for argument receiver, expected str but got Optional[str].
"content": current_content.get("content", {}).get(
"main_body", ""
),
"final_score": evaluation["scores"]["overall"],

Check failure

Code scanning / Pyre

Uninitialized local Error

Uninitialized local [61]: Local variable evaluation is undefined, or not always defined.
"content_metadata": current_content.get(
"content", {}
).get("metadata", {}),
"evaluation": evaluation,

Check failure

Code scanning / Pyre

Uninitialized local Error

Uninitialized local [61]: Local variable evaluation is undefined, or not always defined.
@kyegomez kyegomez merged commit 49b65e4 into kyegomez:master Feb 20, 2025
14 of 25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants