Web_search tools

[harshalmore31]

This pull request introduces several new features and improvements related to web search functionalities. The most significant changes include the addition of search capabilities using Exa.ai, SerpAPI, and Tavily, as well as a comprehensive web search module with enhanced content extraction and formatting.

New Search Capabilities:

• Exa.ai Integration:

  • Added exa_search function to perform web searches using the Exa.ai API and format the results.

• SerpAPI Integration:

  • Implemented serpapi_search function to utilize SerpAPI for Google searches and format the results.

• Tavily Integration:

  • Developed tavily_search function to perform searches using the Tavily API and format the results.

• Web Search:

  • Implemented web search locally with google indexing without an Agent

Web Search Module Enhancements:

• Lazy Loading and Improved Content Extraction:

  • Introduced WebSearch class with lazy loading of required packages, improved HTML preprocessing, and hybrid content extraction methods.

• Environment Configuration:

  • Added new environment variables for Google API key and custom search engine ID in .env.example.