Format: application/x-python-code
1 o
2
3 �h� � @ sf d dl mZmZ d dlZd dlmZ d dlmZ d dlm Z
4 d dl Z d dl Z d dl
5 mZ d dlZd dlZd dlZddlmZmZmZmZmZmZmZmZ e j e j d � e � e �Z d
6 e!d ee!e!f fd d
7 �Z"d ee! fdd�Z#d ee! fdd�Z$d ee! fdd�Z%d ee! fdd�Z&i Z'd a(d ee! fdd�Z)d ee! fdd�Z*d ee! fdd�Z+d ee! fd d �Z,dS ) � )�List�TupleN)�
8 BeautifulSoup)�DDGS)�search)�ValidationError� )�FetchNewsParams�FetchArxivParams�FetchDDGParams�FetchTorrentsParams�FetchWikiParams�FetchIMDBParams�FetchYouTubeParams�FetchFinanceParams)�level�url�returnc C s� zCt j| ddidd�}|�� t|jd�}|jr |jj�� n| }|�d�}d� dd � |D ��}d
9 t
10 � �}t
11 d 7 a
12 | |d �t |< ||fW S t yP | d
13 f Y S w )Nz
14 User-Agentz Mozilla/5.0� )�headers�timeoutz html.parser�p�
15 c s s � | ] }|� � �� V qd S )N)�get_text�strip)�.0r � r �7C:\Users\shubh\Videos\agent\agent\modules\search\web.py� <genexpr>$ s � z scrape_page.<locals>.<genexpr>�scrape_� ��href�text� )
16 �requests�get�raise_for_statusr r# �title�stringr �find_all�join�scrape_counter�
17 scraped_store� Exception)r �resp�soupr( Z
18 paragraphs�snippet� scrape_idr r r � scrape_page s
19
20
21 �r3 c
22 K s� z6t di | ��}g }t|j|jdd�D ] }t|�\}}|�|� d|� d|� �� t|�|jkr3 |W S q|W S tyR } zt� d|� �� g W Y d }~S d }~ww )NT)� num_results�uniquer �Invalid parameters: r )
23 r �
24 google_search�query� max_resultsr3 �append�lenr �logger�error)�kwargs�params�resultsr r( r1 �er r r �
25 fetch_news- s ���rB c
26 K s� z4t di | ��}g }tj|j|jd�}|�� D ]}|j�dd��� }|� |j
27 � d|� d|j � �� q|W S t yP } zt
28 �d|� �� g W Y d }~S d }~ww )N�r8 r9 r � z
29 Abstract: z
30 PDF: r6 r )r
31 �arxiv�Searchr8 r9 r@ �summary�replacer r: r( �pdf_urlr r< r= )r> r? �papersr �rrG rA r r r � fetch_arxiv= s ��rL c
32 K sJ zt di | ��W S ty$ } zt�d|� �� g W Y d }~S d }~ww )Nr6 r )rB r r< r= )r> rA r r r �
33 search_googleK s ��rM c
34 K �` zt di | ��}t|j� d�|jd�W S ty/ } zt�d|� �� g W Y d }~S d }~ww )Nz proxy "torrent" rC r6 r )r rM r8 r9 r r< r= �r> r? rA r r r �fetch_torrentsS � ��rP c K s� zbt d i | ��}t�d� g }tj|j|jd�D ]G}z+tj|dd�}dt� �}td7 a|j|j d�t
35 |< |� |j � d|j � d|j� �� W q t
36 y_ } zt�d |� �� W Y d }~qd }~ww |W S t
37 y~ } zt�d
38 |� �� g W Y d }~S d }~ww ) N�en)r@ F)Z auto_suggestr r r! r zWikipedia subpage error: zWikipedia fetch error: r )r
39 � wikipediaZset_langr r8 r9 �pager, r rG r- r: r( r. r< �warningr= )r> r? r@ r( rT r2 rA r r r �fetch_wikipedia_ s*
40
41 $ ����rV c
42 K rN )Nz site:imdb.comrC zIMDB fetch error: r )r rM r8 r9 r. r< r= rO r r r �
43 fetch_imdbs rQ rW c
44 K rN )Nz site:youtube.comrC zYouTube fetch error: r )r rM r8 r9 r. r< r= rO r r r �
45 fetch_youtube{ rQ rX c
46 K s� zPt d
47 i | ��}g }|j�� �� }|D ]:}t�|�}|j}|�d�p(|�d�p(d}d|� �}dt� �} td7 a||d�t | < |�
48 |� � � d|� d|� �� q|W S t yl }
49 zt
50 �d |
51 � �� g W Y d }
52 ~
53 S d }
54 ~
55 ww ) NZlongBusinessSummaryrG r$ z https://finance.yahoo.com/quote/r r r! r zFinance fetch error: r )r r8 r �split�yfZTicker�infor& r, r- r: �upperr. r< r= ) r> r? r@ Ztickers�symbolZtickerr[ r# r r2 rA r r r �
56 fetch_finance� s&
57
58
59 ��r^ )-�typingr r r% �bs4r Zduckduckgo_searchr Z googlesearchr r7 rE �loggingZpydanticr �jsonrS ZyfinancerZ Z
60 tools.schemasr r
61 r r r
62 r r r � basicConfig�INFO� getLogger�__name__r< �strr3 rB rL rM rP r- r, rV rW rX r^ r r r r �<module> s2 (
63