Format: application/x-python-code
1 o
2
3 ��hE � @ sN d dl Z d dlZd dlZd dlZd dlZd dlmZ ed�ZG dd� d�Z dS )� N)� setup_loggerzrecall_sampler/data_loaderc @ s~ e Zd Zddd�Zdd� Zdd� Zded efd
4 d �Zd ej fd d
5 �Z
6 d ej fdd�Z dej de defdd�Z
7 d efdd�ZdS )�
8 DataLoader� us-east-1c C s || _ || _|| _|| _d S �N)�matches_config�catalog_config�
9 customer_name� region_name)�selfr r r r � r �@C:\Users\shubh\Videos\recall\recall_audit_sampler\data_loader.py�__init__ s
10 zDataLoader.__init__c C s8 t j�� }|jd| jd�}|j|d�}t�|d �}|S )N�secretsmanager)� service_namer )�SecretId� SecretString)�boto3�session�Session�clientr �get_secret_value�json�loads)r
11 � secret_namer r �response�secretr r r �
12 get_secret s
13
14 zDataLoader.get_secretc C sF | � |�}t�d|� �� tjd|d |d |d |�dd�d�}|S ) Nz!Connecting to Postgres database: Z matchlibrary�username�password�host�porti8 )Zdbname�userr r r )r �logger�info�psycopg2�connect�get)r
15 r r �connr r r �connect_postgres s
16
17 �zDataLoader.connect_postgres�
18 sql_file_path�returnc K sp t j�|�s
19 td|� ���t|d�� }|�� }W d � n1 s!w Y tdd| ji|��}|jdi |��S )NzSQL file not found: �rr r ) �os�path�isfile�FileNotFoundError�open�read�dictr �format)r
20 r) �kwargs�fZ sql_template�paramsr r r �render_sql_from_file& s
21 �z DataLoader.render_sql_from_filec C � | � | jd d �}ddlm}m} |�� |dd� �d�}| j| jd |d �}t�d
22 |� �� t �
23 ||�}|� � | � |g d �d � |j
24 rJtd
25 ��|S )N�
26 connectionr r ��datetime� timedelta� ��days�%Y%m%d�sql_file�Z catalog_datez-Loading matches data from Postgres database: )Zcomp_source_storeZcomp_skuZbase_source_store�base_sku�matchZ match_status�matchesz)Matches dataset is empty! Cannot proceed.)r( r r; r< �now�strftimer7 r" r# �pdZread_sql_query�close�validate_schema�empty�
27 ValueError�r
28 r' r; r< Z one_day_agoZ sql_query�dfr r r � load_matches/ s zDataLoader.load_matchesc C r8 )Nr9 r r r: r= r> r@ rA rB z-Loading catalog data from Postgres database: )ZskuZ
29 product_title�sizeZuomZ pack_sizeZbrand�categoryZ product_url�catalogz)Catalog dataset is empty! Cannot proceed.)r( r r; r< rF rG r7 r" r# rH Zread_sqlrI rJ rK rL rM r r r � load_catalog> s zDataLoader.load_catalogrN �required_fields� dataset_namec s, � fdd�|D �}|rt |� d|� ���d S )Nc s g | ] }|� j vr|�qS r )�columns)�.0�col�rN r r �
30 <listcomp>O s z.DataLoader.validate_schema.<locals>.<listcomp>z missing required columns: )rL )r
31 rN rT rU �missingr rY r rJ N s �zDataLoader.validate_schemac C s | � � }| �� }||fS r )rO rS )r
32 �
33 matches_df�
34 catalog_dfr r r �loadS s zDataLoader.loadN)r )�__name__�
35 __module__� __qualname__r
36 r r( �strr7 rH Z DataFramerO rS �listrJ �tupler^ r r r r r
37 s
38 r )
39 r r$ ZpandasrH r r, � logger_setupr r" r r r r r �<module> s