Format: application/x-python-code
1 B
2
3 8T�cw � @ s d dl Z G dd� d�ZdS )� Nc @ s e Zd Zdd� Zddd�ZdS )�
4 DataExtractorc C s || _ || _d S )N)�
5 spark_session� cred_manager)�selfr r � r �G/home/glue_user/workspace/jupyter_workspace/dev/helper/dataextractor.py�__init__ s zDataExtractor.__init__Nc C s� |r|nd}|rd| nd}d|kr\d|� d|d � d|d � d
6 |� �� � }| j�|�}�n�d |kr�| j�|�}|d d
7 |d |d |d |d |d d�}d|� d|d � d|d � d
8 |� �� � }| jj�d�jf |��d|�� � }�n d|k�r�| j�|�}|d } |d }
9 y |d } W n |d } Y nX y |d } W n d} Y nX y |d }
10 W n d}
11 Y nX y |d }W n t
12 d��Y nX d | � d | � d |
13 � �}d|� d|d � d
14 |� �� � }| jj�d ��d |��d|
15 ��d|��d|�� � }|S )!N�*zwhere � ZgluezSELECT z FROM �database�.�table� Z snowflake�accountz.snowflakecomputing.com�user�passwordZschemaZ warehouse)ZsfURLZsfUserZ
16 sfPasswordZ
17 sfDatabaseZsfSchemaZ sfWarehousez net.snowflake.spark.snowflake�queryZaurora�host�base_url�portZ5432�postgresz Password Not Defined in secretzjdbc:postgresql://�:�/Zjdbc�url) �stripr �sqlr �get_credentials�read�format�options�option�load�
18 ValueError)r �sourcer Zpushdown_predicate�columnsr Zdf�credsZ sfOptionsr
19 r r r r r r r r r � extract_data
20 sV (
21 (
22
23
24 zDataExtractor.extract_data)NN)�__name__�
25 __module__� __qualname__r r& r r r r r s r )�boto3r r r r r �<module> s