File: //opt/alt/python37/lib/python3.7/site-packages/ssa/modules/__pycache__/stat_sender.cpython-37.pyc
B
$ �c� �
@ sd d Z dZddlZddlZddlZddlmZ ddlmZ ddl m
Z
m Z ddl m
Z
ddlmZ dd lmZmZ dd
lmZ dd lmZ dd lmZ d
dlmZ d
dlm Z ddl m Z ddl m Z ddl!m"Z"m#Z# ddl$m%Z%m&Z&m'Z'm(Z( G dd� de�Z)e*dk�r`e&� ej+dej,d� ye)� �-� W n4 e k
�r^ Z. ze/e.� e0d
��W ddZ.[.X Y nX dS )zd
This module contains SSA classes for sending e-mails with report
and sending reports to ClickHouse
z ssa.modules� N)� EmailMessage)�partial)�Optional�Any)�get_admin_email)�is_cl_solo_edition)�Session�Response)� HTTPAdapter)�RequestException)�Retry� )�Common)�
DecisionMaker� )� stat_server)�SSAError)�Mailer�render_report_table)� read_sys_id� sentry_init�
duration_cast� format_datec s� e Zd ZdZ� fdd�Zdedd�dd�Zeee d�d d
�Z ee
d �d d
�Z ee
d �dd�Z ee
d�dd�Zeeed�dd��Zeed�dd�Zee
d�dd�Z� ZS ) �StatisticsSenderz4
Send report to ClickHouse and over e-mails
c s� t � �� t�d�| _| j�dt� t� | _dt � d�| _
t � | _ t
dtdg�tddd g�dd
�}t|d �}t� | _| j�d|� t| jjd d
�| j_d S )N� stat_senderz StatisticsSender enabled: %szhttps://z
/api/clos-ssa� �POSTi� i� i� )�total�method_whitelist�status_forcelist�backoff_factor)Z max_retries�
)�timeout)�super�__init__�logging� getLogger�logger�info� __package__r �sys_idr � ch_endpointr � mail_senderr � frozensetr
r �sessionZmountr �request)�selfZ
retry_conf�adapter)� __class__� �H/opt/alt/python37/lib/python3.7/site-packages/ssa/modules/stat_sender.pyr$ * s
zStatisticsSender.__init__N)�report�returnc C s0 |dkrt � �� }| jr"| �|� | �|� dS )zi
Send given report over e-mail
If no report given, get it from DecisionMaker API
N)r �get_json_report� summary_notification_enabled� email_report�clickhouse_report)r0 r5 r3 r3 r4 �send; s
zStatisticsSender.send)� report_viewr6 c C s� | � � }|r�|�d�r�t|�\}}| jj|dt|d �|d�}|jt�|�dd|d � d�d� |j|d d|d � d
�d� | j� |� |S d S ) z4
Create and send e-mail with report
�domainsZ
ssa_report�date)Z recipient�templater> �html�jsonZreport_z.json)�subtype�filenamer@ z.htmlN)
�get_mail_recipient�getr r, �_messager �add_attachmentrA �dumpsZ_send)r0 r< Zmail_toZ report_table�mail�msgr3 r3 r4 r9 G s
z StatisticsSender.email_report)r6 c
C sH y| � � pt� S tk
rB } z| j�dt|�� W dd}~X Y nX dS )z�
Retrieve a recipient's e-mail:
1. get address from a wmt-api utility
2. if command failed or address is empty, get address of server admin
z get_admin_email failed with: %sN)�wmt_api_report_emailr � Exceptionr' �error�str)r0 �er3 r3 r4 rD ` s z#StatisticsSender.get_mail_recipientc C s� t dd�rdnd}y&tjd|� �dgdddd�j�� }W n> tjtttfk
rv } z| j �
dt |�� d S d }~X Y nX yt �
|��d
��d �S t jk
r� } z| j �
d t |�� W d d }~X Y n" tk
r� | j �
d
|� Y nX d S )zD
Retrieve a recipient's e-mail address from WMT API
T)�skip_jwt_checkz wmt-api-solozwmt-apiz&/usr/share/web-monitoring-tool/wmtbin/z --config-get)�check�text�capture_outputzwmt-api utility failed: %sN�configZ report_emailz!wmt-api returned invalid json: %sz(wmt-api returned unexpected response: %s)r �
subprocess�run�stdout�strip�CalledProcessError�AttributeError�OSError�
ValueErrorr' rM rN rA �loadsrE �JSONDecodeError)r0 �_utilZ api_responserO r3 r3 r4 rK l s(
z%StatisticsSender.wmt_api_report_emailc
C s� |� d�r�| j�d| j� y"| jj| j| �| �|��d�}W nN tk
r� } z0| jj d|d| jid� t
dt |�� ��|�W dd}~X Y nX | � |�S | j�d � d
S dS ) z+
Send report to ClickHouse
r= zSending POST request to %s)rA zPOST failed with %s�endpoint)�extraz'Failed to POST data to SSA API server: Nz*Report is empty, not sending to ClickHouseF)
rE r' r( r+ r. Zpost�_ch_pack�clickhouse_formatr rM r rN �_process_response)r0 r< �resprO r3 r3 r4 r: � s
$
z"StatisticsSender.clickhouse_report)�valuer6 c C s
t | d�S )z<
Pack given value into data field of a dict
)�data)�dict)rf r3 r3 r4 rb � s zStatisticsSender._ch_pack)�original_reportr6 c C s` t � }xT|�d�D ]F}|�| j|�d�|�d�|�d�|�d�dd� |�d�D �d �� qW |S )
zd
Format local report for sending to ClickHouse
(required structures differ)
r= �name� slow_urls� slow_reqs�
total_reqsc
S s< g | ]4}|� d �|� d�t|� d��t|� dd��d��qS )rj �reqs_num�average_duration� correlationr )�urlZcount_requests� avg_durationrp )rE r �float)�.0�ur3 r3 r4 �
<listcomp>� s z6StatisticsSender.clickhouse_format.<locals>.<listcomp>�urls)� system_id�domainZcount_slow_urlsZcount_slow_requestsZtotal_requests�details)�listrE �appendr* )r0 ri Z ch_reportry r3 r3 r4 rc � s z"StatisticsSender.clickhouse_format)�responser6 c C s� |j s(| jjd|j|jd|jid� dS | j�d|j|j|j� |�� }|d dkrr| jjd|d d |id� dS | j�d
� d S ) z�
Check received response
:param response: a requests.Response object
:return: True in case of success, False otherwise
z&Unable to connect to server with %s:%sZ resp_text)ra Fz [%s:%s] Response received %s�status�okz Received response with status %sr} z Sent to ClickHouse successfullyT) r r' rM � status_code�reasonrR r( rq rA )r0 r} �resultr3 r3 r4 rd � s z"StatisticsSender._process_response)N)�__name__�
__module__� __qualname__�__doc__r$ rh r; r r r9 rN rD rK �boolr: � staticmethodr rb r{ rc r rd �
__classcell__r3 r3 )r2 r4 r % s r �__main__zstat_sender_standalone.log)rC �level)1r� r) rA r% rU �
email.messager � functoolsr �typingr r �clcommon.cpapir �clcommon.lib.cleditionr Zrequestsr r Zrequests.adaptersr
Zrequests.exceptionsr Z$requests.packages.urllib3.util.retryr �commonr �decision_makerr �internal.constantsr �internal.exceptionsr Zinternal.mailerr r �internal.utilsr r r r r r� � basicConfig�INFOr; �exc�print�
SystemExitr3 r3 r3 r4 �<module> s< 0