File: //opt/alt/python37/lib/python3.7/site-packages/ssa/modules/__pycache__/storage.cpython-37.pyc
B
$ �c� � @ s� d Z ddlZddlmZ ddlmZmZmZmZ ddl Z ddl m
Z
m Z m Z m
Z
mZ ddlmZmZ eG dd� d��Zee d �d
d �Zd d
� Zeeee f d �dd�ZdS )zF
Represents storage where ssa data is collected, stored and extracted
� N)� dataclass)�List�Iterator�Tuple�Dict)�func�cast�case�literal_column�distinct)�
session_scope�
RequestResultc @ s2 e Zd ZU eed< ee ed< eed< eed< dS )�
DomainData� domain_name�domain_total_reqs�is_a_wordpress_domain� urls_numberN)�__name__�
__module__� __qualname__�str�__annotations__r �int�bool� r r �D/opt/alt/python37/lib/python3.7/site-packages/ssa/modules/storage.pyr s
r )�returnc c s� t | ���}|�tjt�dtj�t�tj�t� tj
�t� t tj
����tjt�dtj���tjt�dtj��}tj|dd� d�}xf|D ]^\}}t|�}d}dgd }x,|D ]$\} }
} } }
| |t|
�< t ||
�}q�W t||| |d�V q�W W dQ R X dS ) z7
Iterates data from database domain-by-domain.
z%Hc S s | d S )Nr r )�itemr r r �<lambda>/ � z#iter_domains_data.<locals>.<lambda>)�keyr � )r r r r N)r �queryr
�domainr �strftime�
created_at�Count�id�max� wordpress�countr �path�group_by�order_by� itertools�groupby�tupler r )�engine�dbZresults_by_hourZresults_by_hour_groupedr �groupZdomain_results_by_hourr Zrequests_number_by_hour�_�hourZ requests_numZ is_wordpressZurlsr r r �iter_domains_data s0
r6 c c sx t | ���d}|�tjt�dtj�t�ttj t
j ��� d�t�
tj�� d�t�ttjt
j ��� d���tj|k��tj�|���tjt�dtj���tjt�dtj��}d}dgd dgd dgd }}}x�|D ]~\} }
} } }
|�r(|| k�r(|t||||d�fV dgd dgd dgd }}}| |t|
�< | |t|
�< |
|t|
�< | }q�W | t| |||d�fV W dQ R X dS ) z6
Iterates urls data from database url-by-url.
z%H�url_throttled_reqs�url_total_reqs�
url_slow_reqsNr r! )r+ r7 r8 r9 )r r" r
r+ r r$ r% ZSumr �hitting_limits�
sqlalchemy�Integer�labelr& r' �is_slow_request�filterr# �in_r, r- �dictr )r1 r Z all_pathsr2 � urls_dataZ
previous_pathr7 r8 r9 r+ r5 Zurl_throttled_reqZ
url_total_reqZ url_slow_reqr r r �iter_urls_dataD sH " " rC c c sv t | ��d}|�tjtj��tj|k��tj�}t� t
|�dd� �}x$|D ] \}}|dd� |D �fV qHW W dQ R X dS )zA
Get information about durations of requests url-by-url.
c S s | d S )Nr r )r r r r r � r z#get_url_durations.<locals>.<lambda>c S s g | ] \}}|�qS r r )�.0r4 �durationr r r �
<listcomp>� s z%get_url_durations.<locals>.<listcomp>N) r r" r
r+ rE r? r# r- r. r/ �list)r1 r r2 rB Zdurations_by_pathr r3 r r r �get_url_durations| s
rH )�__doc__r. � dataclassesr �typingr r r r r; r r r r
r �ssa.dbr r
r r6 rC r r rH r r r r �<module> s (8