o /i`@sddlZddlZddlZddlZddlZddlmZmZmZddlm Z m Z m Z m Z m Z ddlmZddlmZddlmZmZmZddlmZmZdd lmZdd lmZmZdd lmZm Z m!Z!m"Z"m#Z#e rdd l$m%Z%dd l&m'Z'ddl(m)Z)ddlm*Z*e+dZ,GdddZ-Gddde-Z.Gddde-Z/Gddde-Z0Gddde-Z1Gddde-Z2Gddde-Z3d$dd d!e e4fd"d#Z5dS)%N)datetime timedeltatimezone) TYPE_CHECKINGAnyOptionalUnioncast)resolve_serializer)DEFAULT_FAILURE_TTL)AbandonedJobErrorInvalidJobOperationNoSuchJobError)Job JobStatus)Queue)BaseDeathPenaltyUnixSignalDeathPenalty)as_text backend_classcurrent_timestampnowparse_composite_key)Redis)Pipeline) Execution) Serializerz rq.registryc@seZdZdZeZeZdZ      dBde de dde e d d e d d e e d e fde e e f ddZddZddZdedefddZedefddZdCdefddZdDd ed!ed"e d#d$edef d%d&ZdEd e ee fd"e d#d'efd(d)ZdFd*e efd+d,ZdGd.ed/ed0ed1edee f d2d3Zd4d5Zd d defd6d7Z dHd8e d e fd9edd fd:d;Z!e"dZ#dId*e ed?e efd@dAZ$dS)J BaseRegistryz Base implementation of a job registry, implemented in Redis sorted set. Each job is stored as a key in the registry, scored by expiration time (unix timestamp). zrq:registry:{0}defaultNname connectionr job_classrqueuer serializerrdeath_penalty_classcCsf|r|j|_|j|_|j|_n ||_||_t||_|j|j|_|r&|nt|_t |d|d|_ dS)Nr%)override) r r!r$r key_templateformatkeyrr"rr%)selfr r!r"r#r$r%r+K/var/www/html/flask_server/venv/lib/python3.10/site-packages/rq/registry.py__init__(s   zBaseRegistry.__init__cCs|jS)z+Returns the number of jobs in this registry)countr*r+r+r,__len__>szBaseRegistry.__len__cCs |j|jko|jjj|jjjkSN)r r!connection_poolconnection_kwargs)r*otherr+r+r,__eq__Bs zBaseRegistry.__eq__itemreturncCs0|}t||jr |j}|j|jtt|duS)z Returns a boolean indicating registry contains the given job instance or job id. Args: item (Union[str, Job]): A Job ID or a Job. N) isinstancer"idr!zscorer)r strr*r6job_idr+r+r, __contains__Hs zBaseRegistry.__contains__cCs |jddS)z{Returns the number of jobs in this registry after running cleanup Returns: int: _description_ Tcleanup) get_job_countr/r+r+r,r.Us zBaseRegistry.countTcCs|r||j|jS)zReturns the number of jobs in this registry after optional cleanup. Args: cleanup (bool, optional): _description_. Defaults to True. Returns: int: _description_ )r@r!zcardr))r*r@r+r+r,rA^s zBaseRegistry.get_job_countrFjobttlpipelinerxxcCsb|dkr|nt|}|dkrd}|dur$tt|j|j|j|i|dS|jj|j|j|i|dS)aAdds a job to a registry with expiry time of now + ttl, unless it's -1 which is set to +inf Args: job (Job): The Job to add, or job_id. ttl (int, optional): The time to live. Defaults to 0. pipeline (Optional['Pipeline'], optional): The Redis Pipeline. Defaults to None. xx (bool, optional): .... Defaults to False. Returns: result (int): The ZADD command result r+infNrF)rr intzaddr)r9r!)r*rCrDrErFscorer+r+r,addks  zBaseRegistry.add delete_jobcCsj|dur|n|j}t||jr|jn|}||j|}|r3t||jr&|}n tj|||jd}| |S)aERemoves job from registry and deletes it if `delete_job == True` Args: job (Job|str): The Job to remove from the registry, or job_id pipeline (Pipeline|None): The Redis Pipeline. Defaults to None. delete_job (bool, optional): If should delete the job.. Defaults to False. Nr!r$) r!r8r"r9zremr)rfetchr$delete)r*rCrErNr!r=result job_instancer+r+r,removes zBaseRegistry.remove timestampcs6|dur|nt}jjd|}fdd|DS)zReturns job ids whose score are less than current timestamp. Returns ids for jobs with an expiry time earlier than timestamp, specified as seconds since the Unix epoch. timestamp defaults to call time if unspecified. Nrcg|]}|qSr+ parse_job_id.0r=r/r+r, z4BaseRegistry.get_expired_job_ids..rr! zrangebyscorer))r*rVrL expired_jobsr+r/r,get_expired_job_idssz BaseRegistry.get_expired_job_idsrGstartenddescr@cs0|rfddjjj|||dDS)aReturns list of all job ids. Args: start (int, optional): start rank. Defaults to 0. end (int, optional): end rank. Defaults to -1. desc (bool, optional): sort in reversed order. Defaults to False. cleanup (bool, optional): whether to perform the cleanup. Defaults to True. Returns: List[str]: list of the job ids in the registry crWr+rXrZr/r+r,r\r]z,BaseRegistry.get_job_ids..rdr@r!zranger)r*rbrcrdr@r+r/r, get_job_idss $zBaseRegistry.get_job_idscCst|j|j|jdS)z3Returns Queue object associated with this registry.rO)rr r!r$r/r+r+r, get_queueszBaseRegistry.get_queuecCs |j|j|j}t|tjS)zkReturns job's expiration time. Args: job (Job): The Job to get the expiration )r!r:r)r9r fromtimestamprutc)r*rCrLr+r+r,get_expiration_timesz BaseRegistry.get_expiration_time job_or_idat_frontcCst||jr |}|j}n |j}|jj||j|d}|j|j|j}|s&t|j -}t |j |j|j|d}d|_ d|_ d|_||j|||d}|Wd|S1s[wY|S)aRequeues the job with the given job ID. Args: job_or_id (Union['Job', str]): The Job or the Job ID at_front (bool, optional): If the Job should be put at the front of the queue. Defaults to False. Raises: InvalidJobOperation: If nothing is returned from the `ZREM` operation. Returns: Job: The Requeued Job. rO)r!r"r$N)rEro)r8r"r$rQr!rPr)r9rrErorigin started_atended_at _exc_infosave _enqueue_jobexecute)r*rnrorCr$rSrEr#r+r+r,requeues(    zBaseRegistry.requeueentrycCst|ts t|}|S)aGeneric function to retrieve the job id from the stored entry. Some Registries might have a different entry format. Args: entry (str): the entry from the registry Returns: str: the job_id parsed from the registry. )r8r;r)ryr+r+r,rYs zBaseRegistry.parse_job_idexception_handlerscCsdS)zThis method is automatically called by `count()` and `get_job_ids()` methods implemented in BaseRegistry. Base registry doesn't have any special cleanup instructionsNr+)r*rVrzr+r+r,r@szBaseRegistry.cleanup)rNNNNN)TrNFNFr1rrGFT)FNN)%__name__ __module__ __qualname____doc__rr"rr%r'r;rtyperrr-r0r5rboolr>propertyrJr.rArMrUfloatralistrirjrrmrx staticmethodrYr@r+r+r+r,rsP    $ $$  " rc@s8eZdZdZdZd+deedeefddZd,d d d d de de de f ddZ   d-d d dee d ed de ddf ddZ d.de de de de deeeeff ddZdede fd d!Zd/de de d ed de de f d"d#Zd0dee efd ed de fd$d%Zed&edefd'd(Zd1de d ed ddfd)d*ZdS)2StartedJobRegistryaM Registry of currently executing jobs. Each queue maintains a StartedJobRegistry. Jobs in this registry are ones that are currently being executed. Jobs are added to registry right before they are executed and removed right after completion (success or failure). Each entry is a {job_id}:{execution_id} z rq:wip:{0}NrVrzc Cs||dur|nt}||}|r|}|j}|D]}z |jj||j|jd}Wn ty4Yqw| |j t t t |r\|D]} | |t t t } | durWd} | s[nqE|joc|jdk} | rm|||qdtjdt jdt} td|jj|j| |jtj|d |j| |d d |j|d d q||jd||WddS1swYdSdS)aRemove abandoned jobs from registry and add them to FailedJobRegistry. Removes jobs with an expiry time earlier than timestamp, specified as seconds since the Unix epoch. timestamp defaults to call time if unspecified. Removed jobs are added to the global failed job queue. Args: timestamp (datetime): The datetime to use as the limit. NrOTrz Moved to z , due to z, at z%s cleanup: %s %srErp) worker_nameF)refresh_job_status)rrarjr!rEr"rQr$rexecute_failure_callbackr%r traceback extract_stack retries_leftretryFailedJobRegistryrrloggerwarning __class__r9 set_statusrFAILED_handle_failureenqueue_dependentszremrangebyscorer)rw) r*rVrzrLjob_idsr#rEr=rChandler fallthroughr exc_stringr+r+r,r@sJ    "zStartedJobRegistry.cleanuprF executionrrErrDrFr7cCs:|dkr|nt|}|dkrd}|j|j|j|i|dS)aAdds an execution to a registry with expiry time of now + ttl, unless it's -1 which is set to +inf Args: execution (Execution): The Execution to add. pipeline (Pipeline): The Redis Pipeline. ttl (int, optional): The time to live. Defaults to 0. xx (bool, optional): .... Defaults to False. Returns: result (int): The ZADD command result rrGrHrI)rrKr) composite_key)r*rrErDrFrLr+r+r, add_execution8s z StartedJobRegistry.add_executionrCrNcCs&|dur|n|j}||j|jdS)aRemoves job from registry and deletes it if `delete_job == True` Args: execution (Execution): The Execution to remove job (Optional[Job]): The Job to remove from the registry pipeline (Optional['Pipeline'], optional): The Redis Pipeline. Defaults to None. delete_job (bool, optional): If should delete the job.. Defaults to False. N)r!rPr)r)r*rrCrErNr!r+r+r,remove_executionJsz#StartedJobRegistry.remove_executionrGTrbrcrdr@cCs,|r|dd|jj|j|||dDS)aHFunction to retrieve a list of tuples where the first item is the job id and the second is the execution id. Args: start (int, optional): start rank. Defaults to 0. end (int, optional): end rank. Defaults to -1. desc (bool, optional): sort in reversed order. Defaults to False. cleanup (bool, optional): whether to perform the cleanup. Defaults to True. Returns: List[Tuple[str, str]]: a list of tuples where the first item is the job id and the second is the execution id. cSsg|]}tt|qSr+)rr)r[ryr+r+r,r\ps z@StartedJobRegistry.get_job_and_execution_ids..rerfrhr+r+r,get_job_and_execution_ids^s z,StartedJobRegistry.get_job_and_execution_idsr6cCs,|}t||jr |j}tt||jddvS)zMethod to check if the item is in the registry. Args: item (Any): Either a Job (instance of job_class) or a job id. Returns: bool: True if the item is in the registry. Fr?)r8r"r9r r;rir<r+r+r,r>ts zStartedJobRegistry.__contains__cCtr1NotImplementedErrorr*rCrDrErFr+r+r,rMzStartedJobRegistry.addcCrr1r)r*rCrErNr+r+r,rUrzStartedJobRegistry.removerycCs"t|ts t|}t|\}}|Sr1)r8r;rr)ryr= _execution_idr+r+r,rYs  zStartedJobRegistry.parse_job_idcCsD|dur|n|j}dd|D}|r |j|jg|RdSdS)zRemoves job executions from the started job registry. Args: job (Job): The Job to remove from the registry pipeline (Optional['Pipeline']): The Redis Pipeline. Defaults to None. NcSsg|]}|jqSr+)r)r[rr+r+r,r\sz8StartedJobRegistry.remove_executions..)r!get_executionsrPr))r*rCrEr! execution_idsr+r+r,remove_executionss z$StartedJobRegistry.remove_executionsr~)rFNNFr}r{r|r1)rrrrr'rrrr@rJrrrrtupler;rrr>rMrrUrrYrr+r+r+r,rsJ  9  $$ rc@s0eZdZdZdZddeedeefddZdS) FinishedJobRegistryz Registry of jobs that have been completed. Jobs are added to this registry after they have successfully completed for monitoring purposes. zrq:finished:{0}NrVrzcC(|dur|nt}|j|jd|dSzRemove expired jobs from registry. Removes jobs with an expiry time earlier than timestamp, specified as seconds since the Unix epoch. timestamp defaults to call time if unspecified. Nrrr!rr)r*rVrzrLr+r+r,r@zFinishedJobRegistry.cleanupr~) rrrrr'rrrr@r+r+r+r,rs rc@sReZdZdZdZddeedeefddZ   dd d d e d ed fddZ dS)rz- Registry of containing failed jobs. z rq:failed:{0}NrVrzcCrrrrr+r+r,r@rzFailedJobRegistry.cleanuprprCrrrErcCs|durt}|dkr |nt|}|r|}n|j}||_|j|ddd|j||d||j|j |i|s?| dSdS)z Adds a job to a registry with expiry time of now + ttl. `ttl` defaults to DEFAULT_FAILURE_TTL if not specified. NrF)rE include_metainclude_result)rDrE) r rr!rErtrur@rKr)r9rw)r*rCrDrrErLpr+r+r,rMs   zFailedJobRegistry.addr~)NrpN) rrrrr'rrrr@r;rMr+r+r+r,rs rc s`eZdZdZdZddeedeefddZdd e d ee d ed d e de f fdd Z Z S)DeferredJobRegistryzH Registry of deferred jobs (waiting for another job to finish). zrq:deferred:{0}NrVrzc Cs|dur|nt}||}|re|jF}|D].}z |jj||j|jd}Wn ty0Yqw|jt j |ddt }|j ||ddq| |jd||WddS1s^wYdSdS)a,Remove expired jobs from registry and add them to FailedJobRegistry. Removes jobs with an expiry time earlier than timestamp, specified as seconds since the Unix epoch. timestamp defaults to call time if unspecified. Removed jobs are added to the failed job registry. NrOrz>Expired in DeferredJobRegistry, moved to FailedJobRegistry at rp)rrErr)rrar!rEr"rQr$rrrrrrrr)rw) r*rVrzrLrrEr=rCexc_infor+r+r,r@s"     "zDeferredJobRegistry.cleanupFrCrDrErrFr7cs"|durd}tt|||||S)zp Adds a job to a registry with expiry time of now + ttl. Defaults to -1 (never expire). NrG)superrrMrrr+r,rMszDeferredJobRegistry.addr~r)rrrrr'rrrr@rrJrrM __classcell__r+r+rr,rs 4rcseZdZdZdZfddZdddded fd d Zdd eeded fd dZ dd eedede e fddZ de de fdefddZZS)ScheduledJobRegistryz% Registry of scheduled jobs. zrq:scheduled:{0}cstj|i||j|_dSr1)rr-raget_jobs_to_enqueue)r*argskwargsrr+r,r-s zScheduledJobRegistry.__init__NrCrrErcCsX|jstttjdkr tjntj d}|j|d}t| }|j |j |j |iS)z Adds job to registry, scored by its execution time (in UTC). If datetime has no tzinfo, it will assume local timezone. r)seconds)tzinfo)rrrtimedaylightaltzonereplacecalendartimegm utctimetupler!rKr)r9)r*rCscheduled_datetimerEtzrVr+r+r,schedules " zScheduledJobRegistry.schedulerVcCs@tdt|dur |n|j}|dur|nt}||jd|S)aRemove jobs whose timestamp is in the past from registry. Args: timestamp (Optional[int], optional): The timestamp. Defaults to None. pipeline (Optional['Pipeline'], optional): The Redis pipeline. Defaults to None. zSScheduledJobRegistry.remove_jobs() is deprecated and will be removed in the future.Nr)warningswarnDeprecationWarningr!rrr))r*rVrEr!rLr+r+r, remove_jobs$s z ScheduledJobRegistry.remove_jobs chunk_sizer7cCs8|dur|nt}|jj|jd|d|d}dd|DS)aGet's a list of job IDs that should be scheduled. Args: timestamp (Optional[int]): _description_. Defaults to None. chunk_size (int, optional): _description_. Defaults to 1000. Returns: jobs (List[str]): A list of Job ids Nr)rbnumcSsg|]}t|qSr+)rrZr+r+r,r\>sz=ScheduledJobRegistry.get_jobs_to_schedule..r^)r*rVrrLjobs_to_scheduler+r+r,get_jobs_to_schedule2s z)ScheduledJobRegistry.get_jobs_to_schedulerncCs>t|tr |j}n|}|j|j|}|sttj|t j dS)a:Returns datetime (UTC) at which job is scheduled to be enqueued Args: job_or_id (Union[Job, str]): The Job instance or Job ID Raises: NoSuchJobError: If the job was not found Returns: datetime (datetime): The scheduled time as datetime object )r) r8rr9r!r:r)rrrkrrl)r*rnr=rLr+r+r,get_scheduled_time@s z'ScheduledJobRegistry.get_scheduled_timer1r~)Nr)rrrrr'r-rrrJrrr;rrrrrr+r+rr,r s  "rc@s$eZdZdZddeefddZdS)CanceledJobRegistryzrq:canceled:{0}NrVcCstr1r)r*rVr+r+r,ra[sz'CanceledJobRegistry.get_expired_job_idsr1)rrrr'rrrar+r+r+r,rXsrr#rrzcCsxt|j|j|j|jdt|j|j|j|jdj|dt|j|j|j|jdt|j|j|j|jddS)zCleans StartedJobRegistry, FinishedJobRegistry and FailedJobRegistry, and DeferredJobRegistry of a queue. Args: queue (Queue): The queue to clean )r r!r"r$)rzN) rr r!r"r$r@rrr)r#rzr+r+r,clean_registries_s" rr1)6rloggingrrrrrrtypingrrrrr rq.serializersr defaultsr exceptionsr rrrCrrr#rtimeoutsrrutilsrrrrrredisr redis.clientr rq.executionsrr getLoggerrrrrrrrrrrr+r+r+r,s<        U..*O