[prev in list] [next in list] [prev in thread] [next in thread]
List: wsf-c-dev
Subject: Re: [Dev] [Siddhi] non-matched or expired events in pattern query
From: Jayesh Senjaliya <jhsonline () gmail ! com>
Date: 2017-09-29 5:47:54
Message-ID: CADFdJVXAD8fsH_Udw5OSKHDMaJe96MRCS_9-rLpGH-L44=V+cA () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Hi Gobinath,
Thanks for the suggestion about absence pattern, but we wont be able to
upgrade to Siddhi 4 anytime soon.
I am basically at the point where I can get all the relevant (subscribe)
events that happened during the given interval of first arrival of publish
events.
here AllPublisher = all registered publisher-subscriber events ( for each
publisher event )
from registeryTable as s join #publisher as p on p.pid == s.pid
select p.pid, s.sid insert into AllPublisher;
from every(a=AllPublisher) -> s=subscriber[pid == a.pid]
within 1 sec
select a.pid, s.sid
insert into completed_jobs_in_1_sec;
now, i need to find out those events from publisher (a) that dint had any
match from s within 1 sec.
which I was expecting to be available with " insert *expired events* into
not_completed_jobs", but looks like expired events are only available when
window is used. I am also looking at the code to see if i should add this.
Thanks
Jayesh
On Wed, Sep 27, 2017 at 4:21 AM, Gobinath <slgobinath@gmail.com> wrote:
> Hi,
>
> If you can use Siddhi 4 snapshot release, it can be done using the new
> feature 'Absent Pattern' added to Siddhi 4. The query to detect the events
> that do not match the condition within 10 seconds is given below:
>
> from every e1=publisher -> not subscriber[e1.pid == pid] for 10 sec
> select e1.pid
> insert into not_completed_jobs_in_time;
>
> The above query waits for 10 seconds from the arrival of every publisher
> event and if there is no subscriber event with an id satisfying the
> condition arrived within that waiting period, the id of the publisher event
> will be inserted into the not_completed_jobs_in_time stream.
>
> I guess the official document for Siddhi 4 is under construction. So you
> can find more details about absent pattern at [1]
>
> Still, Siddhi 4 is not production ready so I wonder whether you can use
> this feature or not.
>
> [1] http://www.javahelps.com/2017/08/detect-absence-of-events-
> wso2-siddhi.html
>
>
>
>
> On Tue, Sep 26, 2017 at 10:05 PM, Jayesh Senjaliya <jhsonline@gmail.com>
> wrote:
>
>> Hi Grainier,
>>
>> ya, i came across that example page, but i think that does not work in my
>> use-case which is as follow.
>>
>> i have a publish event followed by multiple subscribe event for the same
>> publish job.
>> now i want to catch if certain jobs (publish -> subscribe) has been
>> finished with 10 sec.
>> I have all the registered jobs in db table, which i use to gather all the
>> required publish-subscribe job events.
>>
>> define table jobTable( pid string, sid string);
>> define stream pubStream (pid int, status string);
>> define stream subStream (pid int, sid int, status string);
>>
>> -- this will get all the publish-> subscribe jobs events as master list
>> from pubStream as p join jobTable as t
>> on p.pid == t.pid
>> select p.pid, t.sid insert into allPSJobs;
>>
>> -- this is where i need to do intersection where if subStream event is
>> seen within 2 sec then remove that from master list ( allPSJobs ) if not
>> include that in not_completed_jobs_in_time
>>
>> from every ( a=allPSJobs ) -> s= subStream[sid == a.sid and pid==a.pid ]
>> within 2 sec
>> select s.pid, s.sid insert into completed_jobs_in_time;
>>
>>
>> hope that make sense from what i am trying to do.
>>
>> Thanks
>> Jayesh
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Sep 25, 2017 at 8:39 AM, Grainier Perera <grainier@wso2.com>
>> wrote:
>>
>>> Hi Jay,
>>>
>>> You can try something similar to this to get non-matched events during
>>> last 10 secs; You can find some documentation on this as well; link
>>> <https://docs.wso2.com/display/CEP420/Sample+0111+-+Detecting+non-occurrences+with+Patterns>
>>>
>>>
>>>
>>>> define stream publisher (pid string, time string);
>>>> define stream subscriber (pid string, sid string, time string);
>>>
>>>
>>>> from publisher#window.time(10 sec)
>>>> select *
>>>> insert expired events into expired_publisher;
>>>
>>>
>>>> from every pub=publisher -> sub=subscriber[pub.pid == pid] or
>>>> exp=expired_publisher[pub.pid == pid]
>>>> select pub.pid as pid, pub.time as time, sub.pid as subPid
>>>> insert into filter_stream;
>>>
>>>
>>>> from filter_stream [(subPid is null)]
>>>> select pid, time
>>>> insert into not_seen_in_last_10_sec_events;
>>>
>>>
>>> Moreover, I didn't get what you meant by "also is there a way to perform
>>> intersection of events based on grouping or time window ?" can you please
>>> elaborate on this?
>>>
>>> Regards,
>>>
>>> On Mon, Sep 25, 2017 at 11:02 AM, Jayesh Senjaliya <jhsonline@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> is there a way to get events that didnt match within the given time
>>>> frame.
>>>>
>>>> for example:
>>>>
>>>> define stream publisher (pid string, time string);
>>>> define stream subscriber (pid string, sid string, time string);
>>>>
>>>> from every (e1=publisher) -> e2=subscriber[e1.pid == pid]
>>>> within 10 sec
>>>> select e1.pid, e2.sid
>>>> insert into seen_in_last_10_sec_events;
>>>>
>>>>
>>>> so if i have matching event above, i will see it in
>>>> seen_in_last_10_sec_events, but is there a way to get all events or non
>>>> matched events during that last 10 seconds from publisher or subscriber ?
>>>>
>>>> also is there a way to perform intersection of events based on grouping
>>>> or time window ?
>>>>
>>>>
>>>> Thanks
>>>> Jay
>>>>
>>>> _______________________________________________
>>>> Dev mailing list
>>>> Dev@wso2.org
>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>
>>>>
>>>
>>>
>>> --
>>> Grainier Perera
>>> Senior Software Engineer
>>> Mobile : +94716122384 <+94%2071%20612%202384>
>>> WSO2 Inc. | http://wso2.com
>>> lean.enterprise.middleware
>>>
>>
>>
>> _______________________________________________
>> Dev mailing list
>> Dev@wso2.org
>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>
>>
>
>
> --
> *Gobinath** Loganathan*
> Graduate Student,
> Electrical and Computer Engineering,
> Western University.
> Email : slgobinath@gmail.com
> Blog : javahelps.com <http://www.javahelps.com/>
>
>
[Attachment #5 (text/html)]
<div dir="ltr">Hi Gobinath,<div><br></div><div>Thanks for the suggestion about \
absence pattern, but we wont be able to upgrade to Siddhi 4 anytime \
soon.</div><div><br></div><div>I am basically at the point where I can get all the \
relevant (subscribe) events that happened during the given interval of first arrival \
of publish events.</div><div><br></div><div>here AllPublisher = all registered \
publisher-subscriber events ( for each publisher event \
)</div><div><br></div><div><div>from registeryTable as s join #publisher as p on \
p.pid == s.pid</div><div>select p.pid, s.sid insert into \
AllPublisher;</div></div><div><br></div><div><br></div><div><div>from \
every(a=AllPublisher) -> s=subscriber[pid == a.pid]</div><div>within 1 \
sec</div><div>select a.pid, s.sid</div><div>insert into \
completed_jobs_in_1_sec;</div></div><div><br></div><div>now, i need to find out those \
events from publisher (a) that dint had any match from s within 1 \
sec.</div><div><br></div><div>which I was expecting to be available with " \
insert <b>expired events</b> into not_completed_jobs", but looks like expired \
events are only available when window is used. I am also looking at the code to see \
if i should add this.</div><div><br></div><div>Thanks</div><div>Jayesh</div><div><br>< \
/div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><div \
class="gmail_extra"><br><div class="gmail_quote">On Wed, Sep 27, 2017 at 4:21 AM, \
Gobinath <span dir="ltr"><<a href="mailto:slgobinath@gmail.com" \
target="_blank">slgobinath@gmail.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" \
style="font-family:verdana,sans-serif">Hi,</div><div class="gmail_default" \
style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" \
style="font-family:verdana,sans-serif">If you can use Siddhi 4 snapshot release, it \
can be done using the new feature 'Absent Pattern' added to Siddhi 4. The \
query to detect the events that do not match the condition within 10 seconds is given \
below:</div><div class="gmail_default" \
style="font-family:verdana,sans-serif"><br></div><div class="gmail_default"><span \
style="color:rgb(102,102,102)"><span style="font-family:monospace,monospace">from \
every e1=publisher -> not subscriber[e1.pid == pid] for 10 sec<br>select \
e1.pid<br>insert into not_completed_jobs_in_time;</span></span></div><div \
class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div \
class="gmail_default" style="font-family:verdana,sans-serif">The above query waits \
for 10 seconds from the arrival of every publisher event and if there is no \
subscriber event with an id satisfying the condition arrived within that waiting \
period, the id of the publisher event will be inserted into the \
not_completed_jobs_in_time stream.</div><div class="gmail_default" \
style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" \
style="font-family:verdana,sans-serif">I guess the official document for Siddhi 4 is \
under construction. So you can find more details about absent pattern at \
[1]</div><div class="gmail_default" \
style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" \
style="font-family:verdana,sans-serif">Still, Siddhi 4 is not production ready so I \
wonder whether you can use this feature or not.</div><div class="gmail_default" \
style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" \
style="font-family:verdana,sans-serif">[1] <a \
href="http://www.javahelps.com/2017/08/detect-absence-of-events-wso2-siddhi.html" \
target="_blank">http://www.javahelps.com/2017/<wbr>08/detect-absence-of-events-<wbr>wso2-siddhi.html</a><br></div><div \
class="gmail_default" style="font-family:verdana,sans-serif"> <br></div><div \
class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div \
class="gmail_default" style="font-family:verdana,sans-serif"><br></div></div><div \
class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On Tue, Sep 26, \
2017 at 10:05 PM, Jayesh Senjaliya <span dir="ltr"><<a \
href="mailto:jhsonline@gmail.com" target="_blank">jhsonline@gmail.com</a>></span> \
wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px \
#ccc solid;padding-left:1ex"><div dir="ltr">Hi Grainier,<div><br></div><div>ya, i \
came across that example page, but i think that does not work in my use-case which is \
as follow.</div><div><br></div><div>i have a publish event followed by multiple \
subscribe event for the same publish job.</div><div>now i want to catch if certain \
jobs (publish -> subscribe) has been finished with 10 sec.</div><div>I have all \
the registered jobs in db table, which i use to gather all the required \
publish-subscribe job events.</div><div><br></div><div><div>define table jobTable( \
pid string, sid string);</div></div><div><div>define stream pubStream (pid int, \
status string);</div><div>define stream subStream (pid int, sid int, status \
string);</div></div><div><br></div><div>-- this will get all the publish-> \
subscribe jobs events as master list</div><div><div>from pubStream as p join \
jobTable as t</div><div><span style="white-space:pre-wrap"> </span>on p.pid == \
t.pid</div><div>select p.pid, t.sid insert into \
allPSJobs;</div></div><div><br></div><div>-- this is where i need to do intersection \
where if subStream event is seen within 2 sec then remove that from master list ( \
allPSJobs ) if not include that in \
not_completed_jobs_in_time</div><div><br></div><div><div>from every ( a=allPSJobs ) \
-> s= subStream[sid == a.sid and pid==a.pid ]</div><div>within 2 sec \
</div><div>select s.pid, s.sid insert into \
completed_jobs_in_time;</div></div><div><br></div><div><br></div><div>hope that make \
sense from what i am trying to \
do.</div><div><br></div><div>Thanks</div><div>Jayesh</div><div><br></div><div><br></di \
v><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><div \
class="gmail_extra"><br><div class="gmail_quote">On Mon, Sep 25, 2017 at 8:39 AM, \
Grainier Perera <span dir="ltr"><<a href="mailto:grainier@wso2.com" \
target="_blank">grainier@wso2.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">Hi Jay,<div><br></div><div>You can try \
something similar to this to get non-matched events during last 10 secs; You can find \
some documentation on this as well; <a \
href="https://docs.wso2.com/display/CEP420/Sample+0111+-+Detecting+non-occurrences+with+Patterns" \
target="_blank">link</a> </div><span><div><br></div><blockquote class="gmail_quote" \
style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><br><font face="monospace, monospace">define \
stream publisher (pid string, time string);<br></font><font face="monospace, \
monospace">define stream subscriber (pid string, sid string, time string);</font> \
</blockquote></span><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><font face="monospace, \
monospace"><br></font><font face="monospace, monospace">from publisher#window.time(10 \
sec)<br></font><font face="monospace, monospace">select *<br></font><font \
face="monospace, monospace">insert expired events into expired_publisher;</font> \
</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><font face="monospace, \
monospace"><br></font><font face="monospace, monospace">from every pub=publisher \
-> sub=subscriber[pub.pid == pid] or exp=expired_publisher[pub.pid == \
pid]<br></font><font face="monospace, monospace">select pub.pid as pid, pub.time as \
time, sub.pid as subPid<br></font><font face="monospace, monospace">insert into \
filter_stream;</font> </blockquote><blockquote class="gmail_quote" style="margin:0px \
0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><font \
face="monospace, monospace"><br></font><font face="monospace, monospace">from \
filter_stream [(subPid is null)]<br></font><font face="monospace, monospace">select \
pid, time<br></font><font face="monospace, monospace">insert into \
not_seen_in_last_10_sec_events<wbr>;</font></blockquote><div><br></div><div>Moreover, \
I didn't get what you meant by "also is there a way to perform intersection \
of events based on grouping or time window ?" can you please elaborate on \
this?</div><div><br></div><div>Regards, </div></div><div \
class="gmail_extra"><br><div class="gmail_quote"><div><div \
class="m_7644740098464793087m_3615473517557963974h5">On Mon, Sep 25, 2017 at 11:02 \
AM, Jayesh Senjaliya <span dir="ltr"><<a href="mailto:jhsonline@gmail.com" \
target="_blank">jhsonline@gmail.com</a>></span> wrote:<br></div></div><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div><div \
class="m_7644740098464793087m_3615473517557963974h5"><div \
dir="ltr">Hi,<div><br></div><div>is there a way to get events that didnt match within \
the given time frame.</div><div><br></div><div>for example: \
</div><div><br></div><div><font style="color:rgb(80,0,80);font-size:12.8px" \
face="monospace, monospace">define stream publisher (pid string, time \
string);<br></font><font style="color:rgb(80,0,80);font-size:12.8px" face="monospace, \
monospace">define stream subscriber (pid string, sid string, time \
string);</font><br></div><div><font style="color:rgb(80,0,80);font-size:12.8px" \
face="monospace, monospace"><br></font></div><div><div>from every (e1=publisher) \
-> e2=subscriber[e1.pid == pid]</div><div>within 10 sec</div><div>select e1.pid, \
e2.sid</div><div>insert into \
seen_in_last_10_sec_events;</div></div><div><br></div><div><br></div><div>so if i \
have matching event above, i will see it in seen_in_last_10_sec_events, but is there \
a way to get all events or non matched events during that last 10 seconds from \
publisher or subscriber ?</div><div><br></div><div>also is there a way to perform \
intersection of events based on grouping or time window \
?</div><div><br></div><div><br></div><div>Thanks</div><div>Jay</div></div> \
<br></div></div>______________________________<wbr>_________________<br> Dev mailing \
list<br> <a href="mailto:Dev@wso2.org" target="_blank">Dev@wso2.org</a><br>
<a href="http://wso2.org/cgi-bin/mailman/listinfo/dev" rel="noreferrer" \
target="_blank">http://wso2.org/cgi-bin/mailma<wbr>n/listinfo/dev</a><br> \
<br></blockquote></div><span \
class="m_7644740098464793087m_3615473517557963974HOEnZb"><font \
color="#888888"><br><br clear="all"><span class="m_7644740098464793087HOEnZb"><font \
color="#888888"><div><br></div>-- <br><div \
class="m_7644740098464793087m_3615473517557963974m_1715430795549809218gmail_signature" \
data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div \
dir="ltr"><span><font color="#888888">Grainier Perera<br>Senior Software \
Engineer<br></font></span></div><div><span><font color="#888888">Mobile : <a \
href="tel:+94%2071%20612%202384" value="+94716122384" \
target="_blank">+94716122384</a><br></font></span></div><div dir="ltr"><span><font \
color="#888888">WSO2 Inc. | <a href="http://wso2.com" \
target="_blank">http://wso2.com</a><br>lean.enterprise.middleware</font></span><br></div></div></div></div></div></div>
</font></span></font></span></div>
</blockquote></div><br></div>
<br>______________________________<wbr>_________________<br>
Dev mailing list<br>
<a href="mailto:Dev@wso2.org" target="_blank">Dev@wso2.org</a><br>
<a href="http://wso2.org/cgi-bin/mailman/listinfo/dev" rel="noreferrer" \
target="_blank">http://wso2.org/cgi-bin/mailma<wbr>n/listinfo/dev</a><br> \
<br></blockquote></div><br><br clear="all"><br>-- <br></div></div><div \
class="m_7644740098464793087gmail_signature" data-smartmail="gmail_signature"><div \
dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div \
dir="ltr"><div dir="ltr"><div style="font-size:12.8px"><font face="verdana, \
sans-serif" size="1"><b><font color="#444444"><u>Gobinath</u></font></b></font><b \
style="font-family:verdana,sans-serif;font-size:x-small"><font color="#444444"> \
Loganathan</font></b></div><div dir="ltr"><div dir="ltr"><font face="verdana, \
sans-serif" size="1" color="#444444">Graduate Student, </font></div><div \
dir="ltr"><font face="verdana, sans-serif" size="1" color="#444444">Electrical and \
Computer Engineering,</font></div><div dir="ltr"><font face="verdana, sans-serif" \
size="1" color="#444444">Western University.</font></div></div><div \
style="font-size:12.8px"><font face="verdana, sans-serif" color="#666666" \
size="1">Email : <a href="mailto:slgobinath@gmail.com" \
target="_blank">slgobinath@gmail.com</a></font></div><div dir="ltr" \
style="font-size:12.8px"><font face="verdana, sans-serif" color="#666666" \
size="1">Blog : <a href="http://www.javahelps.com/" \
target="_blank">javahelps.com</a><div \
style="display:inline-block;width:16px;height:16px"> \
</div></font></div></div></div></div></div></div></div></div></div></div> </div>
</blockquote></div><br></div>
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic