'Re: [webkit-dev] EWS Comments on Bugzilla (Was: EWS now parses error logs in case of build failure)'

[prev in list] [next in list] [prev in thread] [next in thread] 

List:       webkit-dev
Subject:    Re: [webkit-dev] EWS Comments on Bugzilla (Was: EWS now parses error logs in case of build failure)
From:       Alexey Proskuryakov <ap () webkit ! org>
Date:       2019-11-05 17:04:17
Message-ID: E9C5B943-492F-41AD-A543-DABA5E7B519E () webkit ! org
[Download RAW message or body]

[Attachment #2 (multipart/alternative)]

> 4 нояб. 2019 г., в 1:37 PM, Ryosuke Niwa <rniwa@webkit.org> \
> написал(а): 
> 
> On Mon, Nov 4, 2019 at 9:40 AM Alexey Proskuryakov <ap@webkit.org \
> <mailto:ap@webkit.org>> wrote: 
> Can you elaborate on that, how exactly is e-mailing on first failure useful to \
> reviewers? 
> Getting rid of Bugzilla comments was one of the goals of EWS rewrite, based on \
> engineering feedback about noise in bugs and in e-mail, and I wholeheartedly agree \
> with this feedback. So I think that comments are generally undesirable. 
> Since I don't understand what your precise scenario is, I may be make straw man \
> arguments below, but here are some things that I think make the proposed behavior \
> unhelpful (add a comment on first failure, or when all EWSes pass). 
> 1. EWS comments in Bugzilla are so annoying that some people take the radical step \
> of manually hiding them. EWS history is archived anyway, there is no need to look \
> into comments for it. 
> 2. There are often many people CC'ed on the bug to whom EWS data is irrelevant or \
> even mysterious (e.g. reporters, web developers or non-reviewers). The noise is a \
> slight annoyance, discouraging further participation in the project. 
> 3. I believe that for most reviewers, the mode of operation is one of the two: (1) \
> do it when pinged directly, or (2) go over the review queue when one has the time. \
> Getting EWS comments helps neither. 
> 4. Commenting when all EWSes pass is not very practical - it's too often that we \
> have some stragglers that take days (or forever). I don't think that we can make it \
> reliable even if we start actively policing EWS responsiveness. 
> 5. The reviewer likely wants to know the state of multiple EWSes if they are going \
> to wait for EWS at all. What exactly are they going to do after getting an e-mail \
> that one EWS failed? 
> I often use a EWS failure as a signal to wait reviewing a patch. Otherwise, a bug \
> mail will stay in my inbox as one of items to get to. 
> I can see the usefulness in the somewhat unusual case of a super urgent patch. We \
> may want multiple people to watch it, so that members of CC list would go and ask \
> the patch author to update it with more urgency than e-mail allows for. I think \
> that opt-in is a better mechanism for that, so that people who opted in would \
> receive information about each EWS data point. 
> I think there is a value in knowing that a patch isn't ready instead of having to \
> open the bug to realize that.

So just to clarify, 

- a major part of how you get to review bugs is by being CC'ed, and you review them \
                when you have the time to read bugmail;
- and you don't open the bug in Bugzilla if there is already an EWS failure by the \
time you read the e-mail where review is requested?

That's clearly a valid benefit. In my mind, it probably doesn't outweigh the \
downsides. On the other hand, yours is a voice of someone who reviews way more \
patches than Maciej and me combined these days, so maybe more e-mail is an overall \
benefit to many of the reviewers.

- Alexey

> - R. Niwa
> > 3 нояб. 2019 г., в 6:58 PM, Maciej Stachowiak <mjs@apple.com \
> > <mailto:mjs@apple.com>> написал(а): 
> > 
> > I think they are useful to actual and potential reviewers. Direct email to the \
> > patch author is not something anyone can Cc themselves on, and is not archived, \
> > so seems like a strictly worse form of communication. 
> > > On Nov 2, 2019, at 9:34 AM, Alexey Proskuryakov <ap@apple.com \
> > > <mailto:ap@apple.com>> wrote: 
> > > 
> > > My preference is still e-mailing the patch author directly (possibly, also \
> > > having an option to opt in for anyone). Bugzilla comments will always be \
> > > irrelevant for most people CC'ed on the bug, and they are almost always \
> > > undesirable to keep within the discussion flow. 
> > > - Alexey
> > > 
> > > > 1 нояб. 2019 г., в 18:28, Aakash Jain <aakash_jain@apple.com \
> > > > <mailto:aakash_jain@apple.com>> написал(а): 
> > > > Sounds good. I prefer the single comment when the first failure occur. That \
> > > > way notification would be sent as soon as the first failure happens. 
> > > > I'll implement that (assuming it's acceptable to everyone).
> > > > 
> > > > Thanks
> > > > Aakash
> > > > 
> > > > > On Nov 1, 2019, at 8:35 PM, Maciej Stachowiak <mjs@apple.com \
> > > > > <mailto:mjs@apple.com>> wrote: 
> > > > > 
> > > > > How about only a single comment when the first failure occurs? (Or else \
> > > > > when all bots pass, if there is never a failure.) 
> > > > > This should help the author, the reviewer, and anyone else cc'd, without \
> > > > > being too spammy. 
> > > > > > On Nov 1, 2019, at 5:20 PM, Aakash Jain <aakash_jain@apple.com \
> > > > > > <mailto:aakash_jain@apple.com>> wrote: 
> > > > > > Hi Ryosuke,
> > > > > > 
> > > > > > Many people didn't like the noise by the EWS comments, and we removed the \
> > > > > > comments as per previous discussion in: \
> > > > > > https://lists.webkit.org/pipermail/webkit-dev/2019-June/030683.html \
> > > > > > <https://lists.webkit.org/pipermail/webkit-dev/2019-June/030683.html>. 
> > > > > > I agree with your point that having some kind of notification might be \
> > > > > > useful. 
> > > > > > I proposed some ideas in \
> > > > > > https://lists.webkit.org/pipermail/webkit-dev/2019-September/030798.html \
> > > > > > <https://lists.webkit.org/pipermail/webkit-dev/2019-September/030798.html>, \
> > > > > > but didn't get much feedback. If we can all agree on a solution, I can \
> > > > > > look into implementing it. 
> > > > > > Thanks
> > > > > > Aakash
> > > > > > 
> > > > > > > On Oct 30, 2019, at 1:03 AM,
> > > > > > > - R. Niwa
> > > > > > > <rniwa@webkit.org <mailto:rniwa@webkit.org>> wrote:
> > > > > > > 
> > > > > > > These enhancements are great. There is one feature of the old EWS that \
> > > > > > > I really miss, which is that I used to get emails when some EWS failed. \
> > > > > > > With new EWS, I have to keep checking back the bugzilla to see if any \
> > > > > > > of them have failed periodically. 
> > > > > > > Can we add a feature to opt into such an email notification? Maybe a \
> > > > > > > flag on a patch or JSON configuration file somewhere. 
> > > > > > > - R. Niwa
> > > > > > > 
> > > > > > > On Tue, Oct 29, 2019 at 4:05 PM Aakash Jain <aakash_jain@apple.com \
> > > > > > > <mailto:aakash_jain@apple.com>> wrote: Hi Everyone,
> > > > > > > 
> > > > > > > I am happy to announce another EWS feature.
> > > > > > > 
> > > > > > > From now on, in case of build failure, EWS will parse the errors and \
> > > > > > > display them in a separate 'errors' log. You wouldn't have to search \
> > > > > > > through thousands of lines of logs to find the error message. 
> > > > > > > For example, in https://ews-build.webkit.org/#/builders/16/builds/6054 \
> > > > > > > <https://ews-build.webkit.org/#/builders/16/builds/6054>, in step #7 \
> > > > > > > WebKit failed to compile. Complete logs (stdio) are 38,000+ lines, and \
> > > > > > > the error is not at the end of the logs. Normally, it requires some \
> > > > > > > searching through the logs to find the relevant errors. But now, there \
> > > > > > > is another 'errors' log, which contains just the relevant 11 lines \
> > > > > > > (containing error and few related lines to provide additional context). \
> > > > > > >  Hopefully this would save some time and efforts previously spent on \
> > > > > > > searching through the large logs. 
> > > > > > > Note that this information is not displayed in status-bubble tool-tip, \
> > > > > > > since this might be lot of text to display in the tooltip. My further \
> > > > > > > plan is to make this information more readily available, by adding it \
> > > > > > > to a custom designed page which will open on clicking the status bubble \
> > > > > > > https://webkit.org/b/197522 <https://webkit.org/b/197522> 
> > > > > > > Please let me know if you notice any issues or have any feedback.
> > > > > > > 
> > > > > > > Thanks
> > > > > > > Aakash
> > > > > > > 
> > > > > > > Reference: https://webkit.org/b/203418 <https://webkit.org/b/203418>
> > > > > > > _______________________________________________
> > > > > > > webkit-dev mailing list
> > > > > > > webkit-dev@lists.webkit.org <mailto:webkit-dev@lists.webkit.org>
> > > > > > > https://lists.webkit.org/mailman/listinfo/webkit-dev \
> > > > > > >                 <https://lists.webkit.org/mailman/listinfo/webkit-dev>
> > > > > > > -- 
> > > > > > > - R. Niwa
> > > > > > 
> > > > > > _______________________________________________
> > > > > > webkit-dev mailing list
> > > > > > webkit-dev@lists.webkit.org <mailto:webkit-dev@lists.webkit.org>
> > > > > > https://lists.webkit.org/mailman/listinfo/webkit-dev \
> > > > > > <https://lists.webkit.org/mailman/listinfo/webkit-dev>
> > > > > 
> > > > 
> > > > _______________________________________________
> > > > webkit-dev mailing list
> > > > webkit-dev@lists.webkit.org <mailto:webkit-dev@lists.webkit.org>
> > > > https://lists.webkit.org/mailman/listinfo/webkit-dev \
> > > > <https://lists.webkit.org/mailman/listinfo/webkit-dev>
> > > 
> > > - Alexey
> > > 
> > 
> 
> 
> _______________________________________________
> webkit-dev mailing list
> webkit-dev@lists.webkit.org <mailto:webkit-dev@lists.webkit.org>
> https://lists.webkit.org/mailman/listinfo/webkit-dev \
> <https://lists.webkit.org/mailman/listinfo/webkit-dev>

[Attachment #5 (unknown)]

<html><head><meta http-equiv="Content-Type" content="text/html; \
charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; \
line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote \
type="cite" class=""><div class="">4 нояб. 2019 г., в 1:37 PM, Ryosuke Niwa \
&lt;<a href="mailto:rniwa@webkit.org" class="">rniwa@webkit.org</a>&gt; \
написал(а):</div><br class="Apple-interchange-newline"><div class=""><div \
dir="ltr" class=""><br class=""><div class="gmail_quote"><div dir="ltr" \
class="gmail_attr">On Mon, Nov 4, 2019 at 9:40 AM Alexey Proskuryakov &lt;<a \
href="mailto:ap@webkit.org" class="">ap@webkit.org</a>&gt; wrote:<br \
class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div \
style="word-wrap:break-word;line-break:after-white-space" class=""><div class=""><br \
class=""></div><div class="">Can you elaborate on that, how exactly is e-mailing on \
first failure useful to reviewers?</div><div class=""><br class=""></div><div \
class="">Getting rid of Bugzilla comments was one of the goals of EWS rewrite, based \
on engineering feedback about noise in bugs and in e-mail, and I wholeheartedly agree \
with this feedback. So I think that comments are generally undesirable.</div><div \
class=""><br class=""></div><div class="">Since I don't understand what your precise \
scenario is, I may be make straw man arguments below, but here are some things that I \
think make the proposed behavior unhelpful (add a comment on first failure, or when \
all EWSes pass).</div><div class=""><br class=""></div><div class="">1. EWS comments \
in Bugzilla are so annoying that some people take the radical step of manually hiding \
them. EWS history is archived anyway, there is no need to look into comments for \
it.</div><div class=""><br class=""></div><div class="">2. There are often many \
people CC'ed on the bug to whom EWS data is irrelevant or even mysterious (e.g. \
reporters, web developers or non-reviewers). The noise is a slight annoyance, \
discouraging further participation in the project.</div><div class=""><br \
class=""></div><div class="">3. I believe that for most reviewers, the mode of \
operation is one of the two: (1) do it when pinged directly, or (2) go over the \
review queue when one has the time. Getting EWS comments helps neither.</div><div \
class=""><br class=""></div><div class="">4. Commenting when all EWSes pass is not \
very practical - it's too often that we have some stragglers that take days (or \
forever). I don't think that we can make it reliable even if we start actively \
policing EWS responsiveness.</div><div class=""><br class=""></div><div class="">5. \
The reviewer likely wants to know the state of multiple EWSes if they are going to \
wait for EWS at all. What exactly are they going to do after getting an e-mail that \
one EWS failed?</div></div></blockquote><div class=""><br class=""></div><div \
class="">I often use a EWS failure as a signal to wait reviewing a patch. Otherwise, \
a bug mail will stay in my inbox as one of items to get to.</div><div class=""><br \
class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div \
style="word-wrap:break-word;line-break:after-white-space" class=""><div class="">I \
can see the usefulness in the somewhat unusual case of a super urgent patch. We may \
want multiple people to watch it, so that members of CC list would go and ask the \
patch author to update it with more urgency than e-mail allows for. I think that \
opt-in is a better mechanism for that, so that people who opted in would receive \
information about each EWS data point.</div></div></blockquote><div class=""><br \
class=""></div><div class="">I think there is a value in knowing that a patch isn't \
ready instead of having to open the bug to realize \
that.</div></div></div></div></blockquote><div><br class=""></div><div>So just to \
clarify,&nbsp;</div><div><br class=""></div><div>- a major part of how you get to \
review bugs is by being CC'ed, and you review them when you have the time to read \
bugmail;</div><div>- and you don't open the bug in Bugzilla if there is already an \
EWS failure by the time you read the e-mail where review is requested?</div><div><br \
class=""></div><div>That's clearly a valid benefit. In my mind, it probably doesn't \
outweigh the downsides. On the other hand, yours is a voice of someone who reviews \
way more patches than Maciej and me combined these days, so maybe more e-mail is an \
overall benefit to many of the reviewers.</div><div><br class=""></div><div><div \
class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: \
after-white-space;" class=""><div class="">- Alexey</div><div class=""><br \
class=""></div></div><br class="Apple-interchange-newline"></div></div><br \
class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div \
class="gmail_quote"><div class="">- R. Niwa</div><blockquote class="gmail_quote" \
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div \
style="word-wrap:break-word;line-break:after-white-space" class=""><div \
class=""><blockquote type="cite" class=""><div class="">3 нояб. 2019 г., в 6:58 \
PM, Maciej Stachowiak &lt;<a href="mailto:mjs@apple.com" target="_blank" \
class="">mjs@apple.com</a>&gt; написал(а):</div><br class=""><div \
class=""><div class=""><br class="">I think they are useful to actual and potential \
reviewers. Direct email to the patch author is not something anyone can Cc themselves \
on, and is not archived, so seems like a strictly worse form of communication.<br \
class=""><br class=""><blockquote type="cite" class="">On Nov 2, 2019, at 9:34 AM, \
Alexey Proskuryakov &lt;<a href="mailto:ap@apple.com" target="_blank" \
class="">ap@apple.com</a>&gt; wrote:<br class=""><br class=""><br class="">My \
preference is still e-mailing the patch author directly (possibly, also having an \
option to opt in for anyone). Bugzilla comments will always be irrelevant for most \
people CC'ed on the bug, and they are almost always undesirable to keep within the \
discussion flow.<br class=""><br class="">- Alexey<br class=""><br \
class=""><blockquote type="cite" class="">1 нояб. 2019 г., в 18:28, Aakash Jain \
&lt;<a href="mailto:aakash_jain@apple.com" target="_blank" \
class="">aakash_jain@apple.com</a>&gt; написал(а):<br class=""><br \
class="">Sounds good. I prefer the single comment when the first failure occur. That \
way notification would be sent as soon as the first failure happens.<br class=""><br \
class="">I'll implement that (assuming it's acceptable to everyone).<br class=""><br \
class="">Thanks<br class="">Aakash<br class=""><br class=""><blockquote type="cite" \
class="">On Nov 1, 2019, at 8:35 PM, Maciej Stachowiak &lt;<a \
href="mailto:mjs@apple.com" target="_blank" class="">mjs@apple.com</a>&gt; wrote:<br \
class=""><br class=""><br class="">How about only a single comment when the first \
failure occurs? (Or else when all bots pass, if there is never a failure.)<br \
class=""><br class="">This should help the author, the reviewer, and anyone else \
cc'd, without being too spammy.<br class=""><br class=""><blockquote type="cite" \
class="">On Nov 1, 2019, at 5:20 PM, Aakash Jain &lt;<a \
href="mailto:aakash_jain@apple.com" target="_blank" \
class="">aakash_jain@apple.com</a>&gt; wrote:<br class=""><br class="">Hi Ryosuke,<br \
class=""><br class="">Many people didn't like the noise by the EWS comments, and we \
removed the comments as per previous discussion in: <a \
href="https://lists.webkit.org/pipermail/webkit-dev/2019-June/030683.html" \
target="_blank" class="">https://lists.webkit.org/pipermail/webkit-dev/2019-June/030683.html</a>.<br \
class=""><br class="">I agree with your point that having some kind of notification \
might be useful.<br class=""><br class="">I proposed some ideas in <a \
href="https://lists.webkit.org/pipermail/webkit-dev/2019-September/030798.html" \
target="_blank" class="">https://lists.webkit.org/pipermail/webkit-dev/2019-September/030798.html</a>, \
but didn't get much feedback. If we can all agree on a solution, I can look into \
implementing it.<br class=""><br class="">Thanks<br class="">Aakash<br class=""><br \
class=""><blockquote type="cite" class="">On Oct 30, 2019, at 1:03 AM,<br clear="all" \
class=""><div class=""><div dir="ltr" class="gmail_signature" \
data-smartmail="gmail_signature">- R. Niwa</div></div> &lt;<a \
href="mailto:rniwa@webkit.org" target="_blank" class="">rniwa@webkit.org</a>&gt; \
wrote:<br class=""><br class="">These enhancements are great. There is one feature of \
the old EWS that I really miss, which is that I used to get emails when some EWS \
failed. With new EWS, I have to keep checking back the bugzilla to see if any of them \
have failed periodically.<br class=""><br class="">Can we add a feature to opt into \
such an email notification? Maybe a flag on a patch or JSON configuration file \
somewhere.<br class=""><br class="">- R. Niwa<br class=""><br class="">On Tue, Oct \
29, 2019 at 4:05 PM Aakash Jain &lt;<a href="mailto:aakash_jain@apple.com" \
target="_blank" class="">aakash_jain@apple.com</a>&gt; wrote:<br class="">Hi \
Everyone,<br class=""><br class="">I am happy to announce another EWS feature.<br \
class=""><br class="">From now on, in case of build failure, EWS will parse the \
errors and display them in a separate 'errors' log. You wouldn't have to search \
through thousands of lines of logs to find the error message.<br class=""><br \
class="">For example, in <a \
href="https://ews-build.webkit.org/#/builders/16/builds/6054" target="_blank" \
class="">https://ews-build.webkit.org/#/builders/16/builds/6054</a>, in step #7 \
WebKit failed to compile. Complete logs (stdio) are 38,000+ lines, and the error is \
not at the end of the logs. Normally, it requires some searching through the logs to \
find the relevant errors. But now, there is another 'errors' log, which contains just \
the relevant 11 lines (containing error and few related lines to provide additional \
context).<br class=""><br class="">Hopefully this would save some time and efforts \
previously spent on searching through the large logs.<br class=""><br class="">Note \
that this information is not displayed in status-bubble tool-tip, since this might be \
lot of text to display in the tooltip. My further plan is to make this information \
more readily available, by adding it to a custom designed page which will open on \
clicking the status bubble <a href="https://webkit.org/b/197522" target="_blank" \
class="">https://webkit.org/b/197522</a><br class=""><br class="">Please let me know \
if you notice any issues or have any feedback.<br class=""><br class="">Thanks<br \
class="">Aakash<br class=""><br class="">Reference: <a \
href="https://webkit.org/b/203418" target="_blank" \
class="">https://webkit.org/b/203418</a><br \
class="">_______________________________________________<br class="">webkit-dev \
mailing list<br class=""><a href="mailto:webkit-dev@lists.webkit.org" target="_blank" \
class="">webkit-dev@lists.webkit.org</a><br class=""><a \
href="https://lists.webkit.org/mailman/listinfo/webkit-dev" target="_blank" \
class="">https://lists.webkit.org/mailman/listinfo/webkit-dev</a><br class="">-- <br \
class="">- R. Niwa<br class=""></blockquote><br \
class="">_______________________________________________<br class="">webkit-dev \
mailing list<br class=""><a href="mailto:webkit-dev@lists.webkit.org" target="_blank" \
class="">webkit-dev@lists.webkit.org</a><br class=""><a \
href="https://lists.webkit.org/mailman/listinfo/webkit-dev" target="_blank" \
class="">https://lists.webkit.org/mailman/listinfo/webkit-dev</a><br \
class=""></blockquote><br class=""></blockquote><br \
class="">_______________________________________________<br class="">webkit-dev \
mailing list<br class=""><a href="mailto:webkit-dev@lists.webkit.org" target="_blank" \
class="">webkit-dev@lists.webkit.org</a><br class=""><a \
href="https://lists.webkit.org/mailman/listinfo/webkit-dev" target="_blank" \
class="">https://lists.webkit.org/mailman/listinfo/webkit-dev</a><br \
class=""></blockquote><br class="">- Alexey<br class=""><br class=""></blockquote><br \
class=""></div></div></blockquote></div><br class=""><div class=""> <div \
style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: \
none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class=""><div \
class=""><br class=""></div></div></div></div>_______________________________________________<br \
class=""> webkit-dev mailing list<br class="">
<a href="mailto:webkit-dev@lists.webkit.org" target="_blank" \
class="">webkit-dev@lists.webkit.org</a><br class=""> <a \
href="https://lists.webkit.org/mailman/listinfo/webkit-dev" rel="noreferrer" \
target="_blank" class="">https://lists.webkit.org/mailman/listinfo/webkit-dev</a><br \
class=""> </blockquote></div></div>
</div></blockquote></div><br class=""><div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: \
start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; \
word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; \
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div \
class=""><br class=""></div></div></div></body></html>

_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

[prev in list] [next in list] [prev in thread] [next in thread] 