'Re: [webkit-dev] EWS Comments on Bugzilla (Was: EWS now parses error logs in case of build failure)'

[prev in list] [next in list] [prev in thread] [next in thread] 

List:       webkit-dev
Subject:    Re: [webkit-dev] EWS Comments on Bugzilla (Was: EWS now parses error logs in case of build failure)
From:       Alexey Proskuryakov <ap () webkit ! org>
Date:       2019-11-04 17:39:47
Message-ID: B5BD4841-127C-4909-AA3A-3B7CF853004E () webkit ! org
[Download RAW message or body]

[Attachment #2 (multipart/alternative)]

Can you elaborate on that, how exactly is e-mailing on first failure useful to \
reviewers?

Getting rid of Bugzilla comments was one of the goals of EWS rewrite, based on \
engineering feedback about noise in bugs and in e-mail, and I wholeheartedly agree \
with this feedback. So I think that comments are generally undesirable.

Since I don't understand what your precise scenario is, I may be make straw man \
arguments below, but here are some things that I think make the proposed behavior \
unhelpful (add a comment on first failure, or when all EWSes pass).

1. EWS comments in Bugzilla are so annoying that some people take the radical step of \
manually hiding them. EWS history is archived anyway, there is no need to look into \
comments for it.

2. There are often many people CC'ed on the bug to whom EWS data is irrelevant or \
even mysterious (e.g. reporters, web developers or non-reviewers). The noise is a \
slight annoyance, discouraging further participation in the project.

3. I believe that for most reviewers, the mode of operation is one of the two: (1) do \
it when pinged directly, or (2) go over the review queue when one has the time. \
Getting EWS comments helps neither.

4. Commenting when all EWSes pass is not very practical - it's too often that we have \
some stragglers that take days (or forever). I don't think that we can make it \
reliable even if we start actively policing EWS responsiveness.

5. The reviewer likely wants to know the state of multiple EWSes if they are going to \
wait for EWS at all. What exactly are they going to do after getting an e-mail that \
one EWS failed?

6. More bugmail delays response, especially for active project members who are CC'ed \
on a lot of bugs. I personally started reading bugmail more frequently now, knowing \
that there is more signal and less noise.

I can see the usefulness in the somewhat unusual case of a super urgent patch. We may \
want multiple people to watch it, so that members of CC list would go and ask the \
patch author to update it with more urgency than e-mail allows for. I think that \
opt-in is a better mechanism for that, so that people who opted in would receive \
information about each EWS data point.

- Alexey

> 3 нояб. 2019 г., в 6:58 PM, Maciej Stachowiak <mjs@apple.com> \
> написал(а): 
> 
> I think they are useful to actual and potential reviewers. Direct email to the \
> patch author is not something anyone can Cc themselves on, and is not archived, so \
> seems like a strictly worse form of communication. 
> > On Nov 2, 2019, at 9:34 AM, Alexey Proskuryakov <ap@apple.com> wrote:
> > 
> > 
> > My preference is still e-mailing the patch author directly (possibly, also having \
> > an option to opt in for anyone). Bugzilla comments will always be irrelevant for \
> > most people CC'ed on the bug, and they are almost always undesirable to keep \
> > within the discussion flow. 
> > - Alexey
> > 
> > > 1 нояб. 2019 г., в 18:28, Aakash Jain <aakash_jain@apple.com> \
> > > написал(а): 
> > > Sounds good. I prefer the single comment when the first failure occur. That way \
> > > notification would be sent as soon as the first failure happens. 
> > > I'll implement that (assuming it's acceptable to everyone).
> > > 
> > > Thanks
> > > Aakash
> > > 
> > > > On Nov 1, 2019, at 8:35 PM, Maciej Stachowiak <mjs@apple.com> wrote:
> > > > 
> > > > 
> > > > How about only a single comment when the first failure occurs? (Or else when \
> > > > all bots pass, if there is never a failure.) 
> > > > This should help the author, the reviewer, and anyone else cc'd, without \
> > > > being too spammy. 
> > > > > On Nov 1, 2019, at 5:20 PM, Aakash Jain <aakash_jain@apple.com> wrote:
> > > > > 
> > > > > Hi Ryosuke,
> > > > > 
> > > > > Many people didn't like the noise by the EWS comments, and we removed the \
> > > > > comments as per previous discussion in: \
> > > > > https://lists.webkit.org/pipermail/webkit-dev/2019-June/030683.html. 
> > > > > I agree with your point that having some kind of notification might be \
> > > > > useful. 
> > > > > I proposed some ideas in \
> > > > > https://lists.webkit.org/pipermail/webkit-dev/2019-September/030798.html, \
> > > > > but didn't get much feedback. If we can all agree on a solution, I can look \
> > > > > into implementing it. 
> > > > > Thanks
> > > > > Aakash
> > > > > 
> > > > > > On Oct 30, 2019, at 1:03 AM, Ryosuke Niwa <rniwa@webkit.org> wrote:
> > > > > > 
> > > > > > These enhancements are great. There is one feature of the old EWS that I \
> > > > > > really miss, which is that I used to get emails when some EWS failed. \
> > > > > > With new EWS, I have to keep checking back the bugzilla to see if any of \
> > > > > > them have failed periodically. 
> > > > > > Can we add a feature to opt into such an email notification? Maybe a flag \
> > > > > > on a patch or JSON configuration file somewhere. 
> > > > > > - R. Niwa
> > > > > > 
> > > > > > On Tue, Oct 29, 2019 at 4:05 PM Aakash Jain <aakash_jain@apple.com> \
> > > > > > wrote: Hi Everyone,
> > > > > > 
> > > > > > I am happy to announce another EWS feature.
> > > > > > 
> > > > > > From now on, in case of build failure, EWS will parse the errors and \
> > > > > > display them in a separate 'errors' log. You wouldn't have to search \
> > > > > > through thousands of lines of logs to find the error message. 
> > > > > > For example, in https://ews-build.webkit.org/#/builders/16/builds/6054, \
> > > > > > in step #7 WebKit failed to compile. Complete logs (stdio) are 38,000+ \
> > > > > > lines, and the error is not at the end of the logs. Normally, it requires \
> > > > > > some searching through the logs to find the relevant errors. But now, \
> > > > > > there is another 'errors' log, which contains just the relevant 11 lines \
> > > > > > (containing error and few related lines to provide additional context). 
> > > > > > Hopefully this would save some time and efforts previously spent on \
> > > > > > searching through the large logs. 
> > > > > > Note that this information is not displayed in status-bubble tool-tip, \
> > > > > > since this might be lot of text to display in the tooltip. My further \
> > > > > > plan is to make this information more readily available, by adding it to \
> > > > > > a custom designed page which will open on clicking the status bubble \
> > > > > > https://webkit.org/b/197522 
> > > > > > Please let me know if you notice any issues or have any feedback.
> > > > > > 
> > > > > > Thanks
> > > > > > Aakash
> > > > > > 
> > > > > > Reference: https://webkit.org/b/203418
> > > > > > _______________________________________________
> > > > > > webkit-dev mailing list
> > > > > > webkit-dev@lists.webkit.org
> > > > > > https://lists.webkit.org/mailman/listinfo/webkit-dev
> > > > > > -- 
> > > > > > - R. Niwa
> > > > > 
> > > > > _______________________________________________
> > > > > webkit-dev mailing list
> > > > > webkit-dev@lists.webkit.org
> > > > > https://lists.webkit.org/mailman/listinfo/webkit-dev
> > > > 
> > > 
> > > _______________________________________________
> > > webkit-dev mailing list
> > > webkit-dev@lists.webkit.org
> > > https://lists.webkit.org/mailman/listinfo/webkit-dev
> > 
> > - Alexey
> > 
> 

[Attachment #5 (unknown)]

<html><head><meta http-equiv="Content-Type" content="text/html; \
charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; \
line-break: after-white-space;" class=""><div class=""><br class=""></div><div \
class="">Can you elaborate on that, how exactly is e-mailing on first failure useful \
to reviewers?</div><div class=""><br class=""></div><div class="">Getting rid of \
Bugzilla comments was one of the goals of EWS rewrite, based on engineering feedback \
about noise in bugs and in e-mail, and I wholeheartedly agree with this feedback. So \
I think that comments are generally undesirable.</div><div class=""><br \
class=""></div><div class="">Since I don't understand what your precise scenario is, \
I may be make straw man arguments below, but here are some things that I think make \
the proposed behavior unhelpful (add a comment on first failure, or when all EWSes \
pass).</div><div class=""><br class=""></div><div class="">1. EWS comments in \
Bugzilla are so annoying that some people take the radical step of manually hiding \
them. EWS history is archived anyway, there is no need to look into comments for \
it.</div><div class=""><br class=""></div><div class="">2. There are often many \
people CC'ed on the bug to whom EWS data is irrelevant or even mysterious (e.g. \
reporters, web developers or non-reviewers). The noise is a slight annoyance, \
discouraging further participation in the project.</div><div class=""><br \
class=""></div><div class="">3. I believe that for most reviewers, the mode of \
operation is one of the two: (1) do it when pinged directly, or (2) go over the \
review queue when one has the time. Getting EWS comments helps neither.</div><div \
class=""><br class=""></div><div class="">4. Commenting when all EWSes pass is not \
very practical - it's too often that we have some stragglers that take days (or \
forever). I don't think that we can make it reliable even if we start actively \
policing EWS responsiveness.</div><div class=""><br class=""></div><div class="">5. \
The reviewer likely wants to know the state of multiple EWSes if they are going to \
wait for EWS at all. What exactly are they going to do after getting an e-mail that \
one EWS failed?</div><div class=""><br class=""></div><div class="">6. More bugmail \
delays response, especially for active project members who are CC'ed on a lot of \
bugs. I personally started reading bugmail more frequently now, knowing that there is \
more signal and less noise.</div><div class=""><br class=""></div><div class="">I can \
see the usefulness in the somewhat unusual case of a super urgent patch. We may want \
multiple people to watch it, so that members of CC list would go and ask the patch \
author to update it with more urgency than e-mail allows for. I think that opt-in is \
a better mechanism for that, so that people who opted in would receive information \
about each EWS data point.</div><br class=""><div><div class=""><div \
style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: \
after-white-space;" class=""><div class="">- Alexey</div><div class=""><br \
class=""></div></div><br class="Apple-interchange-newline"></div><blockquote \
type="cite" class=""><div class="">3 нояб. 2019 г., в 6:58 PM, Maciej \
Stachowiak &lt;<a href="mailto:mjs@apple.com" class="">mjs@apple.com</a>&gt; \
написал(а):</div><br class="Apple-interchange-newline"><div class=""><div \
class=""><br class="">I think they are useful to actual and potential reviewers. \
Direct email to the patch author is not something anyone can Cc themselves on, and is \
not archived, so seems like a strictly worse form of communication.<br class=""><br \
class=""><blockquote type="cite" class="">On Nov 2, 2019, at 9:34 AM, Alexey \
Proskuryakov &lt;<a href="mailto:ap@apple.com" class="">ap@apple.com</a>&gt; \
wrote:<br class=""><br class=""><br class="">My preference is still e-mailing the \
patch author directly (possibly, also having an option to opt in for anyone). \
Bugzilla comments will always be irrelevant for most people CC'ed on the bug, and \
they are almost always undesirable to keep within the discussion flow.<br \
class=""><br class="">- Alexey<br class=""><br class=""><blockquote type="cite" \
class="">1 нояб. 2019 г., в 18:28, Aakash Jain &lt;<a \
href="mailto:aakash_jain@apple.com" class="">aakash_jain@apple.com</a>&gt; \
написал(а):<br class=""><br class="">Sounds good. I prefer the single comment \
when the first failure occur. That way notification would be sent as soon as the \
first failure happens.<br class=""><br class="">I'll implement that (assuming it's \
acceptable to everyone).<br class=""><br class="">Thanks<br class="">Aakash<br \
class=""><br class=""><blockquote type="cite" class="">On Nov 1, 2019, at 8:35 PM, \
Maciej Stachowiak &lt;<a href="mailto:mjs@apple.com" class="">mjs@apple.com</a>&gt; \
wrote:<br class=""><br class=""><br class="">How about only a single comment when the \
first failure occurs? (Or else when all bots pass, if there is never a failure.)<br \
class=""><br class="">This should help the author, the reviewer, and anyone else \
cc'd, without being too spammy.<br class=""><br class=""><blockquote type="cite" \
class="">On Nov 1, 2019, at 5:20 PM, Aakash Jain &lt;<a \
href="mailto:aakash_jain@apple.com" class="">aakash_jain@apple.com</a>&gt; wrote:<br \
class=""><br class="">Hi Ryosuke,<br class=""><br class="">Many people didn't like \
the noise by the EWS comments, and we removed the comments as per previous discussion \
in: <a href="https://lists.webkit.org/pipermail/webkit-dev/2019-June/030683.html" \
class="">https://lists.webkit.org/pipermail/webkit-dev/2019-June/030683.html</a>.<br \
class=""><br class="">I agree with your point that having some kind of notification \
might be useful.<br class=""><br class="">I proposed some ideas in <a \
href="https://lists.webkit.org/pipermail/webkit-dev/2019-September/030798.html" \
class="">https://lists.webkit.org/pipermail/webkit-dev/2019-September/030798.html</a>, \
but didn't get much feedback. If we can all agree on a solution, I can look into \
implementing it.<br class=""><br class="">Thanks<br class="">Aakash<br class=""><br \
class=""><blockquote type="cite" class="">On Oct 30, 2019, at 1:03 AM, Ryosuke Niwa \
&lt;<a href="mailto:rniwa@webkit.org" class="">rniwa@webkit.org</a>&gt; wrote:<br \
class=""><br class="">These enhancements are great. There is one feature of the old \
EWS that I really miss, which is that I used to get emails when some EWS failed. With \
new EWS, I have to keep checking back the bugzilla to see if any of them have failed \
periodically.<br class=""><br class="">Can we add a feature to opt into such an email \
notification? Maybe a flag on a patch or JSON configuration file somewhere.<br \
class=""><br class="">- R. Niwa<br class=""><br class="">On Tue, Oct 29, 2019 at 4:05 \
PM Aakash Jain &lt;<a href="mailto:aakash_jain@apple.com" \
class="">aakash_jain@apple.com</a>&gt; wrote:<br class="">Hi Everyone,<br \
class=""><br class="">I am happy to announce another EWS feature.<br class=""><br \
class="">From now on, in case of build failure, EWS will parse the errors and display \
them in a separate 'errors' log. You wouldn't have to search through thousands of \
lines of logs to find the error message.<br class=""><br class="">For example, in <a \
href="https://ews-build.webkit.org/#/builders/16/builds/6054" \
class="">https://ews-build.webkit.org/#/builders/16/builds/6054</a>, in step #7 \
WebKit failed to compile. Complete logs (stdio) are 38,000+ lines, and the error is \
not at the end of the logs. Normally, it requires some searching through the logs to \
find the relevant errors. But now, there is another 'errors' log, which contains just \
the relevant 11 lines (containing error and few related lines to provide additional \
context).<br class=""><br class="">Hopefully this would save some time and efforts \
previously spent on searching through the large logs.<br class=""><br class="">Note \
that this information is not displayed in status-bubble tool-tip, since this might be \
lot of text to display in the tooltip. My further plan is to make this information \
more readily available, by adding it to a custom designed page which will open on \
clicking the status bubble <a href="https://webkit.org/b/197522" \
class="">https://webkit.org/b/197522</a><br class=""><br class="">Please let me know \
if you notice any issues or have any feedback.<br class=""><br class="">Thanks<br \
class="">Aakash<br class=""><br class="">Reference: <a \
href="https://webkit.org/b/203418" class="">https://webkit.org/b/203418</a><br \
class="">_______________________________________________<br class="">webkit-dev \
mailing list<br class=""><a href="mailto:webkit-dev@lists.webkit.org" \
class="">webkit-dev@lists.webkit.org</a><br \
class="">https://lists.webkit.org/mailman/listinfo/webkit-dev<br class="">-- <br \
class="">- R. Niwa<br class=""></blockquote><br \
class="">_______________________________________________<br class="">webkit-dev \
mailing list<br class=""><a href="mailto:webkit-dev@lists.webkit.org" \
class="">webkit-dev@lists.webkit.org</a><br \
class="">https://lists.webkit.org/mailman/listinfo/webkit-dev<br \
class=""></blockquote><br class=""></blockquote><br \
class="">_______________________________________________<br class="">webkit-dev \
mailing list<br class=""><a href="mailto:webkit-dev@lists.webkit.org" \
class="">webkit-dev@lists.webkit.org</a><br \
class="">https://lists.webkit.org/mailman/listinfo/webkit-dev<br \
class=""></blockquote><br class="">- Alexey<br class=""><br class=""></blockquote><br \
class=""></div></div></blockquote></div><br class=""><div class=""> <div \
style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; \
text-indent: 0px; text-transform: none; white-space: normal; widows: auto; \
word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; \
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div \
class=""><br class=""></div></div></div></body></html>

_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

[prev in list] [next in list] [prev in thread] [next in thread] 