[prev in list] [next in list] [prev in thread] [next in thread]
List: logback-dev
Subject: [logback-dev] [JIRA] (LOGBACK-1046) logs appenders with prudent mode do not recover after "no space
From: "Nadav Wexler (JIRA)" <noreply-jira () qos ! ch>
Date: 2015-02-01 14:38:12
Message-ID: JIRA.12001.1422801435828.1692.1422801492030 () pixie
[Download RAW message or body]
[Attachment #2 (text/html)]
<style>
/* Changing the layout to use less space for mobiles */
@media screen and (max-device-width: 480px), screen and \
(-webkit-min-device-pixel-ratio: 2) { #email-body { min-width: 30em !important; }
#email-page { padding: 8px !important; }
#email-banner { padding: 8px 8px 0 8px !important; }
#email-avatar { margin: 1px 8px 8px 0 !important; padding: 0 !important; }
#email-fields { padding: 0 8px 8px 8px !important; }
#email-gutter { width: 0 !important; }
}
</style>
<div id="email-body">
<table id="email-wrap" align="center" border="0" cellpadding="0" cellspacing="0" \
style="background-color:#f0f0f0;color:#000000;width:100%;"> <tr valign="top">
<td id="email-page" style="padding:16px !important;">
<table align="center" border="0" cellpadding="0" cellspacing="0" \
style="background-color:#ffffff;border:1px solid #bbbbbb;color:#000000;width:100%;"> \
<tr valign="top"> <td bgcolor="#003366" \
style="background-color:#003366;color:#ffffff;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;line-height:1;"><img \
src="http://jira.qos.ch/s/en_USb9v8he-418945332/850/25/_/jira-logo-scaled.png" alt="" \
style="vertical-align:top;" /></td> </tr><tr valign="top">
<td id="email-banner" style="padding:32px 32px 0 32px;">
<table align="left" border="0" cellpadding="0" cellspacing="0" \
width="100%" style="width:100%;"> <tr valign="top">
<td style="color:#505050;font-family:Arial,FreeSans,Helvetica,sans-serif;padding:0;">
<img id="email-avatar" src="http://jira.qos.ch/secure/useravatar?avatarId=10122" \
alt="" height="48" width="48" border="0" align="left" style="padding:0;margin: 0 16px \
16px 0;" />
<div id="email-action" style="padding: 0 0 8px \
0;font-size:12px;line-height:18px;"> <a class="user-hover" rel="nadavwe" \
id="email_nadavwe" href="http://jira.qos.ch/secure/ViewProfile.jspa?name=nadavwe" \
style="color:#326ca6;">Nadav Wexler</a> created <img \
src="http://jira.qos.ch/images/icons/issuetypes/bug.png" height="16" width="16" \
border="0" align="absmiddle" alt="Bug"> <a \
style='color:#326ca6;text-decoration:none;' \
href='http://jira.qos.ch/browse/LOGBACK-1046'>LOGBACK-1046</a> </div>
<div id="email-summary" \
style="font-size:16px;line-height:20px;padding:2px 0 16px 0;"> <a \
style='color:#326ca6;text-decoration:none;' \
href='http://jira.qos.ch/browse/LOGBACK-1046'><strong>logs appenders with prudent \
mode do not recover after "no space left on device" error</strong></a> \
</div> </td>
</tr>
</table>
</td>
</tr>
<tr valign="top">
<td id="email-fields" style="padding:0 32px 32px 32px;">
<table border="0" cellpadding="0" cellspacing="0" \
style="padding:0;text-align:left;width:100%;" width="100%"> <tr valign="top">
<td id="email-gutter" style="width:64px;white-space:nowrap;"></td>
<td>
<table border="0" cellpadding="0" cellspacing="0" width="100%">
<tr valign="top">
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 \
10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Issue Type:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 \
0 10px 0;width:100%;">
<img src="http://jira.qos.ch/images/icons/issuetypes/bug.png" \
height="16" width="16" border="0" align="absmiddle" alt="Bug"> Bug </td>
</tr> <tr valign="top">
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 \
10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Assignee:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 \
0 10px 0;width:100%;"> <a class="user-hover" rel="logback-dev@qos.ch" \
id="email_logback-dev@qos.ch" \
href="http://jira.qos.ch/secure/ViewProfile.jspa?name=logback-dev%40qos.ch" \
style="color:#326ca6;">Logback dev list</a> </td>
</tr> <tr valign="top">
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 \
10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Components:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 \
0 10px 0;width:100%;"> logback-core </td>
</tr>
<tr valign="top">
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 \
10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Created:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 \
0 10px 0;width:100%;"> 01/Feb/15 3:37 PM
</td>
</tr> <tr valign="top">
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 \
10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Description:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 \
0 10px 0;width:100%;"> <p style='margin-top:0;margin-bottom:10px;'>Hey,<br/>
I've seen that this error has been around for some time and I hope that this \
description is complete and will be helpful in reproducing and fixing.</p>
<p style='margin-top:0;margin-bottom:10px;'>System:<br/>
a logback.xml with a file appender with prudent=true.<br/>
the log path should be to a volume with little available space.</p>
<p style='margin-top:0;margin-bottom:10px;'>Scenario:<br/>
start writing to the log file. as soon as the space is depleted, errors start \
happening:<br/> 15:44:40,595 |-ERROR in \
c.q.l.c.recovery.ResilientFileOutputStream@1944673755 - IO failure while writing to \
file <span class="error">[/Volumes/TESTVOL/logs/my-log.2015-02-01.log]</span> \
java.io.IOException: No space left on device<br/> at java.io.IOException: No space \
left on device<br/> at at java.io.FileOutputStream.writeBytes(Native Method)<br/>
at at java.io.FileOutputStream.write(FileOutputStream.java:345)<br/>
at at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)<br/>
at at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)<br/>
at at ch.qos.logback.core.recovery.ResilientOutputStreamBase.flush(ResilientOutputStreamBase.java:79)<br/>
<span class="error">[...]</span><br/>
15:44:51,064 |-INFO in c.q.l.c.recovery.ResilientFileOutputStream@1944673755 - \
Attempting to recover from IO failure on file <span \
class="error">[/Volumes/TESTVOL/logs/my-log.2015-02-01.log]</span><br/> \
15:44:51,064 |-INFO in c.q.l.c.recovery.ResilientFileOutputStream@1944673755 - \
Recovered from IO failure on file <span \
class="error">[/Volumes/TESTVOL/logs/my-log.2015-02-01.log]</span><br/> \
15:44:51,064 |-ERROR in ch.qos.logback.core.rolling.RollingFileAppender<span \
class="error">[MY_LOG]</span> - IO failure in appender \
java.nio.channels.ClosedChannelException<br/> at \
java.nio.channels.ClosedChannelException<br/> at at \
sun.nio.ch.FileLockImpl.release(FileLockImpl.java:58)</p>
<p style='margin-top:0;margin-bottom:10px;'>and then:<br/>
15:44:51,069 |-WARN in ch.qos.logback.core.rolling.RollingFileAppender<span \
class="error">[MY_LOG]</span> - Attempted to append to non started appender \
<span class="error">[MY_LOG]</span>.</p>
<p style='margin-top:0;margin-bottom:10px;'>Debugging:<br/>
i have investigated this issue and found the culprit to be line 204 in \
FileAppender:<br/> finally {<br/>
if (fileLock != null) </p>
{
---> fileLock.release();
}
<p style='margin-top:0;margin-bottom:10px;'> <span \
class="error">[...]</span></p>
<p style='margin-top:0;margin-bottom:10px;'>the problem is that when the original \
IOException was thrown, the channel was closed as part of the attemptRecovery method \
in ResilientOutputStreamBase.<br/> the release will throw a ClosedChannelException if \
the file channel is closed.<br/> the appender is then set to started=false in \
OutputStreamAppender subAppend Method and stays this way until restarted.</p>
<p style='margin-top:0;margin-bottom:10px;'>Fix suggestion:<br/>
the easy fix here is changing the guard of the release:<br/>
finally {<br/>
if (fileLock != null && fileChannel.isOpen()) </p>
{
fileLock.release();
}
<p style='margin-top:0;margin-bottom:10px;'> <span \
class="error">[...]</span></p>
<p style='margin-top:0;margin-bottom:10px;'>this prevents the release from throwing \
the exception.</p>
<p style='margin-top:0;margin-bottom:10px;'>for now, an easy mitigation (if possible) \
is to set prudent=false.</p>
<p style='margin-top:0;margin-bottom:10px;'>hope this helps and the bug will be \
fixed.</p>
</td>
</tr>
\
<tr valign="top"> <td \
style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 \
10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Project:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 \
0 10px 0;width:100%;">
<a style="color:#326ca6;" \
href="http://jira.qos.ch/browse/LOGBACK">logback</a> </td>
</tr> <tr valign="top">
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 \
10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Priority:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 \
0 10px 0;width:100%;"> <img \
src="http://jira.qos.ch/images/icons/priorities/major.png" height="16" width="16" \
border="0" align="absmiddle" alt="Major"> Major </td>
</tr>
<tr valign="top">
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 \
10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Reporter:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 \
0 10px 0;width:100%;"> <a class="user-hover" rel="nadavwe" id="email_nadavwe" \
href="http://jira.qos.ch/secure/ViewProfile.jspa?name=nadavwe" \
style="color:#326ca6;">Nadav Wexler</a> </td>
</tr> </table>
</td>
</tr>
</table>
</td>
</tr>
</table>
</td><!-- End #email-page -->
</tr>
<tr valign="top">
<td style="color:#505050;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:10px;line-height:14px;padding: \
0 16px 16px 16px;text-align:center;"> This message is automatically generated by \
JIRA.<br />
If you think it was sent incorrectly, please contact your JIRA \
administrators<br />
For more information on JIRA, see: <a style='color:#326ca6;' \
href='http://www.atlassian.com/software/jira'>http://www.atlassian.com/software/jira</a>
</td>
</tr>
</table><!-- End #email-wrap -->
</div><!-- End #email-body -->
_______________________________________________
logback-dev mailing list
logback-dev@qos.ch
http://mailman.qos.ch/mailman/listinfo/logback-dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic