[prev in list] [next in list] [prev in thread] [next in thread]
List: squid-users
Subject: Re: [squid-users] All Adaptation ICAPs go down at the same time
From: "Eliezer Croitoru" <ngtech1ltd () gmail ! com>
Date: 2021-04-19 9:07:08
Message-ID: 000001d734fb$6081bd70$21853850$ () gmail ! com
[Download RAW message or body]
This is a multipart message in MIME format.
[Attachment #2 (multipart/alternative)]
This is a multipart message in MIME format.
Hey Roie,
From the output I assume it's a dns resolution issue.
In the past I remember that Docker was updating the hosts file with the relevant \
names but it's not working the same way now.
Currently Docker is using a local network dns service which is being accessed via \
127.0.0.53.
From I remember Squid is resolving the icap service name only at startup or reload.
Lately Alex published a testable patch that might fix specific issues with icap \
services which are resolved by dns. ( sorry I don't remember the bug report)
I assume you can try to test this patch first.
If these services are static to some degree you might be able to create a script that \
updates the hosts file and reload squid on each change.
When using the hosts file it's possible that some issues will disappear.
There is also another possibility which is a malformed ICAP response or wrong \
sessions handling which cause this issue.
You might be able to use tcpdump from either the host or the container side to \
capture traffic when these goes down.
Depends on your preference of debug level you might even be able to debug specific \
debug_options like for ICAP services and/or requests to the degree you might be able \
to see what happens on the basic level of the ICAP encapsulation.
If you really need help with a diagnosis and a solution you might be able to use Alex \
and the measurement factory.
All The Bests,
Eliezer
From: squid-users <squid-users-bounces@lists.squid-cache.org> On Behalf Of roie \
rachamim
Sent: Monday, April 12, 2021 12:54 PM
To: squid-users@lists.squid-cache.org
Subject: [squid-users] All Adaptation ICAPs go down at the same time
Hi,
Our setup includes squid that runs in docker container with several ICAP servers in \
additional containers.
From time to time we see in cache.log the following messages:
2021/04/12 00:22:39| optional ICAP service is down after an options fetch failure: \
icap://icap1.proxy:14590/censor [down,!opt] 2021/04/12 00:22:39| optional ICAP \
service is down after an options fetch failure: icap://icap2.proxy:1344/request \
[down,!opt] 2021/04/12 00:22:39| optional ICAP service is down after an options fetch \
failure: icap://icap3.proxy:14590/response [down,!opt]
2021/04/12 06:10:45| optional ICAP service is down after an options fetch failure: \
icap://icap1.proxy:14590/censor [down,!opt] 2021/04/12 06:10:45| optional ICAP \
service is down after an options fetch failure: icap://icap2.proxy:1344/request \
[down,!opt] 2021/04/12 06:10:45| optional ICAP service is down after an options fetch \
failure: icap://icap3.proxy:14590/response [down,!opt]
We're trying to understand why it happens to all ICAPs at once. This happens in 4.14 \
and in 5.0.4
Any thoughts about what might cause this ?
Many Thanks,
Roie
[Attachment #5 (text/html)]
<html xmlns:v="urn:schemas-microsoft-com:vml" \
xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" \
xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type \
content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 \
(filtered medium)"><style><!-- /* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple \
style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal>Hey \
Roie,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>From \
the output I assume it's a dns resolution issue.<o:p></o:p></p><p class=MsoNormal>In \
the past I remember that Docker was updating the hosts file with the relevant names \
but it's not working the same way now.<o:p></o:p></p><p class=MsoNormal>Currently \
Docker is using a local network dns service which is being accessed via \
127.0.0.53.<o:p></o:p></p><p class=MsoNormal>From I remember Squid is resolving the \
icap service name only at startup or reload.<o:p></o:p></p><p class=MsoNormal>Lately \
Alex published a testable patch that might fix specific issues with icap services \
which are resolved by dns. ( sorry I don't remember the bug report)<o:p></o:p></p><p \
class=MsoNormal>I assume you can try to test this patch first.<o:p></o:p></p><p \
class=MsoNormal>If these services are static to some degree you might be able to \
create a script that updates the hosts file and reload squid on each \
change.<o:p></o:p></p><p class=MsoNormal>When using the hosts file it's possible that \
some issues will disappear.<o:p></o:p></p><p class=MsoNormal><br>There is also \
another possibility which is a malformed ICAP response or wrong sessions handling \
which cause this issue.<o:p></o:p></p><p class=MsoNormal>You might be able to use \
tcpdump from either the host or the container side to capture traffic when these goes \
down.<o:p></o:p></p><p class=MsoNormal>Depends on your preference of debug level you \
might even be able to debug specific debug_options like for ICAP services<br>and/or \
requests to the degree you might be able to see what happens on the basic level of \
the ICAP encapsulation.<o:p></o:p></p><p class=MsoNormal>If you really need help with \
a diagnosis and a solution you might be able to use Alex and the measurement \
factory.<br><br><o:p></o:p></p><p class=MsoNormal>All The Bests,<o:p></o:p></p><p \
class=MsoNormal>Eliezer<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><div \
style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p \
class=MsoNormal><b>From:</b> squid-users \
<squid-users-bounces@lists.squid-cache.org> <b>On Behalf Of </b>roie \
rachamim<br><b>Sent:</b> Monday, April 12, 2021 12:54 PM<br><b>To:</b> \
squid-users@lists.squid-cache.org<br><b>Subject:</b> [squid-users] All Adaptation \
ICAPs go down at the same time<o:p></o:p></p></div><p \
class=MsoNormal><o:p> </o:p></p><div><div><p \
class=MsoNormal>Hi,<o:p></o:p></p></div><div><p \
class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal \
style='margin-bottom:12.0pt'>Our setup includes squid that runs in docker container \
with several ICAP servers in additional containers.<o:p></o:p></p></div><div><p \
class=MsoNormal>From time to time we see in cache.log the following \
messages:<br>2021/04/12 00:22:39| optional ICAP service is down after an options \
fetch failure: icap://icap1.proxy:14590/censor [down,!opt]<br>2021/04/12 00:22:39| \
optional ICAP service is down after an options fetch failure: \
icap://icap2.proxy:1344/request [down,!opt]<br>2021/04/12 00:22:39| optional ICAP \
service is down after an options fetch failure: icap://icap3.proxy:14590/response \
[down,!opt]<br><br>2021/04/12 06:10:45| optional ICAP service is down after an \
options fetch failure: icap://icap1.proxy:14590/censor [down,!opt]<br>2021/04/12 \
06:10:45| optional ICAP service is down after an options fetch failure: \
icap://icap2.proxy:1344/request [down,!opt]<br>2021/04/12 06:10:45| optional ICAP \
service is down after an options fetch failure: icap://icap3.proxy:14590/response \
[down,!opt]<o:p></o:p></p></div><div><p \
class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal \
style='margin-bottom:12.0pt'>We're trying to understand why it happens to all ICAPs \
at once. This happens in 4.14 and in 5.0.4<o:p></o:p></p></div><div><p \
class=MsoNormal style='margin-bottom:12.0pt'>Any thoughts about what might cause this \
?<o:p></o:p></p></div><div><p class=MsoNormal>Many \
Thanks,<o:p></o:p></p></div><div><p \
class=MsoNormal>Roie<o:p></o:p></p></div></div></div></body></html>
[Attachment #6 (text/plain)]
_______________________________________________
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic