[prev in list] [next in list] [prev in thread] [next in thread] 

List:       squid-users
Subject:    [squid-users] R:  squid stores multiple copies of identical ETags
From:       "Tabacchiera, Stefano" <stefano.tabacchiera () IGT ! com>
Date:       2020-06-30 9:10:57
Message-ID: DM6PR01MB3817CBD17D7EB746D6B28EB2E46F0 () DM6PR01MB3817 ! prod ! exchangelabs ! com
[Download RAW message or body]

[Attachment #2 (text/plain)]

> > I mean: squid would store a new copy of the object while leaving the

> > old copy deletion to cleanup task?



> Some parts of the cleanup process may be delegated. The details depend on the \
> cache_dir type. I do not know or remember aufs specifics, but I suspect that all \
> ufs-based cache_dirs, including aufs, use lazy garbage collection >(under normal \
> circumstances). The cache_swap_low and cache_swap_high directives should determine \
> what is "normal".



Alex, you were absolutely right.

I managed to reproduce the case.

On test environment I set up "cache_swap_low 1" and "cache_swap_low 2" and enabled \
the store_log.

Then I tailed the store_log and watched the evolution of cache_dir, while running \
squidclient toward origin server every 100ms.



Store.log:

1593504359.704 SWAPOUT 00 000098C5 1865A3A26D411E7C0D8D87770720E405  200 1593504359 \
1593504061        -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json

1593504359.858 SWAPOUT 00 000098C6 1865A3A26D411E7C0D8D87770720E405  200 1593504359 \
1593504061        -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json

1593504360.015 SWAPOUT 00 000098C7 1865A3A26D411E7C0D8D87770720E405  200 1593504359 \
1593504061        -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json

1593504360.170 SWAPOUT 00 000098C8 1865A3A26D411E7C0D8D87770720E405  200 1593504359 \
1593504061        -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json

1593504360.324 SWAPOUT 00 000098C9 1865A3A26D411E7C0D8D87770720E405  200 1593504359 \
1593504061        -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json

1593504360.476 SWAPOUT 00 000098CA 1865A3A26D411E7C0D8D87770720E405  200 1593504359 \
1593504061        -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json

1593504360.634 SWAPOUT 00 000098CB 1865A3A26D411E7C0D8D87770720E405  200 1593504360 \
1593504061        -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json

1593504360.788 SWAPOUT 00 000098CC 1865A3A26D411E7C0D8D87770720E405  200 1593504360 \
1593504061        -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json

1593504360.941 SWAPOUT 00 000098CD 1865A3A26D411E7C0D8D87770720E405  200 1593504360 \
1593504061        -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json

1593504361.096 SWAPOUT 00 000098CE 1865A3A26D411E7C0D8D87770720E405  200 1593504360 \
1593504061        -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json

1593504361.249 SWAPOUT 00 000098CF 1865A3A26D411E7C0D8D87770720E405  200 1593504360 \
1593504061        -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json

1593504361.403 SWAPOUT 00 000098D0 1865A3A26D411E7C0D8D87770720E405  200 1593504360 \
1593504061        -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json

1593504361.556 SWAPOUT 00 000098D1 1865A3A26D411E7C0D8D87770720E405  200 1593504360 \
1593504061        -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json

1593504361.607 RELEASE 00 000098D1 1865A3A26D411E7C0D8D87770720E405   ?         ?     \
?         ? ?/? ?/? ? ?



The cached object was gradually appearing in cache_dir, until the "RELEASE" line \
showed up in the store.log.

At this right moment, all copies of the object stored on disk were deleted.



So I'm assuming that only one object on disk (the last one retrieved) is the object \
referenced as "active" by squid, all the rest being trashable.

Since the client is forcing a "no-cache" header, squid does what the client is asking \
for, and every time it stores the object on disk.

I'm also assuming that IF another client asked the same object without the "no-cache" \
header, squid would serve the latest cached object on disk.

If I'm right so far, squid never "overwrites" the old copy of an object on disk. \
Instead, it stores a new one, marking it as "active", and let the deletion task to \
(a)ufs threads.



Could this this way?



Thanks!

ST



____________________________________________________________________________________ \
La presente comunicazione ed i suoi allegati e' destinata esclusivamente ai \
destinatari. Qualsiasi suo utilizzo, comunicazione o diffusione non autorizzata e' \
proibita. Se ha ricevuto questa comunicazione per errore, la preghiamo di darne \
immediata comunicazione al mittente e di cancellare tutte le informazioni \
erroneamente acquisite. Grazie This message and its attachments are intended only for \
use by the addressees. Any use, re-transmission or dissemination not authorized of it \
is prohibited. If you received this e-mail in error, please inform the sender \
immediately and delete all the material. Thank you.


[Attachment #3 (text/html)]

<html xmlns:v="urn:schemas-microsoft-com:vml" \
xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" \
xmlns="http://www.w3.org/TR/REC-html40"> <head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
	{font-family:"Lucida Console";
	panose-1:2 11 6 9 4 5 4 2 2 4;}
/* Style Definitions */
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
	{mso-style-priority:99;
	mso-style-link:"Testo normale Carattere";
	margin:0cm;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;
	mso-fareast-language:EN-US;}
span.TestonormaleCarattere
	{mso-style-name:"Testo normale Carattere";
	mso-style-priority:99;
	mso-style-link:"Testo normale";
	font-family:"Calibri",sans-serif;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-family:"Calibri",sans-serif;
	mso-fareast-language:EN-US;}
@page WordSection1
	{size:612.0pt 792.0pt;
	margin:70.85pt 2.0cm 2.0cm 2.0cm;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="IT" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoPlainText"><span lang="EN-US">&gt;&gt; I mean: squid would store a new \
copy of the object while leaving the <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">&gt;&gt; old copy deletion to cleanup \
task?<o:p></o:p></span></p> <p class="MsoPlainText"><span \
lang="EN-US"><o:p>&nbsp;</o:p></span></p> <p class="MsoPlainText"><span \
lang="EN-US">&gt;Some parts of the cleanup process may be delegated. </span>The \
details depend on the cache_dir type. <span lang="EN-US">I do not know or remember \
aufs specifics, but I suspect that all ufs-based cache_dirs, including aufs, use lazy \
garbage collection &gt;(under normal circumstances). </span>The cache_swap_low and \
cache_swap_high directives should determine what is \
&quot;normal&quot;.<o:p></o:p></p> <p class="MsoPlainText"><span \
style="color:black"><o:p>&nbsp;</o:p></span></p> <p class="MsoPlainText"><span \
lang="EN-US" style="color:black">Alex, you were absolutely \
right.<o:p></o:p></span></p> <p class="MsoPlainText"><span lang="EN-US" \
style="color:black">I managed to reproduce the case.<o:p></o:p></span></p> <p \
class="MsoPlainText"><span lang="EN-US" style="color:black">On test environment I set \
up &quot;cache_swap_low 1&quot; and &quot;cache_swap_low 2&quot; and enabled the \
store_log.<o:p></o:p></span></p> <p class="MsoPlainText"><span lang="EN-US" \
style="color:black">Then I tailed the store_log and watched the evolution of \
cache_dir, while running squidclient toward origin server every \
100ms.<o:p></o:p></span></p> <p class="MsoPlainText"><span lang="EN-US" \
style="color:black"><o:p>&nbsp;</o:p></span></p> <p class="MsoPlainText"><span \
lang="EN-US" style="color:black">Store.log:<o:p></o:p></span></p> <p \
class="MsoPlainText"><span lang="EN-US" \
style="font-size:9.0pt;font-family:&quot;Lucida \
Console&quot;;color:black">1593504359.704 SWAPOUT 00 000098C5 \
1865A3A26D411E7C0D8D87770720E405&nbsp; 200 1593504359 \
1593504061&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json<o:p></o:p></span></p> <p \
class="MsoPlainText"><span lang="EN-US" \
style="font-size:9.0pt;font-family:&quot;Lucida \
Console&quot;;color:black">1593504359.858 SWAPOUT 00 000098C6 \
1865A3A26D411E7C0D8D87770720E405&nbsp; 200 1593504359 \
1593504061&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json<o:p></o:p></span></p> <p \
class="MsoPlainText"><span lang="EN-US" \
style="font-size:9.0pt;font-family:&quot;Lucida \
Console&quot;;color:black">1593504360.015 SWAPOUT 00 000098C7 \
1865A3A26D411E7C0D8D87770720E405&nbsp; 200 1593504359 \
1593504061&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json<o:p></o:p></span></p> <p \
class="MsoPlainText"><span lang="EN-US" \
style="font-size:9.0pt;font-family:&quot;Lucida \
Console&quot;;color:black">1593504360.170 SWAPOUT 00 000098C8 \
1865A3A26D411E7C0D8D87770720E405&nbsp; 200 1593504359 \
1593504061&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json<o:p></o:p></span></p> <p \
class="MsoPlainText"><span lang="EN-US" \
style="font-size:9.0pt;font-family:&quot;Lucida \
Console&quot;;color:black">1593504360.324 SWAPOUT 00 000098C9 \
1865A3A26D411E7C0D8D87770720E405&nbsp; 200 1593504359 \
1593504061&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json<o:p></o:p></span></p> <p \
class="MsoPlainText"><span lang="EN-US" \
style="font-size:9.0pt;font-family:&quot;Lucida \
Console&quot;;color:black">1593504360.476 SWAPOUT 00 000098CA \
1865A3A26D411E7C0D8D87770720E405&nbsp; 200 1593504359 \
1593504061&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json<o:p></o:p></span></p> <p \
class="MsoPlainText"><span lang="EN-US" \
style="font-size:9.0pt;font-family:&quot;Lucida \
Console&quot;;color:black">1593504360.634 SWAPOUT 00 000098CB \
1865A3A26D411E7C0D8D87770720E405&nbsp; 200 1593504360 \
1593504061&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json<o:p></o:p></span></p> <p \
class="MsoPlainText"><span lang="EN-US" \
style="font-size:9.0pt;font-family:&quot;Lucida \
Console&quot;;color:black">1593504360.788 SWAPOUT 00 000098CC \
1865A3A26D411E7C0D8D87770720E405&nbsp; 200 1593504360 \
1593504061&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json<o:p></o:p></span></p> <p \
class="MsoPlainText"><span lang="EN-US" \
style="font-size:9.0pt;font-family:&quot;Lucida \
Console&quot;;color:black">1593504360.941 SWAPOUT 00 000098CD \
1865A3A26D411E7C0D8D87770720E405&nbsp; 200 1593504360 \
1593504061&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json<o:p></o:p></span></p> <p \
class="MsoPlainText"><span lang="EN-US" \
style="font-size:9.0pt;font-family:&quot;Lucida \
Console&quot;;color:black">1593504361.096 SWAPOUT 00 000098CE \
1865A3A26D411E7C0D8D87770720E405&nbsp; 200 1593504360 \
1593504061&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json<o:p></o:p></span></p> <p \
class="MsoPlainText"><span lang="EN-US" \
style="font-size:9.0pt;font-family:&quot;Lucida \
Console&quot;;color:black">1593504361.249 SWAPOUT 00 000098CF \
1865A3A26D411E7C0D8D87770720E405&nbsp; 200 1593504360 \
1593504061&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json<o:p></o:p></span></p> <p \
class="MsoPlainText"><span lang="EN-US" \
style="font-size:9.0pt;font-family:&quot;Lucida \
Console&quot;;color:black">1593504361.403 SWAPOUT 00 000098D0 \
1865A3A26D411E7C0D8D87770720E405&nbsp; 200 1593504360 \
1593504061&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json<o:p></o:p></span></p> <p \
class="MsoPlainText"><span lang="EN-US" \
style="font-size:9.0pt;font-family:&quot;Lucida \
Console&quot;;color:black">1593504361.556 SWAPOUT 00 000098D1 \
1865A3A26D411E7C0D8D87770720E405&nbsp; 200 1593504360 \
1593504061&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -1 text/plain 544275/544275 GET \
http://xxx.xxx.xxx.xxx/blah/FEED.json<o:p></o:p></span></p> <p \
class="MsoPlainText"><span lang="EN-US" \
style="font-size:9.0pt;font-family:&quot;Lucida \
Console&quot;;color:red">1593504361.607 RELEASE 00 000098D1 \
1865A3A26D411E7C0D8D87770720E405&nbsp;&nbsp; \
?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \
?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \
?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ? ?/? ?/? ? ?<o:p></o:p></span></p> \
<p class="MsoPlainText"><span lang="EN-US" \
style="color:black"><o:p>&nbsp;</o:p></span></p> <p class="MsoPlainText"><span \
lang="EN-US" style="color:black">The cached object was gradually appearing in \
cache_dir, until the "RELEASE" line showed up in the store.log.<o:p></o:p></span></p> \
<p class="MsoPlainText"><span lang="EN-US" style="color:black">At this right moment, \
all copies of the object stored on disk were deleted.<o:p></o:p></span></p> <p \
class="MsoPlainText"><span lang="EN-US" \
style="color:black"><o:p>&nbsp;</o:p></span></p> <p class="MsoPlainText"><span \
lang="EN-US" style="color:black">So I'm assuming that only one object on disk (the \
last one retrieved) is the object referenced as "active" by squid, all the rest being \
trashable.<o:p></o:p></span></p> <p class="MsoPlainText"><span lang="EN-US" \
style="color:black">Since the client is forcing a "no-cache" header, squid does what \
the client is asking for, and every time it stores the object on \
disk.<o:p></o:p></span></p> <p class="MsoPlainText"><span lang="EN-US" \
style="color:black">I'm also assuming that IF another client asked the same object \
without the "no-cache" header, squid would serve the latest cached object on \
disk.<o:p></o:p></span></p> <p class="MsoPlainText"><span lang="EN-US" \
style="color:black">If I'm right so far, squid never "overwrites" the old copy of an \
object on disk. Instead, it stores a new one, marking it as "active", and let the \
deletion task to (a)ufs threads.<o:p></o:p></span></p> <p class="MsoPlainText"><span \
lang="EN-US" style="color:black"><o:p>&nbsp;</o:p></span></p> <p \
class="MsoPlainText"><span lang="EN-US" style="color:black">Could this this \
way?<o:p></o:p></span></p> <p class="MsoPlainText"><span lang="EN-US" \
style="color:black"><o:p>&nbsp;</o:p></span></p> <p class="MsoPlainText"><span \
lang="EN-US" style="color:black">Thanks!<o:p></o:p></span></p> <p \
class="MsoPlainText"><span lang="EN-US" style="color:black">ST<o:p></o:p></span></p> \
<p class="MsoPlainText"><span lang="EN-US" \
style="color:black"><o:p>&nbsp;</o:p></span></p> </div>
____________________________________________________________________________________ \
La presente comunicazione ed i suoi allegati e' destinata esclusivamente ai \
destinatari. Qualsiasi suo utilizzo, comunicazione o diffusione non autorizzata e' \
proibita. Se  ha ricevuto questa comunicazione per errore, la preghiamo di darne \
immediata comunicazione al mittente e di cancellare tutte le informazioni \
erroneamente acquisite. Grazie This message and its attachments are intended only for \
use by the addressees. Any use,  re-transmission or dissemination not authorized of \
it is prohibited. If you received this e-mail in error, please inform the sender \
immediately and delete all the material. Thank you. </body>
</html>


[Attachment #4 (unknown)]

_______________________________________________
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users


[prev in list] [next in list] [prev in thread] [next in thread] 

Configure | About | News | Add a list | Sponsored by KoreLogic