[prev in list] [next in list] [prev in thread] [next in thread]
List: postgresql-general
Subject: Re: PostgreSQL GSSAPI Windows AD
From: Tumasgiu Rossini <rossini.t () gmail ! com>
Date: 2023-05-26 20:40:04
Message-ID: CAJD9AWy+Mh=9mkywDtzHmWPD12DrsdW9yzEDqBGxuEGf4K-CjA () mail ! gmail ! com
[Download RAW message or body]
Have you tried your tickets on the client machine ? From my (little)
understanding, the postgresql server complain that the client initiated the
communication with an ticket signed with a different key (kvno 3 vs. 4).
Hope it help.
For information here the differences from your setup with mine (debian 10 /
AD 2012) :
1) postgresql server not joined in domain
2) keytab generated with
ktpass -out postgres.keytab ^
-princ POSTGRES/debby@dom.local ^
-mapUser DOM\postgres ^
-rndpass ^
-mapOp set ^
-maxpass ^
-crypto AES256-SHA1 ^
-ptype KRB5_NT_PRINCIPAL
Le ven. 26 mai 2023 Ã 20:35, Jean-Philippe Chenel <jp.chenel@live.ca> a
écrit :
> Dear Tumasgiu Rossini,
>
> When I do the ktpass command on Windows AD, I can see that there is no
> other AD account mapped, otherwise it will raise an exception (Failed to
> set property 'servicePrincipalName').
>
> *Here is the klist command:*
> root@SFADAPGDDF02:/# klist -k /etc/postgresql/postgres.keytab
> KVNO Principal
> ----
> --------------------------------------------------------------------------
> 4 postgres/UBUNTU.ad.corp.com@AD.CORP.COM
>
> *Windows AD command:*
> PS C:\Users\Administrateur> get-aduser pgsql_ubuntu -properties
> msDS-KeyVersionNumber
>
> DistinguishedName : CN=pgsql_ubuntu,CN=Managed Service
> Accounts,DC=ad,DC=corp,DC=com
> Enabled : True
> GivenName : pgsql_ubuntu
> msDS-KeyVersionNumber : 4
> Name : pgsql_ubuntu
> ObjectClass : user
> ObjectGUID : dcaadc3c-2faf-44cf-a558-2a441cca690c
> SamAccountName : pgsql_ubuntu
> SID : S-1-5-21-1388463811-2779960163-2428466526-1204
> Surname :
> UserPrincipalName : postgres/UBUNTU.ad.corp.com@AD.CORP.COM
>
> If I look at the postgresql.log, I saw another kvno number. This one is
> matching the user trying to connect.
>
> 2023-05-26 18:30:08.576 UTC [4033] jp.chenel@template1 LOG: accepting
> GSS security context failed
> 2023-05-26 18:30:08.576 UTC [4033] jp.chenel@template1 DETAIL:
> Unspecified GSS failure. Minor code may provide more information: Request
> ticket server postgres/sfadapgddf02.ad.sygifcorp.com@AD.SYGIFCORP.COM not
> found in keytab (ticket kvno 3)
>
> Like I said, if I make a new keytab, just changing "-pass postgres",
> connections will work again. How to change this password ! For security
> reason, I don't want to let this password.
>
> With best regards,
> ------------------------------
> *De :* Tumasgiu Rossini <rossini.t@gmail.com>
> *Envoyé :* 26 mai 2023 12:09
> *À :* Jean-Philippe Chenel <jp.chenel@live.ca>
> *Objet :* Re: PostgreSQL GSSAPI Windows AD
>
> Hi,
>
> are you sure that there is no other ad account mapped to the postgres/
> UBUNTU.ad.corp.com@AD.CORP.COM principal ?
>
> Also you should check that the kvnos of both your keytab and your ad
> account matches, with the following commands :
>
> in linux for the keytab
> klist /path/to/the/keytab
>
> and in Windows for the account
> get-aduser <username> -properties msDS-KeyVersionNumber
>
>
> Le jeu. 25 mai 2023 Ã 23:51, Jean-Philippe Chenel <jp.chenel@live.ca> a
> écrit :
>
> Hi,
>
> I've recently updated from PostgreSQL 9.6 to 14 and also ubuntu 16.04 to
> 22.04.
> I've made all the installation required for postgresql to connect in
> GSSAPI authentication to a Windows domain.
>
> Something is going wrong and I don't know why.
> When I change the mapped user password from "postgres" to anything else,
> the connection stop to work
>
> Log of postgres:
> Unspecified GSS failure. Minor code may provide more information: Request
> ticket server postgres/ubuntu.ad.corp.com@AD.CORP.COM not found in keytab
> (ticket kvno 3)
>
> Here is the ktpass command (Windows AD):
>
> working:
> ktpass -out postgres.keytab -princ postgres/UBUNTU.ad.corp.com@AD.CORP.COM
> -mapUser AD\pgsql_ubuntu -pass postgres -mapOp add -crypto AES256-SHA1
> -ptype KRB5_NT_PRINCIPAL
>
> not working:
> ktpass -out postgres.keytab -princ postgres/UBUNTU.ad.corp.com@AD.CORP.COM
> -mapUser AD\pgsql_ubuntu -pass other_password -mapOp add -crypto
> AES256-SHA1 -ptype KRB5_NT_PRINCIPAL
>
> I put the keytab on the postgres server, the keytab file is referenced in
> the postgresql.conf file.
>
> Here is the full procedure:
>
> 1. Create user in AD for postgresql mapping (pgsql_ubuntu), always
> valid, support AES256
> 2. Create another user for connection testing
> 3. run ktpass command
> 4. put the keytab file on the pg server in /etc/postgresql, chown to
> postgres and chmod 600
> 5. postgresql.conf krb_server_keyfile =
> '/etc/postgresql/postgres.keytab'
> 6. pg_hba is configured to connect over gss
> 7. ubuntu server (postgres) is added to domain with this command:
> sudo realm join server.ad.corp.com -U Administrateur
>
>
> I don't know why it works when the password is "postgres" and why I can't
> change it.
>
> With best regards,
>
>
[Attachment #3 (text/html)]
<div dir="ltr"><br><div class="gmail_default" style="font-size:large">Have you tried \
your tickets on the client machine ? From my (little) understanding, the postgresql \
server complain that the client initiated the communication with an ticket signed \
with a different key (kvno 3 vs. 4). Hope it help.<br></div><div \
class="gmail_default" style="font-size:large"><br></div><div style="font-size:large" \
class="gmail_default">For information here the differences from your setup with mine \
(debian 10 / AD 2012) :</div><div style="font-size:large" class="gmail_default">1) \
postgresql server not joined in domain</div><div style="font-size:large" \
class="gmail_default">2) keytab generated with <br></div><div style="font-size:large" \
class="gmail_default"> ktpass -out postgres.keytab ^</div><div \
style="font-size:large" class="gmail_default"> -princ POSTGRES/debby<a \
class="gmail_plusreply" id="plusReplyChip-1">@dom.local</a> ^</div><div \
style="font-size:large" class="gmail_default"> -mapUser DOM\postgres \
^</div><div style="font-size:large" class="gmail_default"> -rndpass \
^</div><div style="font-size:large" class="gmail_default"> -mapOp set \
^</div><div style="font-size:large" class="gmail_default"> -maxpass \
^</div><div style="font-size:large" class="gmail_default"> -crypto \
AES256-SHA1 ^</div><div style="font-size:large" class="gmail_default"> \
-ptype KRB5_NT_PRINCIPAL
</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" \
class="gmail_attr">Le ven. 26 mai 2023 Ã 20:35, Jean-Philippe Chenel <<a \
href="mailto:jp.chenel@live.ca">jp.chenel@live.ca</a>> a écrit \
:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div \
class="msg-2278426881114518341">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Consolas,Courier,monospace;color:rgb(0,0,0)">Dear \
T</span><span style="font-family:Consolas,Courier,monospace;font-size:14px;display:inline;color:rgb(0,0,0);background-color:rgb(255,255,255)">umasgiu
Rossini,</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:14px;display:inline;color:rgb(66,66,66);background-color:rgb(255,255,255)"><br>
</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Consolas,Courier,monospace;font-size:14px;display:inline;color:rgb(0,0,0);background-color:rgb(255,255,255)">When \
I do the ktpass command on Windows AD, I can see that there is no other AD account \
mapped, otherwise it will raise an exception (Failed to set property \
'servicePrincipalName').</span></div> <div \
style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> \
<span style="font-size:14px;display:inline;color:rgb(66,66,66);background-color:rgb(255,255,255)"><br>
</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Consolas,Courier,monospace;font-size:14px;display:inline;color:rgb(0,0,0);background-color:rgb(255,255,255)"><b>Here \
is the klist command:</b></span></div> <div \
style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> \
<span style="font-family:Consolas,Courier,monospace;font-size:14px;display:inline;color:rgb(0,0,0);background-color:rgb(255,255,255)">root@SFADAPGDDF02:/# \
klist -k /etc/postgresql/postgres.keytab</span><span \
style="font-size:14px;display:inline;color:rgb(66,66,66);background-color:rgb(255,255,255)">
<div><span style="font-family:Consolas,Courier,monospace;color:rgb(0,0,0)">KVNO \
Principal</span><br> </div>
<div><span style="font-family:Consolas,Courier,monospace;color:rgb(0,0,0)">---- \
--------------------------------------------------------------------------</span></div>
<div><span style="font-family:Consolas,Courier,monospace;color:rgb(0,0,0)"> 4 \
postgres/<a href="mailto:UBUNTU.ad.corp.com@AD.CORP.COM" \
target="_blank">UBUNTU.ad.corp.com@AD.CORP.COM</a></span></div> <br>
</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Consolas,Courier,monospace;font-size:14px;display:inline;color:rgb(0,0,0);background-color:rgb(255,255,255)"><b>Windows \
AD command:</b></span></div> <div \
style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> \
<span style="font-family:Consolas,Courier,monospace;font-size:14px;display:inline;color:rgb(0,0,0);background-color:rgb(255,255,255)">PS \
C:\Users\Administrateur> get-aduser pgsql_ubuntu -properties \
msDS-KeyVersionNumber</span><span \
style="font-size:14px;display:inline;color:rgb(66,66,66);background-color:rgb(255,255,255)">
<div><br>
</div>
<div><span style="font-family:Consolas,Courier,monospace;color:rgb(0,0,0)">DistinguishedName \
: CN=pgsql_ubuntu,CN=Managed Service Accounts,DC=ad,DC=corp,DC=com</span></div> \
<div><span style="font-family:Consolas,Courier,monospace;color:rgb(0,0,0)">Enabled \
: True</span></div> <div><span \
style="font-family:Consolas,Courier,monospace;color:rgb(0,0,0)">GivenName \
: pgsql_ubuntu</span></div> <div><span \
style="font-family:Consolas,Courier,monospace;color:rgb(0,0,0)">msDS-KeyVersionNumber \
: 4</span></div> <div><span \
style="font-family:Consolas,Courier,monospace;color:rgb(0,0,0)">Name \
: pgsql_ubuntu</span></div> <div><span \
style="font-family:Consolas,Courier,monospace;color:rgb(0,0,0)">ObjectClass \
: user</span></div> <div><span \
style="font-family:Consolas,Courier,monospace;color:rgb(0,0,0)">ObjectGUID \
: dcaadc3c-2faf-44cf-a558-2a441cca690c</span></div> <div><span \
style="font-family:Consolas,Courier,monospace;color:rgb(0,0,0)">SamAccountName \
: pgsql_ubuntu</span></div> <div><span \
style="font-family:Consolas,Courier,monospace;color:rgb(0,0,0)">SID \
: S-1-5-21-1388463811-2779960163-2428466526-1204</span></div> <div><span \
style="font-family:Consolas,Courier,monospace;color:rgb(0,0,0)">Surname \
:</span></div> </span></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Consolas,Courier,monospace;font-size:14px;display:inline;color:rgb(0,0,0);background-color:rgb(255,255,255)">UserPrincipalName \
: postgres/<a href="mailto:UBUNTU.ad.corp.com@AD.CORP.COM" \
target="_blank">UBUNTU.ad.corp.com@AD.CORP.COM</a></span><span \
style="font-size:14px;display:inline;color:rgb(66,66,66);background-color:rgb(255,255,255)"><br>
</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Consolas,Courier,monospace;font-size:14px;display:inline;color:rgb(0,0,0);background-color:rgb(255,255,255)"><br>
</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Consolas,Courier,monospace;font-size:14px;display:inline;color:rgb(0,0,0);background-color:rgb(255,255,255)">If \
I look at the postgresql.log, I saw another kvno number. This one is matching the \
user trying to connect.</span></div> <div \
style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> \
<span style="font-family:Consolas,Courier,monospace;font-size:14px;display:inline;color:rgb(0,0,0);background-color:rgb(255,255,255)"><br>
</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Consolas,Courier,monospace;font-size:14px;display:inline;color:rgb(0,0,0);background-color:rgb(255,255,255)">2023-05-26 \
18:30:08.576 UTC [4033] jp.chenel@template1 LOG: accepting GSS security context \
failed<br> 2023-05-26 18:30:08.576 UTC [4033] jp.chenel@template1 DETAIL: \
Unspecified GSS failure. Minor code may provide more information: Request ticket \
server postgres/<a href="mailto:sfadapgddf02.ad.sygifcorp.com@AD.SYGIFCORP.COM" \
target="_blank">sfadapgddf02.ad.sygifcorp.com@AD.SYGIFCORP.COM</a> not found in \
keytab (ticket kvno 3)<br> </span></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Consolas,Courier,monospace;font-size:14px;display:inline;color:rgb(0,0,0);background-color:rgb(255,255,255)"><br>
</span></div>
<div style="color:rgb(0,0,0)"><font face="Consolas, Courier, monospace"><span \
style="font-size:14px">Like I said, if I make a new keytab, just changing "-pass \
postgres", connections will work again. How to change this password ! For \
security reason, I don't want to let this password.</span></font></div> <div \
style="color:rgb(0,0,0)"><font face="Consolas, Courier, monospace"><span \
style="font-size:14px"><br> </span></font></div>
<div style="color:rgb(0,0,0)"><font face="Consolas, Courier, monospace"><span \
style="font-size:14px">With best regards,</span></font></div> <div \
id="m_5220537146120859373appendonsend"></div> <hr \
style="display:inline-block;width:98%"> <div id="m_5220537146120859373divRplyFwdMsg" \
dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" \
color="#000000"><b>De :</b> Tumasgiu Rossini <<a href="mailto:rossini.t@gmail.com" \
target="_blank">rossini.t@gmail.com</a>><br> <b>Envoyé :</b> 26 mai 2023 \
12:09<br> <b>À :</b> Jean-Philippe Chenel <<a href="mailto:jp.chenel@live.ca" \
target="_blank">jp.chenel@live.ca</a>><br> <b>Objet :</b> Re: PostgreSQL GSSAPI \
Windows AD</font> <div> </div>
</div>
<div>
<div dir="ltr">
<div style="font-size:large">Hi,</div>
<div style="font-size:large"><br>
</div>
<div style="font-size:large">are you sure that there is no other ad account mapped to \
the <span style="font-size:12pt;margin:0px">postgres/<a \
href="mailto:UBUNTU.ad.corp.com@AD.CORP.COM" \
target="_blank">UBUNTU.ad.corp.com@AD.CORP.COM</a></span> principal ?</div> <div \
style="font-size:large"><br> </div>
<div style="font-size:large">Also you should check that the kvnos of both your keytab \
and your ad account matches, with the following commands :</div> <div \
style="font-size:large"><br> </div>
<div style="font-size:large">in linux for the keytab </div>
<div style="font-size:large"> klist /path/to/the/keytab<br>
</div>
<div style="font-size:large"><br>
</div>
<div style="font-size:large">and in Windows for the account
</div>
<div style="font-size:large"> get-aduser <username> -properties \
msDS-KeyVersionNumber <br>
</div>
<div style="font-size:large"><br>
</div>
</div>
<br>
<div>
<div dir="ltr">Le jeu. 25 mai 2023 Ã 23:51, Jean-Philippe Chenel <<a \
href="mailto:jp.chenel@live.ca" target="_blank">jp.chenel@live.ca</a>> a écrit \
:<br> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"> <div>
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi,</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I've recently updated from PostgreSQL 9.6 to 14 and also ubuntu 16.04 to \
22.04.</div> <div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I've made all the installation required for postgresql to connect in GSSAPI \
authentication to a Windows domain.</div> <div \
style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> \
<br> </div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Something is going wrong and I don't know why.</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
When I change<span style="display:inline;background-color:rgb(255,255,255)"> the \
mapped user </span>password from "postgres" to anything else, the \
connection stop to work</div> <div \
style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> \
<br> </div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Log of postgres:</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Unspecified GSS failure. Minor code may provide more information: Request ticket \
server postgres/<a href="mailto:ubuntu.ad.corp.com@AD.CORP.COM" \
target="_blank">ubuntu.ad.corp.com@AD.CORP.COM</a> not found in keytab (ticket kvno \
3)</div> <div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Here is the ktpass command (Windows AD):</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
working:</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
ktpass -out postgres.keytab -princ postgres/<a \
href="mailto:UBUNTU.ad.corp.com@AD.CORP.COM" \
target="_blank">UBUNTU.ad.corp.com@AD.CORP.COM</a> -mapUser AD\pgsql_ubuntu -pass \
postgres -mapOp add -crypto AES256-SHA1 -ptype KRB5_NT_PRINCIPAL<br> </div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
not working:</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:12pt;margin:0px">ktpass -out postgres.keytab -princ \
postgres/<a href="mailto:UBUNTU.ad.corp.com@AD.CORP.COM" \
target="_blank">UBUNTU.ad.corp.com@AD.CORP.COM</a> -mapUser AD\pgsql_ubuntu -pass \
other_password -mapOp add -crypto AES256-SHA1
-ptype KRB5_NT_PRINCIPAL</span><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I put the keytab on the postgres server, the keytab file is referenced in the \
postgresql.conf file.</div> <div \
style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> \
<br> </div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Here is the full procedure:</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<ol>
<li style="list-style-type:"1. ""><span>Create user in AD for postgresql \
mapping (<span style="text-align:start;display:inline;background-color:rgb(255,255,255)">pgsql_ubuntu), \
always valid, support AES256</span><br> </span></li><li \
style="list-style-type:"2. ""><span><span \
style="text-align:start;display:inline;background-color:rgb(255,255,255)">Create \
another user for connection testing</span></span></li><li \
style="list-style-type:"3. ""><span><span \
style="text-align:start;display:inline;background-color:rgb(255,255,255)">run ktpass \
command</span></span></li><li style="list-style-type:"4. ""><span><span \
style="text-align:start;display:inline;background-color:rgb(255,255,255)">put the \
keytab file on the pg server in /etc/postgresql, chown to postgres and chmod \
600</span></span></li><li style="list-style-type:"5. ""><span><span \
style="text-align:start;display:inline;background-color:rgb(255,255,255)">postgresql.conf \
krb_server_keyfile = '/etc/postgresql/postgres.keytab'</span></span></li><li \
style="list-style-type:"6. ""><span><span \
style="text-align:start;display:inline;background-color:rgb(255,255,255)">pg_hba is \
configured to connect over gss</span></span></li><li style="list-style-type:"7. \
""><span><span style="text-align:start;display:inline;background-color:rgb(255,255,255)">ubuntu \
server (postgres) is added to domain with this command:<br> sudo realm join <a \
href="http://server.ad.corp.com" target="_blank">server.ad.corp.com</a> -U \
Administrateur<br> <br>
</span></span></li></ol>
<div><span><span style="text-align:start;display:inline;background-color:rgb(255,255,255)">I \
don't know why it works when the password is "postgres" and why I \
can't change it.</span></span></div> <div><span><span \
style="text-align:start;display:inline;background-color:rgb(255,255,255)"><br> \
</span></span></div> <div><span><span \
style="text-align:start;display:inline;background-color:rgb(255,255,255)">With best \
regards,</span></span></div> </div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div></blockquote></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic