[prev in list] [next in list] [prev in thread] [next in thread]
List: forgerock-openidm
Subject: [OpenIDM] RE : RE : User creation error OpenDJ
From: "Belleville-Rioux, Vincent" <rioux.vincent () uqam ! ca>
Date: 2015-02-09 13:38:09
Message-ID: 0AD36E0192997748BD28BC68B14C75D0164534A2 () Lettre ! gst ! uqam ! ca
[Download RAW message or body]
Hi Matthias,
I ran a 500K batch import this week-end without a single error this week-end after \
removing the patch operation. Before that, I had a duplicate create and a stack \
trace about it for every new managed/user created when it tried to auto-sync to \
OpenDJ.
The patch was triggered this way :
We add managed/users manually through scripting.
sync.json defines a mapping from managed/user to opendj/account. We rely on \
auto-sync to create the accounts, and run a recon dialy in case some didn't get \
through in realtime.
In that mapping, we generate a few attributes for OpenDJ (displayName and such), \
among which the uidNumber attribute. We want this attribute to be unique among all \
OpenDJ accounts so I made a transform script that :
1 - If source.uidNumber is undefined, continue and assign a new uidNumber
2 - Make a random number from 1 million to 1 100 million (boundaries chosen to stay \
within a 32-bit integer while avoiding possible conflicts). 3 - Verify that this \
number is NOT already assigned through a openidm.query to OpenDJ directly. 4 - Assign \
the number through a patch.push. 5 - Return the uidNumber.
Now, it is clear to me that this transform script should not have been patching users \
directly as OpenIDM will receive the correct uidNumber and apply it to the account \
correctly.
It can be classified as a case of configuration error, but I still believe the \
behavior is weird - duplicate creates to the external system should not happen \
because if our OpenDJ installation was setup in such a way to allow those duplicates, \
we might have a rather large mess to cleanup now.
Thanks,
Vincent
________________________________
De : openidm-bounces@forgerock.org [openidm-bounces@forgerock.org] de la part de \
Matthias Tristl [matthias.tristl@forgerock.com] Date d'envoi : 9 février 2015 01:44
À : openidm@forgerock.org
Objet : Re: [OpenIDM] RE : User creation error OpenDJ
Hi Vincent,
I agree, a patch should not trigger a create and I would be surprised if it does. \
Should be very easy to test in your set up, right?
On the other hand, can you explain why you use a patch as part of a synchronisation? \
That does not make much sense (if the target of the patch is the same as the sunc \
targeting) for me except for actions like LINK or UNLINK in which case the patch \
command should be part of a onLink (...) script.
Can you explain?
Matthias
Matthias Tristl : ForgeRock INC
e: matthias.tristl@forgerock.com<mailto:matthias.tristl@forgerock.com>
t: +47 47707662
w: forgerock.com<http://forgerock.com>
On Fri, Feb 6, 2015 at 11:42 PM, Belleville-Rioux, Vincent \
<rioux.vincent@uqam.ca<mailto:rioux.vincent@uqam.ca>> wrote: I think I hit an \
undiscovered bug.
I found what's happening :
Consider the following scenario :
We create a managed/user through the REST interface.
This triggers an auto-sync to a LDAP connector.
There is a transform script defined on an attribute between managed/user and the \
target opendj. This transform script calls an openidm.patch directly.
Now, what happens :
The account is created in the LDAP system correctly. However we see a duplicate \
CREATE action being triggered by OpenIDM. The second one fails as the user already \
exists in the LDAP system.
Why this happens :
I *think* that what happens is that the openidm.patch call results in a CREATE action \
on the target system because the user has not been created yet when it is evaluated. \
Then, the actual CREATE action triggered by the autosync from the addition of the \
managed/user will be tried later, and fail.
I'd classify this as a bug because calling a patch on a non-existant user should not \
result in a create. It should throw the proper error.
Also, if my understanding of what happens isn't right and the patch is done *after* \
the user has been created, then it should still not result in a duplicate CREATE \
event.
Vincent
________________________________
De : openidm-bounces@forgerock.org<mailto:openidm-bounces@forgerock.org> \
[openidm-bounces@forgerock.org<mailto:openidm-bounces@forgerock.org>] de la part de \
Gael Allioux [gael.allioux@forgerock.com<mailto:gael.allioux@forgerock.com>] Date \
d'envoi : 6 février 2015 14:28 À : \
openidm@forgerock.org<mailto:openidm@forgerock.org> Objet : Re: [OpenIDM] User \
creation error OpenDJ
On 02/06/2015 08:22 PM, Belleville-Rioux, Vincent wrote:
Anybody has an idea what would cause OpenIDM to try and add a user two times to a \
single external system (OpenDJ) ? See the following OpenDJ log coming from an \
automatic sync from OpenIDM :
[06/Feb/2015:14:04:43 -0500] ADD REQ conn=3 op=145 msgID=146 \
dn="uid=qo55068416,dc=mydc" [06/Feb/2015:14:04:43 -0500] ADD RES conn=3 op=145 \
msgID=146 result=0 etime=33 [06/Feb/2015:14:04:43 -0500] SEARCH REQ conn=3 op=146 \
msgID=147 base="uid=qo55068416,dc=mydc" scope=baseObject filter="(objectClass=*)" \
attrs="entryUUID" [06/Feb/2015:14:04:43 -0500] SEARCH RES conn=3 op=146 msgID=147 \
result=0 nentries=1 etime=2 [06/Feb/2015:14:04:43 -0500] SEARCH REQ conn=3 op=147 \
msgID=148 base="" scope=baseObject filter="(objectClass=*)" attrs="subschemaSubentry" \
[06/Feb/2015:14:04:43 -0500] SEARCH RES conn=3 op=147 msgID=148 result=0 nentries=1 \
etime=2 [06/Feb/2015:14:04:43 -0500] SEARCH REQ conn=3 op=148 msgID=149 \
base="dc=mydc" scope=wholeSubtree \
filter="(&(&(objectClass=top)(objectClass=person)(objectClass=organizationalPerson)(ob \
jectClass=inetOrgPerson)(objectClass=posixAccount)(objectClass=eduPerson))(entryUUID=290262d3-1fa8-4b12-b59b-314132bc72d3))" \
attrs="cn,description,displayName,eduPersonAffiliation,eduPersonAssurance,eduPersonEnt \
itlement,eduPersonNickname,eduPersonOrgDN,eduPersonOrgUnitDN,eduPersonPrimaryAffiliati \
on,eduPersonPrimaryOrgUnitDN,eduPersonPrincipalName,eduPersonScopedAffiliation,eduPers \
onTargetedID,employeeNumber,employeeType,entryUUID,givenName,objectClass,sn,uid,userName"
[06/Feb/2015:14:04:43 -0500] SEARCH RES conn=3 op=148 msgID=149 result=0 nentries=1 \
etime=3 [06/Feb/2015:14:04:43 -0500] SEARCH REQ conn=3 op=149 msgID=150 \
base="dc=mydc" scope=wholeSubtree filter="(uniqueMember=uid=qo55068416,dc=mydc)" \
attrs="1.1" [06/Feb/2015:14:04:43 -0500] SEARCH RES conn=3 op=149 msgID=150 result=0 \
nentries=0 etime=0 [06/Feb/2015:14:04:44 -0500] SEARCH REQ conn=3 op=150 msgID=151 \
base="" scope=baseObject filter="(objectClass=*)" attrs="subschemaSubentry" \
[06/Feb/2015:14:04:44 -0500] SEARCH RES conn=3 op=150 msgID=151 result=0 nentries=1 \
etime=2 [06/Feb/2015:14:04:44 -0500] ADD REQ conn=3 op=151 msgID=152 \
dn="uid=qo55068416,dc=mydc" [06/Feb/2015:14:04:44 -0500] ADD RES conn=3 op=151 \
msgID=152 result=68 message="The entry uid=qo55068416,dc=mydc cannot be added because \
an entry with that name already exists" etime=2
Is it because of the (uniqueMember=uid=qo55068416,dc=mydc) search? (Also, what \
causes that search?) this is group membership search. Are you dealing with groups?
2x ADD can sometimes happen when no correlation query is defined.
would you share your sync.json?
Thanks,
Vincent
_______________________________________________
OpenIDM mailing list
OpenIDM@forgerock.org<mailto:OpenIDM@forgerock.org>
https://lists.forgerock.org/mailman/listinfo/openidm
_______________________________________________
OpenIDM mailing list
OpenIDM@forgerock.org<mailto:OpenIDM@forgerock.org>
https://lists.forgerock.org/mailman/listinfo/openidm
[Attachment #3 (text/html)]
<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" id="owaParaStyle"></style>
</head>
<body fpstyle="1" ocsi="0">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">Hi \
Matthias, <div><br>
</div>
<div>I ran a 500K batch import this week-end without a single error this week-end \
after removing the patch operation. Before that, I had a duplicate create and a \
stack trace about it for every new managed/user created when it tried to auto-sync to \
OpenDJ.</div> <div><br>
</div>
<div><span style="font-size: 10pt;">The patch was triggered this way :</span></div>
<div><br>
</div>
<div>We add managed/users manually through scripting.</div>
<div><br>
</div>
<div>sync.json defines a mapping from managed/user to opendj/account. We rely \
on auto-sync to create the accounts, and run a recon dialy in case some didn't get \
through in realtime.</div> <div><br>
</div>
<div>In that mapping, we generate a few attributes for OpenDJ (displayName and such), \
among which the uidNumber attribute. We want this attribute to be unique among \
all OpenDJ accounts so I made a transform script that :</div> <div><br>
</div>
<div>1 - If source.uidNumber is undefined, continue and assign a new uidNumber</div>
<div>2 - Make a random number from 1 million to 1 100 million (boundaries chosen to \
stay within a 32-bit integer while avoiding possible conflicts).</div> <div>3 - \
Verify that this number is NOT already assigned through a openidm.query to OpenDJ \
directly.</div> <div>4 - Assign the number through a patch.push.</div>
<div>5 - Return the uidNumber.</div>
<div>
<div><br>
</div>
<div>Now, it is clear to me that this transform script should not have been patching \
users directly as OpenIDM will receive the correct uidNumber and apply it to the \
account correctly. </div> <div><br>
</div>
<div>It can be classified as a case of configuration error, but I still believe the \
behavior is weird - duplicate creates to the external system should not happen \
because if our OpenDJ installation was setup in such a way to allow those duplicates, \
we might have a rather large mess to cleanup now.</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>Vincent</div>
<div><br>
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div style=""><br>
</div>
</div>
</div>
</div>
<div style="font-family: Times New Roman; color: #000000; font-size: 16px">
<hr tabindex="-1">
<div id="divRpF371080" style="direction: ltr;"><font face="Tahoma" size="2" \
color="#000000"><b>De :</b> openidm-bounces@forgerock.org \
[openidm-bounces@forgerock.org] de la part de Matthias Tristl \
[matthias.tristl@forgerock.com]<br> <b>Date d'envoi :</b> 9 février 2015 01:44<br>
<b>À :</b> openidm@forgerock.org<br>
<b>Objet :</b> Re: [OpenIDM] RE : User creation error OpenDJ<br>
</font><br>
</div>
<div></div>
<div>
<div dir="ltr">Hi Vincent,
<div><br>
</div>
<div>I agree, a patch should not trigger a create and I would be surprised if it \
does. Should be very easy to test in your set up, right?</div> <div><br>
</div>
<div>On the other hand, can you explain why you use a patch as part of a \
synchronisation? That does not make much sense (if the target of the patch is the \
same as the sunc targeting) for me except for actions like LINK or UNLINK in which \
case the patch command should be part of a onLink (...) script.</div>
<div><br>
</div>
<div>Can you explain?</div>
<div><br>
</div>
<div>Matthias</div>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div class="gmail_signature">
<div dir="ltr">
<div><br>
</div>
<div><br>
</div>
<div>Matthias Tristl : ForgeRock INC<br>
e: <a href="mailto:matthias.tristl@forgerock.com" \
target="_blank">matthias.tristl@forgerock.com</a><br>
t: +47 47707662<br>
w: <a href="http://forgerock.com" target="_blank">forgerock.com</a></div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On Fri, Feb 6, 2015 at 11:42 PM, Belleville-Rioux, Vincent
<span dir="ltr"><<a href="mailto:rioux.vincent@uqam.ca" \
target="_blank">rioux.vincent@uqam.ca</a>></span> wrote:<br> <blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; \
padding-left:1ex"> <div bgcolor="#FFFFFF">
<div style="direction:ltr; font-family:Tahoma; color:#000000; font-size:10pt">I think \
I hit an undiscovered bug. <div><br>
</div>
<div>I found what's happening :</div>
<div><br>
</div>
<div>Consider the following scenario :</div>
<div><br>
</div>
<div>We create a managed/user through the REST interface.</div>
<div><br>
</div>
<div>This triggers an auto-sync to a LDAP connector.</div>
<div><br>
</div>
<div>There is a transform script defined on an attribute between managed/user and the \
target opendj. This transform script calls an openidm.patch directly.</div> \
<div><br> </div>
<div>Now, what happens :</div>
<div><br>
</div>
<div>The account is created in the LDAP system correctly. However we see a \
duplicate CREATE action being triggered by OpenIDM. The second one fails as the \
user already exists in the LDAP system.</div> <div><br>
</div>
<div>Why this happens :</div>
<div><br>
</div>
<div>I *think* that what happens is that the openidm.patch call results in a CREATE \
action on the target system because the user has not been created yet when it is \
evaluated. Then, the actual CREATE action triggered by the autosync from the \
addition of the managed/user will be tried later, and fail.</div>
<div><br>
</div>
<div>I'd classify this as a bug because calling a patch on a non-existant user should \
not result in a create. It should throw the proper error.</div> <div><br>
</div>
<div>Also, if my understanding of what happens isn't right and the patch is done \
*after* the user has been created, then it should still not result in a duplicate \
CREATE event.</div> <div><br>
</div>
<div>Vincent</div>
<div>
<div><br>
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div><br>
</div>
</div>
</div>
</div>
<div style="font-family:Times New Roman; color:#000000; font-size:16px">
<hr>
<div style="direction:ltr"><font face="Tahoma" color="#000000"><b>De :</b> <a \
href="mailto:openidm-bounces@forgerock.org" target="_blank"> \
openidm-bounces@forgerock.org</a> [<a href="mailto:openidm-bounces@forgerock.org" \
target="_blank">openidm-bounces@forgerock.org</a>] de la part de Gael Allioux [<a \
href="mailto:gael.allioux@forgerock.com" \
target="_blank">gael.allioux@forgerock.com</a>]<br> <b>Date d'envoi :</b> 6 février \
2015 14:28<br> <b>À :</b> <a href="mailto:openidm@forgerock.org" \
target="_blank">openidm@forgerock.org</a><br> <b>Objet :</b> Re: [OpenIDM] User \
creation error OpenDJ<br> </font><br>
</div>
<div>
<div class="h5">
<div></div>
<div><br>
<div>On 02/06/2015 08:22 PM, Belleville-Rioux, Vincent wrote:<br>
</div>
<blockquote type="cite">
<div style="direction:ltr; font-family:Tahoma; color:#000000; font-size:10pt">
<div><span style="font-size:10pt">Anybody has an idea what would cause OpenIDM to try \
and add a user two times to a single external system (OpenDJ) ? See the \
following OpenDJ log coming from an automatic sync from OpenIDM :</span></div> \
<div><br> <div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div>
<div>[06/Feb/2015:14:04:43 -0500] <b>ADD REQ conn=3 op=145 msgID=146 \
dn="uid=qo55068416,dc=mydc"</b></div> <div>[06/Feb/2015:14:04:43 -0500] ADD \
RES conn=3 op=145 msgID=146 result=0 etime=33</div> <div>[06/Feb/2015:14:04:43 -0500] \
SEARCH REQ conn=3 op=146 msgID=147 base="<b>uid=qo55068416,dc=mydc</b>" \
scope=baseObject filter="(objectClass=*)" attrs="entryUUID"</div> \
<div>[06/Feb/2015:14:04:43 -0500] SEARCH RES conn=3 op=146 msgID=147 result=0 <b> \
nentries=1 </b>etime=2</div> <div>[06/Feb/2015:14:04:43 -0500] SEARCH REQ conn=3 \
op=147 msgID=148 base="" scope=baseObject \
filter="(objectClass=*)" attrs="subschemaSubentry"</div> \
<div>[06/Feb/2015:14:04:43 -0500] SEARCH RES conn=3 op=147 msgID=148 result=0 \
nentries=1 etime=2</div> <div>[06/Feb/2015:14:04:43 -0500] SEARCH REQ conn=3 op=148 \
msgID=149 base="dc=mydc" scope=wholeSubtree \
filter="(&(&(objectClass=top)(objectClass=person)(objectClass=organizatio \
nalPerson)(objectClass=inetOrgPerson)(objectClass=posixAccount)(objectClass=eduPerson))(entryUUID=290262d3-1fa8-4b12-b59b-314132bc72d3))"
attrs="cn,description,displayName,eduPersonAffiliation,eduPersonAssurance,eduPe \
rsonEntitlement,eduPersonNickname,eduPersonOrgDN,eduPersonOrgUnitDN,eduPersonPrimaryAf \
filiation,eduPersonPrimaryOrgUnitDN,eduPersonPrincipalName,eduPersonScopedAffiliation, \
eduPersonTargetedID,employeeNumber,employeeType,entryUUID,givenName,objectClass,sn,uid,userName"</div>
<div>[06/Feb/2015:14:04:43 -0500] SEARCH RES conn=3 op=148 msgID=149 result=0 \
nentries=1 etime=3</div> <div>[06/Feb/2015:14:04:43 -0500] SEARCH REQ conn=3 op=149 \
msgID=150 base="dc=mydc" scope=wholeSubtree \
filter="(uniqueMember=uid=qo55068416,dc=mydc)" attrs="1.1"</div> \
<div>[06/Feb/2015:14:04:43 -0500] SEARCH RES conn=3 op=149 msgID=150 result=0 <b \
style="background-color:rgb(255,255,0)"> nentries=0</b> etime=0</div>
<div>[06/Feb/2015:14:04:44 -0500] SEARCH REQ conn=3 op=150 msgID=151 \
base="" scope=baseObject filter="(objectClass=*)" \
attrs="subschemaSubentry"</div> <div>[06/Feb/2015:14:04:44 -0500] SEARCH \
RES conn=3 op=150 msgID=151 result=0 nentries=1 etime=2</div> \
<div>[06/Feb/2015:14:04:44 -0500] ADD REQ conn=3 op=151 msgID=152 \
dn="uid=qo55068416,dc=mydc"</div> <div>[06/Feb/2015:14:04:44 -0500] <b>ADD \
RES conn=3 op=151 msgID=152 result=68 message="The entry uid=qo55068416,dc=mydc \
cannot be added because an entry with that name already exists" \
etime=2</b></div> <div><b><br>
</b></div>
<div>Is it because of the (uniqueMember=uid=qo55068416,dc=mydc) search? (Also, \
what causes that search?)</div> </div>
</div>
</div>
</div>
</div>
</blockquote>
this is group membership search. Are you dealing with groups?<br>
<br>
2x ADD can sometimes happen when no correlation query is defined.<br>
<br>
would you share your sync.json?<br>
<br>
<br>
<br>
<blockquote type="cite">
<div style="direction:ltr; font-family:Tahoma; color:#000000; font-size:10pt">
<div>
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>Vincent</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset target="_blank"></fieldset> <br>
<pre>_______________________________________________
OpenIDM mailing list
<a href="mailto:OpenIDM@forgerock.org" target="_blank">OpenIDM@forgerock.org</a>
<a href="https://lists.forgerock.org/mailman/listinfo/openidm" \
target="_blank">https://lists.forgerock.org/mailman/listinfo/openidm</a> </pre>
</blockquote>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
_______________________________________________<br>
OpenIDM mailing list<br>
<a href="mailto:OpenIDM@forgerock.org" target="_blank">OpenIDM@forgerock.org</a><br>
<a href="https://lists.forgerock.org/mailman/listinfo/openidm" \
target="_blank">https://lists.forgerock.org/mailman/listinfo/openidm</a><br> <br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</body>
</html>
_______________________________________________
OpenIDM mailing list
OpenIDM@forgerock.org
https://lists.forgerock.org/mailman/listinfo/openidm
--===============3886218366594061819==--
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic