[prev in list] [next in list] [prev in thread] [next in thread]
List: wsf-java-dev
Subject: Re: [Dev] [Hiera] Deeper hash merge instead of Native hash merge?
From: Rajkumar Rajaratnam <rajkumarr () wso2 ! com>
Date: 2015-12-28 6:30:24
Message-ID: CAFgrWyQ54dku5o-nFpPHQGgmJngxpqHJEHBChS5NVfkf7U0+Jg () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Hi Akila,
Thanks for your thought. Lets ship native hash merge data structures by
default. Those who wants deeper merging can alter the merge behavior in
hiera.yaml (:merge_behavior: deeper) and data structures in data yaml
files. This doesn't involve any changes in puppet modules.
Thanks,
Raj.
On Fri, Dec 25, 2015 at 9:44 PM, Akila Ravihansa Perera <ravihansa@wso2.com>
wrote:
> Hi Raj,
>
> It is true that native hash merging has its own limitations which would
> lead to data duplication in some cases. But we need to analyze the
> trade-off between data duplication against readability and trace-ability
> aspects. IMHO, it's okay to have few data duplication if it makes things
> more simple and less error prone.
>
> My concern is that what if this accidentally leads to unexpected data
> merging? I understand that native merging will merge the top level keys as
> well, but that is less susceptible for human errors compared to N-level
> deep merge.
>
> Also have a look at Puppet official docs about deep merging [1]. Looks
> like it is mainly intended for complex data structures and converting them
> to Puppet resources.
>
> We can ask the users to create their own Hiera data structures with deeper
> merging if they are comfortable with it. But given the additional
> complexity involved in configuring and installing deep_merge library [2],
> shall we ship native merge data structures by default?
>
> [1]
> https://docs.puppetlabs.com/hiera/3.0/lookup_types.html#deep-merging-in-hiera
> [2]
> https://docs.puppetlabs.com/hiera/3.0/lookup_types.html#deep-merging-in-hiera
>
> Thanks.
>
> On Fri, Dec 25, 2015 at 10:45 AM, Rajkumar Rajaratnam <rajkumarr@wso2.com>
> wrote:
>
>> Hi Devs,
>>
>> We are using *Native* hash merge right now. *It merges only the
>> top-level keys and values in each source hash. If the same key exists in
>> both a lower priority source and a higher priority source, the higher
>> priority value will be used. *But it doesn't allow us to override a
>> nested hash.
>>
>> Lets take an example,
>>
>> *hiera.yaml*
>>
>> ---
>> :hierarchy:
>> - "wso2/%{::product_name}/%{::product_version}/default"
>> - wso2/common
>> :backends:
>> - yaml
>> :yaml:
>> :datadir: "hieradata/%{::environment}"
>>
>> *wso2/common.yaml*
>>
>> ---
>> # Deployment synchronization configuration
>> wso2::dep_sync :
>> enabled : false
>> auto_checkout : true
>> auto_commit : false
>> repository_type : svn
>> svn :
>> url : http://svnrepo.example.com/repos/
>> user : username
>> password : password
>> append_tenant_id: true
>>
>> *wso2/wso2esb/4.9.0/default.yaml*
>>
>> ---
>> wso2::dep_sync :
>> enabled : true
>> svn :
>> url : http://rajkumar.com/repos/
>>
>>
>> *Native* hash merge would return the following;
>>
>> {
>> "enabled" =>true,
>> "auto_checkout" =>true,
>> "auto_commit" =>false,
>> "repository_type" =>"svn",
>> "svn" => {
>> "url" =>"http://rajkumar.com/repos/"
>> }
>> }
>>
>> *Deeper* hash merge would return the following;
>>
>> {
>> "enabled" =>true,
>> "auto_checkout" =>true,
>> "auto_commit" =>false,
>> "repository_type" =>"svn",
>> "svn" => {
>> "url" =>"http://rajkumar.com/repos/",
>> "user" =>"username",
>> "password" =>"password",
>> "append_tenant_id" =>true
>> }
>> }
>>
>>
>> If we use *Native* hash merging, we need to copy the complete nested
>> hash ("svn" in above example) and override it. If we use *Deeper* hash
>> merging, we can copy only a part of a nested hash and override it. As you
>> can see, *Deeper* hash merge reduces data duplication.
>>
>> Thoughts please.
>>
>> Thanks,
>> Raj.
>>
>> --
>> Rajkumar Rajaratnam
>> Committer & PMC Member, Apache Stratos
>> Software Engineer, WSO2
>>
>> Mobile : +94777568639
>> Blog : rajkumarr.com
>>
>
>
>
> --
> Akila Ravihansa Perera
> WSO2 Inc.; http://wso2.com/
>
> Blog: http://ravihansa3000.blogspot.com
>
--
Rajkumar Rajaratnam
Committer & PMC Member, Apache Stratos
Software Engineer, WSO2
Mobile : +94777568639
Blog : rajkumarr.com
[Attachment #5 (text/html)]
<div dir="ltr"><div class="gmail_default" \
style="font-family:tahoma,sans-serif;font-size:small">Hi Akila,<br><br></div><div \
class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">Thanks \
for your thought. Lets ship native hash merge data structures by default. Those who \
wants deeper merging can alter the merge behavior in hiera.yaml (:merge_behavior: \
deeper) and data structures in data yaml files. This doesn't involve any changes \
in puppet modules. <br><br></div><div class="gmail_default" \
style="font-family:tahoma,sans-serif;font-size:small">Thanks,<br></div><div \
class="gmail_default" \
style="font-family:tahoma,sans-serif;font-size:small">Raj.<br></div></div><div \
class="gmail_extra"><br><div class="gmail_quote">On Fri, Dec 25, 2015 at 9:44 PM, \
Akila Ravihansa Perera <span dir="ltr"><<a href="mailto:ravihansa@wso2.com" \
target="_blank">ravihansa@wso2.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">Hi Raj,<div><br></div><div>It is true that \
native hash merging has its own limitations which would lead to data duplication in \
some cases. But we need to analyze the trade-off between data duplication against \
readability and trace-ability aspects. IMHO, it's okay to have few data \
duplication if it makes things more simple and less error prone. \
</div><div><br></div><div>My concern is that what if this accidentally leads to \
unexpected data merging? I understand that native merging will merge the top level \
keys as well, but that is less susceptible for human errors compared to N-level deep \
merge.</div><div><br></div><div>Also have a look at Puppet official docs about deep \
merging [1]. Looks like it is mainly intended for complex data structures and \
converting them to Puppet resources.</div><div><br></div><div>We can ask the users to \
create their own Hiera data structures with deeper merging if they are comfortable \
with it. But given the additional complexity involved in configuring and installing \
deep_merge library [2], shall we ship native merge data structures by \
default?</div><div><br></div><div>[1] <a \
href="https://docs.puppetlabs.com/hiera/3.0/lookup_types.html#deep-merging-in-hiera" \
target="_blank">https://docs.puppetlabs.com/hiera/3.0/lookup_types.html#deep-merging-in-hiera</a></div><div>[2] \
<a href="https://docs.puppetlabs.com/hiera/3.0/lookup_types.html#deep-merging-in-hiera" \
target="_blank">https://docs.puppetlabs.com/hiera/3.0/lookup_types.html#deep-merging-in-hiera</a></div><div><br></div><div>Thanks.</div></div><div \
class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On Fri, Dec 25, \
2015 at 10:45 AM, Rajkumar Rajaratnam <span dir="ltr"><<a \
href="mailto:rajkumarr@wso2.com" target="_blank">rajkumarr@wso2.com</a>></span> \
wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px \
#ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" \
style="font-family:tahoma,sans-serif;font-size:small">Hi Devs,<br><br></div><div \
class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">We are \
using <b>Native</b> hash merge right now. <i>It merges only the top-level keys and \
values in each source hash. If the same key exists in both a lower priority source \
and a higher priority source, the higher priority value will be used. </i>But it \
doesn't allow us to override a nested hash.<br><br></div><div \
class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">Lets take \
an example, <br><br></div><div class="gmail_default" \
style="font-family:tahoma,sans-serif;font-size:small"><u>hiera.yaml</u><br><pre \
style="background-color:rgb(43,43,43);color:rgb(169,183,198);font-family:"Source \
Code Pro";font-size:10.5pt">---<br><span \
style="color:rgb(204,120,50);font-weight:bold">:hierarchy:</span><span \
style="color:rgb(106,135,89)"><br></span><span style="color:rgb(106,135,89)"> \
</span>- <span style="color:rgb(106,135,89)">"wso2/%{::product_name}/%{::product_version}/default"<br></span><span \
style="color:rgb(106,135,89)"> </span>- wso2/common<br><span \
style="color:rgb(204,120,50);font-weight:bold">:backends:<br></span><span \
style="color:rgb(204,120,50);font-weight:bold"> </span>- yaml<br><span \
style="color:rgb(204,120,50);font-weight:bold">:yaml:<br></span><span \
style="color:rgb(204,120,50);font-weight:bold"> :datadir: </span><span \
style="color:rgb(106,135,89)">"hieradata/%{::environment}"</span></pre><u>wso2/common.yaml</u><br><pre \
style="background-color:rgb(43,43,43);color:rgb(169,183,198);font-family:"Source \
Code Pro";font-size:10.5pt">---<br><span \
style="color:rgb(98,151,85);font-style:italic"># Deployment synchronization \
configuration<br></span>wso2::dep_sync :<br> <span \
style="color:rgb(204,120,50);font-weight:bold">enabled : </span>false<br> <span \
style="color:rgb(204,120,50);font-weight:bold">auto_checkout : </span>true<br> \
<span style="color:rgb(204,120,50);font-weight:bold">auto_commit : </span>false<br> \
<span style="color:rgb(204,120,50);font-weight:bold">repository_type : </span>svn<br> \
<span style="color:rgb(204,120,50);font-weight:bold">svn :<br></span><span \
style="color:rgb(204,120,50);font-weight:bold"> url : </span><a \
href="http://svnrepo.example.com/repos/" \
target="_blank">http://svnrepo.example.com/repos/</a><br> <span \
style="color:rgb(204,120,50);font-weight:bold">user : </span>username<br> <span \
style="color:rgb(204,120,50);font-weight:bold">password : </span>password<br> \
<span style="color:rgb(204,120,50);font-weight:bold">append_tenant_id: \
</span>true</pre><u>wso2/wso2esb/4.9.0/default.yaml</u><br><pre \
style="background-color:rgb(43,43,43);color:rgb(169,183,198);font-family:"Source \
Code Pro";font-size:10.5pt">---<br>wso2::dep_sync :<br> <span \
style="color:rgb(204,120,50);font-weight:bold">enabled : </span>true<br> <span \
style="color:rgb(204,120,50);font-weight:bold">svn :<br></span><span \
style="color:rgb(204,120,50);font-weight:bold"> url : </span><a \
href="http://rajkumar.com/repos/" \
target="_blank">http://rajkumar.com/repos/</a><br></pre><br></div><div \
class="gmail_default" \
style="font-family:tahoma,sans-serif;font-size:small"><b>Native</b> hash merge would \
return the following;<br><pre \
style="background-color:rgb(43,43,43);color:rgb(169,183,198);font-family:"Source \
Code Pro";font-size:10.5pt">{<br> <span \
style="color:rgb(106,135,89)">"enabled" </span>=>true,<br> <span \
style="color:rgb(106,135,89)">"auto_checkout" </span>=>true,<br> \
<span style="color:rgb(106,135,89)">"auto_commit" </span>=>false,<br> \
<span style="color:rgb(106,135,89)">"repository_type" </span>=><span \
style="color:rgb(106,135,89)">"svn"</span>,<br> <span \
style="color:rgb(106,135,89)">"svn" </span>=> {<br> \
"url" =>"<a href="http://rajkumar.com/repos/" \
target="_blank">http://rajkumar.com/repos/</a>"<br> \
}<br>}</pre><div><span><b>Deeper</b> hash merge would return the \
following;</span><br><pre \
style="background-color:rgb(43,43,43);color:rgb(169,183,198);font-family:"Source \
Code Pro";font-size:10.5pt">{<br> <span \
style="color:rgb(106,135,89)">"enabled" </span>=>true,<br> <span \
style="color:rgb(106,135,89)">"auto_checkout" </span>=>true,<br> \
<span style="color:rgb(106,135,89)">"auto_commit" </span>=>false,<br> \
<span style="color:rgb(106,135,89)">"repository_type" </span>=><span \
style="color:rgb(106,135,89)">"svn"</span>,<br> <span \
style="color:rgb(106,135,89)">"svn" </span>=> {<br> \
"url" =>"<a href="http://rajkumar.com/repos/" \
target="_blank">http://rajkumar.com/repos/</a>",<br> "user" \
=>"username",<br> "password" \
=>"password",<br> "append_tenant_id" =>true<br> \
}<br>}</pre><br>If we use <b>Native</b> hash merging, we need to copy the complete \
nested hash ("svn" in above example) and override it. If we use \
<b>Deeper</b> hash merging, we can copy only a part of a nested hash and override it. \
As you can see, <b>Deeper</b> hash merge reduces data duplication. \
</div><br></div><div class="gmail_default" \
style="font-family:tahoma,sans-serif;font-size:small">Thoughts please. <br></div><div \
class="gmail_default" \
style="font-family:tahoma,sans-serif;font-size:small"><br></div><div \
class="gmail_default" \
style="font-family:tahoma,sans-serif;font-size:small">Thanks,<br></div><div \
class="gmail_default" \
style="font-family:tahoma,sans-serif;font-size:small">Raj.<span><font \
color="#888888"><br clear="all"></font></span></div><span><font \
color="#888888"><br>-- <br><div><div dir="ltr"><div><div dir="ltr"><div><div \
dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><span \
style="color:rgb(102,102,102)"><font face="verdana, sans-serif">Rajkumar \
Rajaratnam<br></font></span></div><div><span style="color:rgb(102,102,102)"><font \
face="verdana, sans-serif">Committer & PMC Member, Apache \
Stratos<br></font></span></div><div><span style="color:rgb(102,102,102)"><font \
face="verdana, sans-serif">Software Engineer, \
WSO2<br><br></font></span></div><div><span style="color:rgb(102,102,102)"><font \
face="verdana, sans-serif">Mobile : <a href="tel:%2B94777568639" value="+94777568639" \
target="_blank">+94777568639</a><br></font></span></div><div><span \
style="color:rgb(102,102,102)"><font face="verdana, sans-serif">Blog : <a \
href="http://rajkumarr.com" \
target="_blank">rajkumarr.com</a><br></font></span></div><div><span \
style="color:rgb(102,102,102)"></span></div></div></div></div></div></div></div></div></div></div></div></div></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span \
class="HOEnZb"><font color="#888888">-- <br><div><div dir="ltr"><div><div \
dir="ltr"><div>Akila Ravihansa Perera<br>WSO2 Inc.<span \
style="color:rgb(136,136,136);font-size:12.8000001907349px">; </span><a \
href="http://wso2.com/" style="color:rgb(17,85,204);font-size:12.8000001907349px" \
target="_blank">http://wso2.com/</a><br><br>Blog: <a \
href="http://ravihansa3000.blogspot.com" \
target="_blank">http://ravihansa3000.blogspot.com</a></div></div></div></div></div> \
</font></span></div> </blockquote></div><br><br clear="all"><br>-- <br><div \
class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div \
dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><span \
style="color:rgb(102,102,102)"><font face="verdana, sans-serif">Rajkumar \
Rajaratnam<br></font></span></div><div><span style="color:rgb(102,102,102)"><font \
face="verdana, sans-serif">Committer & PMC Member, Apache \
Stratos<br></font></span></div><div><span style="color:rgb(102,102,102)"><font \
face="verdana, sans-serif">Software Engineer, \
WSO2<br><br></font></span></div><div><span style="color:rgb(102,102,102)"><font \
face="verdana, sans-serif">Mobile : +94777568639<br></font></span></div><div><span \
style="color:rgb(102,102,102)"><font face="verdana, sans-serif">Blog : <a \
href="http://rajkumarr.com" \
target="_blank">rajkumarr.com</a><br></font></span></div><div><span \
style="color:rgb(102,102,102)"></span></div></div></div></div></div></div></div></div></div></div></div></div></div>
</div>
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic