[prev in list] [next in list] [prev in thread] [next in thread]
List: mifos-developer
Subject: Re: [Mifos-developer] Questions around data migration/ initial
From: "Amy Bensinger (Contractor)" <abensinger () grameenfoundation ! org>
Date: 2007-12-24 7:20:56
Message-ID: 9DD845C1ED0D5D40B4B56DF5A4B1EB0E024B88D9 () gfmail ! gfusa ! org
[Download RAW message or body]
--===============1913454093==
Content-class: urn:content-classes:message
Content-Type: multipart/alternative;
boundary="----_=_NextPart_001_01C845FD.81FD5A6A"
This is a multi-part message in MIME format.
We have other deployments that are migrating from ledger-based systems
to Mifos. (IIRM, for example).
The answers below are a little more vague than I'd like to provide, but
I'll try to come up with something more complete. In brief, though:
1) Some portion of the GK tools may be useful to you (it looks like some
files may have been taken down-I'll see if I can find them or re-post
them). However, ass you are going to 1.1, the schema changes and
additions will mean you'll need different processes.
The most important step-and you may be well into this portion-is getting
the data mapping and data clean-up correct. In my experience, the
mechanics of the SQL are not nearly as difficult as mapping and cleaning
the data, whether you write custom scripts or use an existing ETL tool.
Mapping requires a pretty thorough understanding of how both the legacy
and new systems are used. It sounds like you already have some good ETL
ideas, but perhaps others can chime in?
2) You can import historical data into the system.
a. Again, the mapping is the key in this area-i.e. understanding
how you will use the historical data for reporting. If the data will
truly be used for reporting and decision-making, it's worth keeping. If
you find-as can often be the case in data migration scenarios-that you
can report adequately off archived data and/or the data will not be used
frequently, the extra effort may not be worthwhile. If your goal is to
use loan history for evaluation of future loans for existing clients,
then you will want to investigate migrating the results of previous
loans into Mifos.
b. My assumption would be that you want to migrate all active loans
into the system. This process is possible, and looking at GK mapping
may be extremely helpful.
3) We have been discussing this process. I think now that 1.1 is
coming out and Mifos will be more stable and better built, the processes
will get more repeatable. As always (like GK has done), we'd love the
successes and non-successes to be documented for the community.
Again-these answers are quick. I'll see what other information I (or
others!) can provide.
Thanks --aB
________________________________
From: mifos-developer-bounces@lists.sourceforge.net
[mailto:mifos-developer-bounces@lists.sourceforge.net] On Behalf Of
Soham Dhakal
Sent: Monday, December 24, 2007 11:27 AM
To: 'Mifos functional discussions'; 'Developer'
Subject: [Mifos-developer] Questions around data migration/ initial
setup ofmifos
Hi All,
As part of our detailed gap analysis, we are trying to figure out ways
to get the existing data (manual) into mifos. I searched on the website
and also the mailing list to get some ideas. I found a link
http://mifos.org/knowledge/deployment-project-pages/grameen-koota/data-m
igration
but it does not have anything listed. Are there some other documentation
with regards to this. I am looking at Kettle to see how to integrate it
but i am not too familiar with it.
Here are some things we are trying to address, please provide your input
as to how these are being accomplished by others..
1. We would like to import all the client information from a csv file
(client demographics, Group information, Center information, Branch
information), or something similar, has anyone written scripts to do
this?
2. How to import existing loans into the system
a. Old loans that have been completed. - Do we import all the
previous loan cycles (does mifos even allow this, I mean how will all of
the different transactions related to a loan be updated)?
b. Loans that are ongoing but have installments remaining.
3. Should we have a standard process for migration into Mifos? What i
mean is that regardless of the existing system, we have standard formats
(csv, xml?) that can be imported into mifos through import scripts. So
the only thing MFIs would have to consider is dumping out their existing
data in the format recognized by mifos.
Thanks
Soham
[Attachment #3 (text/html)]
<html xmlns:v="urn:schemas-microsoft-com:vml" \
xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:blue;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:Calibri;
color:blue;
font-weight:normal;
font-style:normal;
text-decoration:none none;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
/* List Definitions */
@list l0
{mso-list-id:195385418;
mso-list-type:hybrid;
mso-list-template-ids:-1624055502 67698705 67698713 67698715 67698703 67698713 \
67698715 67698703 67698713 67698715;} @list l0:level1
{mso-level-start-at:2;
mso-level-text:"%1\)";
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1
{mso-list-id:620457070;
mso-list-type:hybrid;
mso-list-template-ids:867968150 67698703 67698713 67698715 67698703 67698713 \
67698715 67698703 67698713 67698715;} @list l1:level1
{mso-level-tab-stop:.25in;
mso-level-number-position:left;
margin-left:.25in;
text-indent:-.25in;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
-->
</style>
</head>
<body lang=EN-US link=blue vlink=blue>
<div class=Section1>
<p class=MsoNormal><font size=2 color=blue face=Calibri><span style='font-size:
11.0pt;font-family:Calibri;color:blue'>We have other deployments that are
migrating from ledger-based systems to Mifos. (IIRM, for \
example).<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Calibri><span style='font-size:
11.0pt;font-family:Calibri;color:blue'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Calibri><span style='font-size:
11.0pt;font-family:Calibri;color:blue'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Calibri><span style='font-size:
11.0pt;font-family:Calibri;color:blue'>The answers below are a little more
vague than I’d like to provide, but I’ll try to come up with
something more complete. In brief, though: <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Calibri><span style='font-size:
11.0pt;font-family:Calibri;color:blue'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Calibri><span style='font-size:
11.0pt;font-family:Calibri;color:blue'>1) Some portion of the GK tools may be
useful to you (it looks like some files may have been taken down—I’ll
see if I can find them or re-post them). However, ass you are going to 1.1, the
schema changes and additions will mean you’ll need different \
processes.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Calibri><span style='font-size:
11.0pt;font-family:Calibri;color:blue'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Calibri><span style='font-size:
11.0pt;font-family:Calibri;color:blue'>The most important step—and you may
be well into this portion—is getting the data mapping and data clean-up
correct. In my experience, the mechanics of the SQL are not nearly as
difficult as mapping and cleaning the data, whether you write custom scripts or
use an existing ETL tool. Mapping requires a pretty thorough understanding of
how both the legacy and new systems are used. It sounds like you already have
some good ETL ideas, but perhaps others can chime in?<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Calibri><span style='font-size:
11.0pt;font-family:Calibri;color:blue'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Calibri><span style='font-size:
11.0pt;font-family:Calibri;color:blue'><o:p> </o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in;text-indent:-.25in;mso-list:l0 level1 \
lfo2'><![if !supportLists]><font size=2 color=blue face=Calibri><span \
style='font-size:11.0pt;font-family:Calibri; color:blue'><span \
style='mso-list:Ignore'>2)<font size=1 face="Times New Roman"><span style='font:7.0pt \
"Times New Roman"'> \
</span></font></span></span></font><![endif]><font size=2 color=blue \
face=Calibri><span style='font-size:11.0pt;font-family:Calibri; color:blue'>You can \
import historical data into the system. <o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:1.0in;text-indent:-.25in;mso-list:l0 level2 \
lfo2'><![if !supportLists]><font size=2 color=blue face=Calibri><span \
style='font-size:11.0pt;font-family:Calibri; color:blue'><span \
style='mso-list:Ignore'>a.<font size=1 face="Times New Roman"><span style='font:7.0pt \
"Times New Roman"'> \
</span></font></span></span></font><![endif]><font size=2 color=blue \
face=Calibri><span style='font-size:11.0pt;font-family:Calibri; color:blue'>Again, \
the mapping is the key in this area—i.e. understanding how you will use the \
historical data for reporting. If the data will truly be used for reporting and \
decision-making, it’s worth keeping. If you find—as can often be \
the case in data migration scenarios—that you can report adequately off \
archived data and/or the data will not be used frequently, the extra effort may not \
be worthwhile. If your goal is to use loan history for evaluation of future \
loans for existing clients, then you will want to investigate migrating the results \
of previous loans into Mifos.<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:1.0in;text-indent:-.25in;mso-list:l0 level2 \
lfo2'><![if !supportLists]><font size=2 color=blue face=Calibri><span \
style='font-size:11.0pt;font-family:Calibri; color:blue'><span \
style='mso-list:Ignore'>b.<font size=1 face="Times New Roman"><span style='font:7.0pt \
"Times New Roman"'> \
</span></font></span></span></font><![endif]><font size=2 color=blue \
face=Calibri><span style='font-size:11.0pt;font-family:Calibri; color:blue'>My \
assumption would be that you want to migrate all active loans into the system. \
This process is possible, and looking at GK mapping may be extremely \
helpful.<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in;text-indent:-.25in;mso-list:l0 level1 \
lfo2'><![if !supportLists]><font size=2 color=blue face=Calibri><span \
style='font-size:11.0pt;font-family:Calibri; color:blue'><span \
style='mso-list:Ignore'>3)<font size=1 face="Times New Roman"><span style='font:7.0pt \
"Times New Roman"'> \
</span></font></span></span></font><![endif]><font size=2 color=blue \
face=Calibri><span style='font-size:11.0pt;font-family:Calibri; color:blue'>We have \
been discussing this process. I think now that 1.1 is coming out and Mifos will \
be more stable and better built, the processes will get more repeatable. As \
always (like GK has done), we’d love the successes and non-successes to be \
documented for the community.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Calibri><span style='font-size:
11.0pt;font-family:Calibri;color:blue'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Calibri><span style='font-size:
11.0pt;font-family:Calibri;color:blue'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Calibri><span style='font-size:
11.0pt;font-family:Calibri;color:blue'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Calibri><span style='font-size:
11.0pt;font-family:Calibri;color:blue'>Again—these answers are quick. \
I’ll see what other information I (or others!) can \
provide.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Calibri><span style='font-size:
11.0pt;font-family:Calibri;color:blue'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Calibri><span style='font-size:
11.0pt;font-family:Calibri;color:blue'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Calibri><span style='font-size:
11.0pt;font-family:Calibri;color:blue'>Thanks --aB<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Calibri><span style='font-size:
11.0pt;font-family:Calibri;color:blue'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Calibri><span style='font-size:
11.0pt;font-family:Calibri;color:blue'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Calibri><span style='font-size:
11.0pt;font-family:Calibri;color:blue'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Calibri><span style='font-size:
11.0pt;font-family:Calibri;color:blue'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Calibri><span style='font-size:
11.0pt;font-family:Calibri;color:blue'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=blue face=Calibri><span style='font-size:
11.0pt;font-family:Calibri;color:blue'><o:p> </o:p></span></font></p>
<div>
<div class=MsoNormal align=center style='text-align:center'><font size=3
face="Times New Roman"><span style='font-size:12.0pt'>
<hr size=2 width="100%" align=center tabindex=-1>
</span></font></div>
<p class=MsoNormal><b><font size=2 face=Tahoma><span style='font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font size=2
face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'>
mifos-developer-bounces@lists.sourceforge.net
[mailto:mifos-developer-bounces@lists.sourceforge.net] <b><span
style='font-weight:bold'>On Behalf Of </span></b>Soham Dhakal<br>
<b><span style='font-weight:bold'>Sent:</span></b> Monday, December 24, 2007
11:27 AM<br>
<b><span style='font-weight:bold'>To:</span></b> 'Mifos functional
discussions'; 'Developer'<br>
<b><span style='font-weight:bold'>Subject:</span></b> [Mifos-developer]
Questions around data migration/ initial setup ofmifos</span></font><o:p></o:p></p>
</div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
<div>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Hi All,</span></font><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>As part of our detailed gap analysis, we are trying to
figure out ways to get the existing data (manual) into mifos. I searched on the
website and also the mailing list to get some ideas. I found a link \
</span></font><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><a
href="http://mifos.org/knowledge/deployment-project-pages/grameen-koota/data-migration"
title="http://mifos.org/knowledge/deployment-project-pages/grameen-koota/data-migrati \
on">http://mifos.org/knowledge/deployment-project-pages/grameen-koota/data-migration</a>
</span></font><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>but it does not have anything listed. Are
there some other documentation with regards to this. I am looking at Kettle to
see how to integrate it but i am not too familiar with \
it.</span></font><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'> <o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Here are some things we are trying to
address, please provide your input as to how these are being accomplished by
others..</span></font><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'> <o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>1. We would like to import all the client
information from a csv file (client demographics, Group information,
Center information, Branch information), or something similar, has anyone
written scripts to do this? </span></font><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'> <o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>2. How to import existing loans into the
system</span></font><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> a. Old loans that have
been completed. - Do we import all the previous loan cycles (does mifos
even allow this, I mean how will all of the different transactions related to a
loan be updated)?</span></font><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> b. Loans that are
ongoing but have installments remaining. </span></font><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'> <o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>3. Should we have a standard process for
migration into Mifos? What i mean is that regardless of the existing system, we
have standard formats (csv, xml?) that can be imported into mifos through
import scripts. So the only thing MFIs would have to consider is dumping out their
existing data in the format recognized by mifos. </span></font><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'> <o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'> <o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Thanks</span></font><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'> <o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Soham</span></font><o:p></o:p></p>
</div>
</div>
</body>
</html>
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
--===============1913454093==--
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic