[prev in list] [next in list] [prev in thread] [next in thread]
List: olpc-sugar
Subject: Re: [Sugar-devel] Draft Proposal for Add AI to Chat activity
From: Sujay R <sujay1844 () gmail ! com>
Date: 2024-04-02 2:25:00
Message-ID: CA+kJ3JOQz3JARbxGDrvUJGbMeW7si4S5zcim78Fdo7aTRi=LeA () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Yea sorry about that code snippet. I don't know why I added that, even I
don't understand it. I just wanted to let you know that I can figure out
the part of integrating the bot into the activity. That's why I added that
random snippet I've removed it.
I'm attaching my final proposal, that I'm submitting, as a reference.
Thank you for your assistance throughout the proposal review process
On Tue, Apr 2, 2024 at 12:04 AM Chihurumnaya Ibiam <ibiam@sugarlabs.org>
wrote:
> Please use the reply-all button to keep the conversation in the list.
>
> *"Please excuse me if I'm wrong, I'm not very familiar with the
> codebase.:"*
>
> You should remove this line, it serves no purpose, I also don't understand
> the code snippet
> you added and why.
>
> --
>
> Ibiam Chihurumnaya
> ibiam@sugarlabs.org
>
>
>
> On Sun, Mar 31, 2024 at 6:10 PM Sujay R <sujay1844@gmail.com> wrote:
>
>> I've taken your advice and added more details about the implementation.
>> I'm sorry if sharing PDFs again and again is irritating. I'm sure you'll
>> let me know of a better method.
>>
>> I have asked you to review multiple versions of my draft proposal. I'm
>> very grateful for you take out your time and help me applying. Can you
>> please review it one last time? I'll submit once you give the green light.
>>
>> Thank you again for your guidance and support throughout this process
>>
>> On Fri, Mar 29, 2024 at 8:10 PM Chihurumnaya Ibiam <ibiam@sugarlabs.org>
>> wrote:
>>
>>> Your timeline is terse, it'll be great if you add some implementation
>>> detail to your proposal as there's
>>> scarcely any.
>>>
>>> --
>>>
>>> Ibiam Chihurumnaya
>>> ibiam@sugarlabs.org
>>>
>>>
>>>
>>> On Thu, Mar 28, 2024 at 7:54 PM Sujay R <sujay1844@gmail.com> wrote:
>>>
>>>> Ok, the cloud sounds good.
>>>>
>>>> And thank you for patiently discussing the requirements with me and
>>>> answering my questions. Now, I think I can come up with a tentative
>>>> timeline.
>>>> I've attached v2 of my proposal with the timeline. Please let me know
>>>> if there are any more changes. If none, I'll submit it on the GSoC website.
>>>> The deadline is right around the corner
>>>>
>>>> On Wed, Mar 27, 2024 at 11:37 PM Chihurumnaya Ibiam <
>>>> ibiam@sugarlabs.org> wrote:
>>>>
>>>>> We were thinking of a cloud option, but we haven't decided yet.
>>>>>
>>>>> --
>>>>>
>>>>> Ibiam Chihurumnaya
>>>>> ibiam@sugarlabs.org
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Mar 26, 2024 at 6:40 PM Sujay R <sujay1844@gmail.com> wrote:
>>>>>
>>>>>> Thanks for explaining that. But it's still not clear to me where the
>>>>>> FOSS LLM should be run. Not on the devices running Sugar, so is cloud the
>>>>>> option you're looking for?
>>>>>>
>>>>>> On Tue, Mar 26, 2024 at 11:02 PM Chihurumnaya Ibiam <
>>>>>> ibiam@sugarlabs.org> wrote:
>>>>>>
>>>>>>> The plan was never really to run an LLM on Sugar as that'll
>>>>>>> drastically increase the size of the activity
>>>>>>> and Sugar itself as Chat is a fructose
>>>>>>> <https://wiki.sugarlabs.org/go/Development_Team/Release/Modules>
>>>>>>> activity.
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Ibiam Chihurumnaya
>>>>>>> ibiam@sugarlabs.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Mar 26, 2024 at 10:54 AM Sujay R <sujay1844@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Sugar runs on a lot of devices including low end devices - 2GB ram
>>>>>>>>> - and we intend to keep it that way, the chat activity is
>>>>>>>>> typically used by more than one Sugar instance, the chatbot should
>>>>>>>>> also be able to run on just one instance.
>>>>>>>>
>>>>>>>>
>>>>>>>> Running LLMs on just 2GB ram is atleast a few years away. So the
>>>>>>>> bot has to be hosted on an API. Cloud is a good option, there is serverless
>>>>>>>> GPU inference and provisioned ones. One that I like is RunPod (serverless
>>>>>>>> pricing <https://www.runpod.io/serverless-gpu> and provisioned
>>>>>>>> pricing <https://www.runpod.io/gpu-instance/pricing>). Local
>>>>>>>> hosting is also an option, for a 7B model, a moderately new (4-5 year old)
>>>>>>>> GPU with 16GB VRAM. Running with lower RAM is possible but with excessive
>>>>>>>> quantisation (rounding off) at the cost of quality and speed.
>>>>>>>>
>>>>>>>>
>>>>>>>>> You can leverage the sugar-datastore if you need to store activity
>>>>>>>>> related data.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Storing the chat history is not an issue. Time complexity for
>>>>>>>> generation(inference) of a transfomer is O(n^2) where n is the number of
>>>>>>>> tokens. So we need to be mindful of how much history do we actually need
>>>>>>>>
>>>>>>>>>
[Attachment #5 (text/html)]
<div dir="ltr"><div class="gmail_default" style="font-size:small"><div \
class="gmail_default">Yea sorry about that code snippet. I don't know why I added \
that, even I don't understand it. I just wanted to let you know that I can figure \
out the part of integrating the bot into the activity. That's why I added that \
random snippet I've removed it.</div><div class="gmail_default"><br></div><div \
class="gmail_default">I'm attaching my final proposal, that I'm submitting, \
as a reference.</div><div class="gmail_default"><br></div><div \
class="gmail_default">Thank you for your assistance throughout the proposal review \
process</div></div></div><br><div class="gmail_quote"><div dir="ltr" \
class="gmail_attr">On Tue, Apr 2, 2024 at 12:04 AM Chihurumnaya Ibiam <<a \
href="mailto:ibiam@sugarlabs.org">ibiam@sugarlabs.org</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div \
dir="ltr"><div>Please use the reply-all button to keep the conversation in the \
list.</div><div><br></div><div><b>"Please excuse me if I'm wrong, I'm not very \
familiar with the codebase.:"</b></div><div><b><br></b></div><div>You should \
remove this line, it serves no purpose, I also don't understand the code \
snippet</div><div>you added and why.<br></div><div><div><div dir="ltr" \
class="gmail_signature"><div dir="ltr"><pre \
style="color:rgb(46,52,54);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px"><span \
style="font-family:monospace,monospace">-- <br></span></pre><div \
style="color:rgb(46,52,54);font-size:14.6667px;font-style:normal;font-weight:normal;le \
tter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;width:71ch"><span \
style="font-family:monospace,monospace"><span></span><span></span>Ibiam \
Chihurumnaya</span></div><div \
style="color:rgb(46,52,54);font-size:14.6667px;font-style:normal;font-weight:normal;le \
tter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;width:71ch"><span \
style="font-family:monospace,monospace"><a href="mailto:ibiam@sugarlabs.org" \
target="_blank">ibiam@sugarlabs.org</a><br></span></div><br></div></div></div><br></div></div><br><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Mar 31, 2024 at \
6:10 PM Sujay R <<a href="mailto:sujay1844@gmail.com" \
target="_blank">sujay1844@gmail.com</a>> wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" \
style="font-size:small">I've taken your advice and added more details about the \
implementation. I'm sorry if sharing PDFs again and again is irritating. I'm \
sure you'll let me know of a better method.<br><br>I have asked you to review \
multiple versions of my draft proposal. I'm very grateful for you take out your \
time and help me applying. Can you please review it one last time? I'll submit \
once you give the green light.</div><div class="gmail_default" \
style="font-size:small"><br></div><div class="gmail_default" \
style="font-size:small">Thank you again for your guidance and support throughout this \
process</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On \
Fri, Mar 29, 2024 at 8:10 PM Chihurumnaya Ibiam <<a \
href="mailto:ibiam@sugarlabs.org" target="_blank">ibiam@sugarlabs.org</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div \
dir="ltr"><div>Your timeline is terse, it'll be great if you add some \
implementation detail to your proposal as there's</div><div>scarcely \
any.<br></div><div><div><div dir="ltr" class="gmail_signature"><div dir="ltr"><pre \
style="color:rgb(46,52,54);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px"><span \
style="font-family:monospace,monospace">-- <br></span></pre><div \
style="color:rgb(46,52,54);font-size:14.6667px;font-style:normal;font-weight:normal;le \
tter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;width:71ch"><span \
style="font-family:monospace,monospace"><span></span><span></span>Ibiam \
Chihurumnaya</span></div><div \
style="color:rgb(46,52,54);font-size:14.6667px;font-style:normal;font-weight:normal;le \
tter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;width:71ch"><span \
style="font-family:monospace,monospace"><a href="mailto:ibiam@sugarlabs.org" \
target="_blank">ibiam@sugarlabs.org</a><br></span></div><br></div></div></div><br></div></div><br><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 28, 2024 at \
7:54 PM Sujay R <<a href="mailto:sujay1844@gmail.com" \
target="_blank">sujay1844@gmail.com</a>> wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" \
style="font-size:small">Ok, the cloud sounds good.</div><div class="gmail_default" \
style="font-size:small"><br></div><div class="gmail_default" \
style="font-size:small">And thank you for patiently discussing the requirements with \
me and answering my questions. Now, I think I can come up with a tentative \
timeline.</div><div class="gmail_default" style="font-size:small">I've attached \
v2 of my proposal with the timeline. Please let me know if there are any more \
changes. If none, I'll submit it on the GSoC website. The deadline is right \
around the corner</div></div><br><div class="gmail_quote"><div dir="ltr" \
class="gmail_attr">On Wed, Mar 27, 2024 at 11:37 PM Chihurumnaya Ibiam <<a \
href="mailto:ibiam@sugarlabs.org" target="_blank">ibiam@sugarlabs.org</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">We were \
thinking of a cloud option, but we haven't decided yet.<br \
clear="all"><div><div><div dir="ltr" class="gmail_signature"><div dir="ltr"><pre \
style="color:rgb(46,52,54);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px"><span \
style="font-family:monospace,monospace">-- <br></span></pre><div \
style="color:rgb(46,52,54);font-size:14.6667px;font-style:normal;font-weight:normal;le \
tter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;width:71ch"><span \
style="font-family:monospace,monospace"><span></span><span></span>Ibiam \
Chihurumnaya</span></div><div \
style="color:rgb(46,52,54);font-size:14.6667px;font-style:normal;font-weight:normal;le \
tter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;width:71ch"><span \
style="font-family:monospace,monospace"><a href="mailto:ibiam@sugarlabs.org" \
target="_blank">ibiam@sugarlabs.org</a><br></span></div><br></div></div></div><br></div></div><br><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 26, 2024 at \
6:40 PM Sujay R <<a href="mailto:sujay1844@gmail.com" \
target="_blank">sujay1844@gmail.com</a>> wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" \
style="font-size:small">Thanks for explaining that. But it's still not clear to \
me where the FOSS LLM should be run. Not on the devices running Sugar, so is cloud \
the option you're looking for?</div></div><br><div class="gmail_quote"><div \
dir="ltr" class="gmail_attr">On Tue, Mar 26, 2024 at 11:02 PM Chihurumnaya Ibiam \
<<a href="mailto:ibiam@sugarlabs.org" target="_blank">ibiam@sugarlabs.org</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div \
dir="ltr"><div>The plan was never really to run an LLM on Sugar as that'll \
drastically increase the size of the activity</div><div>and Sugar itself as Chat is a \
<a href="https://wiki.sugarlabs.org/go/Development_Team/Release/Modules" \
target="_blank">fructose</a> activity.</div><div><div><div dir="ltr" \
class="gmail_signature"><div dir="ltr"><pre \
style="color:rgb(46,52,54);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px"><span \
style="font-family:monospace,monospace">-- <br></span></pre><div \
style="color:rgb(46,52,54);font-size:14.6667px;font-style:normal;font-weight:normal;le \
tter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;width:71ch"><span \
style="font-family:monospace,monospace"><span></span><span></span>Ibiam \
Chihurumnaya</span></div><div \
style="color:rgb(46,52,54);font-size:14.6667px;font-style:normal;font-weight:normal;le \
tter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;width:71ch"><span \
style="font-family:monospace,monospace"><a href="mailto:ibiam@sugarlabs.org" \
target="_blank">ibiam@sugarlabs.org</a><br></span></div><br></div></div></div><br></div></div><br><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 26, 2024 at \
10:54 AM Sujay R <<a href="mailto:sujay1844@gmail.com" \
target="_blank">sujay1844@gmail.com</a>> wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div dir="ltr"><blockquote class="gmail_quote" \
style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex">Sugar runs on a lot of devices including low end \
devices - 2GB ram - and we intend to keep it that way, the chat activity \
is<br>typically used by more than one Sugar instance, the chatbot should also be able \
to run on just one instance.</blockquote><div> </div><div><span \
class="gmail_default" style="font-size:small">Running LLMs on just 2GB ram is atleast \
a few years away. So the bot has to be hosted on an API. Cloud is a good option, \
there is serverless GPU inference and provisioned ones. One that I like is RunPod (<a \
href="https://www.runpod.io/serverless-gpu" target="_blank">serverless pricing</a> \
and <a href="https://www.runpod.io/gpu-instance/pricing" target="_blank">provisioned \
pricing</a>). Local hosting is also an option, for a 7B model, a moderately new (4-5 \
year old) GPU with 16GB VRAM. Running with lower RAM is possible but with excessive \
quantisation (rounding off) at the cost of quality and speed.</span></div><div> \
</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px \
solid rgb(204,204,204);padding-left:1ex"><span class="gmail_default" \
style="font-size:small"><div>You can leverage the sugar-datastore if you need to \
store activity related data.</div></span></blockquote><div> </div><div \
class="gmail_default" style="font-size:small">Storing the chat history is not an \
issue. Time complexity for generation(inference) of a transfomer is O(n^2) where n is \
the number of tokens. So we need to be mindful of how much history do we actually \
need</div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px \
0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> \
</blockquote></div></div> </blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
["Sujay_R-Sugar_labs_GSoC_Chat_Activity_v4.pdf" (application/pdf)]
_______________________________________________
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic