[prev in list] [next in list] [prev in thread] [next in thread] 

List:       kde-core-devel
Subject:    Re: aRts as KDE2.0 audio server
From:       Stefan Westerfeld <stefan () space ! twc ! de>
Date:       1999-09-07 22:58:34
[Download RAW message or body]

   Hi!

On Sun, Sep 05, 1999 at 11:04:11PM -0400, Cristian Tibirna wrote:
> > About four weeks ago, Christian Esken asked me whether aRts could replace
> > KAudioServer2 in KDE2.0. He said he has lots of stuff to do, and little
> > time to spend upon completing his KAudioServer2. Sure aRts can replace it,
> > I just need to make a few tests, I answered.
> 
> AFAIK, one of the important goals of Christian was to implement network
> transparency in KAS2.

Oh, well, forgot that:

+ aRts is fully network transparent

Though that doesn't exactly improve the security situation. For instance
KOffice ceased to use CORBA combined with inet ports (using unix domain
sockets now), because of the possible security risk.

Currently, the audioserver protocol consists of a combination of CORBA and
TCP, while TCP is used only where necessary (raw data transport), and CORBA
where possible, as CORBA interfaces are much much easier to maintain,
redefine and extend than some kind of TCP based protocol.

> > PROs
> > ====
> > + aRts can directly compete with other desktop multimedia systems
> >   => Microsoft for instance puts a lot of effort into multimedia, such
> >   as DirectShow/DirectSound and the upcoming DirectMusic stuff which will
> >   be in Win2k. BeOS as well has a nice multimedia API, and Apple provides
> >   its QuickTime stuff. And of course there is Gnome with their GMF. While
> >   I don't know many details about them, the idea is on all sides to provide
> >   easy multimedia application development and good interoperability with
> >   components.
> >   Arts can do that as well, making KDE2.0 a better place for multimedia
> >   developers.
> 
> aRts offers a library set? If so, this is excellent. In the light of what
> you say later, can a stable version of these libs be included in the
> kdelibs module and serve as a base for the kaudioserver while aRts will
> continue development on an own set of advancing libraries?

Yes something along these lines would be good. Currently, aRts has no own
set of libraries, just exposing CORBA interfaces. However, for the sake
of convenience, I intend to do something similar to the esd library
interface sooner or later.

> > + aRts is very modular, extensible & CORBA based
> >   => Integrating a new wave form for instance is a matter of 20 lines of
> >   code. Integrating the audio server stuff was a matter of a few days. You
> >   don't get that with a monolithic approach...
> 
> I acknowledge I don't get this. I would rather understand if you said that
> the modularity of aRts allows to *separate* not to *integrate* the audio
> server stuff.

Well, ok, I'll try to expain in detail what you get with aRts. After all,
this may be the base of the KDE2.0 multimedia framework, so it makes sense
to have a closer look of how it works now.

Arts is based on the assumption that most multimedia tasks can be described
and performed using flow graphs. For instance, the task of playing two mp3s
(here: from external source, such as mpg123+artscat) at the same time could
be described as

  [ AudioServer incoming ]       [ AudioServer incoming ]
           |                               |
		   V                               V
  [    Volume Scaling    ]       [    Volume Scaling    ]
           |                               |
           V                               V
  [   Equalizer module   ]       [  Equalizer module    ]
           |                               |
		   V                               V
  [      Bus Send        ]       [       Bus Send       ]



  [      Bus Get         ]
           |
           V
  [    Volume Scaling    ]
           |
           V
  [   Saturation Filter  ]
           |
		   V
  [      Play module     ]

Ok, thats the rough outline. In a real setup with a recursive delay effect
involved, it's 37 synthesis modules (and perhaps another 20 for the GUI or
so). Note that aRts currently uses Busses on every point where N-to-M
connections with dynamically changing partners must be made.

What you see above is three "structures" executing. A structure is a
diagram that describes what modules are involved, and how they shall
be connected.

If one of the mp3s terminates, one of the structures would leave. Since
the bus is dynamically N-to-M connecting, you wouldn't need to care for
anything else.


So to the implementation of that stuff. First of all, there are two
constraints:

- signal flow must be very very fast (as there are lots of modules)
- GUI modules should be executed seperately from synthesis modules, since
  you can tolerate the GUI not doing anything for half a second, if CPU
  load gets tight, but you can't tolerate this for synthesis (as you'd
  hear clicks)

Arts consists of some components basic concepts to implement all that:

a) Structure (flow graph) description interface:

   - structure descriptions describe structures (which contains modules)
   - module descriptions describe modules (which has ports)
   - port descriptions describe ports (which can be connected)

b) Servers that can execute modules

c) An execution manager that manages structure execution spread among
   servers

Arts further comes with two servers, one for the Qt/KDE GUI (integrated in
artsbuilder currently), and one for synthesis.

The synthesis server implements an optimized signal flow system, that will
effectively do the audio signal related calculation with small modules.


=== Well, now to your modularity question:

The constraints make it impossible to follow a one module, one process (or
worse: one module = one complete KDE application with GUI and synthesis
stuff) scheme.

That means: if you have something that processes signals, it has to be
executed inside the synthesis server. There, again, you have modularization.

* To make a new module, you derive a class, implement two or three functions,
and there you go. I intend to make it .so-objects sooner or later, so you
can install new modules without recompiling.

* Same is true for new visual elements.

* Currently, integrating whole bunches of infrastructure like "midi support"
or "video support" isn't that modular - you need to change the synthesis
server for that. Well - one could try to modularize even further in the
future.

* Making new structures (flow graphs) ... this is of course possible without
recompiling. Structure descriptions are just data files.

* Making composite modules: Structures are reusable as modules.

Thus, integrating the audio server stuff was not possible only using modules,
it needed to introduce new components in the synthesis server for listening
on TCP ports for instance (well, one probably could have made this as module).

Integrating filters, effects, waveforms, fileformats and similar is possible
as module.

> Anyhow, I will ask: would the modularity of aRts allow to separate part of
> its code and make a *small* audio server? (this is again in the light of
> what you say later, about the too large size of aRts compared with media2)

So I read this as: What is the minimum required for KDE?

Well, you'd need at least the signal evaluation system of the synthesis
server, the GUI server, execution manager and flow graph description
components.

The modules you can strip off - it is imaginable that for instance if KDE2.0
shouldn't ship with aRts midi support, you can strip off that as well...
but I guess it doesn't save much. One could save shipping the flow graph
designer (breaking up artsbuilder between the GUI server and the flow
graph designer).

Of course you could also forget the idea that the GUI is composed of small
modules for a lightweight audio server. Thus you could strip off the
GUI server as well.

I mean: what makes the problems is probably mainly the CORBA overhead,
the signal evaluation stuff and such isn't very large (in source), and
I see no reason why it should take memory.

> > + aRts is there & stable
> >   => Arts is under development since nearly two years now. It works all the
> >   time. The flow system is well tested, and the realtime performance is fine.
> >   It's not some "here I have specs that you might want to read"-project, but
> >   something that is implemented and known to work.
> 
> This is the strongest argument.
> 
> > + I can and want to make it happen
> >   => My time is limited and my project to work on is aRts. Thus, after
> >   suggesting that we might want to use GMF for KDE2.0 I realized, that
> >   while it might be a good idea, it wouldn't be the thing to work on for
> >   me, as I can't split myself between aRts and GMF.
> >   On the other hand, working on having aRts in KDE2.0 should be possible
> >   from the time and motivation I have.
> 
> I don't know about GMF, but I experienced a bit with esd, and is quite
> demanding on the processor compared with the OpenSound drivers, which
> makes it slipping when I compile KDE2 :-)

Well - as I said already a few times: I see no way for a resposive and
click/breakfree audioserver without realtime priority. You can of course
throw away one, and get either a non-responsive audioserver or a responsive
audioserver which produces clicks whenever the CPU load increases.

ESD is somewhere in between. It's not highly responsive as aRts, but it
also isn't free from clicks & breaks under load.

Perhaps for some applications - you may argue - responiveness isn't
important (for instance for Coolos mails that make sounds). I'll integrate
a parameter in the next aRts releases where you can tune the buffers
yourself, so everybody can see what happens.

> > POSSIBILITIES
> > =============
> > * later integration with OpenParts/KOM technology
> >   => Integration with KOM (and possible OpenParts+other KOffice technlogogy
> >   like KScript) is possible since aRts is designed for server operation.
> 
> Excellent. Another real application of KDE's component technology.
> 
> > 
> > CONS
> > ====
> > - it would be a bad idea to couple the release cycles of aRts and KDE
> >   => This is a technical problem which should be solvable: I don't want
> >   that during making aRts the KDE2 audio server, people will only be
> >   able to install a new aRts version with every KDE release, that is
> >   every year or so. The aRts releases are currently every one or two
> >   months, and that should stay like that.
> 
> See above. I think the better solution would be to adapt KDE's release
> scheduling to aRts' one :-) Releasing KDE too sparsely hurts us.
> 
> > - aRts is incompatible with other solutions of the problem
> >   => There are a lot of solutions for the same problem around, for instance
> >   esd, GMF, NAS, KAudioServer1/2, and perhaps some ALSA concepts. Arts is
> >   naturally incompatible with all of them.
> 
> (Innocent request) could you please ellaborate?

Well, first thing: aRts needs to be the app that gets the soundcard (for
sensible operation - if anybody of you is willing to write a Synth_PLAY_ESD
module and show me that it's cool - go ahead - but also prove that the
full duplex stuff and realtime responsiveness is still good).

Thus, other programs that also "need to be the app" that get the soundcard,
such as esd, NAS and KAudioServer1/2 can't be run at the same time aRts is
run. Well - one could make them send their output to aRts when modifying
the source.

With GMF I honestly doesn't know what happens - but certainly it won't
integrate with aRts in the first place. Mainly they have an own concept
for filter graphs and processing (mainly the one module one process
scheme), that is not compatible with the aRts view of the world. However,
adapters may be possible... but I can't say, as I honestly don't really
see how the CORBA interfaces in the Gnome CVS solve whatever problem?

About ALSA I can only say that they are trying to provide a sound
architecture for all apps, however at kernel level. While this may be
a similar goal to that of aRts, it has also some disadvantages, such
as linux only, crashs bring system down, development may be harder,...

I am only expert for aRts audio server technology, so I might have
missed something ;)

> > - it wasn't designed to be an audio server
> >   => aRts stands for "analog realtime synthesizer". It is designed to be
> >   just that. Accidentially, it happens to be able to solve the problems
> >   of an audio server easily, due to its flexibility. But of course it
> >   contains lots of code and infrastructure for synthesis, midi handling,
> >   etc.
> 
> See above. Could the synthesis part be separated and include the common
> part in the KDE libs?

Well - see above. The flow system needs to be shipped with KDE2. The flow
system is the core of synthesis. Of course you could strip of something
like Synth_WAVE_SIN or Synth_WAVE_TRI or Synth_PLAY_AKAI, which probably
most people don't need when running KDE.

In fact, the perfect thing would be in fact having the framework, flow
system,  whatever is necessary shipped with KDE, and not touching these
things any more - I am not sure wether this is reachable.

> 
> > - it is quite a bit larger that the current solution
> >   => Compare aRts
> > 
> >   stefan@orion:~/src/kde/kmusic/ksynth>
> >   wc `find . -name \*.cc; find . -name \*.cpp; find . -name \*.h`|tail -1
> >     23818   61614  570556 total
> > 
> >   with the current KAudioServer1 solution (KAudioServer2 is similar)
> > 
> >   stefan@orion:~/src/kde/kdelibs/mediatool> 
> >   wc `find . -name \*.cc; find . -name \*.cpp; find . -name \*.h`|tail -1
> >       766    2846   21718 total
> >   stefan@orion:~/src/kde/kdebase/kaudio> ~/linc
> >   wc `find . -name \*.cc; find . -name \*.cpp; find . -name \*.h`|tail -1
> >      2944   10324   79839 total
> 
> Humble hint: use kdesdk/scripts/cxxmetrics to do the above.
> 
> > - it does require root rights for realtime operation
> >   => Currently, you are supposed to run aRts with realtime priority. There
> >   is a system call (sched_setscheduler) that ensures it gets the right
> >   priority (to avoid your mp3 to stop if you start netscape for instance),
> >   but it requires root rights. Arts can be installed suid root for that,
> >   and there may be a way to redesign some of its routines to run in non-
> >   realtimed mode as well. However, you'll get latency problems them, so
> >   installing suid root would be the best solution.
> 
> This was already addressed by others. Can a user connect to a
> system-wide server started by the root? Then in highly media-centered
> installments of KDE, the admin can take care of this. In other kinds of
> shops (like at home or in non-media univ labs), either it doesn't count if
> the realtime operation isn't perfect or the user can however run thing as
> root at its own risk.

Oh - running as root is really evil. For instance:

Suppose you have a Synth_PLAY_RAW_FILE module that simply plays a raw file
(without wave header). That sounds harmless.

Now suppose you have a clever user.

He makes a structure

[ Synth_PLAY_RAW_FILE /etc/shadow ]
    |
	+-----------------------------------+
    |                                   |
    V                                   V
[ Synth_PLAY ]    [ Synth_DUMP_AS_WAVE /home/user/test.wav ]

and executes that. Well, he'll get some noise on his speakers, but on the
other hand, he'll get the contents of /etc/shadow in a file.

Avoiding that easily by the server is no option, as the two possibilities
would be

- the user must upload all sample data via TCP (which is really bad if
  you have a large piano sample set for instance)

- you check each and every file access that is made (and possible other
  potential harmful actions) in every module... even though you can try
  to do that there will be always module writers which have no clue and
  just use fopen... and it's a huge task

So my solution stays a suid-root + dropping privileges a few lines after
start. An alternative would be a

kmakerealtime

binary, which does just the set_scheduler stuff and then execs
artsserver.bin...

I know you might tell me now that if it can't be run as systemwide root
server securely, it is basically flawed and whatever. However, this is
*not* ESD. It's the possible core of a multimedia framework, and people
need to accept that there might be requirements for multimedia stuff
which might be different than for a file manager.

For instance, I am currently thinking seriously about using the mlockall
call (locking pages in memory) as well, because if you are in a midi
performance, you don't want your samples to get swapped out. While I see
that some of the stronger realtime optimizations probably should be
options, the stuff shall not be shrinkwrapped in a way that closes these
doors for all users.

Note that even Microsoft sent the midi parts of the sourcecode (!!!) of
their windows stuff to EMagic (the makers of AudioLogic), to make them
look at that, and to make sure that Windows installations will work with
multimedia stuff in a sensible manner. (Well, as far as that is possible
for windows ;).

> > - it uses more resources than "handoptimized" lean C solutions like esd
> >   => Arts is based heavily on CORBA and very modular. If you start a
> >   current release, a thousand CORBA objects are built. If you hit a key
> >   on your midi keyboard, thousands of lines of code decide what to build
> >   in memory, why and how, and then dynamically, small aRts modules are
> >   added and connected.
> >   While it is designed to be reasonable fast during all that, of course
> >   some other solutions like esd should be able to achieve higher performance,
> >   while of course being less flexible.
> 
> This is the worse problem we have to confront. This is also the reason for
> which I was asking about possibility to separate the server code.
> 
> > I would like to hear all feedback you can give me. Especially, some kind
> > of decision must be made whether it should be the KDE2.0 audio server, or
> > what conditions must be met by aRts until it can be the KDE2.0 audio server.
> 
> Hope the above helps.
> 
> > Until now, I haven't started Qt2 porting yet, because I am working actively
> > on aRts as synthesizer application for KDE1 (which is very usable there, I
> > guess). And if aRts shouldn't be in KDE2.0 core, it is probably a bad idea
> > to make it KDE2/Qt2 compatible before anyone out there uses that for
> > composing.
> 
> May I dare? in src/synthesizer/audioman_impl.cc#60-66 in CVS,
> socketIOEvent is not declared (should be in the header). In the aRts-3.3
> source tarball this variable doesn't exist (an autodeleted ArtsIOEvent
> called ev is used). But the tarball doesn't want to compile for me because
> of missing socketbind.h or something. I added the socketIOEvent
> declaration in audioman_impl.h and all compiled well and functioned.

Replace

#include <socketbits.h>

by

#include <sys/types.h>
#include <sys/socket.h>

BTW, I just did a CVS commit. Everything should be fine now. But this is
a developer version by definition, no release, so it may burn your computer.
But you could play with the new full duplex support while it does that, so
you at least had some fun with it ;)

> Now I have to learn how to use it for the presentation I give next week
> :-))
> 
> > 
> >   - The GMF whitepaper
> > 
> >         http://developer.gnome.org/doc/whitepapers/GMF/index.html
> > 
>  Is there any code on this? I couldn't find any indices.
Yes, it's in the Gnome CVS, module gmf.

   Cu... Stefan
-- 
  -* Stefan Westerfeld, stefan@space.twc.de (PGP!), Hamburg/Germany
     KDE Developer, project infos at http://space.twc.de/~stefan/kde *-

[prev in list] [next in list] [prev in thread] [next in thread]