Hiding internal DBUS use inside a library

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

Hiding internal DBUS use inside a library

Lennart Poettering-2
Hi!

I am working on a library that makes internal use of DBUS. It connects
to the system bus and does some work there. I want to hide the fact
that the library uses DBUS completely. Unfortunately this is not
possible wth the current API, as it seems. Due to connection recyling
we might get access to a connection some other part of the application
(maybe some other library?) registered and might have installed
watch/timeout handlers for. If we now overwrite those handlers with
our own DBUS goes nuts.

In short: a library that uses the current API function dbus_bus_get()
is simply broken.

Honestly I think that the whole connection recycling thing is a really
bad idea. The fact that one part of a process might call functions
like dbus_connection_set_timeout_functions() or
dbus_connection_set_max_received_size() on a DBusConnection and
overwrite values other parts already set earlier without even knowing about it
is a royal error source.

And when does this connection recycling ever become handy? What is the
gain of the added complexity? Either a connection is created by a
library, in which case recycling is a bad idea due to the reasons
described above. Or it is created by the main program, in which case
one can asssume that the programmer knows what to do and shares
the DBusConnection on his own behalf if he wants to.

Would a patch that removes the connection recyling entirely be
accepted?

In case not: would a patch adding a new function
dbus_bus_get_private() which is identical to dbus_bus_get() but uses
dbus_connection_new_private() internally, be accepted?

Did I miss something? Is there already a way to deal with this cleanly?

Lennart

--
Lennart Poettering; lennart [at] poettering [dot] de
ICQ# 11060553; GPG 0x1A015CC4; http://0pointer.de/lennart/
_______________________________________________
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus
Reply | Threaded
Open this post in threaded view
|

Re: Hiding internal DBUS use inside a library

Colin Walters
On Tue, 2005-08-23 at 21:48 +0200, Lennart Poettering wrote:

> Hi!
>
> I am working on a library that makes internal use of DBUS. It connects
> to the system bus and does some work there. I want to hide the fact
> that the library uses DBUS completely. Unfortunately this is not
> possible wth the current API, as it seems. Due to connection recyling
> we might get access to a connection some other part of the application
> (maybe some other library?) registered and might have installed
> watch/timeout handlers for. If we now overwrite those handlers with
> our own DBUS goes nuts.
Ultimately, you can only have one main loop (well, per thread) in your
application.  Now, I did run into the connection sharing problem between
two threads with separate mainloops.  I have a partial patch to add an
API to get unshared connections, so that e.g. the GLib bindings can have
one connection per mainloop.

> In short: a library that uses the current API function dbus_bus_get()
> is simply broken.

Not necessarily - if the library doesn't install its own handlers on the
connection, but e.g. instead just does dbus_connection_send and returns,
or does dbus_connection_send_with_reply_and_block.  The former
definitely works, the latter *should* work conceptually; whether it
currently does or not is another question :)

> Honestly I think that the whole connection recycling thing is a really
> bad idea. The fact that one part of a process might call functions
> like dbus_connection_set_timeout_functions()

But you're going to have a serious problem anyways if you have two main
loops in your application.  In what situation have you found this to be
a problem?

> or
> dbus_connection_set_max_received_size()

Yeah...somewhat problematic, although this is kind of a specialized
function.  It probably isn't something any library should be invoking.

> And when does this connection recycling ever become handy?

I think one of the goals was to reduce the number of open file
descriptors desktop applications have.

> Would a patch that removes the connection recyling entirely be
> accepted?

I don't think so, not without more discussion.

> In case not: would a patch adding a new function
> dbus_bus_get_private() which is identical to dbus_bus_get() but uses
> dbus_connection_new_private() internally, be accepted?

I have a patch in progress for that which will be needed for the
per-mainloop connection.



_______________________________________________
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Hiding internal DBUS use inside a library

Havoc Pennington
In reply to this post by Lennart Poettering-2
On Tue, 2005-08-23 at 21:48 +0200, Lennart Poettering wrote:
>
> Honestly I think that the whole connection recycling thing is a really
> bad idea. The fact that one part of a process might call functions
> like dbus_connection_set_timeout_functions() or
> dbus_connection_set_max_received_size() on a DBusConnection and
> overwrite values other parts already set earlier without even knowing about it
> is a royal error source.

It's possible we should move the connection sharing up into the
bindings.

With GNOME apps, set_timeout_functions() will always be called by the
GLib main loop stuff, and set_max_received_size() would never be called
(it's a specialized function used only by the message bus).

> And when does this connection recycling ever become handy? What is the
> gain of the added complexity? Either a connection is created by a
> library, in which case recycling is a bad idea due to the reasons
> described above. Or it is created by the main program, in which case
> one can asssume that the programmer knows what to do and shares
> the DBusConnection on his own behalf if he wants to.

The point is to share the connection among code modules. i.e. inside
GNOME, all GNOME libraries and portions of a GNOME app can use the same
connection because it's assumed/defined that DBusConnection is used with
the GLib main loop.

We do the same thing with the X display connection maintained by GTK.
(And KDE does the same with the one for Qt.)

If you don't share the connection you'd have 20 open connections per app
or more quite easily, which is noticeable overhead.

> Did I miss something? Is there already a way to deal with this cleanly?

There is a way to do it with today's libdbus, because nothing about
dbus_bus_get() is magic. You can just call
dbus_connection_open_private() then dbus_bus_register(); the rest of the
code in dbus_bus_get() is just maintaining the shared connection global
variable.

Havoc


_______________________________________________
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus
Reply | Threaded
Open this post in threaded view
|

Re: Hiding internal DBUS use inside a library

Lennart Poettering-2
In reply to this post by Colin Walters
On Thu, 25.08.05 16:02, Colin Walters ([hidden email]) wrote:

> > In short: a library that uses the current API function dbus_bus_get()
> > is simply broken.
>
> Not necessarily - if the library doesn't install its own handlers on the
> connection, but e.g. instead just does dbus_connection_send and returns,
> or does dbus_connection_send_with_reply_and_block.  The former
> definitely works, the latter *should* work conceptually; whether it
> currently does or not is another question :)

Hrm, more advanced libraries (like mine) will need stuff like signals
and thus integration in to the main event loop, and if they want to
hide the internal usage of DBUS they will abstract the main loop API,
hence use dbus_connection_set_watch_functisions().

Just using dbus_connection_send_with_reply_and_block() ain't enough.

> > Honestly I think that the whole connection recycling thing is a really
> > bad idea. The fact that one part of a process might call functions
> > like dbus_connection_set_timeout_functions()
>
> But you're going to have a serious problem anyways if you have two main
> loops in your application.  In what situation have you found this to be
> a problem?

While working on Avahi, a free mDNS/DNS-SD responder for Linux. The
public API of the avahi daemon is published via DBUS. We provide two
libraries: libavahi-core and libavahi-client. The fomer allows the user to
embedd an mDNS stack into his applications (mostly for printers,
routers and the like), the latter is an abstraction around our DBUS
API for normal desktop applications. The two APIs are mostly
identical, making it very easy to switch between them. (just a little
search and replace). libavahi-core doesn't have anything to do with
DBUS. libavahi-client is a DBUS wrapper. Both use the same main loop
abstraction. For libavahi-client we needed to write some glue code to
connect DBUS to our event loop abstraction layer.

I can think of many other situations where dbus connection recycling
becomes a problem:

consider gnome-vfs with a backend which makes use of DBUS (e.g. to
show a list of cameras or something it connects to
org.fd.MagicCameraManager or something) . Consumers of the gnome-vfs API
don't want to know about the internal DBUS usage of an auxiliary
backend. And they don't want that their own DBUS usage is broken by it.

> > or
> > dbus_connection_set_max_received_size()
>
> Yeah...somewhat problematic, although this is kind of a specialized
> function.  It probably isn't something any library should be
> invoking.

dbus_connection_set_timeout_functions() and
dbus_connection_set_watch_functions() are surely functions that
libraries which want to do some kind of event loop abstraction for
asynchronous signals want to use.

> > And when does this connection recycling ever become handy?
>
> I think one of the goals was to reduce the number of open file
> descriptors desktop applications have.

Yes, that's the idea of recycling. But most libraries can't make use
of it (as I tried to explain above) and the main program shouldn't
need more tha a single DBusConnection anyway.

Lennart
--
Lennart Poettering; lennart [at] poettering [dot] de
ICQ# 11060553; GPG 0x1A015CC4; http://0pointer.de/lennart/
_______________________________________________
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus
Reply | Threaded
Open this post in threaded view
|

Re: Hiding internal DBUS use inside a library

Lennart Poettering-2
In reply to this post by Havoc Pennington
On Thu, 25.08.05 16:44, Havoc Pennington ([hidden email]) wrote:

> With GNOME apps, set_timeout_functions() will always be called by the
> GLib main loop stuff, and set_max_received_size() would never be called
> (it's a specialized function used only by the message bus).

Wrong. Our client library for the avahi mdns/dns-sd responder tries to
abstract DBUS from the user. Hence it WILL call
set_{timeout,watch}_functions(), even when used by Gnome apps.

> > And when does this connection recycling ever become handy? What is the
> > gain of the added complexity? Either a connection is created by a
> > library, in which case recycling is a bad idea due to the reasons
> > described above. Or it is created by the main program, in which case
> > one can asssume that the programmer knows what to do and shares
> > the DBusConnection on his own behalf if he wants to.
>
> The point is to share the connection among code modules. i.e. inside
> GNOME, all GNOME libraries and portions of a GNOME app can use the same
> connection because it's assumed/defined that DBusConnection is used with
> the GLib main loop.

If it makes sense to share DBusConenctions in Gnome modules then Gnome
should implement the connection recycling, not the core DBUS system!

> We do the same thing with the X display connection maintained by GTK.
> (And KDE does the same with the one for Qt.)

I doubt that simply copying this behaviour from GTK to DBUS without
second thought makes sense.

> If you don't share the connection you'd have 20 open connections per app
> or more quite easily, which is noticeable overhead.

Is it really such a overhead? I doubt that. If you share the
DBusConnection with too many modules you'll end up iterating though
signal handlers ("module #1, is this signal for you? no? ok, module #2
is this signal for you? no? ..... module #4711 is this signal for you?
no? hmm, i guess i have to discard it then, what a pity for the lost
CPU cycles for nothing"). In contrast when using seperate fds for
different modules only the module that wants the signal will get it.

> > Did I miss something? Is there already a way to deal with this cleanly?
>
> There is a way to do it with today's libdbus, because nothing about
> dbus_bus_get() is magic. You can just call
> dbus_connection_open_private() then dbus_bus_register(); the rest of the
> code in dbus_bus_get() is just maintaining the shared connection global
> variable.

Yes, we already implemented this for Avahi 0.1. However we
still have a compile time variable for the system bus address, which
must be set correctly to match DBUS' own. This is ugly and doesn't
belong in Avahi, but in DBUS.

(the least that could be done is exporting the bus address in the
pkg-config file as variable)

Lennart

--
Lennart Poettering; lennart [at] poettering [dot] de
ICQ# 11060553; GPG 0x1A015CC4; http://0pointer.de/lennart/
_______________________________________________
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus
Reply | Threaded
Open this post in threaded view
|

Re: Hiding internal DBUS use inside a library

Colin Walters
On Thu, 2005-08-25 at 23:07 +0200, Lennart Poettering wrote:
> On Thu, 25.08.05 16:44, Havoc Pennington ([hidden email]) wrote:
>
> > With GNOME apps, set_timeout_functions() will always be called by the
> > GLib main loop stuff, and set_max_received_size() would never be called
> > (it's a specialized function used only by the message bus).
>
> Wrong. Our client library for the avahi mdns/dns-sd responder tries to
> abstract DBUS from the user. Hence it WILL call
> set_{timeout,watch}_functions(), even when used by Gnome apps.

This is confusing to me - what arguments does it give to those
functions?  Is it using an internal mainloop or something?
Why is it useful to call those functions if the application is using a
GLib mainloop?

_______________________________________________
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Hiding internal DBUS use inside a library

John Palmieri-2
In reply to this post by Lennart Poettering-2
On Thu, 2005-08-25 at 22:52 +0200, Lennart Poettering wrote:

> On Thu, 25.08.05 16:02, Colin Walters ([hidden email]) wrote:
>
> > > In short: a library that uses the current API function dbus_bus_get()
> > > is simply broken.
> >
> > Not necessarily - if the library doesn't install its own handlers on the
> > connection, but e.g. instead just does dbus_connection_send and returns,
> > or does dbus_connection_send_with_reply_and_block.  The former
> > definitely works, the latter *should* work conceptually; whether it
> > currently does or not is another question :)
>
> Hrm, more advanced libraries (like mine) will need stuff like signals
> and thus integration in to the main event loop, and if they want to
> hide the internal usage of DBUS they will abstract the main loop API,
> hence use dbus_connection_set_watch_functisions().
>
> Just using dbus_connection_send_with_reply_and_block() ain't enough.

Have you looked at what other complex libraries like HAL does?  It
leaves it up to the programmer to integrate D-Bus with a mainloop and
uses a user supplied connection:

 DBusConnection *dbus_connection;

 dbus_connection = dbus_bus_get (DBUS_BUS_SYSTEM, error);

 if (dbus_error_is_set (error))
     return FALSE;

 dbus_connection_setup_with_g_main (dbus_connection, NULL);

 libhal_ctx_set_dbus_connection (ctx, dbus_connection);

I don't think it is irrational to expose that much DBus in a library.
Besides if an application is using DBus the programmer should know
because semantics change when you are dealing with client/server
functionality.  Hiding it does not give any advantages.
 
--
John (J5) Palmieri <[hidden email]>

_______________________________________________
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus
Reply | Threaded
Open this post in threaded view
|

Re: Hiding internal DBUS use inside a library

Lennart Poettering-2
In reply to this post by Colin Walters
On Thu, 25.08.05 18:00, Colin Walters ([hidden email]) wrote:

> > > With GNOME apps, set_timeout_functions() will always be called by the
> > > GLib main loop stuff, and set_max_received_size() would never be called
> > > (it's a specialized function used only by the message bus).
> >
> > Wrong. Our client library for the avahi mdns/dns-sd responder tries to
> > abstract DBUS from the user. Hence it WILL call
> > set_{timeout,watch}_functions(), even when used by Gnome apps.
>
> This is confusing to me - what arguments does it give to those
> functions?  Is it using an internal mainloop or something?

No, it doesn't use an internal main loop. It just implements some glue
code to abstract the event loop.

The user of our library fills a vtable with his own implementations of
functions for creating and modifying time or IO event sources. We use
those functions internally for all housekeeping we need to do and - in
the case of libavahi-client - to drive an internaly used
DBusConnection. For that DBusConnection we set the timeut/watch
functions to some glue code with translates them to our event loop
abstraction layer. The abstraction layer is actually very thin, since
it resambles the DBUS event loop abstraction a bit.

> Why is it useful to call those functions if the application is using a
> GLib mainloop?

Because Avahi doesn't use GLib and the application doesn't know we use
DBUS internally. Thefore we ask the user to provide us with the vtable
I mentioned above and use that for all our internal work (including DBUS).

Lennart

--
Lennart Poettering; lennart [at] poettering [dot] de
ICQ# 11060553; GPG 0x1A015CC4; http://0pointer.de/lennart/
_______________________________________________
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus
Reply | Threaded
Open this post in threaded view
|

Re: Hiding internal DBUS use inside a library

Lennart Poettering-2
In reply to this post by John Palmieri-2
On Thu, 25.08.05 18:08, John (J5) Palmieri ([hidden email]) wrote:

> > Hrm, more advanced libraries (like mine) will need stuff like signals
> > and thus integration in to the main event loop, and if they want to
> > hide the internal usage of DBUS they will abstract the main loop API,
> > hence use dbus_connection_set_watch_functisions().
> >
> > Just using dbus_connection_send_with_reply_and_block() ain't enough.
>
> Have you looked at what other complex libraries like HAL does?  It

Yes we have. And didn't consider it usable for our needs. Why?

First, we want to retain the freedom to drop the DBUS dependency
without changing the API or ABI. Therefore we cannot export any DBUS
specific types (like DBusConnection) to the user. The public API of
our library doesn't use a single DBus symbol or definition.

Second, because we want to retain maximum compatiblity with our second
library ("libavahi-core") which does in-process what our first library
("libavahi-client") does over DBUS talking to a system daemon. For
avahi-core we need some kind of event loop abstraction anyway, so we
want to use the exact abstracion layer in avahi-clien, too.

> I don't think it is irrational to expose that much DBus in a
> library.

Maybe. But it it doesn't fit our needs and the same is true in many
other cases. Think of any plugins-based system daemon that doesn't use
Glib. Such as Apache or whatever. If an Apache module wants to use
DBUS it has to integrate the DBusConnection with the way Apache does
event dispatching. Therefore it will set the timeout/watch functions
of the DBusConnection to some glue code. If now another apache module
wants to do something similar the connection recycling *WILL* break
it. Unless of course, both modules use exactly the same glue code
implementations AND they are loaded and initialized at the same time.

> Besides if an application is using DBus the programmer should know
> because semantics change when you are dealing with client/server
> functionality.  Hiding it does not give any advantages.

That's not the point.

Lennart

--
Lennart Poettering; lennart [at] poettering [dot] de
ICQ# 11060553; GPG 0x1A015CC4; http://0pointer.de/lennart/
_______________________________________________
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus
Reply | Threaded
Open this post in threaded view
|

Re: Hiding internal DBUS use inside a library

Havoc Pennington
In reply to this post by Lennart Poettering-2
On Thu, 2005-08-25 at 23:07 +0200, Lennart Poettering wrote:

> On Thu, 25.08.05 16:44, Havoc Pennington ([hidden email]) wrote:
>
> > With GNOME apps, set_timeout_functions() will always be called by the
> > GLib main loop stuff, and set_max_received_size() would never be called
> > (it's a specialized function used only by the message bus).
>
> Wrong. Our client library for the avahi mdns/dns-sd responder tries to
> abstract DBUS from the user. Hence it WILL call
> set_{timeout,watch}_functions(), even when used by Gnome apps.
>

Then you should use a private connection. I would suggest that the
shared dbus connection is defined to be in use by the primary main loop
or event system of the application. For GNOME applications, this would
be the GTK main loop, for apps not using a toolkit this would be the
app's own custom event loop.

> If it makes sense to share DBusConenctions in Gnome modules then Gnome
> should implement the connection recycling, not the core DBUS system!

The reason I don't agree is that a non-GNOME library may have legitimate
reasons to use "the dbus connection attached to the primary event loop
of the application" rather than a private connection. There's no reason
to require linking to GNOME libs in order to access the "primary bus
connection."

We could explicitly allow the primary main loop to "grab ownership" with
an API like:

 dbus_bus_register_main_loop (void *cookie);
 
where the code that calls set_watch_functions() would call the above
function. At that point, any future call to set_watch_functions() will
fail and print a warning.

But I think it's sort of overkill.

> > If you don't share the connection you'd have 20 open connections per app
> > or more quite easily, which is noticeable overhead.
>
> Is it really such a overhead? I doubt that.

I don't. There's a fixed limit on number of file descriptors available,
and DBusConnection is not a very small object.

>  If you share the
> DBusConnection with too many modules you'll end up iterating though
> signal handlers ("module #1, is this signal for you? no? ok, module #2
> is this signal for you? no? ..... module #4711 is this signal for you?
> no? hmm, i guess i have to discard it then, what a pity for the lost
> CPU cycles for nothing"). In contrast when using seperate fds for
> different modules only the module that wants the signal will get it.

We could optimize that away pretty well with a hash table, and the
overhead I'm concerned about is primarily space not speed.

> Yes, we already implemented this for Avahi 0.1. However we
> still have a compile time variable for the system bus address, which
> must be set correctly to match DBUS' own. This is ugly and doesn't
> belong in Avahi, but in DBUS.
>
> (the least that could be done is exporting the bus address in the
> pkg-config file as variable)

That is in bugzilla already... feel free to provide a patch...

Havoc


_______________________________________________
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus
Reply | Threaded
Open this post in threaded view
|

Re: Hiding internal DBUS use inside a library

Colin Walters
In reply to this post by Lennart Poettering-2
On Fri, 2005-08-26 at 00:56 +0200, Lennart Poettering wrote:

> The user of our library fills a vtable with his own implementations of
> functions for creating and modifying time or IO event sources. We use
> those functions internally for all housekeeping we need to do and - in
> the case of libavahi-client - to drive an internaly used
> DBusConnection. For that DBusConnection we set the timeut/watch
> functions to some glue code with translates them to our event loop
> abstraction layer. The abstraction layer is actually very thin, since
> it resambles the DBUS event loop abstraction a bit.

Urgh.  I guess given your goals Havoc's suggestion of using
dbus_connection_open_private is right.

This seems like a lot of pain though, both for users of your library who
want to integrate it with their apps using a mainloop like GLib, and
also for you as developers.  I'm not sure I see why you have such a
strong goal of avoiding a D-BUS dependency.  But I don't understand the
compatibility issues between libavahi-client and libavahi-core though...

I hope at some point though that you declare the Avahi D-BUS interface
stable, and support clients using it directly through bindings such as
Python or GLib.  The protocol looks simple enough.



_______________________________________________
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Hiding internal DBUS use inside a library

Colin Walters
In reply to this post by Lennart Poettering-2
On Fri, 2005-08-26 at 01:14 +0200, Lennart Poettering wrote:

> Maybe. But it it doesn't fit our needs and the same is true in many
> other cases. Think of any plugins-based system daemon that doesn't use
> Glib. Such as Apache or whatever. If an Apache module wants to use
> DBUS it has to integrate the DBusConnection with the way Apache does
> event dispatching. Therefore it will set the timeout/watch functions
> of the DBusConnection to some glue code.

This hypothetical Apache module can't set those timeout/watch functions
to anything sane without having a mainloop anyways though, and I doubt
Apache modules are able to run mainloops outside of the Apache core.


_______________________________________________
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Hiding internal DBUS use inside a library

Trent Lloyd
In reply to this post by John Palmieri-2
On Thu, Aug 25, 2005 at 06:08:09PM -0400, John (J5) Palmieri wrote:

> On Thu, 2005-08-25 at 22:52 +0200, Lennart Poettering wrote:
> > On Thu, 25.08.05 16:02, Colin Walters ([hidden email]) wrote:
> >
> > > > In short: a library that uses the current API function dbus_bus_get()
> > > > is simply broken.
> > >
> > > Not necessarily - if the library doesn't install its own handlers on the
> > > connection, but e.g. instead just does dbus_connection_send and returns,
> > > or does dbus_connection_send_with_reply_and_block.  The former
> > > definitely works, the latter *should* work conceptually; whether it
> > > currently does or not is another question :)
> >
> > Hrm, more advanced libraries (like mine) will need stuff like signals
> > and thus integration in to the main event loop, and if they want to
> > hide the internal usage of DBUS they will abstract the main loop API,
> > hence use dbus_connection_set_watch_functisions().
> >
> > Just using dbus_connection_send_with_reply_and_block() ain't enough.
>
> Have you looked at what other complex libraries like HAL does?  It
> leaves it up to the programmer to integrate D-Bus with a mainloop and
> uses a user supplied connection:
>
>  DBusConnection *dbus_connection;
>
>  dbus_connection = dbus_bus_get (DBUS_BUS_SYSTEM, error);
>
>  if (dbus_error_is_set (error))
>      return FALSE;
>
>  dbus_connection_setup_with_g_main (dbus_connection, NULL);
>
>  libhal_ctx_set_dbus_connection (ctx, dbus_connection);

Also consider situations where the applications doesn't know about the
'other' dbusconnection.

For example a shared library makes use of avahi, and the application
makes use of avahi, or two entirely separate parts, 1 is using dbus and
1 is using avahi, or even two entirely separate parts of the code making
use of avahi with two separate main loop adapters.

Additionally, I see this being a problem in threads, although I'm not
sure how much D-BUS does to avoid that, it may well work (although, if
it does I have no idea hwo it handles executing handlers in the otehr
thread, or perhaps thats simply not supported, but it should be), also
consider different event loops in two threads, this might not happen
in glib but glib isn't the whole world. :)

Any which way you look at it, these use cases should not in broken.

In particular, the problem we had was that the system bus address is not
exported in any way, and the claims in dbus-bus.c that its 'hard
coded' are not in fact as hard coded as you'd like it to be, because its
defined on $localstatedir.  (I suspect that comment is out of date)

I will write a patch and submit it to the bug walters pointed
out -- but what would you prefer
 a) dbus_bus_get_private()
 or b) dbus_get_system_bus_address() (This, vs a defintiion, so it can
 check the env too or whatever other behavior decides to be supported
 in future)

Personally, I'm fond of the former but I think the latter should also
exist.

(Oh, when I wrote the patch to use dbus_connection_open_private, I did
make it respect $DBUS_SYSTEM_BUS_ADDRESS if set, which seemed to be in
line with what libdbus would do, but thats pretty much never set)

Trent

>
> I don't think it is irrational to expose that much DBus in a library.
> Besides if an application is using DBus the programmer should know
> because semantics change when you are dealing with client/server
> functionality.  Hiding it does not give any advantages.
>  
> --
> John (J5) Palmieri <[hidden email]>
>
> _______________________________________________
> dbus mailing list
> [hidden email]
> http://lists.freedesktop.org/mailman/listinfo/dbus

--
Trent Lloyd <[hidden email]>
Bur.st Networking Inc.
_______________________________________________
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus
Reply | Threaded
Open this post in threaded view
|

Re: Hiding internal DBUS use inside a library

John Palmieri-2
On Fri, 2005-08-26 at 18:17 +0800, Trent Lloyd wrote:

> On Thu, Aug 25, 2005 at 06:08:09PM -0400, John (J5) Palmieri wrote:
> > On Thu, 2005-08-25 at 22:52 +0200, Lennart Poettering wrote:
> > > On Thu, 25.08.05 16:02, Colin Walters ([hidden email]) wrote:
> > >
> > > > > In short: a library that uses the current API function dbus_bus_get()
> > > > > is simply broken.
> > > >
> > > > Not necessarily - if the library doesn't install its own handlers on the
> > > > connection, but e.g. instead just does dbus_connection_send and returns,
> > > > or does dbus_connection_send_with_reply_and_block.  The former
> > > > definitely works, the latter *should* work conceptually; whether it
> > > > currently does or not is another question :)
> > >
> > > Hrm, more advanced libraries (like mine) will need stuff like signals
> > > and thus integration in to the main event loop, and if they want to
> > > hide the internal usage of DBUS they will abstract the main loop API,
> > > hence use dbus_connection_set_watch_functisions().
> > >
> > > Just using dbus_connection_send_with_reply_and_block() ain't enough.
> >
> > Have you looked at what other complex libraries like HAL does?  It
> > leaves it up to the programmer to integrate D-Bus with a mainloop and
> > uses a user supplied connection:
> >
> >  DBusConnection *dbus_connection;
> >
> >  dbus_connection = dbus_bus_get (DBUS_BUS_SYSTEM, error);
> >
> >  if (dbus_error_is_set (error))
> >      return FALSE;
> >
> >  dbus_connection_setup_with_g_main (dbus_connection, NULL);
> >
> >  libhal_ctx_set_dbus_connection (ctx, dbus_connection);
>
> Also consider situations where the applications doesn't know about the
> 'other' dbusconnection.
>
> For example a shared library makes use of avahi, and the application
> makes use of avahi, or two entirely separate parts, 1 is using dbus and
> 1 is using avahi, or even two entirely separate parts of the code making
> use of avahi with two separate main loop adapters.

So having two mainloops seems weird to me.  A mainloop should provide
integration so anything that would need a mainloop would just use the
controlling applications mainloop.

> Additionally, I see this being a problem in threads, although I'm not
> sure how much D-BUS does to avoid that, it may well work (although, if
> it does I have no idea hwo it handles executing handlers in the otehr
> thread, or perhaps thats simply not supported, but it should be), also
> consider different event loops in two threads, this might not happen
> in glib but glib isn't the whole world. :)

I threads why not just used blocking calls?  The main purpose of a
mainloop is to emulate threads where the complication of threads is not
needed.  I can see the need for a mainloop in threads for some small
number of cases.  So I'm not going to argue with you on the point.

> Any which way you look at it, these use cases should not in broken.
>
> In particular, the problem we had was that the system bus address is not
> exported in any way, and the claims in dbus-bus.c that its 'hard
> coded' are not in fact as hard coded as you'd like it to be, because its
> defined on $localstatedir.  (I suspect that comment is out of date)
>
> I will write a patch and submit it to the bug walters pointed
> out -- but what would you prefer
>  a) dbus_bus_get_private()
>  or b) dbus_get_system_bus_address() (This, vs a defintiion, so it can
>  check the env too or whatever other behavior decides to be supported
>  in future)

Why not both?  dbus_bus_get_private should be the same as dbus_bus_get
where you send in a DBusBusType.  Internally it should use
dbus_get_system_bus_address() for system bus connections.  This should
also be changed in dbus_bus_get to use dbus_get_system_bus_address() and
wherever else it is hard coded.

> Personally, I'm fond of the former but I think the latter should also
> exist.
>
> (Oh, when I wrote the patch to use dbus_connection_open_private, I did
> make it respect $DBUS_SYSTEM_BUS_ADDRESS if set, which seemed to be in
> line with what libdbus would do, but thats pretty much never set)
>
> Trent

Oh and on the core/client thing I was just going off what Lennart had
posted in a previous e-mail about keeping the libraries compatible.  

--
John (J5) Palmieri <[hidden email]>

_______________________________________________
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus
Reply | Threaded
Open this post in threaded view
|

Re: Hiding internal DBUS use inside a library

Lennart Poettering-2
In reply to this post by Colin Walters
On Fri, 26.08.05 00:09, Colin Walters ([hidden email]) wrote:

> On Fri, 2005-08-26 at 01:14 +0200, Lennart Poettering wrote:
>
> > Maybe. But it it doesn't fit our needs and the same is true in many
> > other cases. Think of any plugins-based system daemon that doesn't use
> > Glib. Such as Apache or whatever. If an Apache module wants to use
> > DBUS it has to integrate the DBusConnection with the way Apache does
> > event dispatching. Therefore it will set the timeout/watch functions
> > of the DBusConnection to some glue code.
>
> This hypothetical Apache module can't set those timeout/watch functions
> to anything sane without having a mainloop anyways though, and I doubt
> Apache modules are able to run mainloops outside of the Apache core.

That's not the point. Consider any other plugins-based system which
does have a central thing you'd consider a proper "main loop".

Lennart

--
Lennart Poettering; lennart [at] poettering [dot] de
ICQ# 11060553; GPG 0x1A015CC4; http://0pointer.de/lennart/
_______________________________________________
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus
Reply | Threaded
Open this post in threaded view
|

Re: Hiding internal DBUS use inside a library

Lennart Poettering-2
In reply to this post by Havoc Pennington
On Thu, 25.08.05 20:19, Havoc Pennington ([hidden email]) wrote:

>
> On Thu, 2005-08-25 at 23:07 +0200, Lennart Poettering wrote:
> > On Thu, 25.08.05 16:44, Havoc Pennington ([hidden email]) wrote:
> >
> > > With GNOME apps, set_timeout_functions() will always be called by the
> > > GLib main loop stuff, and set_max_received_size() would never be called
> > > (it's a specialized function used only by the message bus).
> >
> > Wrong. Our client library for the avahi mdns/dns-sd responder tries to
> > abstract DBUS from the user. Hence it WILL call
> > set_{timeout,watch}_functions(), even when used by Gnome apps.
> >
>
> Then you should use a private connection. I would suggest that the
> shared dbus connection is defined to be in use by the primary main loop
> or event system of the application. For GNOME applications, this would
> be the GTK main loop, for apps not using a toolkit this would be the
> app's own custom event loop.

But how does one make sure that the DBusConnection that is returned by
dbus_bus_get() has already been initialized to work with the main
event loop?

An example: my library internally uses the system bus. so it calls
dbus_bus_get() for the system bus. How does it know that
dbus_connection_setup_with_g_main() (or what ever is the right thing
to do for registering the connection with the main loop) has already
been called for that connection?

> We could explicitly allow the primary main loop to "grab ownership" with
> an API like:
>
>  dbus_bus_register_main_loop (void *cookie);
>  
> where the code that calls set_watch_functions() would call the above
> function. At that point, any future call to set_watch_functions() will
> fail and print a warning.

Hmm, the least what could be done is to let set_watch_functions() fail
if the watch functions have already been set previously. Right now
each call succeeds and straneg things happen.

> > > If you don't share the connection you'd have 20 open connections per app
> > > or more quite easily, which is noticeable overhead.
> >
> > Is it really such a overhead? I doubt that.
>
> I don't. There's a fixed limit on number of file descriptors available,
> and DBusConnection is not a very small object.

Ok. That's a good point. I didn't consider memory consumption.

> >  If you share the
> > DBusConnection with too many modules you'll end up iterating though
> > signal handlers ("module #1, is this signal for you? no? ok, module #2
> > is this signal for you? no? ..... module #4711 is this signal for you?
> > no? hmm, i guess i have to discard it then, what a pity for the lost
> > CPU cycles for nothing"). In contrast when using seperate fds for
> > different modules only the module that wants the signal will get it.
>
> We could optimize that away pretty well with a hash table, and the
> overhead I'm concerned about is primarily space not speed.

The current API doesn't allow that this is optimized using a hash
table.

> > Yes, we already implemented this for Avahi 0.1. However we
> > still have a compile time variable for the system bus address, which
> > must be set correctly to match DBUS' own. This is ugly and doesn't
> > belong in Avahi, but in DBUS.
> >
> > (the least that could be done is exporting the bus address in the
> > pkg-config file as variable)
>
> That is in bugzilla already... feel free to provide a patch...

Ok, will do.

Lennart

--
Lennart Poettering; lennart [at] poettering [dot] de
ICQ# 11060553; GPG 0x1A015CC4; http://0pointer.de/lennart/
_______________________________________________
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus
Reply | Threaded
Open this post in threaded view
|

Re: Hiding internal DBUS use inside a library

Lennart Poettering-2
In reply to this post by Colin Walters
On Fri, 26.08.05 00:05, Colin Walters ([hidden email]) wrote:

> > the case of libavahi-client - to drive an internaly used
> > DBusConnection. For that DBusConnection we set the timeut/watch
> > functions to some glue code with translates them to our event loop
> > abstraction layer. The abstraction layer is actually very thin, since
> > it resambles the DBUS event loop abstraction a bit.
>
> Urgh.  I guess given your goals Havoc's suggestion of using
> dbus_connection_open_private is right.

Ok, we implemented that already.

> This seems like a lot of pain though, both for users of your library who
> want to integrate it with their apps using a mainloop like GLib, and
> also for you as developers.  I'm not sure I see why you have such a
> strong goal of avoiding a D-BUS dependency.  But I don't understand the
> compatibility issues between libavahi-client and libavahi-core though...
>
> I hope at some point though that you declare the Avahi D-BUS interface
> stable, and support clients using it directly through bindings such as
> Python or GLib.  The protocol looks simple enough.

Yes, the DBUS API is intended to be stable. It's perfectly OK to use
it directly from any language. In fact the avahi distribution contains
some utilities using the python-dbus to access avahi stuff.

The C wrapper as definied in libavahi-client just makes many things
easier, and resembles the libavahi-core API so that switching between
the two APIs is simplified.

Lennart

--
Lennart Poettering; lennart [at] poettering [dot] de
ICQ# 11060553; GPG 0x1A015CC4; http://0pointer.de/lennart/
_______________________________________________
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus
Reply | Threaded
Open this post in threaded view
|

Re: Hiding internal DBUS use inside a library

Colin Walters
On Mon, 2005-08-29 at 21:14 +0200, Lennart Poettering wrote:

> Yes, the DBUS API is intended to be stable. It's perfectly OK to use
> it directly from any language. In fact the avahi distribution contains
> some utilities using the python-dbus to access avahi stuff.

Ok, cool...I'm wondering then why you need libavahi-client at all; why
not just recommend using bindings appropriate for the application
language like GLib, Python, Qt etc?


_______________________________________________
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus
Reply | Threaded
Open this post in threaded view
|

Re: Hiding internal DBUS use inside a library

Lennart Poettering-2
In reply to this post by John Palmieri-2
On Fri, 26.08.05 12:04, John (J5) Palmieri ([hidden email]) wrote:

> > For example a shared library makes use of avahi, and the application
> > makes use of avahi, or two entirely separate parts, 1 is using dbus and
> > 1 is using avahi, or even two entirely separate parts of the code making
> > use of avahi with two separate main loop adapters.
>
> So having two mainloops seems weird to me.  A mainloop should provide
> integration so anything that would need a mainloop would just use the
> controlling applications mainloop.

Hrmm? Lathiat didn't say two main loops. He said two main loop
adapters.

We use main loop adapters to abstract the used main loop from our
library.

Lennart

--
Lennart Poettering; lennart [at] poettering [dot] de
ICQ# 11060553; GPG 0x1A015CC4; http://0pointer.de/lennart/
_______________________________________________
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus
Reply | Threaded
Open this post in threaded view
|

Re: Hiding internal DBUS use inside a library

Lennart Poettering-2
In reply to this post by Colin Walters
On Mon, 29.08.05 15:19, Colin Walters ([hidden email]) wrote:

>
> On Mon, 2005-08-29 at 21:14 +0200, Lennart Poettering wrote:
>
> > Yes, the DBUS API is intended to be stable. It's perfectly OK to use
> > it directly from any language. In fact the avahi distribution contains
> > some utilities using the python-dbus to access avahi stuff.
>
> Ok, cool...I'm wondering then why you need libavahi-client at all; why
> not just recommend using bindings appropriate for the application
> language like GLib, Python, Qt etc?

Three reasons:

1) libavahi-client is much simpler to use than the bare-bone DBUS API.

2) libavahi-client resembles the API of libavahi-core (our embeddable
   mDNS stack), so it is very easy to port your applications between
   the two APIs. (Basically most functions in libavahi-client  have
   the prefix avahi_client while libavahi-core uses avahi_server. The
   set of functions is otherwise the same starting from the arguments
   and not ending in the semantics.)

3) If people code against libavahi-client they can be sure that as
   soons as avahi offers a non-DBUS-RPC they can use that with the
   same interface. (Don't be alarmed, we won't remove the DBUS support
   in avahi anytime soon, however, for embedded applictions it might
   make sense to have an RPC-API that doesn't require DBUS.)

Lennart

--
Lennart Poettering; lennart [at] poettering [dot] de
ICQ# 11060553; GPG 0x1A015CC4; http://0pointer.de/lennart/
_______________________________________________
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus