Pure Python DBus implementation

[Jakub Piotr Cłapa-2]

Would anybody be interested in such? I have a working SASL module (with
DBUS_COOKIE_SHA1 method implemented), the message protocol and the
unmarshaler and would like to know if the whole project would be
welcomed by the community or if I only do this for myself.
It's not perfect but I tried to closely follow the spec and it
successfully parses the two messages you get from the bus after
connecting and sending Hello (this message alone is currently hardcoded
byte by byte because the marshaler is not done yet ;)

A word about 'why':
- the current bindings are buggy (I send two patches only to discover
   that the marhshaler fails to handle Dictionaries)
- they are bindings after all so they have a quite big dependency (the C
   dbus library)
- they are quite high level and enforce many constraints on how you want
   to write your program and use the protocol
- the protocol is simple and the marshaled data map well to the Python
   data types so writing the implementation was quite simple
- alternative implementation (AFAIK there are no other than mine and the
   reference) allow to polish the corner cases in the protocol (e.g. the
   UNIX socket thing I wrote earlier on this list)

--
Regards,
Jakub Piotr Cłapa
--
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus

[John Palmieri-2]

On Sun, 2005-08-07 at 16:10 +0200, Jakub Piotr Cłapa wrote:
> Would anybody be interested in such? I have a working SASL module (with
> DBUS_COOKIE_SHA1 method implemented), the message protocol and the
> unmarshaler and would like to know if the whole project would be
> welcomed by the community or if I only do this for myself.
> It's not perfect but I tried to closely follow the spec and it
> successfully parses the two messages you get from the bus after
> connecting and sending Hello (this message alone is currently hardcoded
> byte by byte because the marshaler is not done yet ;)

Is this necessary?  Decoding the messages in pure python sounds like a
cool project but I have to think this is just duplication.

> A word about 'why':
> - the current bindings are buggy (I send two patches only to discover
>    that the marhshaler fails to handle Dictionaries)

Dictionaries work fine.  What issues are you having? The word buggy
doesn't help me fix the issues.  The whole thing is in flux so when we
reach 1.0 it won't be buggy.  I fail to see how from scratch
implementation will fix the problem that software will have bugs.

> - they are bindings after all so they have a quite big dependency (the C
>    dbus library)

But any system using dbus will have these loaded into shared memory
anyway.

> - they are quite high level and enforce many constraints on how you want
>    to write your program and use the protocol

This is intentional though you can use the low level bindings.  If the
higher level bindings aren't what you need you need to tell me how you
are using them so I can tailor it to your needs or show you a better way
of doing what you want.

> - the protocol is simple and the marshaled data map well to the Python
>    data types so writing the implementation was quite simple
> - alternative implementation (AFAIK there are no other than mine and the
>    reference) allow to polish the corner cases in the protocol (e.g. the
>    UNIX socket thing I wrote earlier on this list)

I have no objections to an alternate implementation though I do think it
will confuse the end user.  My main concern is you sent two simple
patches and then decided to go off on your own instead of trying to pool
resources and mature the already existing bindings.  Right now I am the
only one working on them with some patches streaming in from other
parties so admittedly I don't always have the time to devote to fixing
everything quickly though they are on my list.  For instance your
problems with DBus types can all be solved with the patch I am doing for
implementing introspection reads.  But that all takes a lot of time
since there is a significant rewrite that needs to be done in the
proxying code.
 
--
John (J5) Palmieri <[hidden email]>

--
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus

[Jakub Piotr Cłapa-2]

John (J5) Palmieri wrote:

> On Sun, 2005-08-07 at 16:10 +0200, Jakub Piotr Cłapa wrote:
>
>>Would anybody be interested in such? I have a working SASL module (with
>>DBUS_COOKIE_SHA1 method implemented), the message protocol and the
>>unmarshaler and would like to know if the whole project would be
>>welcomed by the community or if I only do this for myself.
>>It's not perfect but I tried to closely follow the spec and it
>>successfully parses the two messages you get from the bus after
>>connecting and sending Hello (this message alone is currently hardcoded
>>byte by byte because the marshaler is not done yet ;)
>
> Is this necessary?  Decoding the messages in pure python sounds like a
> cool project but I have to think this is just duplication.

It's 10 times (or more) simpler with Python than with C so it has some
advantages.

>>A word about 'why':
>>- the current bindings are buggy (I send two patches only to discover
>>   that the marhshaler fails to handle Dictionaries)
>
> Dictionaries work fine.  What issues are you having? The word buggy
> doesn't help me fix the issues.  The whole thing is in flux so when we
> reach 1.0 it won't be buggy.  I fail to see how from scratch
> implementation will fix the problem that software will have bugs.

It won't fix such bugs but with Python you can easily read the whole
code that is sending the messages and with libdbus it's really difficult
(the amount of C code is big). The debugging and modification is much
simpler.

I haven't really debugged the problem because I got angry on the whole
thing and almost thrown it you of the window, sorry. (it was about the
same time my girlfriend went to Sweden for 2 weeks and these two things
together were somehow difficult to handle)
It just seemed that I only got the beginning of the body sent and the
rest ignored. The struct which caused this can be seen in my last post.
(the one about the pure python dbus-monitor)

>>- they are bindings after all so they have a quite big dependency (the C
>>   dbus library)
>
> But any system using dbus will have these loaded into shared memory
> anyway.

Good point but I rather thought about Windows and other platforms.
You can tell me that porting C code is easy but it's really hard as hell
compared to porting Python code. ;]

>>- they are quite high level and enforce many constraints on how you want
>>   to write your program and use the protocol
>
> This is intentional though you can use the low level bindings.  If the
> higher level bindings aren't what you need you need to tell me how you
> are using them so I can tailor it to your needs or show you a better way
> of doing what you want.

It's just that it imposes many constraints limiting flexibility. Of
course you get free introspection and such (which is cool) but sometimes
it would be better to try something more low-level especially when
testing some not canonical usage patterns. (Like sending XMPP messages
through DBus)

I wanted to do some raw marshaling (to a buffer), sending arbitrary
messages over the bus and such. It's not a thing you would want in a
finished product but it helps when I was experimenting.

>>- the protocol is simple and the marshaled data map well to the Python
>>   data types so writing the implementation was quite simple
>>- alternative implementation (AFAIK there are no other than mine and the
>>   reference) allow to polish the corner cases in the protocol (e.g. the
>>   UNIX socket thing I wrote earlier on this list)
>
> I have no objections to an alternate implementation though I do think it
> will confuse the end user.  My main concern is you sent two simple
> patches and then decided to go off on your own instead of trying to pool
> resources and mature the already existing bindings.  Right now I am the
> only one working on them with some patches streaming in from other
> parties so admittedly I don't always have the time to devote to fixing
> everything quickly though they are on my list.  For instance your
> problems with DBus types can all be solved with the patch I am doing for
> implementing introspection reads.  But that all takes a lot of time
> since there is a significant rewrite that needs to be done in the
> proxying code.

I understand this and really my intention is not to duplicate your work
or to say that it sucks. It's just that I got somehow irritated when I
tried to get this working and since I didn't like the high-level
approach you used I thought about writing my own implementation. I
thought it will be easier (it was not really trivial but not impossible
too) but I succeeded I think.

DBus protocol seemed to be designed with several implementations in mind
  (you can read it in the spec ;]) and it was one of the things that
encouraged  me to do this.

The whole rewrite was probably not totally rational but ended up quite
well I think. It could (after some cleanups) be useful for people who
want to get a grasp of how DBus works and have some qualities that
compiled code doesn't have (e.g. cheap multiplatform support).

I understand your doubts about it (it seems to negate your work and
build competition behind your back) but I don't have such intentions.

Now, when I already got it working, I have some design ideas which I
would love to discuss. We could also decide whether the two
implementations should co-exists or not. Both approaches have pros and
cons so possibly they can both be used in different situations.

PS. Writing in English is still sometimes difficult for me and I'm
pretty sure I haven't written what I wanted to write, sorry...

--
Regards,
Jakub Piotr Cłapa
--
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus

[Ross Burton]

On Thu, 2005-08-11 at 23:08 +0200, Jakub Piotr Cłapa wrote:
> >>- they are bindings after all so they have a quite big dependency (the C
> >>   dbus library)
> >
> > But any system using dbus will have these loaded into shared memory
> > anyway.
>
> Good point but I rather thought about Windows and other platforms.
> You can tell me that porting C code is easy but it's really hard as hell
> compared to porting Python code. ;]

Again, a moot point.  John's argument was that for any system to be
using DBus, even Windows systems, they must be running the daemon which
depends on the library.  Unless of course you are also going to write a
pure-python daemon implementation.

Ross
--
Ross Burton                                 mail: [hidden email]
                                          jabber: [hidden email]
                                     www: http://www.burtonini.com./
 PGP Fingerprint: 1A21 F5B0 D8D0 CFE3 81D4 E25A 2D09 E447 D0B4 33DF



--
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus

[Jakub Piotr Cłapa-2]

Ross Burton wrote:

> On Thu, 2005-08-11 at 23:08 +0200, Jakub Piotr Cłapa wrote:
>
>>>>- they are bindings after all so they have a quite big dependency (the C
>>>>  dbus library)
>>>
>>>But any system using dbus will have these loaded into shared memory
>>>anyway.
>>
>>Good point but I rather thought about Windows and other platforms.
>>You can tell me that porting C code is easy but it's really hard as hell
>>compared to porting Python code. ;]
>
> Again, a moot point.  John's argument was that for any system to be
> using DBus, even Windows systems, they must be running the daemon which
> depends on the library.  Unless of course you are also going to write a
> pure-python daemon implementation.

DBus can also be used in a peer-to-peer mode and can use TCP/IP sockets
instead of UNIX domain. Writing a messagebus in Python would also be an
interesting idea but I'm not planing to do this anytime soon.

But you're right, it's not that important. The only thing I see in my
implementation that's IMHO better than the current bindings is that it
is much simpler. Of course it doesn't have many features but still it
would be even after adding them.

Maybe it's just a mater of personal liking -- I prefer pure Python over
wrapping C libraries when it is reasonable (mostly from the performance
point of view).

I don't argue this should be the way to go and that the bindings should
be depreciated so if you don't like the idea I will simply use it in my
project (because it works and the bindings currently don't) and when I
finish it I could think about working on the bindings.

--
Regards,
Jakub Piotr Cłapa
--
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus

[Anthony Baxter]

On Friday 12 August 2005 03:39, Jakub Piotr Cłapa wrote:

> DBus can also be used in a peer-to-peer mode and can use TCP/IP sockets
> instead of UNIX domain. Writing a messagebus in Python would also be an
> interesting idea but I'm not planing to do this anytime soon.
>
> But you're right, it's not that important. The only thing I see in my
> implementation that's IMHO better than the current bindings is that it
> is much simpler. Of course it doesn't have many features but still it
> would be even after adding them.
>
> Maybe it's just a mater of personal liking -- I prefer pure Python over
> wrapping C libraries when it is reasonable (mostly from the performance
> point of view).

Hopefully (I haven't had a chance to look at your code yet) it should also
be easier to slot a pure-python implementation into alternate event loops.
There's possibly value in an alternate implementation for testing
the main code for regressions, too.

Anthony
--
Anthony Baxter     <[hidden email]>
It's never too late to have a happy childhood.
--
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus

[Jakub Piotr Cłapa-2]

Anthony Baxter wrote:
> Hopefully (I haven't had a chance to look at your code yet) it should also
> be easier to slot a pure-python implementation into alternate event loops.

It is really easy since it just needs a call to connection.loop_iter
when new data arrive on the socket. I know that's pretty raw and not
really prepared for every thing that can happen to a connection but I'm
not an expert in async-io programming and really must first finish my
main project so I thought I leave it for now. It's dumb easy to change
that so it shouldn't be a problem for anyone.

The tried to loosely model the protocols I use (LineProtocol and
ChunkProtocol) after protocols in Twisted but I don't have much
experience about how to this TheRigthWay(TM) so anyone with more
experience in this would be really welcomed. (it should be easy to
replace it with a BetterThing) :)

> There's possibly value in an alternate implementation for testing
> the main code for regressions, too.

That is what I hoped for too. I already found one problem (not a bug
really, but confusing) with UNIX socket addresses.

--
Regards,
Jakub Piotr Cłapa
--
dbus mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/dbus