Transports
My Role With Spectrum So Far
So it’s been a few months since I buried Kraken in lieu of moving towards Spectrum. I’ve been very happy with it from a server administrator and user perspective, but I haven’t done as much as I’d hope coding wise. A lot of things I was going to fix, Jan fixes in no time flat. [...]
Thinking about Gateways
It’s been a little while since I thought much about my illustrious IM Gateway plugin for Openfire. My previous job kept me a tad too busy with other things for me to really have time for it, or be interested in spending my free time on it. Now, let me prefix this by saying I don’t have a lot of free time right now, and I may not for a while. That said, I had some time to step back and look at the IM Gateway plugin and think about some possibilities.
So first off, I don’t really know the state of the Py*t transports. Last I heard there was no one maintaining them really anymore. That is a shame as I was hoping to see them flourish outside my involvement. If that’s not true, then I certainly apologize — just know it’s because I haven’t kept up well. Regardless, I found that I prefer Java to Python for the most part anyway. So would I take those projects back if they were offered to me? Probably not to be frank.
But the IM Gateway plugin — The fact that it only works with Openfire I feel is a disservice to other server implementations, but overall it’s a very cool implementation that tackled some issues with external implementations. But if you need to restart the IM Gateway plugin, you have to take the server with it. Java seems to have a habit of being very plugin-able friendly but then easy to break things with it’s container. I’m seeing that with an application server I’m working with — it’s not hard for an app to break the application server.
So I began thinking about what core things the IM Gateway plugin does that the Pys did not. Primarily it revolves around direct internal access to user’s rosters so that things could trivially be kept in sync. Things being, nicknames, groups, actual rosters, etc. So pulling back I’m wondering how feasible it would be to do something like this:
1. Strip away the ties of the IM Gateway plugin to Openfire and make the implementation standalone
2. Take that stripped away part and turn it into a small “helper” plugin that works with the external gateway implementation to accomplish the same goals
3. Similarly, write a small “helper” plugin for ejabberd that does the same thing
4. What about other servers I don’t have interest/time to write helper plugins for? Well then the transports would act like any other external transport.
How would these helper plugins do their jobs? Well somehow they’d have to have open communication back and forth with the actual transports to manage rosters, maybe even intercept packets and munge them a bit. A little ugly but hey.
As you can probably tell, over the years I have never come up with some solution that could be formed into an XEP for all of this. At some level I continue to help that I’ll see the light through my experiments here, but I’m not holding my breath. In the meantime I’d like to see solid transports that work for everyone. Writing an ejabberd plugin would give me a real reason to learn erlang and really see what it can do.
Now, if you are someone who is excited about this concept, please understand that I don’t have a lot of time right now. I need to spend my free time on things that make me money for the time being and unless someone is dieing to sponsor this concept, I couldn’t put it ahead of other assorted things I am working on. That is not a beg for money, that is a statement of my situation and why I’m not diving into this “right now”. 
I would love to hear feedback on this btw.
External vs Internal Gateways, Round 2
You may have seen, in the past, my podcast about External vs Internal Gateways. In that podcast I talked about a number of benefits that internal gateways have over external gateways.
It’s not all peaches and cream though. You see, the problem with being so close to the internals of the server with internal gateways is that you tend to hear and trigger far more than you intend. I’m going to pick on Openfire a little here since that’s what the IM Gateway plugin is using. Some of these may simply be something that should be fixed in Openfire. Being one of the developers of Openfire, yes, I will investigate them when I can. =)
One of the issues is, when a roster item changes, I actually get a number of listener events that tell me this occurred. It’s difficult to know what the proper event is. In some cases, I’ll get an event with no groups and then another event with groups. Hard to know which one of those is true. All things that I need to run through and make sure I’m doing things correctly.
Likewise, any time I tweak something about a roster item, I get events telling me it’s changed. Yes, I’m aware, I just did them. =) So in that case I need to ignore events.
It’s all very strange actually. I switched to listening for packets for real IQ packets coming from the client. Guess what, there’s a lot of IQ packets that come from the client. So that’s not really making things easier. In XMPP, groups don’t mean as much. Removing and readding groups is nothing big. In AIM and friends though, it’s dire. You remove groups and you’ve moved things around on your roster a lot. This is primarily because in XMPP, you have a roster item, and an attribute of it is what groups are in it. In AIM you have groups, and an attribute of a group is who’s in it. (for nitpickers, I’m referring to attribute here as “something the entity contains”, not attribute in the literal XML form of the word)
So if you were thinking, wow, internal gateways sound so easy! Think again. =) There’s a lot that goes on behind the scenes that’s more difficult to deal with than you might expect. So to sum it up, I’d like to simply say, there’s pros and cons of both. =) Don’t think an internal gateway is a magic bullet.
When Worlds Collide
You know, when you are working by yourself on an open source project, your schedule is your own. If you decide you don’t want to work on it for a couple of months, that’s your perogative. At some level there’s no rules what-so-ever aside from not doing things that will drive folk away from being interested in your project (assuming you care). One of the biggest things that drives the project, however, is what -you- want. When you are working on an open source project where you are teamed up with others, there’s some checks and balances over what you do and your partners do. However, at the end of the day, it’s the same type of situation, you all decide what the focus of the project will be. I don’t like to put it like this, but no one has a “right” to your time except for you.
When you start working with a company on an open source projects, things change. There are deadlines that the company is pushing for help drive your own timeline. There are things they’d like to see happen that you may or may not agree with. Instead of saying “well unless you submit a patch, it’s not happening”, you tend to work something out instead. It’s an interesting adjustment to “normal open source projects”.
Do I dislike either? No. In fact I’ve quite enjoyed the experience of it. It’s taught me a few things that I wasn’t aware of. Like I was rather blind to some of the requirements that corporations ask for. Some of the things they ask for I would typically have thought “that’s ridiculous” with some of my other projects, but here I see a lot of folk bringing the same issue up and I start to discover that the issue is more commonplace than I would have assumed. I’m not citing any examples here. Just suffice to say there’s things that having a broader knowledge of the corporate world has made me reconsider some of the decisions I might have made with other projects in the past.
So many thanks to Jive Software for giving me this additional experience that I wouldn’t have gotten probably with PyAIMt and PyICQt. =) It’s been fun and I hope it continues to be fun!
Busy busy busy… too busy?
The problem with personal projects is just that, when the person in question gets interested in another project or busy or whatever, development stops. This is what the current status of PyAIMt and PyICQt is. I am having a lot of fun with the IM Gateway plugin for Wildfire, and as a result I don’t feel interested in putting any time into PyAIMt and PyICQt. Are these two dead now? No. I still have “things I’d like to do with them” in my head. It’s just most likely we’ll see a 1.0 version of the plugin before I touch PyAIMt and PyICQt again.
One thing that the IM Gateway plugin has brought me is that I’m now quite comfortable working with other folk on the same source code. That said, I’m trying to weigh the pros and cons of enlisting help with PyAIMt and PyICQt. There are quite a few folk that are interested in using it, a few folk who submit me patches from time to time, and certainly general interest in the transports. So my question posed is… Are there folk out there who are fairly experienced python programmers who would be interested in becoming active developers of PyAIMt and PyICQt? I am not handing off the project and will continue to be the project lead, and will continue to put development time into them as time permits. Anyway, I have not yet decided on this, I’d just like to hear from folk who might be interested in contributing. If I see some interest that’ll help drive my decision. =D
On a semi-related note, I continue to find java with an IDE to be fun and interesting. I think working with java without an IDE would be maddening due to how long it takes to compile and test things in the environment I’m using it in, and it’s certainly saved me a ridiculous amount of time in debugging stupid typos and crap. The IM Gateway plugin itself is going pretty well but unfortunately there’s a lot of work that involves working on support libraries as well. The original developer of Java-JML turned over development to me so I’ve been working on that. That’s pretty fun, I know more about the MSN protocol now than I ever expected I’d know. The Yahoo library is … kind of a pain actually. =/ Right now Yahoo is the biggest source of issues for me. Joscar for AIM/ICQ is a damn fine library, but I want to move to using a more simple API and I can’t seem to figure out how I use Joscar’s simpler API. Perhaps I should like… ask them instead of just blogging about it. I tried figuring it out from Adium X but yeah… =) That’s kind of a weird setup. I need to work on implementing the old school ICQ authentication because, unfortunately, the “new style” AIM auth only works with newer accounts. (how bizarre) I’ve still got a lot to work on with IRC as right now it’s a mightily talkative transport and I doubt most folk want to see all of that. =) My biggest issue at the moment is that I’m fighting with AJAX support in the web interface. Trying to use DWR… seems like it should work fine… doesn’t. I wanted to figure this out on my own but it looks like I shall be deferring to the Jive folk to get it taken care of. So many things to do.
So anyway. Thinking back, I remember that the thing I thought was so awesome about XMPP is that the entire spec was written out and ‘open’. My first endeavour was that I like the Zephyr system from MIT and thought, based off the open protocol, that it would be neat to implement zwgc and friends except to communicate to Jabber/XMPP. This formed into JWGC. Transports came later, etc etc. The great thing is that the openness of the protocol leads to a great feeling of being able to write your own client/server/whatever to learn how it all works and go from there. The bad thing is that the openness of the protocol leads to a great feeling of being able to write your own client/server/whatever to learn how it all works and go from there. =) Now, I don’t think this truly is a bad thing, but it does lead to a -lot- of starter projects written and then ditched. I often see folk come jadmin and jdev asking about where to start in writing their own client/server/whatever, and typically they get a response of “why not just contribute to an existing project?”. If the person’s goal is to just learn, starting from the ground up is not a bad way to go. If the person’s goal is to get a server/client up and running that has X feature that all of the servers don’t have, contributing is definitely the better way to go. What’s the fine line between writing your own and not writing your own? I certainly can’t answer that. Note that I think asking “why not just contribute…” is a good way to go about it because that encourages the person to say “well I just want to learn”, at which point ok, go for it. Or it encourages us to help point the person in the right direction.
Future Improvements?
In working with the IM Gateway Plugin, there has been a pressure to try to improve the way transports are handled in general. I’ve begun the process of seeing what could be done to improve things, but one of the big sticking points seems to be not having the transports in your actual roster.
Now many clients provide an option to simply hide transports from your roster. IMO, this is not a bad way to handle it. Leaves it up to the client to decide whether to hide things or not. It also allows for the end user to decide whether or not to see the transports or not. I, for example, like to be able to see the transports and how they reflect when I’m logged in, what my status is, etc. (because, for example, AIM only does available and away, none of the other statuses, and this is neat to see in my roster because I know that AIM folk don’t know that I’m set to do not disturb) It also allows clients like Psi and such to provide me a way to right click and choose log out. (equiv of sending a directed presence packet)
So how would you go about having a transport detect that you’ve logged in if it’s not in your roster to be telling you so? Well you can cheat for one. The IM Gateway Plugin allows me to use internal listeners. This solution, of course, isn’t going to help external transports a lick. But for the plugin, it’s nice. Another way might be to require some sort of ad-hoc command to log into legacy services. Like instead of auto-logging in, you have to select “log me in”. But then how does it detect status changes and such? More ad-hoc commands? Eww! How many freakin’ ad-hoc commands does it take to screw in a lightbulb. Another way could be to create some sort of ‘trust’ relationship with a transport. So a transport could indicate to it’s attached server that it wants to be notified of any presence changes. Not so sure about that either. I think that was already addressed a long time ago in regards to transports wanting to interact directly with a user’s roster.
Long story short, I think this will be fine for the IM Gateway Plugin to ditch the roster, as it apparantly confuses some folk to see transports in their list, but I don’t think this is viable for other transport implementations.
That said, some issues have come up that -should- help other transports. I’d like to, for example, write up an XEP that focuses on how to handle user preferences. I’m not talking things like, “what theme for your client” but rather things that are handled on the server end. For example, mail notifications. AIM has them. Not everyone wants them. User should be able to go in and choose “no or yes” for mail notifications. =) These are enabled or disabled on the transport end and a client shouldn’t be expected to have to be bothered with ignoring lots of things it doesn’t want in the first place. This is basically an ad-hoc command type of thing, but I’d like to standardize it or make it a “best practices” type of thing. That way client devs could fairly easily go “oh this one supports user prefs, I’ll provide an easy link for that”. Something like that.
Another one is a XEP that I had intended to write a while ago… basically something similar to Jabberd2′s component protocol where the transport itself can connect and tell the server what jid’s it wants to be. This means that the transport could say “Hey, I’m aim.jabber, chatrooms.aim.jabber, and fries.aim.jabber” and the server could go “oh ok, well you got ‘em” or “bugger off, those are taken”. Wildfire actually implemented this in a slightly different way than Jabberd2, but I think the same XEP could address both just fine. I believe I originally called this “Extended Component Protocol” or something like that. Who knows. It’s been a while.
On a side note, I think I figured out what was causing problems with Java-JML. Basically I kept getting a severed connection upon logging in. Turns out I think that the server Java-JML has for “fast login” is not a good one. Theoretically you are supposed to connect to a nexus first and it’ll tell you a login server. I’m doing that now.
BTW, Thanks to the Cenqua peeps for providing me with a Fisheye setup for Java-JML! I’m going to link that off the Java-JML site in a bit here. =) I love me some Fisheye.
IM Gateway Plugin for Wildfire, and other things
Today we released the first beta version of the IM Gateway plugin for Wildfire. I’m quite excited to have a release out, but also a little pensive as I always am with a release. This one I’m expecting quite a lot of bug reports on because well, it’s an early beta and that is the way of things. I wanted to get a version out soon-ish so that folk could play with it and could start getting me some solid bug reports to work from. I never run into the same problems my users run into. Always interesting to see things like… “my friend is using ICQ version 1 and I can’t receive messages from him!”
Anyway, I took a break from developing after fixing a last couple of bugs this weekend to make sure that I didn’t get involved in something that would temporarily delay the release. Spent some time on PyICQ-t in the meantime. Upgraded it to use James’s twistfix stuff and also to kill the pubsub support I was working on, since it wasn’t useful anymore after the recent updates to PEP. I would like to put out a new release of that soon. My typical “get everyone on the same page” release. Besides, right now they don’t work with Twisted 2.4. =/ (right now meaning, the current releases)
I have been feeling like crap for about a week so I think I may try to calm it down and just relax until I am better. No sense in making myself sicker. Besides, that will give folk time to comment. In the meantime I will do something leisurely like.. get out a release of the Pys. lol Why is that leisurely? Well because it’s already done, it’s just a matter of I need to put out the release. PyAIMt will be somewhat different in that I need to port over my bazillion changes to it. I may not dive into that until I’m feeling better. Then again, honestly, it’s kinda mindless work, so it might help to get my mind off being sick.
Wildfire plugin updates, and the Py’s
The Wildfire gateway plugin is coming along. Not as fast as I’d like at the moment, but I keep running into minor stumbling blocks outside the context of the actual plugin (read: getting sick, that sort of thing). That said, I’m switching internal libraries for MSN. The one I was using does not behave well in a multi-user environment. Furthermore, it had some kind of annoying debugging that there wasn’t an easy way to turn off. I’ve switched to a more recent looking library. It was between that and another one whose documentation was all in… I believe Korean. All things considered, the function call names were in english, so I could have muddled through, but if both look good, why not choose the one I can read. We’ll see how that goes. This one requires a little more knowledge of the underlying concepts of MSN’s IM. It’ll be fine. =)
As for the Py’s, I’ve done nothing with them for a bit now. The pubsub stuff I was working on was causing some file system problems and, on top of that, the recent updates to pep makes my work semi-useless. No worries, I’m glad to see it doable in a much nicer fashion. James is working hard on some wonderful new revampings to the underlying code for PyMSNt that I’m looking forward to pulling on over to PyICQt and PyAIMt. I will, however, watch and wait until things look like they’re good to go. I don’t have a gauge for this, so I’ll just post it aloud here… do people want me to put out a PyICQ-t and PyAIM-t with all of my current updates that’s an actual release? (underlying code will be cleaned up to lose the pubsub stuff first if I do that) Otherwise, I’ll be waiting for some bigger updates.
It’s interesting to note that looking at code in one language sometimes helps you clarify concepts in your head in another. Just the way joscar does a few things made me go “oohhhh yeah I could do that with pyaim as well”. *shrug* I thought it was neat…
PyAIMt and PyICQt security releases
I have just released minor code change updates for PyICQt and PyAIMt. These updates fix a security hole in the MySQL XDB backend. I highly recommend you update immediately if you are using the MySQL XDB backend. If you are not, then you can ignore these releases.
I’ve been trying to make it a personal policy not to post every release on my blog unless I simply have some blog type observations to make about the releases. However, this one is a security release set and I want to get the word out as much as I can.
Pending transport releases
PyAIM and PyICQ have bother undergone a lot of fixes recently. The only problem is, I have noticed that something is wrong with the buddy icon communication between two people. That being . . . . I send and receive the buddy icon -every time- a message is sent. (yikes!) It’s supposed to be storing a hash of the icon so it doesn’t ask for it again. Something is clearly wrong with the hash… I want to fix this before I put out either release because it’s horrible horrible horrible abuse of bandwidth to just send those every time. =/ Stay tuned…