I thought I'd spend a little time walking through my recent experience in developing
JDigest, a Notification Digester for the
Atlassian Jira issue tracker.
Inspiration
One day I had several people ask me how to reduce their inbox noise from Jira, well, there just isn't anything you can do about that, if you're involved, you get notified. Looking through the open issues against the
Jira product (in Jira of course!), I found
JRA-1369 in Feb 2009. This was set the seed in my my to develop a solution. The problem with these kinds of things is that you never really know if Atlassian is going to 'fix' it in the near future themselves, making the effort redundant.
Proof of concept
The path to a proof of concept helped me in avoid some wrong turns, initially I thought that providing Jira with a new MailQueue implementation would be what was required, but quickly found it was just an over-complication. The route adopted was to create a custom
Event Listener, enabling normal non-issue related email to be mailed direct through the existing MailQueue, and issue-related email to be digested, if appropriate.
Unpicking the default Issue Event handler
Jira currently ships with a 'Default' issue event listener implementation, which is hard coded in several places, the likely reasons were to ensure that the event listener got setup before any issue related activity could occur, so as to not loose events.
Fixing the code to not use the Default issue event listener wasn't too hard, just modifying
ConsistencyCheckImpl.java to not specifically check for the default listener, and
MailListener.java to fix its read only nature to allow system admins to remove it.
Making source code changes to Jira is not something to be taken lightly and concerned me on two fronts, firstly, requiring potential customers to make source code changes could damage adoption, and second, every customization has to be verified on the next release, and Atlassian release often. Initially I made changes by hand, but after the 20th install I wanted something more repeatable so I wrote a
Bash script and made use of
Sed to programmatically fix the files. Even though this needed slight tweaks for newer versions it will save time in the long term support of various Jira versions.
In order to help the wary, I prebuilt a pre-modified classes that can be used to overwrite the existing classes.
Since then I've found
JRA-19957 which asks if this can be fixed, after all, only system admins can modify listeners, its scheduled for 4.1.x so fingers crossed, that this jiggery pokery can be removed soon.
Initial implementation
Exactly what to persist was interesting, I could either (a) try to persist the actual issue event (and its related required data references) and reconstitute later, in order to render the related email content, or (b) render the email, gut the content and persist the rendered output, meaning the reconstituion later would be relatively simple. The initial implementation used RAM to store the actual issue events as they came in. Of course, this isn't scaleable - on a busy site the memory requirements for this would be prohibitive, but was still useful nevertheless, to prove the theory. Choosing (b) seemed to follow the
KISS principle.
Gutting the email in HTML and Text forms, and later reconstituting was relatively easy through a
Velocity template, and will enable end-user styling, but did present a problem - users could change their email preferences half way through a digest period, so half would be in HTML and half would be in text, oh dear, sorry about that.
By now something in the order of 4 months part time had passed, but at least I was getting digest emails!
Hunting around, it seemed there were two choices, either make use of
PropertySet storage within Jira or extend the
Jira Database Schema. The choise wasn't clear cut, on the one hand, PropertySet is already present, if used, would ensure queued digest data would not be lost, would be included in backups etc, on the other hand, using
XStream (awesome code) to translate objects and fields into XML (even handling Lists etc) whilst certainly being usable, may represent a performance hit. Fortuantely the versatility of how JDigest can be enabled will give admins some options on reducing that.
Meta Data
The digest data was being stored per-user, also being stored was meta data about priority, issue type and project. A user could, through a user preference page see what was in store for them. In addition to that though, I also wanted to provide Project Leads with an idea as to what issue activity had generated on a per-project basis (thinking of future Flot charting of events by project), and also on a global basis, enabling the system admin to flush all data. So, even more meta data was required. Given I'd already chosen the PropertySet storage mechanism, that meant event more serialization of additional meta-data, all in response to each and every issue event. This could turn out to be the achiles heal in terms of performance for larger environments.
Dynamic UI?
Jira 4.x is going dynamic with jQuery under the hood, so after getting the various views behaving as expected, I wired in
jQuery support for all the forum submissions and issue status interactions, allowing a targetted refresh. This mean a raft of
Servlets to handle the data without
XWork Actions interfering.
What? User want flexibility?
8 months in, I start thinking about user configuration, after all, enabling Digesting globally is great, but what about things that the
User decides are hot-topics that need up-to-the-minute delivery? Conceiving a UI that wasn't a massive amount of work led me to the idea of a dynamic table, fortunately, enter
DataTables an awsome jQuery library, that can be setup to also use
jEditable (
example)
to yield a client side single (and even multi value select) table editor with ajax calls to persist data. I shall be contributing back to this project heavily!
Licensing
Wow, I thought I was done, 9 months in and I had a working plugin, but. What to do about licensing? After all, Im trying to get payback for the last 9months of burned evenings and weekends. I looked around and initially couldn't find anything, I was specifically interested in Eclipse tooling, but there was simply no such plugin - aha I though, I could write one? But first I looked around some more, I found a reference to the CustomWare
License Creator Plugin but it didnt work for me, looking into the sourcecode I found it was based on
TrueLicense (IIRC). So I tookup that challenge and wrote some unit tests to validate what I though was correct operation, mostly successful but I became painfully aware of the lack of automation, so I donned my
Eclipse (
docs,
articles)
Plugin hat and wrote the automation I wanted, including a runtime Java
Maven2 build enabled project generation facility that would auto create a unit test with necessary byte[] data pre-loaded.
With a nice Eclipse plugin to create per-product licensing projects, a storage system (flat files) to store generated licenses in, and Eclipse editors to allow license retrieval and validation as well as the ability to autogenerate the necessary bootstrap license validation code, I have a really useful and Licensable product.
To obfuscate or to not obfuscate, that is the question
Whatever you do, obfuscation can be bypassed if you put enough effort into it, and thats the point, making it a
PITA for anyone to mess with it, and if they are that intent, heck, fair enough. There are routes, perhaps native compilation of certain aspects of the licensing, but really, obfuscation doesnt really solve the problem, just stops people using
JAD to dump raw java code out potentially enabling simple bytecode tweaks to void licensing. See
this great Javaworld article.
There is a cost to obfuscation, I spent 3weeks or so fine-tuning my use of
ProGuard and a maven build plugin
proguard-maven-plugin to get to grips with the finessing required. Other steps can also be taken to detect tampering, classes can be fingerprinted, and that data secreted to stop casual modification. All of these measures can be circumvented but again, the point is to make it too much grief to want to do.
The development of the Eclipse licensing plugin, provision of a runtime library for third party use, maveninzing the build added something like 3months (part time!) I have yet to wrap that up as a licensed product as a plugin, and as an Eclipse
Rich Client Platform application. It cost me time, but gave a reusable platform that has already had 'best bang for buck' techniques applied.
The build
A repeatable build with no manual tweaks is a must, despite the upfront costs in getting it just so. Being able to just hit go and know that everything that should be done will be done will save time tenfold later. This particular case had quite a complex pom.xml file, containing several plugin executions not considered 'normal'. As its not possible to target the
order of plugin execution for a given
Maven Lifecycle state, I needed to spread out the sequence of plugin executions at the Lifecycle level, slightly misusing Maven, yet achieving what I needed. Maven3 apparently has this fixed.
Summary
The JDigest plugin is a simple concept but converting into a reality took a fair bit of time, and adding in pesky licensing issues almost doubled development time. Of course,
next time, that 50% will be more like 5%, probably. It's been a great learning experience and has set me up nicely for future endeavours, I hope this journey will be interesting to some!
p.s. I'm now affiliated with AppFusions, interesting developments ahead!
