omega.palle.net

Sometimes I stumble upon people with what I think is a useless job. Today the office tower I work in here in Bangkok came to upgrade the parking ticket validation machine. Two people came. One person was the engineer, who did the upgrade. It took all of 1 minute. Before he could do that, the other person had to talk a lot about what was happening, carrying a clipboard. After the job was done, someone from my office had to sign the clipboard. The clipboard person didn’t do anything else.

Why oh why do they need to send two people? Couldn’t the engineer have carried the clipboard themselves? Did the “manager" really need to follow the engineer around?

I’m trying a new way to blog, I’m not sure how well it will work.. lets see!

So JIRA5 happened. I still used the old User class, so I had to update the plugin. Since I only touch Java when I have to, I decided to fix up a few things while in there, which resulted in quite a few fixes:

  • Split the list into several Actions.

First of all this is the correct thing to do. Secondly, it allowed the commits and comments to be included propperly in the All tab of an issue.

  • Adjust the design and markup to be in line with the rest of actions

JIRA has some weird markup going on there, but at least now our stuff fits with the feel of the All tab.

Going forward there are a couple of things that I want to fix:

  • Make a connection between github user and jira user.

Most likely this will be resolved in git-indexer, by having it ask a different service for the correct jira (or in our case, crowd) username.

  • Group commit comments under the commits

Most of our users (or at least the ones bothering with feedback), want commit comments to show nested beneath the commit it self, not in the timeline like it is now. I get bonus points for making this a front end toggle on the github tab :p

It didn't take long for me to notice small stupid things that needed fixing. Now I have deployed 1.4 of the jira plugin and version N of git-indexer to production internally.

git-indexer changes

Yesterday I merged the new comment part, which uses the v3 api github provides to fetch comments. For now it is pretty stupid, supports a max of 100 comments pr repo etc, but it is better than the feed, which was bound to miss stuff. This also gave me the propper HTML of the comments, and it gave me the opportunity to format things a bit better on the JIRA-side.

jira-plugin changes

I added gravatars, and format changes. Also I finally bothered googeling the jackson error I kept getting whenever I changed the JSON feed, so now it does not die if I decide to add more fields to the JSON. This results in me being able to more easily update the two pieces independently, which is always great. I love decoupling!

So, once again, at work, I had a small itch, and I decided to scratch it :) We have a build-system that does CI for us, and it relies on githubs wonderful post commit hooks to know when to run CI. Then we have the ops team, who thinks builder should be behind the firewall, and as such, not really reachable by github. We convinced them to punch a tiny hole in the firewall, based on a post I found with some IPs.

Then this week, we started seing commits not triggering CI anymore. At first I was an arrogant bastard and blamed it on the devs of course, but once several of them complained, I started looking into it. Turns out that GitHub has more
IPs pushing post-commit hooks now.

I tried to find some sort of source of IPs somewhere, but I couldn't. So I wrote a small service again, of course :p

I called it "push-me-please", and it is quite simple. I configured a dummy-repo with a post-commit hook to postbin.org. The service keeps pushing the "test hooks" button, and polling the feed from postbin, picking out any new IPs. It also shows the ips on a webpage. Real simple, real quick, but did the job. I discovered some new ips and new ranges I have to pester ops about, to make them widen the hole a bit :)

The code can be found here: push-me-please.

The current list of IPs:

  • 173.203.140.220
  • 173.203.140.221
  • 207.97.227.233
  • 207.97.227.234
  • 207.97.227.235
  • 207.97.227.236
  • 207.97.227.242
  • 207.97.227.254

So this one time, at work, we used Subversion for source control managment. That worked reasonably well when the number of devs was limited, and the number of geographical locations was limited. Now we have a situation where we have people in at least 3 different countries, some with quite some distance between them. Subversion was becomming harder and harder to use for stuff like branches, which we also wanted to do more of. So when we opened our office in Bangkok, we decided that anything they needed, we would move to Git. We ended up using github for git hosting, and that has worked quite well :)

Anyways, we also use JIRA for issue tracking, and since the dawn of man kind, we have been putting Issue IDs in our commit messages. When using Subversion all was well, JIRA (and later FishEye) worked flawlessly together with it, and showed source code changes in JIRA with the issues.

If we had continued to use FishEye, it has eventually gotten some “ok" git-support, but it is still a lot of stuff to maintain. We ended up not using FishEye much. It was to cumbersome for code reviews, and too slow for source code browsing alone. Now we just do code reviews as comments on github.

But that left us without links from JIRA -> github. That wasn't a fun situation. Github has no search for private repos (and even the public one is kinda sucky). The only git plugin I found was outdated, and crashed our jira, needed manual configuration every time we added a repo (and we have 132 now and counting, so thats a bit of a hassle).

It was time to break out the old java skills and develop a soliution fiting our needs.

It ended up with two parts (for now :p)

git-indexer

git-indexer is a node.js based app (my second!). It has several parts in it.

  1. The first part polls our repo lists from github once every now and then, checks for new repos, adds them to our local DB.
  2. The second part clones any repos we don't already have clones of
  3. The third part pulls those repos once in a while, gets all the commits and looks for bug IDs in the messages. If it finds that, it stores that commit as an event assosiated with that Bug-ID
  4. The fourth part polls the rss-feed (for now, will rewrite to v3 api later), looking for comments. If it finds a comment, it checks if it is connected to a commit that is connected to a bug-id. If so, it stores the commit as en event as well
  5. The last part is a WebServer that takes a bug-id, and returns JSON of all events back

JIRA-plugin (github-jira-plugin)

It's quite simple, it has one config option (WS url), and then adds an issue tab panel where it lists out all commits and comments.

GitHub tab panel in action

Summary and code

As you can see it is neither pretty nor complete at the moment. The comments get fcuked in the RSS from GitHub, and there is no way to get old comments at the moment. The comment issue can be resolved by rewriting parts of git-indexer to the new v3 api from github, and that is on the TODO.

The code can be found here: git-indexer and github-jira-plugin

There are probably several issues with all of the above, but we deployed it to our prod jira yesterday, and version 1.2 seems to have gotten rid of the worst problems :p

At work we had a situation where we needed developers to be able to run a preset number of jobs, to maintain the development-instances we have of some products. We didn't want to give them shell access, so instead I wrote a small node.js-based web-server that they can access, select a job, and run it. The source code is at github if you want to peek.

The cool bits

The cool bits in this is the use of socket.io, or at least that is the cool bit for me, since I have been wanting to play with web sockets or something like it for a while.

What I loved about socket.io was how easy it made it. For me it sort of removed the seperation between client and server. Fire events on the server, listen for them on the client. I use the custom event stuff, not only send packages, because, lets face it, why not when it's there? :)

The main part of the website is a console-window, where the output of any running process shows up, live, to all users on the web-page. Each instance of dev-admin is only allowed to run one script at a time, so it also locks the form-controls, stopping people from running more jobs while one is running.

Running scripts

We wanted the system to be flexible, so what I ended up with was something that could handle groups of scripts, and even let you present forms via the web frontend to configure those scripts. Our concrete need was drupal, so the scripts that is in the repo now as examples are all based around drupal and drush.

In nodejs I spawn a bash process and talk to it via STDIN (no kiddin', I do :p). The first hurdle was that the output didn't look very good. All input would be sent at once, and then all the output would appear as it was running. Not very satisfactory for debugging jobs, I can tell you that much.

The decission was that ordering was more important than complex scripts, so we went with one line, one statement. What we do then is we output another command (hidden from the user interface) that will send a special sequence on STDOUT once the command is finished. We then look for this sequence before issuing the next command and so on.

So the order is great now, what about drush and it's ANSI output? Well, the solution was to write a really simple ANSI parser in javascript, that re-writes any ANSI formating sequences to `` with classes.

The result? pretty output, yay!

really cool looking terminal

Once the process is done, the border changes color, depending on success or failure (If one of the commands in the script fail, we also exit imideatly). Here is an example of success!

Success!

I can't really claim that the rebirth went well, can I? It's been more than 6 months since the last post, which didn't have any content anyways.

But I do want to talk about some of the stuff I am doing recently, because I think it is kinda cool :p

So, I have decided to attempt one more time to "rebirth" my site.