Thursday, October 09, 2014

Custom multiseries trend using API

The initial goal was to create a trend of event data related to the time of day or day of week when the event occured. Later on, it seemed like a good idea to display different timeframes on the same trend.

The end result shows a trend, calculating an average value of a metric (buildtime duration, in this example) for all events that occured in the same time interval (day of week, in this example), for different timeframes (last week, month and year, in this example), which are displayed as different series in the same chart, to be able to compare them and visually notice an evolution or an anomally.

This trend is part of the Buildtime Trend project, you can see the code in action here.

Read on to see how it is done.
The service and API is used to store, analyse and visualise the event data. I'd like to refer to the tutorials on how to create a query and generate a chart.

Generate and group by time intervals


First of all, the event data has a timestamp, so in a simplified example, an event would look like this :

  { id: "1234abcd", duration: "97", timestamp: "2014-10-09T18:32:14Z"}

But to group events on time intervals, like day of week, or hour (time of day), the timestamp  has to be split into its components (thanks to Ryan Spraetz of for the suggested workaround), for example :

  {id: "1234abcd",
    duration: "97",
    timestamp: {
      isotimestamp: "2014-10-09T18:32:14Z",
      day_of_week: 4,
      hour_24: 18,
      hour_12 : 6,
      hour_AMPM : PM,

Look here for the code to split the timestamp (in Python) and a full example of a split timestamp.

A query to group events by day of week, calculating an average value of duration, for all events of the last week, would look like this :

var queryLastWeek = new Keen.Query("average", {
  eventCollection: "builds",
  timeframe: "last_week",
  targetProperty: "duration",
  groupBy: "timestamp.day_of_week"
Using an example from the tutorial, you could easily create a chart with one series of data.
If timeframe is changed to 'last_month' or 'last_year', you get the same query for a longer timeframe.

Combine several queries in one chart

So now we have 3 queries : queryLastWeek, queryLastMonth and queryLastYear

Which are passed as parameters to the method, where the result of the 3 queries are merged to one array by method mergeSeries (see below). This merged array (chart_data) is passed to keen.Visualisation to draw the chart you can see at the top of this post :
var request =[queryLastWeek, queryLastMonth, queryLastYear], function() {
  series_captions = ["Last week", "Last month", "Last year"];
  index_captions = ["Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"];
  chart_data = mergeSeries(,
  // draw chart
  window.chart = new Keen.Visualization(
    {result: chart_data},
       chartType: "columnchart",
       title: "Average buildtime per day of week",
       chartOptions: {
       vAxis: { title: "duration [s]" },
       hAxis: { title: "Day of week" }

You can find the full code here.

Merge data series

First this methods creates a multilevel array with i rows (one for each series, in this example i = 3 (week, month, year)) and j columns (one for each index value in the query, in this example j = 7 : 'Sun' to 'Sat').
Then the methods takes the data array, with the results of all queries as a parameter, loops over the result from each query and assigns the values to the corresponding index in a multilevel array. As a result all values corresponding to 'Monday' will be in the same place in the array.
function mergeSeries(data, index_captions, value_caption, series_captions) {
  chart_data = [];
  // create and populate data array
  for (i = 0; i < index_captions.length; i++) {
    chart_data[i]={caption: index_captions[i]};
    // populate all series
    for (j = 0; j < series_captions.length; j++) {
      chart_data[i][series_captions[j]] = 0;
  // loop over all query result sets
  for (j = 0; j < data.length; j++) {
    timeframe_result = data[j].result;
    timeframe_caption = series_captions[j];
    // copy query data into the populated array
    for (i = 0; i < timeframe_result.length; i++) {
      index = parseInt(timeframe_result[i][value_caption])
      chart_data[index][timeframe_caption] = timeframe_result[i]["result"];
  return chart_data;

Some improvements

Some ideas to make it more efficiently:
  • A special 'groupby' parameter for timestamps as part of the API, would avoid splitting a timestamp and storing all the components in the database
  • Currently, 3 almost identical queries are created to generate the results for the different timeframes. It would be more efficient to repeat the same query several times with only the timeframe changing. Still something to investigate.

Friday, September 19, 2014

How I got more relaxed by no longer commuting by car

Yesterday was car free day, at least in Belgium, and by a happy coincidence I came across an article that pointed out a correlation between mental well-being and the means of transportation when commuting to work. It turns out that not using a car, fe. going by bike, on foot or by public transport increases your mental health. The author wonders about the reason for this.

I'm not a psychologist, nor have I done scientific research to investigate this, but from my personal experience, I can think of a few reasons why not driving by car to commute is better for you.

A few years ago, I was commuting daily by car. Construction works were going on for a few months, so every morning I spent 20-30 minutes in a traffic jam (on top of the 30 minute drive it took me to get to work).
Those 20-30 minutes of waiting, driving slowly, accelerating and breaking again, more waiting, ... well, it annoyed me, and I guess a lot of other people don't like traffic jams either.

A few months later, I was told the contract of my company lease car was about to end and I would get a new one.
Then I started wondering if I really liked spending that much time in traffic jams every day, 50-60 minutes of doing nothing else but stare at the car in front of me. So I started looking for alternatives. It turned out there was a train station at walking distance from my office, and it would take me 50-60 minutes to get from home to work. No gain in travel time (and it would take me less time by car if there would be no traffic jam), but I would spend about 45 minutes in a train, not having to pay attention to the cars in front of me, not having the stress and boredom of waiting in a traffic jam. I could listen to some music, read a bit, take a nap, stare out the window enjoying the scenery passing by or having a chat with a fellow commuter.
So instead of spending about an hour getting annoyed and stressed, I could relax while the train driver got me to work and I could get some things done in the mean time.

So I declined the offer of a new lease car and decided to commute by train. I couldn't have made a better decision. From that moment on I arrived more relaxed at work and at home. Of course, commuting by train can be stressful as well : delayed or cancelled trains, crowded with noisy people. But I was lucky to have a quiet commuter train in the morning, and I could usually avoid rush-hour in the evening, so I usually had a comfortable commute, arriving at work or at home much more relaxed.

Commuting by train can be annoying as well, if you have to cope with long commutes, multiple stop-overs, delays and crowded trains on a daily basis, as I experienced a few years later on another job (but at least I could still doze off or read a bit).
But I was relieved, when I found a job closer to home that would take me 20 minutes by bike. No reading this time while commuting, but having the daily physical exercise and cruising past rows of waiting cars was enjoyable (I'm not gloating, actually, I took a route through the car free city center, so I didn't see that much cars on my way to work), but I knew that if I would go to work by car I would end up in a traffic jam and it would take me much longer to get to work.

I don't use the car that much anymore, only for longer drives, places that are hard to reach by public transport, or when transporting heavy or bulky loads. And I like it. I can't imagine losing multiple hours waiting in traffic jams every week.
Overall I'm more relaxed because I don't get annoyed waiting, can do some enjoyable things while commuting or get some physical exercise (which is also known to reduce stress levels).

A a consequence you have to make some compromises and it will take some extra planning, but it's worth it.

Wednesday, August 06, 2014

Buildtime trend v0.1 released

Everyone likes graphs, trends and stats. This is also true when building software. Did you ever want to know how the total duration of your build process evolved over time? Did you ever wonder why your build process took so much longer all of a sudden, trying to figure out what part of the build process was responsible for this?

Build time trend example (click for realtime stats)

Buildtime trend lets you measure the time of all parts of the build process and creates nice graphs showing how long each build stage takes, and more.
You can easily integrate it with your Travis CI build process, or integrate it with your build process.
The timing data of your build process is then stored, analysed and visualised, using the API, resulting in nice graphs and charts.

In a future version, Buildtime trend will be able to get timing data from Travis CI, without having to manipulate the build config file. Support for other CI environments is on the roadmap as well. And of course, a lot more stats and charts and trends.

Suggestions and comments are welcome. Follow us on Twitter or submit a feature requests to the issue tracker.

Update : An example script in release v0.1 didn't handle extracting the release tarball well, it is fixed in v0.1.2

Saturday, July 12, 2014

GetBack GPS for Android v0.3 is released

Get the latest version on F-Droid.

Arrow pointing to destination and compass indicating North

GetBack GPS for Android v0.3 is now available. It adds a realtime compass and detects your travel direction. This way the arrow pointing to your destination will still point in the right direction, even when travelling backwards.
And there are some small UI improvements.

The app is now also available in Danish, Brazilian Portuguese, Traditional Chinese and French. A big thanks to the translators and supporting services!

A complete list of features and available languages is available on the website.

Friday, July 11, 2014

Remove bars from Android screenshots

How to remove the Status Bar at the top and return/home/screens button bar on the bottom/side from Android screenshots, using commandline ImageMagick.

Portrait screenshot

Command :

convert -size 1080x1920 -extract 1080x1700+0+75 original_screenshot.png screenshot_after.png

1080x1920 is the original size of the image (portrait)
1080x1700 is the new size, +0+75 the x and y offset where the extraction starts
(sizes may differ if you have a phone with a different screensize and resolution)

Original portrait screenshot, before cropping
After cropping

Landscape screenshot

Command :

convert -size 1920x1080 -extract 1792x1005+0+75 original_screenshot.png screenshot_after.png

1920x1080 is the original size of the image (landscape)
1792x1005 is the new size, +0+75 the x and y offset where the extraction starts

(sizes may differ if you have a phone with a different screensize and resolution)


Original screenshot (landscape), before cropping

After cropping

Monday, June 23, 2014

File recovery using foremost

Quick note on using foremost to retrieve files from a harddrive with a faulty partition table :

sudo foremost -dT -i /dev/md2 -t jpeg -o /path/to/restore2/

This scans /dev/md2 hd, looking for jpeg files and puts the restored files in /path/to/restore2/
Foremost can retrieve a lot of different file types (including images, MS Office files, PDF's and zipfiles), look at the man pages for a complete list. It is also possible to define your own filters.

Modify (or copy first) the foremost.conf file and add a line  for every file type fe. :

ex1   y   4096000  \x50\x4b\x03          \x74\x78\x74
ex2   y   4096000  \x50\x4b\x03???\x44   \x74\x78\x74

where you specify :
  • file extension
  • case sensitive pattern (if ASCII codes are used in the pattern)
  • maximum size of a file in bytes
  • starting pattern
  • end pattern
The pattern can be in hex code or ASCII, use a ? as a wildcard for one character. In the second example above, a wildcard for 3 characters is added.

Now run foremost, using the custom config file :

sudo foremost -dT -i /dev/md2 -c /tmp/foremost.conf -o /path/to/restore2/

This scans /dev/md2 hd, using the rules in /tmp/foremost.conf and puts the restored files in /path/to/restore2/

Thursday, March 06, 2014

Google Summer of Code 2014 meetup at Ghent University

Wednesday evening, March 5th, 19u, about 30 students of Ghent university (Belgium) showed up for an info session about Google Summer of Code 2014. Some of them already heard about it, but for most of them it was new.

Google Summer of Code is a yearly initiative of Google, this year held for the 10th time, to introduce students in higher education to Open Source development, by offering them a stipend of 5500 USD for 12 weeks of coding on an Open Source project during the summer. It gives the students realworld experience, writing code that will be used by many users worldwide, collaborating with the teams of the Open Source organisation they will contribute to, improving their teamwork and communication skills.

After a brief introduction about the program, 4 (of the 190 selected for GSoC 2014) Open Source organisations presented themselves.

Sage is an Open Source collection of mathematical tools, ranging from solving differential equations to plotting functions. Some of their projects this year focus on improving the user interface, but there is also a need for adding mathematical tools, fe. for knot theory.

GNU Octave is also related to mathematics, but is mostly used for simulations and numerical computations. Their ideas pages contains several projects, one of them is adding a library for doing finite element simulations.
Both these projects are interesting for students with an affinity for maths and programming, but also for students wanting to improve the GUI or add other supporting tools.

Next up was the phpMyAdmin project, a webbased user interface for MySQL databases. Having participated to GSoC for many years, the list of projects to work on range from improving the AJAX error reporting tool that was added in GSoC 2013, to also report PHP errors, over a tool to normalize and check the structure of a database to interface improvements. If you are proficient in PHP or JavaScript and interested in databases, one of these projects might be for you.

MuseScore was the last project to present itself. It is a free and Open Source music annotation software, but there is also a version for tablets and an online music sheet library to share your compositions. Their ideas for this year's GSoC range from support for visually impaired users to tools for adding lyrics to a music score. Interested students should be proficient with C++ and Qt.

All projects mentioned that communicating with the development team and the mentors, using either the IRC channel or the mailing list is very important and the best way to get into contact with them.

At the reception afterwards, the mentors of the Open Source projects talked to the students answering their questions about GSoC, on how to apply, how to write a good proposal, and many more question that are also answered in the GSoC FAQ.

Thanks to the Open Source organisations (SAGE, GNU Octave, phpMyAdmin and MuseScore) for their talks and sharing their experience, and to the student associations (VTK, Zeus WPI, Ceneka) and UGent for their help in organising the event.

Sunday, February 09, 2014

An impression of FOSDEM 2014

Sunday evening, 19:30, leaving Brussels by train, heading home.
48 hours earlier I arrived here for FOSDEM : another amazing weekend of Open Source, sharing knowledge, meeting people, getting new ideas, geekiness, and well, also a bit of beer.

Opening talk of FOSDEM 2014
Opening talk of FOSDEM 2014
With more than 500 talks and presentations spread over 2 days in 22 rooms held at the ULB in Brussels, with 5-10 thousand visitors, it is impossible to attend and see everything. Luckily, thanks to an amazing team of origanisers and volunteers everything went smoothly and all talks are recorded and the videos will be available soon.

Some highlights, some inspiration and some ideas to work on :

Tuesday, January 21, 2014

going to FOSDEM 2014

It's FOSDEM 2014 again in two weeks time. And I'm looking forward to it : attending lectures, meeting people, talking about Open Source, ...

After having a quick look at the schedule with more than 500 (!) lectures over a period of a weekend, I made a selection of interesting talks, because obviously, it is impossible to attend them all :

Wednesday, January 01, 2014

This was 2013 and plans for 2014!

It's this time of the year again, reviewing the cool things that happened, learning from the stuff that didn't go as planned and adjusting the goals of last year accordingly.

2013 was busy, here are some achievements :
  • visited FOSDEM 2013 and did a talk on phpMyAdmin
  • visited Newline 0x03
  • co-organised a GSoC info session at UGent
  • first Wagner experience : Parsifal, long but musically beautiful
  • phpMyAdmin is accepted as a mentoring organisation for GSoC 2013 and has 6 students. I'll be a co-mentor and co-org admin this year
  • got promoted to a higher pay grade
  • did a citytrip to Berlin, visited LinuxTag 2013 and moderated a few sessions. I've also got some ideas for my Android app while attending the Android session.
  • finished a few yellow routes at Bleau.
  • did some outdoor rock climbing in Fontainebleau.
  • did a presentation introducing Latex to my colleagues.
  • visited Debconf13 held in Vaumarcus, Switzerland, were phpMyAdmin had a team meeting and did a phpMyAdmin tutorial.
  • went to GSoC Mentor Summit 2013 at Google Headquarters, Mountain View, California, organised a Key Signing Party and  a session on international acceptance of Open Source licenses.
  • did some climbing in Yosemite National Park
  • released GetBack GPS, my first Android app
  • prepared sabayon for the first time.
  • made more than 2000 contributions (commits and created issues) in one year on github.
  • travelled to the Philipines.
Check out my plans for 2014 :
  • Android app development : After releasing an Android app this year for the first time, I plan to explore this path a bit more. Improving the GetBack GPS app a bit more, I still have some ideas as you can see on the roadmap, getting some more experience on automated testing, test driven development (TDD), Java and object oriented programming along the way.
    Another Android project for this year will use Bluetooth communication, in a small project I will work on with a friend.
  • Bouldering/climbing : Last year I started climbing in the gym and I did some outdoor climbing as well. I utterly enjoy this, so I definitely want to spend some more time on it. Hopefully with some outdoor climbing trips during the upcoming year.
  • Visit FOSDEM and LinuxTag.
I'm looking forward to an exciting year.
I wish you and your family the same, with good health and successful projects.