Thursday, October 09, 2014

Custom multiseries trend using Keen.io API

The initial goal was to create a trend of event data related to the time of day or day of week when the event occured. Later on, it seemed like a good idea to display different timeframes on the same trend.

The end result shows a trend, calculating an average value of a metric (buildtime duration, in this example) for all events that occured in the same time interval (day of week, in this example), for different timeframes (last week, month and year, in this example), which are displayed as different series in the same chart, to be able to compare them and visually notice an evolution or an anomally.

This trend is part of the Buildtime Trend project, you can see the code in action here.

Read on to see how it is done.
The Keen.io service and API is used to store, analyse and visualise the event data. I'd like to refer to the Keen.io tutorials on how to create a query and generate a chart.

Generate and group by time intervals

 

First of all, the event data has a timestamp, so in a simplified example, an event would look like this :

  { id: "1234abcd", duration: "97", timestamp: "2014-10-09T18:32:14Z"}

But to group events on time intervals, like day of week, or hour (time of day), the timestamp  has to be split into its components (thanks to Ryan Spraetz of Keen.io for the suggested workaround), for example :

  {id: "1234abcd",
    duration: "97",
    timestamp: {
      isotimestamp: "2014-10-09T18:32:14Z",
      day_of_week: 4,
      hour_24: 18,
      hour_12 : 6,
      hour_AMPM : PM,
      ... 
    }
  }


Look here for the code to split the timestamp (in Python) and a full example of a split timestamp.

A query to group events by day of week, calculating an average value of duration, for all events of the last week, would look like this :

var queryLastWeek = new Keen.Query("average", {
  eventCollection: "builds",
  timeframe: "last_week",
  targetProperty: "duration",
  groupBy: "timestamp.day_of_week"
}); 
 
Using an example from the Keen.io tutorial, you could easily create a chart with one series of data.
If timeframe is changed to 'last_month' or 'last_year', you get the same query for a longer timeframe.

Combine several queries in one chart


So now we have 3 queries : queryLastWeek, queryLastMonth and queryLastYear

Which are passed as parameters to the Keen.io client.run method, where the result of the 3 queries are merged to one array by method mergeSeries (see below). This merged array (chart_data) is passed to keen.Visualisation to draw the chart you can see at the top of this post :
var request = client.run([queryLastWeek, queryLastMonth, queryLastYear], function() {
  series_captions = ["Last week", "Last month", "Last year"];
  index_captions = ["Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"];
  chart_data = mergeSeries(
    this.data,
    index_captions,
    "timestamp.day_of_week",
    series_captions
  );
  // draw chart
  window.chart = new Keen.Visualization(
    {result: chart_data},
    document.getElementById("chart_avg_buildtime_weekday"),
    {
       chartType: "columnchart",
       title: "Average buildtime per day of week",
       chartOptions: {
       vAxis: { title: "duration [s]" },
       hAxis: { title: "Day of week" }
    }
  });
});

You can find the full code here.

Merge data series

First this methods creates a multilevel array with i rows (one for each series, in this example i = 3 (week, month, year)) and j columns (one for each index value in the query, in this example j = 7 : 'Sun' to 'Sat').
Then the methods takes the Keen.io data array, with the results of all queries as a parameter, loops over the result from each query and assigns the values to the corresponding index in a multilevel array. As a result all values corresponding to 'Monday' will be in the same place in the array.
function mergeSeries(data, index_captions, value_caption, series_captions) {
  chart_data = [];
  // create and populate data array
  for (i = 0; i < index_captions.length; i++) {
    chart_data[i]={caption: index_captions[i]};
    // populate all series
    for (j = 0; j < series_captions.length; j++) {
      chart_data[i][series_captions[j]] = 0;
    }
  }
  // loop over all query result sets
  for (j = 0; j < data.length; j++) {
    timeframe_result = data[j].result;
    timeframe_caption = series_captions[j];
    // copy query data into the populated array
    for (i = 0; i < timeframe_result.length; i++) {
      index = parseInt(timeframe_result[i][value_caption])
      chart_data[index][timeframe_caption] = timeframe_result[i]["result"];
    }
  }
  return chart_data;
}

Some improvements

Some ideas to make it more efficiently:
  • A special 'groupby' parameter for timestamps as part of the Keen.io API, would avoid splitting a timestamp and storing all the components in the database
  • Currently, 3 almost identical queries are created to generate the results for the different timeframes. It would be more efficient to repeat the same query several times with only the timeframe changing. Still something to investigate.

Friday, September 19, 2014

How I got more relaxed by no longer commuting by car

Yesterday was car free day, at least in Belgium, and by a happy coincidence I came across an article that pointed out a correlation between mental well-being and the means of transportation when commuting to work. It turns out that not using a car, fe. going by bike, on foot or by public transport increases your mental health. The author wonders about the reason for this.

I'm not a psychologist, nor have I done scientific research to investigate this, but from my personal experience, I can think of a few reasons why not driving by car to commute is better for you.

A few years ago, I was commuting daily by car. Construction works were going on for a few months, so every morning I spent 20-30 minutes in a traffic jam (on top of the 30 minute drive it took me to get to work).
Those 20-30 minutes of waiting, driving slowly, accelerating and breaking again, more waiting, ... well, it annoyed me, and I guess a lot of other people don't like traffic jams either.

A few months later, I was told the contract of my company lease car was about to end and I would get a new one.
Then I started wondering if I really liked spending that much time in traffic jams every day, 50-60 minutes of doing nothing else but stare at the car in front of me. So I started looking for alternatives. It turned out there was a train station at walking distance from my office, and it would take me 50-60 minutes to get from home to work. No gain in travel time (and it would take me less time by car if there would be no traffic jam), but I would spend about 45 minutes in a train, not having to pay attention to the cars in front of me, not having the stress and boredom of waiting in a traffic jam. I could listen to some music, read a bit, take a nap, stare out the window enjoying the scenery passing by or having a chat with a fellow commuter.
So instead of spending about an hour getting annoyed and stressed, I could relax while the train driver got me to work and I could get some things done in the mean time.

So I declined the offer of a new lease car and decided to commute by train. I couldn't have made a better decision. From that moment on I arrived more relaxed at work and at home. Of course, commuting by train can be stressful as well : delayed or cancelled trains, crowded with noisy people. But I was lucky to have a quiet commuter train in the morning, and I could usually avoid rush-hour in the evening, so I usually had a comfortable commute, arriving at work or at home much more relaxed.

Commuting by train can be annoying as well, if you have to cope with long commutes, multiple stop-overs, delays and crowded trains on a daily basis, as I experienced a few years later on another job (but at least I could still doze off or read a bit).
But I was relieved, when I found a job closer to home that would take me 20 minutes by bike. No reading this time while commuting, but having the daily physical exercise and cruising past rows of waiting cars was enjoyable (I'm not gloating, actually, I took a route through the car free city center, so I didn't see that much cars on my way to work), but I knew that if I would go to work by car I would end up in a traffic jam and it would take me much longer to get to work.

I don't use the car that much anymore, only for longer drives, places that are hard to reach by public transport, or when transporting heavy or bulky loads. And I like it. I can't imagine losing multiple hours waiting in traffic jams every week.
Overall I'm more relaxed because I don't get annoyed waiting, can do some enjoyable things while commuting or get some physical exercise (which is also known to reduce stress levels).

A a consequence you have to make some compromises and it will take some extra planning, but it's worth it.

Wednesday, August 06, 2014

Buildtime trend v0.1 released

Everyone likes graphs, trends and stats. This is also true when building software. Did you ever want to know how the total duration of your build process evolved over time? Did you ever wonder why your build process took so much longer all of a sudden, trying to figure out what part of the build process was responsible for this?

Build time trend example (click for realtime stats)

Buildtime trend lets you measure the time of all parts of the build process and creates nice graphs showing how long each build stage takes, and more.
You can easily integrate it with your Travis CI build process, or integrate it with your build process.
The timing data of your build process is then stored, analysed and visualised, using the Keen.io API, resulting in nice graphs and charts.

In a future version, Buildtime trend will be able to get timing data from Travis CI, without having to manipulate the build config file. Support for other CI environments is on the roadmap as well. And of course, a lot more stats and charts and trends.

Suggestions and comments are welcome. Follow us on Twitter or submit a feature requests to the issue tracker.

Update : An example script in release v0.1 didn't handle extracting the release tarball well, it is fixed in v0.1.2

Saturday, July 12, 2014

GetBack GPS for Android v0.3 is released

Get the latest version on F-Droid.

Arrow pointing to destination and compass indicating North

GetBack GPS for Android v0.3 is now available. It adds a realtime compass and detects your travel direction. This way the arrow pointing to your destination will still point in the right direction, even when travelling backwards.
And there are some small UI improvements.

The app is now also available in Danish, Brazilian Portuguese, Traditional Chinese and French. A big thanks to the translators and supporting services!

A complete list of features and available languages is available on the website.


Friday, July 11, 2014

Remove bars from Android screenshots

How to remove the Status Bar at the top and return/home/screens button bar on the bottom/side from Android screenshots, using commandline ImageMagick.

Portrait screenshot

Command :

convert -size 1080x1920 -extract 1080x1700+0+75 original_screenshot.png screenshot_after.png

1080x1920 is the original size of the image (portrait)
1080x1700 is the new size, +0+75 the x and y offset where the extraction starts
(sizes may differ if you have a phone with a different screensize and resolution)


Result:
Original portrait screenshot, before cropping
After cropping

Landscape screenshot

Command :

convert -size 1920x1080 -extract 1792x1005+0+75 original_screenshot.png screenshot_after.png

1920x1080 is the original size of the image (landscape)
1792x1005 is the new size, +0+75 the x and y offset where the extraction starts

(sizes may differ if you have a phone with a different screensize and resolution)


Result:

Original screenshot (landscape), before cropping

After cropping

Monday, June 23, 2014

File recovery using foremost

Quick note on using foremost to retrieve files from a harddrive with a faulty partition table :

sudo foremost -dT -i /dev/md2 -t jpeg -o /path/to/restore2/

This scans /dev/md2 hd, looking for jpeg files and puts the restored files in /path/to/restore2/
Foremost can retrieve a lot of different file types (including images, MS Office files, PDF's and zipfiles), look at the man pages for a complete list. It is also possible to define your own filters.

Modify (or copy first) the foremost.conf file and add a line  for every file type fe. :

ex1   y   4096000  \x50\x4b\x03          \x74\x78\x74
ex2   y   4096000  \x50\x4b\x03???\x44   \x74\x78\x74

where you specify :
  • file extension
  • case sensitive pattern (if ASCII codes are used in the pattern)
  • maximum size of a file in bytes
  • starting pattern
  • end pattern
The pattern can be in hex code or ASCII, use a ? as a wildcard for one character. In the second example above, a wildcard for 3 characters is added.

Now run foremost, using the custom config file :


sudo foremost -dT -i /dev/md2 -c /tmp/foremost.conf -o /path/to/restore2/

This scans /dev/md2 hd, using the rules in /tmp/foremost.conf and puts the restored files in /path/to/restore2/

Thursday, March 06, 2014

Google Summer of Code 2014 meetup at Ghent University

Wednesday evening, March 5th, 19u, about 30 students of Ghent university (Belgium) showed up for an info session about Google Summer of Code 2014. Some of them already heard about it, but for most of them it was new.

Google Summer of Code is a yearly initiative of Google, this year held for the 10th time, to introduce students in higher education to Open Source development, by offering them a stipend of 5500 USD for 12 weeks of coding on an Open Source project during the summer. It gives the students realworld experience, writing code that will be used by many users worldwide, collaborating with the teams of the Open Source organisation they will contribute to, improving their teamwork and communication skills.

After a brief introduction about the program, 4 (of the 190 selected for GSoC 2014) Open Source organisations presented themselves.

Sage is an Open Source collection of mathematical tools, ranging from solving differential equations to plotting functions. Some of their projects this year focus on improving the user interface, but there is also a need for adding mathematical tools, fe. for knot theory.

GNU Octave is also related to mathematics, but is mostly used for simulations and numerical computations. Their ideas pages contains several projects, one of them is adding a library for doing finite element simulations.
Both these projects are interesting for students with an affinity for maths and programming, but also for students wanting to improve the GUI or add other supporting tools.

Next up was the phpMyAdmin project, a webbased user interface for MySQL databases. Having participated to GSoC for many years, the list of projects to work on range from improving the AJAX error reporting tool that was added in GSoC 2013, to also report PHP errors, over a tool to normalize and check the structure of a database to interface improvements. If you are proficient in PHP or JavaScript and interested in databases, one of these projects might be for you.

MuseScore was the last project to present itself. It is a free and Open Source music annotation software, but there is also a version for tablets and an online music sheet library to share your compositions. Their ideas for this year's GSoC range from support for visually impaired users to tools for adding lyrics to a music score. Interested students should be proficient with C++ and Qt.

All projects mentioned that communicating with the development team and the mentors, using either the IRC channel or the mailing list is very important and the best way to get into contact with them.

At the reception afterwards, the mentors of the Open Source projects talked to the students answering their questions about GSoC, on how to apply, how to write a good proposal, and many more question that are also answered in the GSoC FAQ.

Thanks to the Open Source organisations (SAGE, GNU Octave, phpMyAdmin and MuseScore) for their talks and sharing their experience, and to the student associations (VTK, Zeus WPI, Ceneka) and UGent for their help in organising the event.