Tuesday, November 24, 2015

Buildtime Trend v0.3 is out

Visualise what's trending in your build process

Buildtime Trend Logo
I'm happy to inform you that Buildtime Trend v0.3 is released. Those of you using Buildtime Trend as a Service had a running preview of all the new features :

  • introduction of a worker queue to make processing build job logs more scalable
  • dashboard chart data can be filtered on build properties 
  • several new dashboard charts and layout improvements
  • enable Keen.io query caching to improve chart loading speed
  • the dashboard takes url parameters to set the refresh rate and the default settings for time interval and filter properties
  • a statistics dashboard is added to monitor usage of Buildtime Trend as a Service
Dashboard example
Dashboard example
More new features, improvements and changes can be found in the release notes and the Changelog files of the project components :
You can check out the dashboards of the projects that are already using Buildtime Trend as as Service :

Do you want to enable Buildtime Trend for your the build process of your project on Travis CI? It is easy to set up.

Buildtime Trend as a Service is currently available for free for Open Source projects, thanks to the kind people of Keen.io.


If you like Buildtime Trend, you are welcome to support the project, by making a donation. Donations will help pay for the hosting and support further development.

You can help make the project better : we welcome any kind of contributions.

Tuesday, April 21, 2015

Gorges du Tarn 2015

It was an amazing week, climbing in Gorges du Tarn with Bleau Climbing team during the second week of the Easter holiday.
Beautiful weather, nice people, good atmosphere, a lot of climbing, some personal bests and climbing improvements on both a physical and mental level.

Some impressions :

  • Flashed my first 6a lead climbing on rock : Nique le lière in the Figues au Cul sector.
  • Cooked spaghetti for approx. 50 people with a great team (Peter, Lieven, Hanne, Anneke, Jasper and Tim)
  • Finished a lot of 5s on sight or flashed them.
  • Discovered that my tent isn't waterproof anymore. Luckily there was a bridge that kept part of the camping site dry, so I moved my tent there during the rainy night. Yes, I slept under a bridge. ;)
  • Almost finished a 6a+ on Noir Désir and started working in a very exciting 6c on La Muse, two projects for next time, so I know what I'll be training for the next few months. :)
Great trip, looking forward to the next one!

Sunday, February 22, 2015

Buildtime Trend v0.2 released!

Visualise what's trending in your build process

Buildtime Trend Logo
What started as a few scripts to gain some insight in the duration of stages in a build process, has evolved into project Buildtime Trend, that generates and gathers timing data of build processes. The aggregated data is used to create charts to visualise trends of a build process.

The major new futures are the support for parsing Travis CI build log files to retrieve timing data and the introduction of the project as a service that gathers Travis CI generated timing data, hosts a dashboard with different charts and offers shield badges with different metrics.

Try it out!

The hosted service supports Open Source projects (public on GitHub) running their builds on Travis CI. Thanks to the kind people of Keen.io hosting the aggregated data, the hosted service is currently available for free for Open Source projects.
Get started! It's easy to set up in a few steps.

A bit more about Buildtime Trend

Dashboard example
Dashboard example
Buildtime Trend is an Open Source project that generates and gathers timing data of build processes. The aggregated data is used to create charts to visualise trends of the build process.
These trends can help you gain insight in your build process : which stages take most time? Which stages are stable or have a fluctuating duration? Is there a decrease or increase in average build duration over time?
With these insights you can improve the stability of your build process and make it more efficient.

The generation of timing data is done with either a client or using Buildtime Trend as a Service.
The Python based client generates custom timing tags for any shell based build process and can easily be integrated. A script processes the generated timing tags when the build is finished, and stores the results.
Buildtime Trend as a Service gets timing and build related data by parsing the logfiles of a buildprocess. Currently, Travis CI is supported. Simply trigger the service at the end of a Travis CI build and the parsing, aggregating and storing of the data is done automatically.

The aggregated build data is used to generate a dashboard with charts powered by the Keen.io API and data store.

Check out the website for more information about the project, follow us on Twitter, or subscribe to the community mailing list.

Tuesday, January 13, 2015

What I learned from 365 days of contributing to Open Source projects

Today I reached a 365 day commit streak on GitHub, with over 2300 commits to Open Source projects in that period. In this post I'd like to share my experiences during this past year and the lessons I learned.

Github contribution overview

It started one year ago, on January 14th, 2014, the day I returned from a two week trip to Malaysia. I didn't take a laptop or smartphone on that trip, in order to be 'disconnected' from PC, internet and E-mail for some time. I've taken a habit to have a 'unplugged' holiday about once a year.
I had a streak of over 100 days running before I left on holiday, and had reached 2000+ commits during that year, so I was eager to start committing again, to keep the running total of commits close to 2000 and to start building a commit streak again.
I don't remember if I had a clear goal at that time of the total consecutive days of committing to Open Source projects I aspired to reach. I guess at first I wanted to match the previous record and see how much further I could get.

At the end of June, I got inspired by Josh (@dzello) from Keen.io who had pledged to commit to Open Source projects for 365 days in a row. I had an extensive streak going on at that time, so I decided to try and reach a 365 day streak as well.

Until then it had been fairly easy to keep the streak going. Doing at least one commit every day is not that hard, and I usually did more than one. I was working on my Android app and I started working on what I called 'my side project' at first. In the last few months my focus has shifted to that 'side project', making it my main project basically, but that's a different story.
So I had plenty to do and I was well motivated to work on my projects regularly, so it was easy to keep committing daily.

Until summer I didn't do any long trips, so I was either at home at some point during the day or had my laptop with me (when going to Berlin for LinuxTag, for example) so finding a few minutes every day to do at least one commit wasn't a big challenge. Although sometimes, it required some planning.
If I had a social, cultural or sports activity planned in the evening after work, I planned for an 'easy' commit on those days, usually cleaning up some code, fixing coding style, writing a small test, improving documentation or doing a few translations. I kept the bigger work, implementing a new feature, figuring out an API or framework, or doing some bigger refactoring for days when I had more time available.

Then summer arrived and I planned to go on a climbing trip for a week. I was in doubt if I would have time and opportunity to keep committing during the trip. But in the end I decided to take my laptop and give it a try. I worked on bash script to crop and scale Android screenshots that week, something I could easily develop and test without access to internet. On some days I barely managed to contribute, finishing the commit only a few minutes before the end of the day, but in the end I managed to keep the streak going.

With this hurdle taken I imagined reaching the 365 day goal was achievable. I didn't know back then I was to go on a few long weekends to Fontainebleau during the Fall, but again I took my laptop with me on the trip and found time to contribute some code.

In the end of October I went to California for the Google Summer of Code 10 year Reunion and I had the opportunity to meet Josh, and Justin, also from Keen.io. Josh  published a blogpost by that time explaining why he ended his commit streak at Burning Man, and why he wasn't planning on starting the streak again.

I read his post, and it got me thinking. Is this continuous streak a good thing? Sure, it was a motivation on its own, you want to keep going because you don't want to break the streak, you'd have to start all over again for a long time to reach the same number of consecutive days.
Some days you don't have time, or you've had a rough day and don't feel like turning on your PC and doing the required commit for the day, but would rather do something else, forgetting about that ongoing streak.

But I decided to go for it and finish the pledge to reach 365 days of committing to Open Source projects. I found out that my motivation wasn't only in keeping the streak going, but mostly in making progress with my projects.

Now that I've reached the 365 day goal I've come to some conclusions:

  • Setting a goal helps to keep you going, but it doesn't necessarily has to be a streak of consecutive days commiting code. Josh mentioned this in his blog post as well, there are other metrics or goals you can aspire. Choose one that works for you and that is realistic.
  • A minimum of one commit per day worked for me, and usually I reached more (6,4 per day on average, with a maximum of 27 at some point). But it would be harder for me to set goal of, for example, 30 commits per week. It would put pressure on me every week to reach that goal and it would be counterproductive, for me at least.
  • Now that I've reached this 365 day goal I will keep the streak going as long as I'm comfortable with. As I mentioned, the one commit a day goal works for me as long as it doesn't interfere with my other commitments : I have a day job and regularly enjoy social, cultural and sports activities.
  • Next time I'm going away for a few days, being it a holiday or a climbing trip I will not take my laptop with me. It will break my streak, but I'm OK with that. Disconnecting from everything for a few days every once in a while is more important to me. Afterwards I'll probably pick it up again and start a new streak, but it doesn't matter how long it will be. I'm glad I reached the 365 days goal, but I don't necessary feel the need to repeat it.
  • As mentioned earlier, I found that my motivation lies in working on my projects, in seeing progress as I continue adding features and improving them.
  • I noticed a few similarities in motivation for doing sport (and other) activities. When you're running, for example, it helps to set a goal (a total distance or an average pace to reach) or a metric (once or twice a week, no matter how long the distance or time spent). Choose one that works for you to keep you motivated. But also allow yourself to take break if you need one and pick up later again. In the end you should enjoy the activity, setting a goal is just a way to keep you going, but the fun of doing the activity should drive you.
Many thanks to Josh to inspiring me to reach these conclusions, to all who supported me during the past year and good luck to everyone aspiring a goal, being it a commit streak or something else.

Kudos to those who still have a very long commit streak going!

Thursday, January 01, 2015

Happy 2015!

I wish you all an exciting 2015 in good health, with a lot of fun and big achievements in your projects! Good luck and joy to all my friends who are expecting a child this year!

Reviewing 2014

Well, 2014 was quite busy. Looking at my plans for 2014 a lot more happened than I expected : I learned Python when I started a new project, visited 7 capital cities and 3 continents, did a few climbing trips and got to visit San Francisco and Yosemite again :

  • celebrated New Year in Cebu, Philipines
  • visited Malaysia (Kuala Lumpur, Melakka, ...), Singapore and Hong Kong
  • bought a Google Nexus 5
  • attended FOSDEM 2014
  • resigned as a phpMyAdmin team member, creating some more time for other projects
  • finished a first blue route at Bleau
  • coorganised an info session about Google Summer of Code at Ghent University
  • started working on a tool to generate a trend graph of a build process, my first Python experience
  • attended Newline 0x04 and talked about reducing iptables configuration complexity
  • attended LinuxTag 2014 and talked about 'Reducing iptables configuration complexity' (presentation slides)
  • Finished a 10km race (Gentse stadsloop) in less than 55 minutes
  • Visited Maker Faire Paris : A lot of interesting stuff, mainly 3D printers, but some with a twist : pancake maker, an industrial welding robot mounted with an extruder, lots of hinges, bearings and other stuff printed ready to be used : no need to assemble
  • First time to visit Paris, I finally got to see the Eifel tower, Avenue des Champs Elysées, Arc de triomphe, Le Louvre (outside), Notre Dame, La Seine, Musée d'Orsay, Musée Rodin, Montmartre and Sacré Coeur
  • Released version 0.3 of Get Back GPS
  • Participated in a climbing training in L'Argentière-La Bessée and received a certificate for lead-climbing  and multi-pitch climbing (KVB3), had a lot of fun : climbing a lot and sending several 5b's and 5c's lead-climbing, some of them on sight.
  • Visited Amsterdam, the 6th capital city I went to this year
  • performed a lead-climber's fall at Klimax II
  • did a 55km cycling trip around Ghent
  • Donated blood for the first time
  • Released the first version of Buildtime Trend, a tool to create visual trends of  a software build process, written in Python and JavaScript
  • GetBack GPS had more than 10 contributors during 1 month, most of them are translating the app
  • Ran 16km/10mi for the first time
  • Spent a weekend in Fontainebleau with Maxime, Seba and Wolf, opening some routes.
  • Went to San Francisco to attend the GSoC 10th Anniversary Reunion and spent a few days in San Francisco and did a trip to Yosemite with Madhura Jayaratne. Shared a few beers with Justin and Josh of Keen.io, talking about Open Source commit streaks and Buildtime Trend.
  • Released GetBack GPS 0.4, introducing 7 more languages.
  • Spent a long weekend climbing in Fontainebleau with Peter, Dorinne, Cécile, Stef, Aline, Senne, Nik, Chris, Andreas, Luke and Barbara, climbing several yellow and blue routes.
  • and a second blood donation
  • finished a red climbing route
  • traveled to Lisbon.
Things to do in 2015:
  • Continue working on Buildtime Trend : offer it as a service (SaaS), add more stats, support more CI environments, ... 
  • Climb as much as possible, both indoor and outdoor, with trips to France during the Easter and summer holidays, and weekend trips to Fontainebleau or one of the climbing areas in the south of Belgium.
  • Visit Open Source and related conferences : FOSDEM, LinuxTag, EuroPython, DebConf and KeenCon.
I'm looking forward to where 2015 will bring me. Looking back at 2014 it is bound to be an exciting year again.

Thursday, October 09, 2014

Custom multiseries trend using Keen.io API

The initial goal was to create a trend of event data related to the time of day or day of week when the event occured. Later on, it seemed like a good idea to display different timeframes on the same trend.

The end result shows a trend, calculating an average value of a metric (buildtime duration, in this example) for all events that occured in the same time interval (day of week, in this example), for different timeframes (last week, month and year, in this example), which are displayed as different series in the same chart, to be able to compare them and visually notice an evolution or an anomally.

This trend is part of the Buildtime Trend project, you can see the code in action here.

Read on to see how it is done.
The Keen.io service and API is used to store, analyse and visualise the event data. I'd like to refer to the Keen.io tutorials on how to create a query and generate a chart.

Generate and group by time intervals


First of all, the event data has a timestamp, so in a simplified example, an event would look like this :

  { id: "1234abcd", duration: "97", timestamp: "2014-10-09T18:32:14Z"}

But to group events on time intervals, like day of week, or hour (time of day), the timestamp  has to be split into its components (thanks to Ryan Spraetz of Keen.io for the suggested workaround), for example :

  {id: "1234abcd",
    duration: "97",
    timestamp: {
      isotimestamp: "2014-10-09T18:32:14Z",
      day_of_week: 4,
      hour_24: 18,
      hour_12 : 6,
      hour_AMPM : PM,

Look here for the code to split the timestamp (in Python) and a full example of a split timestamp.

A query to group events by day of week, calculating an average value of duration, for all events of the last week, would look like this :

var queryLastWeek = new Keen.Query("average", {
  eventCollection: "builds",
  timeframe: "last_week",
  targetProperty: "duration",
  groupBy: "timestamp.day_of_week"
Using an example from the Keen.io tutorial, you could easily create a chart with one series of data.
If timeframe is changed to 'last_month' or 'last_year', you get the same query for a longer timeframe.

Combine several queries in one chart

So now we have 3 queries : queryLastWeek, queryLastMonth and queryLastYear

Which are passed as parameters to the Keen.io client.run method, where the result of the 3 queries are merged to one array by method mergeSeries (see below). This merged array (chart_data) is passed to keen.Visualisation to draw the chart you can see at the top of this post :
var request = client.run([queryLastWeek, queryLastMonth, queryLastYear], function() {
  series_captions = ["Last week", "Last month", "Last year"];
  index_captions = ["Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"];
  chart_data = mergeSeries(
  // draw chart
  window.chart = new Keen.Visualization(
    {result: chart_data},
       chartType: "columnchart",
       title: "Average buildtime per day of week",
       chartOptions: {
       vAxis: { title: "duration [s]" },
       hAxis: { title: "Day of week" }

You can find the full code here.

Merge data series

First this methods creates a multilevel array with i rows (one for each series, in this example i = 3 (week, month, year)) and j columns (one for each index value in the query, in this example j = 7 : 'Sun' to 'Sat').
Then the methods takes the Keen.io data array, with the results of all queries as a parameter, loops over the result from each query and assigns the values to the corresponding index in a multilevel array. As a result all values corresponding to 'Monday' will be in the same place in the array.
function mergeSeries(data, index_captions, value_caption, series_captions) {
  chart_data = [];
  // create and populate data array
  for (i = 0; i < index_captions.length; i++) {
    chart_data[i]={caption: index_captions[i]};
    // populate all series
    for (j = 0; j < series_captions.length; j++) {
      chart_data[i][series_captions[j]] = 0;
  // loop over all query result sets
  for (j = 0; j < data.length; j++) {
    timeframe_result = data[j].result;
    timeframe_caption = series_captions[j];
    // copy query data into the populated array
    for (i = 0; i < timeframe_result.length; i++) {
      index = parseInt(timeframe_result[i][value_caption])
      chart_data[index][timeframe_caption] = timeframe_result[i]["result"];
  return chart_data;

Some improvements

Some ideas to make it more efficiently:
  • A special 'groupby' parameter for timestamps as part of the Keen.io API, would avoid splitting a timestamp and storing all the components in the database
  • Currently, 3 almost identical queries are created to generate the results for the different timeframes. It would be more efficient to repeat the same query several times with only the timeframe changing. Still something to investigate.

Friday, September 19, 2014

How I got more relaxed by no longer commuting by car

Yesterday was car free day, at least in Belgium, and by a happy coincidence I came across an article that pointed out a correlation between mental well-being and the means of transportation when commuting to work. It turns out that not using a car, fe. going by bike, on foot or by public transport increases your mental health. The author wonders about the reason for this.

I'm not a psychologist, nor have I done scientific research to investigate this, but from my personal experience, I can think of a few reasons why not driving by car to commute is better for you.

A few years ago, I was commuting daily by car. Construction works were going on for a few months, so every morning I spent 20-30 minutes in a traffic jam (on top of the 30 minute drive it took me to get to work).
Those 20-30 minutes of waiting, driving slowly, accelerating and breaking again, more waiting, ... well, it annoyed me, and I guess a lot of other people don't like traffic jams either.

A few months later, I was told the contract of my company lease car was about to end and I would get a new one.
Then I started wondering if I really liked spending that much time in traffic jams every day, 50-60 minutes of doing nothing else but stare at the car in front of me. So I started looking for alternatives. It turned out there was a train station at walking distance from my office, and it would take me 50-60 minutes to get from home to work. No gain in travel time (and it would take me less time by car if there would be no traffic jam), but I would spend about 45 minutes in a train, not having to pay attention to the cars in front of me, not having the stress and boredom of waiting in a traffic jam. I could listen to some music, read a bit, take a nap, stare out the window enjoying the scenery passing by or having a chat with a fellow commuter.
So instead of spending about an hour getting annoyed and stressed, I could relax while the train driver got me to work and I could get some things done in the mean time.

So I declined the offer of a new lease car and decided to commute by train. I couldn't have made a better decision. From that moment on I arrived more relaxed at work and at home. Of course, commuting by train can be stressful as well : delayed or cancelled trains, crowded with noisy people. But I was lucky to have a quiet commuter train in the morning, and I could usually avoid rush-hour in the evening, so I usually had a comfortable commute, arriving at work or at home much more relaxed.

Commuting by train can be annoying as well, if you have to cope with long commutes, multiple stop-overs, delays and crowded trains on a daily basis, as I experienced a few years later on another job (but at least I could still doze off or read a bit).
But I was relieved, when I found a job closer to home that would take me 20 minutes by bike. No reading this time while commuting, but having the daily physical exercise and cruising past rows of waiting cars was enjoyable (I'm not gloating, actually, I took a route through the car free city center, so I didn't see that much cars on my way to work), but I knew that if I would go to work by car I would end up in a traffic jam and it would take me much longer to get to work.

I don't use the car that much anymore, only for longer drives, places that are hard to reach by public transport, or when transporting heavy or bulky loads. And I like it. I can't imagine losing multiple hours waiting in traffic jams every week.
Overall I'm more relaxed because I don't get annoyed waiting, can do some enjoyable things while commuting or get some physical exercise (which is also known to reduce stress levels).

A a consequence you have to make some compromises and it will take some extra planning, but it's worth it.