What is a MapKnitter

Mapknitter is a free and open-source software created and run by Public Lab. It lets people upload their own aerial images, position them in a web interface over some existing map data, share it, and export for print. It includes a bunch of sub-components - which contains a JavaScript UI- but the core application is written in Rails.

The challenge has been accepted!

My GSoC project was about upgrading Mapknitter project from Rails 3.2 to Rails 6.0. Initially, the project was running on Rails 3.2. So, to get a smoother upgrade, some steps should be taken before.

I knew that this would involve a lot of work and challenges to overcome, but I got the chance to work with some stuff I've never imagined I would.

The upgrade to Rails 4.2.11

This initial upgrade was not only about the basics of an update of this kind. It involved a lot of preparation before it, this means writing a few more tests and refactoring the CI behavior.

Kaustubh Nair took charge of the testing part, and I took charge of the CI part, this has given me a chance to almost entirely change the way Travis CI works inside Mapknitter.

Since opening a PR to main branch was taking a little longer than expected, we have adopted a new strategy in order to work faster. Along with the mentors we came up with the following: Create a development branch where we ourselves could merge our changes/hotfixes/upgrades/improvements to such branch then see how it went on the CI tools, and, if successful, keep working with the upgrade or something else. With this we could work faster, since we wouldn't need to wait for the approval of the merge. Along with that we have opened a PR from this development branch to main branch, to make our work more transparent to the mentors and to ourselves. This strategy has been proven to be very effective since, as I mentioned before, it's one of the things that made possible for us to be ahead of schedule.

During this part of the program, I have mostly worked on setting a proper environment for us to work on top of. That's why I have focused not only on establishing a stable Travis CI but also on stable docker environments.

During the setups, I have realized that Travis had been using our production environment, which is not good at all. So we have successfully split the development/test/production environments and made some improvements to Travis integration by having parallel jobs (what caused the builds to be ~2x faster).

You can find more of my work here and some more improvements here.

The upgrade to Rails 5.2.3

Rails 5.2.3 was the one that took more time and had me involved in a lot of different stuff. Part of this was because we have decided to skip a few versions and went straight ahead with 5.2.3. Great decision by the way!

During this upgrade, I had the opportunity to work with Jenkins. I changed some of its workflows and refactored some of its scripts here.

Besides all of this, I got to refactor some of the Rails' Asset Pipeline workflow, this involved entirely using Yarn (totally deprecated Bower) and Sprockets. Also helped to set up some system tests here.

Skipping some Rails versions and going straight forward with Rails 5.2.3 came out to be a very smart decision because even though it took a quite long time to finish, actually got us pretty ahead of schedule when we finished it. This made our friend Vidit Chitkara able to start his work on top of our Rails 5 branch, then he started setting ActionCable.

Besides Yarn and ActionCable, with latest Rails we were able to remove some deprecated initializers and files, improve the database setting and its integration with Mapknitter core and Travis.

In this phase of the project, we were also able to properly use Rubocop linter and stylesheet, following the same standards used in Plots2 project. Which makes the PublicLab's ecosystem more cohesive and uniform. Now we have a whole single job to keep track of the asset pipeline in Travis, used to keep track of changes made across the libs used in Mapknitter and its changes influence on our codebase.

We have also set Rails' Bootsnap to improve the loading time of our app, since it caches some expensive operations, like HTTP requests.

There was a massive refactoring of our routes since this version of Rails doesn't allow abstract routing anymore, due to security issues. So by refactoring, we've made Mapknitter less sensitive to security breaches. Since we're talking security, we've enforced the usage of strong params to HTTP requests too, which wraps our "payloaded" objects better for requesting.

All of the described above has been majorly done at publiclab/mapknitter#685

Upgrade to Rails 6.0.0

To get Rails 6.0 running we had to upgrade our Ruby version. Complicated right?! Not really, just had to update our Gemfile and Dockerfile with the desired Ruby version - in our case 2.6.3 - and that's it. Also, a bunch of improvements have been gotten, such as deprecating old patterns in favor of newer ones, using the latest sintax and such.

This upgrade came out to be smoothest of the upgrade series I had been working on. Rudely I just had to run the update task provided by the Rails team - thank God for that task - and also kept on checking the RailsDiff website - a loyal friend to help in this kind of upgrade. Along with that, I removed a bunch of unwanted files that had been floating through the project.

Upcoming work

Is worth to mention that when I finished the Rails 6.0.0 upgrade, I realized that would be super if we had Webpacker setup. This would involve many, many changes in our codebase. But I went straight ahead with this - motivated by our favorable schedule. In simple words, it would merely involve removing Sprockets and starting using Webpacker. Piece of cake, right?

Not really!

But in fact, I got pretty far with this. Actually, you can see everything I accomplished here. But this has taken more time than I expected, and after a friendly discussion with my mentors we decided to keep Webpacker implementation as a future work. But I took a lot of advantage of this actually. In fact, getting involved in this Rails 6 + Webpacker thing made me learn SO MUCH about the Rails Asset Pipeline y' all cannot imagine. Not only this, but I got the chance to get my good friend Arthur Diniz (Debian GSoC 2019 mentee) to collaborate with us of PublicLab, he gave us valuable info y' all can get here.

Some other cool stuff accomplished

Rubocop as part of the development flow

After a lot of negotiation, along with my mentors and colleagues, we got to use Rubocop to standardize more our coding and present to the newcomers some good coding practices. Y'all can get most of this here. One nice thing about this is that we used the same Rubocop settings as the ones used in Plots2 project, so we contribute across projects we can actually see patterns that make contributing easier.

Reduce CI runtime

Since I'm a good old DevOps enthusiast, solving this type of issue is my thing. With this MR I have helped to reduce our build time DRASTICALLY. See the comparative for yourself:

Before caching:

After caching:

Pretty neat, huh?!

Playing with Jenkins

One of the many wonderful experiences that GSoC 2019 have given to me is the opportunity to try tools I have only heard of. One of these tools was the automation server Jenkins.

As I said before, I'm a lover of DevOps, so nothing more pleasing to such creature than having the opportunity to get along with Jenkins. In fact, I got to make some improvements to the continuous delivery method provided by my coder friends of PublicLab. You can get some of the work I have made here, here and here.

Setting ActionCable

If you guys didn't know my GSoC 2019 was entirely collaborative. Actually in all of the upgrades mentioned before I had the help of Kaustubh Nair and many others.

Anyway, setting Action Cable was not in my project scope at all, but Vidit Chitkara(another GSoC 2019 mentee) realized that for the Mapknitter Image Management and Synchronous Editing project we needed ActionCable up and running, and consequently Rails 5. This had us joining forces to get both projects rolling.

Visual representation of Vidit Chitkara and me joining forces

So, to get ActionCable we needed Rails 5 on MapKnitter asap and stable. So Kaustubh and I had it finished, and afterwards I hopped on publiclab/mapknitter#805 to get it on top of our Rails 5 Merge Request. For ActionCable we had to get a Puma server running, but MapKnitter already uses Passenger as its server. So we need to open another tab in our terminal and bundle exec puma -C config/puma.rb right?

NOPE!

We all know this doesn't work on production, so I had to setup Foreman, a powerful gem that allows you to describe a set of processes required to run your application. I had used it before in other projects, so I knew it would work. And it did!

Some more collaborative stuff

I have accomplished a lot, thanks to many people that helped me. I will list below and quickly describe some more Merges I did with the help of good samaritans.

Fixing mismatching dependencies for Yarn

Believe it or not, I spent almost 2 weeks working on this issue. Came out to be 4-5 lines of code. This made me a little disappointed. xD

Fixing Maps loading

I am not the biggest fan of Javascript. But thank God for Sasha Boginsky (one of my mentors), she has helped me a lot during the project, and publiclab/mapknitter#885 was just one the merges she has braved to help me. Not only her but Jeffrey Warren, Gaurav Sachdeva, Sebastian Silva, and many others.

Plots2

It's clear that my GSoC project was on top of Mapknitter, but during the program, I also got the chance to work on Plots2 project as well. I had the opportunity to fix a few issues there, among those, I could help to set ChromeDriver to system tests here and to fix a minor issue regarding the version of Google Chrome browser we also use for system tests here.

It's worth to mention that I could quickly fix those bugs in Plots2 thanks to a similar issue I got when working on the Noosfero project, right here and here.

The end?

This was a HUGE article, I know, but it just had to be. This was my first time being a GSoC mentee. Contributing to such a large organization as PublicLab was just a fantastic experience! I'll carry on the lessons I've learned for the rest of my life.

Well guys, that's it! GSoC 2019 has been a great experience to me, definitely want to participate again next year.

Special thanks to the entire PublicLab community for trusting me such hard task, but also the Noosfero team and LAPPIS team.

Important Links

• My first contribution ever to PublicLab

• The Development branch MR

• Travis CI Improvement

• Splitting the work environments

• Setting Up Rubocop

• Rails 4 Upgrade

• Rails 5 Upgrade

• Jenkins Improvement part 1

• Jenkins Improvement part 2

• Jenkins Improvement part 3

• Fixing Map Loading and the Rails' Assets usage

• ActionCable and Foreman setup

• Rails 6 Upgrade

• Rails 6 Upgrade with WebPacker

• Kaustubh's Report on GSoC

• Vidit's Report on GSoC

This is not a bug, it's a Feature - M. Richard