What I did

Spectral Workbench is an open-source tool to perform low-cost spectral analysis and to share those results online. It consists of a Ruby on Rails web application for publishing, archiving, discussing, and analyzing spectra online.

Mapknitter Exporter Sinatra is minimal Sinatra app to run MapKnitter exports in the cloud, using the mapknitter-exporter gem.

In my Google Summer of Code project I have gotten the chance to work across all three of these repositories and it was plenty of work, trust me.

A little more about my proposal can be found here

Preparing the environment

This kind of work requires a lot of preparation, by this I mean prepare the environment for us to work on, and that’s what I did in Spectral Workbench, Exporter gem and Mapknitter Exporter Sinatra cli projects.

Here I have basically rewrite the docker workflow of the project. Since it was a pretty old repository with very legacy code and practices, it required some restructuring on the configuration files. For instance, the MySQL instance was not dockerized at the time I started the project, and there was no automation that aided the developer to easily get to code.

Still in Spectral, some sort of preparation was required at the continuous integration tool as well, at first it was needed to set up “local” builds for Travis, so that we could obtain a testing environment that simulated faithfully the development/production environment. Besides this, previously it was configured for the CI tool to run all tests at once - what seems to be a nice idea at first, but at long range caused the builds to last “forever”. Then, I have split the test running to execute each test suite separately, thus taking advantage of the parallelism implemented. Here and here you can find more about the mentioned changes.

When it comes to the Exporter Gem project, the preparation consisted in getting the continuous integration tool (Travis CI) properly set up so it could reflect the project’s environment precisely. And also it included some performance upgrades since it has been also configured a multistage build so that the CI pipelines could run in parallel, and thus faster. This entire work is available at publiclab/mapknitter-exporter#470.

A similar work to this has been developed at the Mapknitter Exporter Sinatra client project. The preparation for the upcoming updates involved significant changes in the Travis CI tool, this reflected on some performance upgrades since it was configured parallel builds and a script that sets up the packages needed for the application to run. It is also important to note that cache has been configured for every CI tool in each repository, that way, allied to the parallelism, the builds run significantly faster than before. All of this work can be found here.

It is important to note that this work regarding the CI tool parallelization and better setup caused the builds to be around 2x faster than before, in all three repositories (Spectral, Exporter Gem and Mapknitter Exporter Sinatra).

The Rails and DevOps upgrades

After getting a nice test coverage, a proper CI and developing environment we could start with the upgrade itself. It's widely embraced as a good practice to open smaller Pull Requests containing smaller and significant parts of code as in a granular upgrade. One of my plans for the project was obtaining a stable test coverage, by increasing the test coverage for the Spectral Workbench. It is important to say that this part of the plan was not accomplished the way it was planned, the test coverage was being achieved along with the rest of the main project and it did not make part of the “Preparation” phase of the project.

I have embraced the same work strategy from last year that consisted in creating a development branch where I could merge the changes/hotfixes/upgrades/improvements to such branch then see how it went on the CI tools, and, if successful, keep working with the upgrade or something else. With this I could work faster, since it wouldn't need to wait for the merge approval. Along with that, opening a PR from this development branch to the main branch made my work more transparent to the mentors.

At this part of the project I have configured Ruby 2.6, Rails 4.2 and Node 13. This PR already introduced countless benefits for the Spectral project, for example, Strong Parameters feature that was introduced in Rails 4 prevents assigning request parameters to objects unless they have been explicitly permitted. Practically that meant removing the attr_accessible and attr_protected calls from models and adding a new method to the respective controllers to handle the parameters.

The main advantage of all of this is that they are defined in the controller and can be dynamically assigned in run time attr_accessible was a more static and monolithic way to whitelist attributes, thus making Strong Parameters more flexible. Since every model has been refactored at the previously mentioned PR to give support to the Strong Parameters approach we now have a more modern and trustful code. Talking about security, the usage of strong params to HTTP requests wraps our "payloaded" objects better for requesting, which made the requests better structured.

The majority of gems have also been upgraded, now supporting newer Ruby and Rails. And another good thing that came out of this too was that after the upgrade was complete, it was required a staging environment so that we could test out the changes that were made on the cloud. So along with the help of PublicLab’s sysadmin Sebastian, we have set up an automated staging environment, in a way that when I pushed to a staging branch, I could follow up the Jenkins build on the console and hit an endpoint to verify live the changes I have made.

The Node version has been upgraded too, thus giving a better support for the javascript dependencies manager at the time, which was Bower.

In the meantime my good friend Ruturaj was giving me ultra help with tests. All of his work is on the list below:

He has been of great help for me, his help and friendship is the living proof of the great community the PublicLab is. He has taken a lot of his time to help me and made me very thankful. Reminded me of the good spirit that it is working with open source software.

Visual representation of me and Ruturaj teaming up on the project.

With Ruturaj’s help we got to increase the test coverage by more than 20%, Spectral Workbench went from ~60% to a little more than ~80%. This could be measured right after I got to include and configure SimpleCov to get the test coverage percentages here.

At the Exporter gem some of my work consisted in refactoring the code base, to do that I counted with the help of Rubocop, a very well known tool in PublicLab’s ecosystem of projects. With Rubocop we got to standardize our coding and present to the newcomers some good coding practices. In a practical manner, one nice thing about this, is that we used the same Rubocop settings as the ones used in Plots2, Mapknitter and Spectral Workbench. Here is where I have Rubocop for the Exporter Gem, here is for Spectral Workbench and here is where I have set Rubocop for Mapknitter Exporter Sinatra, all of them following the same coding rules, thus making the PublicLab's ecosystem more cohesive and uniform.

Later, upgrading to Rails 5 was one task that took a little less time than upgrading to Rails 4, but it had me involved in a lot of different stuff.

During this upgrade, there was a massive refactoring of our routes since this version of Rails doesn't allow abstract routing anymore, due to security issues. So by refactoring, Spectral got less sensitive to security breaches. Also, during this upgrade I have removed Bower completely and set up the latest compatible Yarn, which meant a very significant upgrade for Spectral since Bower has been deprecated and no longer maintained. The entire Rails 5 upgrade can be found here.

When it comes to testing, a main request of my mentor Jeffrey Warren for this project was the configuration and inclusion of some system tests, and I got to that here. It is important to notice that this involved configuring ChromeDriver and its adapter gems and also implementing a couple of tests to validate the configuration, which worked very well by the way.

At this point a massive refactoration work has been made at the Mapknitter Exporter Sinatra client at publiclab/mapknitter-exporter-sinatra#56. Here I'm just on phase one of Sinatra's modular code styling. According to the Sinatra docs: “When a classic style application is run, all Sinatra::Application public class methods are exported to the top-level.” Also, using the classical style prevents you from running more than one Sinatra application per Ruby process – all calls to these top-level methods are handled by Sinatra::Application, functioning as a singleton. We can avoid these potentially-confusing scoping problems by reorganizing our application into what Sinatra calls the “modular” style - which is what I'm starting to do here. Besides mitigating scoping issues, we could reduce code duplication(by using lambdas) and some other advantages.

And finally, one of my last great contributions was the configuration of GitPod for Spectral Workbench and Mapknitter Exporter Sinatra. GitPod is an online IDE which can be launched from any GitHub page; within seconds, GitPod provides you with a fully working development environment, including a VSCode-powered IDE and a cloud-based Linux container configured specifically for the project. Having GitPod configured provided immense benefits for the community; the configuration for both projects can be found here for Spectral Workbench and here for the Mapknitter Exporter Sinatra.

Other contributions

Remove `command` from compose files

Here I have helped in PublicLab’s Plots2 project that was failing to start the Passenger web server, causing the application not to start. Link: publiclab/plots2#8111

Links for every contribution I made on the GSoC period

Spectral Workbench

Mapknitter Exporter Sinatra

Exporter Gem

Plots2

https://github.com/publiclab/plots2/pull/8111

4 Comments

Thank you @alaxallves , this is a fun-to-read narrative of the events of your fellowship that really shows the great amount of significant work that you thoughtfully undertook! Also thanks must go to Ruturaj!

Thanks a lot for the opportunity @liz. Looking forward now to present this to the community in September 14th.

Reply to this comment...

This is epic!! Thanks a ton Alax for your excellent work, and thanks to Ruturaj as well.

Also - your anime memes are becoming a tradition here! <3

Thanks a ton Jeff! Trully appreciate the opportunity you and the PL team have given me. <3

Public Lab Research note

Public Lab

Research note

GSoC 2020: Mapknitter Export Module and Spectral Workbench upgrades

What I did

Preparing the environment

The Rails and DevOps upgrades